BUG #15858: could not stat file - over 4GB
The following bug has been logged on the website:
Bug reference: 15858
Logged by: William Allen
Email address: williamedwinallen@live.com
PostgreSQL version: 11.3
Operating system: Windows Server 2012 R2
Description:
Issue using copy from command for files over 4GB.
ERROR: could not stat file "E:\file.txt": Unknown error
SQL state: XX000
On Tue, Jun 18, 2019 at 10:02:53AM +0000, PG Bug reporting form wrote:
Issue using copy from command for files over 4GB.
ERROR: could not stat file "E:\file.txt": Unknown error
SQL state: XX000
Windows is known for having limitations in its former implementations
of stat(), and the various _stat structures they use make actually
that much harder from a compatibility point of view:
/messages/by-id/1803D792815FC24D871C00D17AE95905CF5099@g01jpexmbkw24
Nobody has actually dug enough into this set of issues to get a patch
out of the ground, which basically requires more tweaks that one may
think at first sight (look at pgwin32_safestat() in src/port/dirmod.c
for example).
--
Michael
On Wed, Jun 19, 2019 at 3:26 AM Michael Paquier <michael@paquier.xyz> wrote:
Windows is known for having limitations in its former implementations
of stat(), and the various _stat structures they use make actually
that much harder from a compatibility point of view:
/messages/by-id/1803D792815FC24D871C00D17AE95905CF5099@g01jpexmbkw24
Going through this discussion it is not clear to me if there was a
consensus about the shape of an acceptable patch. Would something like
the attached be suitable?
Regards,
Juan José Santamaría Flecha
Attachments:
0001-support-for-large-files-on-Win32.patchapplication/octet-stream; name=0001-support-for-large-files-on-Win32.patchDownload
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index f4841fb..3b4d2bd 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -52,7 +52,19 @@
#include <direct.h>
#include <sys/utime.h> /* for non-unicode version */
#undef near
+/*
+ * We need to define _CRT_NO_TIME_T in order to prevent the definition of
+ * struct stat and replace it with struct _stat64
+ */
+#if defined(_MSC_VER)
+#define HAVE_STRUCT_STAT64
+#define _CRT_NO_TIME_T
+#include <sys/types.h>
+#include <sys/stat.h>
+#undef _CRT_NO_TIME_T
+#else
#include <sys/stat.h> /* needed before sys/stat hacking below */
+#endif
/* Must be here to avoid conflicting with prototype in windows.h */
#define mkdir(a,b) mkdir(a)
@@ -251,6 +263,34 @@ typedef int pid_t;
* We must pull in sys/stat.h before this part, else our overrides lose.
*/
#define lstat(path, sb) stat(path, sb)
+#if defined(HAVE_STRUCT_STAT64)
+struct stat /* It is actually _stat64 */
+{
+ _dev_t st_dev;
+ _ino_t st_ino;
+ unsigned short st_mode;
+ short st_nlink;
+ short st_uid;
+ short st_gid;
+ _dev_t st_rdev;
+ __int64 st_size;
+ __time64_t st_atime;
+ __time64_t st_mtime;
+ __time64_t st_ctime;
+};
+static __inline int
+fstat(int const _FileName, struct stat* const _Stat)
+{
+ Assert(sizeof(struct stat) == sizeof(struct _stat64));
+ return _fstat64(_FileName, (struct _stat64*)_Stat);
+}
+static __inline int
+stat(char const* const _FileName, struct stat* const _Stat)
+{
+ Assert(sizeof(struct stat) == sizeof(struct _stat64));
+ return _stat64(_FileName, (struct _stat64*)_Stat);
+}
+#endif
/*
* stat() is not guaranteed to set the st_size field on win32, so we
diff --git a/src/port/dirmod.c b/src/port/dirmod.c
index d793240..01cb85d 100644
--- a/src/port/dirmod.c
+++ b/src/port/dirmod.c
@@ -395,11 +395,10 @@ pgwin32_safestat(const char *path, struct stat *buf)
return -1;
}
- /*
- * XXX no support for large files here, but we don't do that in general on
- * Win32 yet.
- */
- buf->st_size = attr.nFileSizeLow;
+ if (sizeof(buf->st_size) < sizeof(uint64))
+ buf->st_size = attr.nFileSizeLow;
+ else
+ buf->st_size = ((uint64) attr.nFileSizeHigh) << 32 | attr.nFileSizeLow;
return 0;
}
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo.santamaria@gmail.com> writes:
On Wed, Jun 19, 2019 at 3:26 AM Michael Paquier <michael@paquier.xyz> wrote:
Windows is known for having limitations in its former implementations
of stat(), and the various _stat structures they use make actually
that much harder from a compatibility point of view:
/messages/by-id/1803D792815FC24D871C00D17AE95905CF5099@g01jpexmbkw24
Going through this discussion it is not clear to me if there was a
consensus about the shape of an acceptable patch. Would something like
the attached be suitable?
I think there's general agreement that the correct fix involves somehow
mapping stat() to _stat64() and mapping "struct stat" to "struct __stat64"
to go along with that. Beyond that, things get murky.
1. Can we assume that _stat64() and struct __stat64 exist on every Windows
version and build toolchain that we care about? Windows itself is
probably OK --- googling found a (non-authoritative) statement that these
were introduced in Windows 2K. But it's less clear whether they'll work
on builds with Cygwin, or Mingw, or Mingw-64, or how far back that support
goes. I found one statement that Mingw declares them only "#if
__MSVCRT_VERSION__ >= 0x0601".
2. Mapping stat() to _stat64() seems easy enough: we already declare
stat(a,b) as a macro on Windows, so just change it to something else.
3. What about the struct name? I proposed just "define stat __stat64",
but Robert thought that was too cute, and he's got a point --- in
particular, it's not clear to me how nicely it'd play to have both
function and object macros for the same name "stat". I see you are
proposing fixing this angle by suppressing the system definition of
struct stat and then defining it ourselves with the same contents as
struct __stat64. That might work. Ordinarily I'd be worried about
bit-rot in a struct that has to track a system definition, but Microsoft
are so religiously anal about never breaking ABI that it might be safe
to assume we don't have to worry about that.
I don't like the specific way you're proposing suppressing the system
definition of struct stat, though. "#define _CRT_NO_TIME_T" seems
like it's going to be a disaster, both because it likely has other
side-effects and because it probably doesn't do what you intend at all
on non-MSVC toolchains. We have precedents for dealing with similar
issues in, eg, plperl; and what those precedents would suggest is
doing something like
#define stat microsoft_native_stat
#include <sys/stat.h>
#undef stat
after which we could do
struct stat {
... same contents as __stat64
};
#define stat(a,b) _stat64(a,b)
Another issue here is that pgwin32_safestat() probably needs revisited
as to its scope and purpose. Its use of GetFileAttributesEx() can
presumably be dropped. I don't actually believe the header comment
claiming that stat() is not guaranteed to update the st_size field;
there's no indication of that in the Microsoft documentation. What
seems more likely is that that's a garbled version of the truth,
that you won't get a correct value of _st_size for files over 4GB.
But the test for ERROR_DELETE_PENDING might be worth keeping. So
that would lead us to
struct stat {
... same contents as __stat64
};
extern int pgwin32_safestat(const char *path, struct stat *buf);
#define stat(a,b) pgwin32_safestat(a,b)
and something like
int
pgwin32_safestat(const char *path, struct stat *buf)
{
int r;
/*
* Don't call stat(), that would just recurse back to here.
* We really want _stat64().
*/
r = _stat64(path, buf);
if (r < 0)
{
if (GetLastError() == ERROR_DELETE_PENDING)
{
/*
* File has been deleted, but is not gone from the filesystem yet.
* This can happen when some process with FILE_SHARE_DELETE has it
* open and it will be fully removed once that handle is closed.
* Meanwhile, we can't open it, so indicate that the file just
* doesn't exist.
*/
errno = ENOENT;
}
}
return r;
}
Not sure if we'd need an explicit cast to override passing struct
stat * to _stat64(). If so, a StaticAssert that sizeof(struct stat)
matches sizeof(struct __stat64) seems like a good idea.
I'd also be very strongly inclined to move pgwin32_safestat into its
own file in src/port and get rid of UNSAFE_STAT_OK. There wouldn't
be a good reason to opt out of using it once we got to this point.
regards, tom lane
I wrote:
Another issue here is that pgwin32_safestat() probably needs revisited
as to its scope and purpose. Its use of GetFileAttributesEx() can
presumably be dropped. I don't actually believe the header comment
claiming that stat() is not guaranteed to update the st_size field;
there's no indication of that in the Microsoft documentation. What
seems more likely is that that's a garbled version of the truth,
that you won't get a correct value of _st_size for files over 4GB.
So after further digging around, it seems that this is wrong. The
existence of pgwin32_safestat() can be traced back to these threads:
/messages/by-id/528853D3C5ED2C4AA8990B504BA7FB850106DF10@sol.transas.com
/messages/by-id/528853D3C5ED2C4AA8990B504BA7FB850106DF2F@sol.transas.com
in which it's stated that
It seems I've found the cause and the workaround of the problem.
MSVC's stat() is implemented by using FindNextFile().
MSDN contains the following suspicious paragraph аbout FindNextFile():
"In rare cases, file attribute information on NTFS file systems
may not be current at the time you call this function. To obtain
the current NTFS file system file attributes, call
GetFileInformationByHandle."
Since we generally cannot open an examined file, we need another way.
I'm wondering though why we adopted the existing coding in the face of
that observation. Couldn't the rest of struct stat be equally out of
date?
In short it seems like maybe we should be doing something similar to the
patch that Sergey actually submitted in that discussion:
/messages/by-id/528853D3C5ED2C4AA8990B504BA7FB850658BA5C@sol.transas.com
which reimplements stat() from scratch on top of GetFileAttributesEx(),
and thus doesn't require any assumptions at all about what's available
from the toolchain's <sys/stat.h>.
regards, tom lane
On Wed, Jun 19, 2019 at 8:02 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
In short it seems like maybe we should be doing something similar to the
patch that Sergey actually submitted in that discussion:/messages/by-id/528853D3C5ED2C4AA8990B504BA7FB850658BA5C@sol.transas.com
I will not have much time for this list in the next couple of weeks,
so I will send this patch in its current WIP state rather than
stalling without a reply.
Most of its functionality comes from Sergey's patch with some cosmetic
changes, and the addition of the 64 bits struct stat and fstat().
Regards,
Juan José Santamaría Flecha
Attachments:
0001-WIP-support-for-large-files-on-Win32-v2.patchapplication/octet-stream; name=0001-WIP-support-for-large-files-on-Win32-v2.patchDownload
From 80fb66c1feb783ee6366aa3e1315395205060021 Mon Sep 17 00:00:00 2001
From: Juan Jose Santamaria Flecha <juanjo.santamaria@gmail.com>
Date: Mon, 24 Jun 2019 12:56:13 -0400
Subject: [PATCH] WIP support for large files on Win32 . Workaround for Windows
<sys/stat.h> shortcomings, reimplement fstat()/stat() on top of
GetFileAttributesEx() and use 64 bits struct stat.
---
configure | 11 ++++
src/include/port/win32_port.h | 62 +++++++++++++++---
src/port/dirmod.c | 52 ---------------
src/port/stat_pg_fixed.c | 149 ++++++++++++++++++++++++++++++++++++++++++
4 files changed, 214 insertions(+), 60 deletions(-)
create mode 100644 src/port/stat_pg_fixed.c
diff --git a/configure b/configure
index 8d47071..e8dcfcb 100755
--- a/configure
+++ b/configure
@@ -16204,6 +16204,12 @@ fi
esac
case " $LIBOBJS " in
+ *" stat_pg_fixed.$ac_objext "* ) ;;
+ *) LIBOBJS="$LIBOBJS stat_pg_fixed.$ac_objext"
+ ;;
+esac
+
+ case " $LIBOBJS " in
*" kill.$ac_objext "* ) ;;
*) LIBOBJS="$LIBOBJS kill.$ac_objext"
;;
@@ -16282,6 +16288,11 @@ if test "$PORTNAME" = "cygwin"; then
;;
esac
+ case " $LIBOBJS " in
+ *" stat_pg_fixed.$ac_objext "* ) ;;
+ *) LIBOBJS="$LIBOBJS stat_pg_fixed.$ac_objext"
+ ;;
+esac
fi
ac_fn_c_check_func "$LINENO" "syslog" "ac_cv_func_syslog"
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index f4841fb..fd1620b 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -52,7 +52,12 @@
#include <direct.h>
#include <sys/utime.h> /* for non-unicode version */
#undef near
-#include <sys/stat.h> /* needed before sys/stat hacking below */
+/* needed before sys/stat hacking below */
+#define fstat microsoft_native_fstat
+#define stat microsoft_native_stat
+#include <sys/stat.h>
+#undef fstat
+#undef stat
/* Must be here to avoid conflicting with prototype in windows.h */
#define mkdir(a,b) mkdir(a)
@@ -254,15 +259,56 @@ typedef int pid_t;
/*
* stat() is not guaranteed to set the st_size field on win32, so we
- * redefine it to our own implementation that is.
+ * redefine it to our own implementation. See src/port/stat_pg_fixed.c
*
- * Some frontends don't need the size from stat, so if UNSAFE_STAT_OK
- * is defined we don't bother with this.
+ * The struct stat is 32 bit in MSVC so we redefine it as a copy of struct
+ * _stat64. This also fixes the struct size across Cygwin and MINGW builds.
*/
-#ifndef UNSAFE_STAT_OK
-extern int pgwin32_safestat(const char *path, struct stat *buf);
-#define stat(a,b) pgwin32_safestat(a,b)
-#endif
+struct stat /* It is actually _stat64 */
+{
+ _dev_t st_dev;
+ _ino_t st_ino;
+ unsigned short st_mode;
+ short st_nlink;
+ short st_uid;
+ short st_gid;
+ _dev_t st_rdev;
+ __int64 st_size;
+ __time64_t st_atime;
+ __time64_t st_mtime;
+ __time64_t st_ctime;
+};
+extern int _pgfstat64(int fileno, struct stat *buf);
+extern int _pgstat64(char const *name, struct stat *buf);
+#if defined(_MSC_VER)
+static __inline int __CRTDECL
+fstat(int _Filenumber, struct stat* _Stat)
+{
+ _STATIC_ASSERT(sizeof(struct stat) == sizeof(struct __stat64));
+ return _pgfstat64(_Filenumber, _Stat);
+}
+
+static __inline int __CRTDECL
+stat(char* const _Filename, struct stat* _Stat)
+{
+ _STATIC_ASSERT(sizeof(struct stat) == sizeof(struct __stat64));
+ return _pgstat64(_Filename, _Stat);
+}
+#else /* defined(__MINGW32__) || defined(__MINGW64__) || defined(__CYGWIN__) */
+static __inline__ int
+fstat(int _Filenumber, struct stat* _Stat)
+{
+ Assert(sizeof(struct stat) == sizeof(struct __stat64));
+ return _pgfstat64(_Filenumber, _Stat);
+}
+
+static __inline__ int
+stat(char* const _Filename, struct stat* _Stat)
+{
+ Assert(sizeof(struct stat) == sizeof(struct __stat64));
+ return _pgstat64(_Filename, _Stat);
+}
+#endif /* defined(_MSC_VER) */
/* These macros are not provided by older MinGW, nor by MSVC */
#ifndef S_IRUSR
diff --git a/src/port/dirmod.c b/src/port/dirmod.c
index d793240..ec4b47f 100644
--- a/src/port/dirmod.c
+++ b/src/port/dirmod.c
@@ -353,55 +353,3 @@ pgwin32_is_junction(const char *path)
return ((attr & FILE_ATTRIBUTE_REPARSE_POINT) == FILE_ATTRIBUTE_REPARSE_POINT);
}
#endif /* defined(WIN32) && !defined(__CYGWIN__) */
-
-
-#if defined(WIN32) && !defined(__CYGWIN__)
-
-#undef stat
-
-/*
- * The stat() function in win32 is not guaranteed to update the st_size
- * field when run. So we define our own version that uses the Win32 API
- * to update this field.
- */
-int
-pgwin32_safestat(const char *path, struct stat *buf)
-{
- int r;
- WIN32_FILE_ATTRIBUTE_DATA attr;
-
- r = stat(path, buf);
- if (r < 0)
- {
- if (GetLastError() == ERROR_DELETE_PENDING)
- {
- /*
- * File has been deleted, but is not gone from the filesystem yet.
- * This can happen when some process with FILE_SHARE_DELETE has it
- * open and it will be fully removed once that handle is closed.
- * Meanwhile, we can't open it, so indicate that the file just
- * doesn't exist.
- */
- errno = ENOENT;
- return -1;
- }
-
- return r;
- }
-
- if (!GetFileAttributesEx(path, GetFileExInfoStandard, &attr))
- {
- _dosmaperr(GetLastError());
- return -1;
- }
-
- /*
- * XXX no support for large files here, but we don't do that in general on
- * Win32 yet.
- */
- buf->st_size = attr.nFileSizeLow;
-
- return 0;
-}
-
-#endif
diff --git a/src/port/stat_pg_fixed.c b/src/port/stat_pg_fixed.c
new file mode 100644
index 0000000..98de7c4
--- /dev/null
+++ b/src/port/stat_pg_fixed.c
@@ -0,0 +1,149 @@
+/*-------------------------------------------------------------------------
+ *
+ * stat_pg_fixed.c
+ * Replacements for <sys/stat.h> functions using GetFileAttributesEx on
+ * Win32.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/port/stat_pg_fixed.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifdef WIN32
+
+#include "c.h"
+#include <tchar.h>
+#if defined(__CYGWIN__)
+#include <w32api/windows.h>
+#else
+#include <windows.h>
+#endif
+
+static const unsigned __int64 EpochShift = UINT64CONST(116444736000000000);
+
+/* Converts FILETIME to time_t */
+static __time64_t
+cvt_ft2ut(FILETIME const * ft)
+{
+ ULARGE_INTEGER unified_ft;
+
+ unified_ft.LowPart = ft->dwLowDateTime;
+ unified_ft.HighPart = ft->dwHighDateTime;
+
+ if (unified_ft.QuadPart < EpochShift)
+ return -1;
+
+ unified_ft.QuadPart -= EpochShift;
+ unified_ft.QuadPart /= 10 * 1000 * 1000;
+
+ return ((__int64) unified_ft.HighPart) << 32 |
+ (__int64) unified_ft.LowPart;
+}
+
+static unsigned short
+cvt_attr2uxmode(int attr, const _TCHAR * name)
+{
+ unsigned short uxmode;
+ const _TCHAR * p;
+
+ uxmode = (unsigned short)((attr & FILE_ATTRIBUTE_DIRECTORY) ?
+ (_S_IFDIR | _S_IEXEC) : (_S_IFREG));
+
+ uxmode |= (unsigned short)(attr & FILE_ATTRIBUTE_READONLY) ?
+ (_S_IREAD) : (_S_IREAD | _S_IWRITE);
+
+ if (p = _tcsrchr(name, _T('.')))
+ {
+ if (! _tcsicmp(p, _T(".exe")) ||
+ ! _tcsicmp(p, _T(".cmd")) ||
+ ! _tcsicmp(p, _T(".bat")) ||
+ ! _tcsicmp(p, _T(".com")))
+ uxmode |= _S_IEXEC;
+ }
+
+ uxmode |= (uxmode & 0700) >> 3;
+ uxmode |= (uxmode & 0700) >> 6;
+
+ return uxmode;
+}
+
+int
+_pgstat64(char const *name, struct stat *buf)
+{
+ WIN32_FILE_ATTRIBUTE_DATA faData;
+
+ if (NULL == name || NULL == buf)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+
+ if (!GetFileAttributesEx(name, GetFileExInfoStandard, & faData))
+ {
+ DWORD error = GetLastError();
+ if (error== ERROR_DELETE_PENDING)
+ {
+ /*
+ * File has been deleted, but is not gone from the filesystem yet.
+ * This can happen when some process with FILE_SHARE_DELETE has it
+ * open and it will be fully removed once that handle is closed.
+ * Meanwhile, we can't open it, so indicate that the file just
+ * doesn't exist.
+ */
+ errno = ENOENT;
+ }
+ _dosmaperr(error);
+ return -1;
+ }
+
+ memset(buf, 0, sizeof(* buf));
+
+ if (faData.ftLastWriteTime.dwLowDateTime ||
+ faData.ftLastWriteTime.dwHighDateTime)
+ buf->st_mtime = cvt_ft2ut( & faData.ftLastWriteTime );
+
+ if (faData.ftLastAccessTime.dwLowDateTime ||
+ faData.ftLastAccessTime.dwHighDateTime)
+ buf->st_atime = cvt_ft2ut(& faData.ftLastAccessTime);
+ else
+ buf->st_atime = buf->st_mtime;
+
+ if (faData.ftCreationTime.dwLowDateTime ||
+ faData.ftCreationTime.dwHighDateTime)
+ buf->st_ctime = cvt_ft2ut(& faData.ftCreationTime);
+ else
+ buf->st_ctime = buf->st_mtime;
+
+ buf->st_mode = cvt_attr2uxmode(faData.dwFileAttributes, name);
+ buf->st_nlink = 1;
+
+ buf->st_size = ((__int64) faData.nFileSizeHigh) << 32 |
+ (__int64)(faData.nFileSizeLow);
+
+ return 0;
+}
+
+int
+_pgfstat64(int fileno, struct stat *buf)
+{
+ const char filepath[MAX_PATH];
+ FILE_NAME_INFO nameInfo;
+
+ if (GetFileInformationByHandleEx((HANDLE) _get_osfhandle(fileno),
+ FileNameInfo,
+ &nameInfo,
+ sizeof(nameInfo)))
+ {
+ sprintf(filepath, "%ws", nameInfo.FileName);
+ return _pgstat64(filepath, buf);
+ }
+ else
+ return -1;
+}
+
+#endif /* WIN32 */
--
2.11.0
On Tue, Jun 25, 2019 at 12:00:45PM +0200, Juan José Santamaría Flecha wrote:
I will not have much time for this list in the next couple of weeks,
so I will send this patch in its current WIP state rather than
stalling without a reply.Most of its functionality comes from Sergey's patch with some cosmetic
changes, and the addition of the 64 bits struct stat and fstat().
The former patch was rather impressive. Or scary. Or both. At which
extent have you tested it? I think that we really need to make sure
of a couple of things which satisfy our needs:
1) Are we able to fix the issues with stat() calls on files larger
than 2GB and report a correct size?
2) Are we able to detect properly that files pending for deletion are
discarded with ENOENT?
3) Are frontends able to use the new layer?
It seems to me that you don't need the configure changes.
Instead of stat_pg_fixed which is confusing because it only involves
Windows, I would rename the new file to stat.c or win32_stat.c. The
location in src/port/ is adapted. I would also move out of
win32_port.h the various inline declarations and keep only raw
declarations. That could be much cleaner.
The code desperately needs more comments to help understand its
logic. Don't we have in the tree an equivalent of cvt_ft2ut? What
does cvt_attr2uxmode do? It would be nice to avoid conversion
wrappers as much as possible, and find out system-related equivalents
if any, and actually if necessary.
+static unsigned short
+cvt_attr2uxmode(int attr, const _TCHAR * name)
This looks rather bug-prone...
I think that this stuff has not been tested and would break at
compilation. If src/tools/msvc/Mkvcbuild.pm is not changed, then the
new file won't get included in the compiled set.
--
Michael
On Wed, Jun 26, 2019 at 4:23 AM Michael Paquier <michael@paquier.xyz> wrote:
The former patch was rather impressive. Or scary. Or both. At which
extent have you tested it? I think that we really need to make sure
of a couple of things which satisfy our needs:
I wanted to make a quick test on the previous patch. So let me state
what have I tested and what I have not: it builds and pass tests in
Windows and Cygwin, but I have not setup a MinGW environment.
1) Are we able to fix the issues with stat() calls on files larger
than 2GB and report a correct size?
I have successfuly tested a COPY with large files.
2) Are we able to detect properly that files pending for deletion are
discarded with ENOENT?
Cannot reproduce reliably, but is using the same logic as pgwin32_safestat().
3) Are frontends able to use the new layer?
After removing UNSAFE_STAT_OK, is this still an issue?
It seems to me that you don't need the configure changes.
The changes in configuration are meant for gcc compilations in Windows
(Cygwin and Mingw).
Instead of stat_pg_fixed which is confusing because it only involves
Windows, I would rename the new file to stat.c or win32_stat.c. The
location in src/port/ is adapted. I would also move out of
win32_port.h the various inline declarations and keep only raw
declarations. That could be much cleaner.
Ok.
The code desperately needs more comments to help understand its
logic. Don't we have in the tree an equivalent of cvt_ft2ut? What
does cvt_attr2uxmode do? It would be nice to avoid conversion
wrappers as much as possible, and find out system-related equivalents
if any, and actually if necessary.
I have only found something similar in ./src/port/gettimeofday.c, but
not sure if this patch should touch that code.
+static unsigned short +cvt_attr2uxmode(int attr, const _TCHAR * name) This looks rather bug-prone...
I wanted to keep as much of the original code as possible, but if this
is found as a viable solution, what shape should it have?
I think that this stuff has not been tested and would break at
compilation. If src/tools/msvc/Mkvcbuild.pm is not changed, then the
new file won't get included in the compiled set.
The previous patch was broken, taken from the wrong local branch
(sorry about that). The attached is still a WIP but it has to do the
things above-mentioned.
Regards,
Juan José Santamaría Flecha
Attachments:
0001-WIP-support-for-large-files-on-Win32-v3.patchapplication/octet-stream; name=0001-WIP-support-for-large-files-on-Win32-v3.patchDownload
From 028a5af1c1f392af98102bd7a403df51e19d78fc Mon Sep 17 00:00:00 2001
From: Juan Jose Santamaria Flecha <juanjo.santamaria@gmail.com>
Date: Fri, 28 Jun 2019 03:53:44 -0400
Subject: [PATCH] WIP support for large files on Win32 . Workaround for Windows
<sys/stat.h> shortcomings, reimplement fstat()/stat() on top of
GetFileAttributesEx() and use 64 bits struct stat.
---
configure | 11 +++
src/include/port/win32_port.h | 35 +++++++---
src/port/dirmod.c | 52 --------------
src/port/win32_stat.c | 154 ++++++++++++++++++++++++++++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 2 +-
5 files changed, 193 insertions(+), 61 deletions(-)
create mode 100644 src/port/win32_stat.c
diff --git a/configure b/configure
index 8d47071..4bcec22 100755
--- a/configure
+++ b/configure
@@ -16204,6 +16204,12 @@ fi
esac
case " $LIBOBJS " in
+ *" win32_stat.$ac_objext "* ) ;;
+ *) LIBOBJS="$LIBOBJS win32_stat.$ac_objext"
+ ;;
+esac
+
+ case " $LIBOBJS " in
*" kill.$ac_objext "* ) ;;
*) LIBOBJS="$LIBOBJS kill.$ac_objext"
;;
@@ -16282,6 +16288,11 @@ if test "$PORTNAME" = "cygwin"; then
;;
esac
+ case " $LIBOBJS " in
+ *" win32_stat.$ac_objext "* ) ;;
+ *) LIBOBJS="$LIBOBJS win32_stat.$ac_objext"
+ ;;
+esac
fi
ac_fn_c_check_func "$LINENO" "syslog" "ac_cv_func_syslog"
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index f4841fb..d84e2d8 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -52,7 +52,12 @@
#include <direct.h>
#include <sys/utime.h> /* for non-unicode version */
#undef near
-#include <sys/stat.h> /* needed before sys/stat hacking below */
+/* needed before sys/stat hacking below */
+#define fstat microsoft_native_fstat
+#define stat microsoft_native_stat
+#include <sys/stat.h>
+#undef fstat
+#undef stat
/* Must be here to avoid conflicting with prototype in windows.h */
#define mkdir(a,b) mkdir(a)
@@ -254,15 +259,29 @@ typedef int pid_t;
/*
* stat() is not guaranteed to set the st_size field on win32, so we
- * redefine it to our own implementation that is.
+ * redefine it to our own implementation. See src/port/win32_stat.c
*
- * Some frontends don't need the size from stat, so if UNSAFE_STAT_OK
- * is defined we don't bother with this.
+ * The struct stat is 32 bit in MSVC so we redefine it as a copy of struct
+ * _stat64. This also fixes the struct size across Cygwin and MINGW builds.
*/
-#ifndef UNSAFE_STAT_OK
-extern int pgwin32_safestat(const char *path, struct stat *buf);
-#define stat(a,b) pgwin32_safestat(a,b)
-#endif
+struct stat /* It is actually _stat64 */
+{
+ _dev_t st_dev;
+ _ino_t st_ino;
+ unsigned short st_mode;
+ short st_nlink;
+ short st_uid;
+ short st_gid;
+ _dev_t st_rdev;
+ __int64 st_size;
+ __time64_t st_atime;
+ __time64_t st_mtime;
+ __time64_t st_ctime;
+};
+extern int _pgfstat64(int fileno, struct stat *buf);
+extern int _pgstat64(char const *name, struct stat *buf);
+#define fstat(fileno, sb) _pgfstat64(fileno, sb)
+#define stat(path, sb) _pgstat64(path, sb)
/* These macros are not provided by older MinGW, nor by MSVC */
#ifndef S_IRUSR
diff --git a/src/port/dirmod.c b/src/port/dirmod.c
index d793240..ec4b47f 100644
--- a/src/port/dirmod.c
+++ b/src/port/dirmod.c
@@ -353,55 +353,3 @@ pgwin32_is_junction(const char *path)
return ((attr & FILE_ATTRIBUTE_REPARSE_POINT) == FILE_ATTRIBUTE_REPARSE_POINT);
}
#endif /* defined(WIN32) && !defined(__CYGWIN__) */
-
-
-#if defined(WIN32) && !defined(__CYGWIN__)
-
-#undef stat
-
-/*
- * The stat() function in win32 is not guaranteed to update the st_size
- * field when run. So we define our own version that uses the Win32 API
- * to update this field.
- */
-int
-pgwin32_safestat(const char *path, struct stat *buf)
-{
- int r;
- WIN32_FILE_ATTRIBUTE_DATA attr;
-
- r = stat(path, buf);
- if (r < 0)
- {
- if (GetLastError() == ERROR_DELETE_PENDING)
- {
- /*
- * File has been deleted, but is not gone from the filesystem yet.
- * This can happen when some process with FILE_SHARE_DELETE has it
- * open and it will be fully removed once that handle is closed.
- * Meanwhile, we can't open it, so indicate that the file just
- * doesn't exist.
- */
- errno = ENOENT;
- return -1;
- }
-
- return r;
- }
-
- if (!GetFileAttributesEx(path, GetFileExInfoStandard, &attr))
- {
- _dosmaperr(GetLastError());
- return -1;
- }
-
- /*
- * XXX no support for large files here, but we don't do that in general on
- * Win32 yet.
- */
- buf->st_size = attr.nFileSizeLow;
-
- return 0;
-}
-
-#endif
diff --git a/src/port/win32_stat.c b/src/port/win32_stat.c
new file mode 100644
index 0000000..63fbf82
--- /dev/null
+++ b/src/port/win32_stat.c
@@ -0,0 +1,154 @@
+/*-------------------------------------------------------------------------
+ *
+ * win32_stat.c
+ * Replacements for <sys/stat.h> functions using GetFileAttributesEx on
+ * Win32.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/port/win32_stat.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifdef WIN32
+
+#include "c.h"
+#include <tchar.h>
+#if defined(__CYGWIN__)
+#include <w32api/windows.h>
+#else
+#include <windows.h>
+#endif
+
+static const unsigned __int64 EpochShift = UINT64CONST(116444736000000000);
+
+/*
+ * Converts a FILETIME struct into a 64 bit time_t
+ */
+static __time64_t
+filetime_to_time(FILETIME const * ft)
+{
+ ULARGE_INTEGER unified_ft;
+
+ unified_ft.LowPart = ft->dwLowDateTime;
+ unified_ft.HighPart = ft->dwHighDateTime;
+
+ if (unified_ft.QuadPart < EpochShift)
+ return -1;
+
+ unified_ft.QuadPart -= EpochShift;
+ unified_ft.QuadPart /= 10 * 1000 * 1000;
+
+ return ((__int64) unified_ft.HighPart) << 32 |
+ (__int64) unified_ft.LowPart;
+}
+
+/*
+ * Converts WIN32 file attributes to unix mode
+ */
+static unsigned short
+fileattr_to_unixmode(int attr, const _TCHAR * name)
+{
+ unsigned short uxmode;
+ const _TCHAR * p;
+
+ uxmode = (unsigned short)((attr & FILE_ATTRIBUTE_DIRECTORY) ?
+ (_S_IFDIR | _S_IEXEC) : (_S_IFREG));
+
+ uxmode |= (unsigned short)(attr & FILE_ATTRIBUTE_READONLY) ?
+ (_S_IREAD) : (_S_IREAD | _S_IWRITE);
+
+ if (p = _tcsrchr(name, _T('.')))
+ {
+ if (! _tcsicmp(p, _T(".exe")) ||
+ ! _tcsicmp(p, _T(".cmd")) ||
+ ! _tcsicmp(p, _T(".bat")) ||
+ ! _tcsicmp(p, _T(".com")))
+ uxmode |= _S_IEXEC;
+ }
+
+ uxmode |= (uxmode & 0700) >> 3;
+ uxmode |= (uxmode & 0700) >> 6;
+
+ return uxmode;
+}
+
+/*
+ * GetFileAttributesEx minimum supported version: Windows XP and
+ * Windows Server 2003
+ */
+int
+_pgstat64(char const *name, struct stat *buf)
+{
+ WIN32_FILE_ATTRIBUTE_DATA faData;
+
+ if (NULL == name || NULL == buf)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+
+ if (!GetFileAttributesEx(name, GetFileExInfoStandard, &faData))
+ {
+ DWORD error = GetLastError();
+ if (error== ERROR_DELETE_PENDING)
+ {
+ /*
+ * File has been deleted, but is not gone from the filesystem yet.
+ * This can happen when some process with FILE_SHARE_DELETE has it
+ * open and it will be fully removed once that handle is closed.
+ * Meanwhile, we can't open it, so indicate that the file just
+ * doesn't exist.
+ */
+ errno = ENOENT;
+ }
+ _dosmaperr(error);
+ return -1;
+ }
+
+ if (faData.ftLastWriteTime.dwLowDateTime ||
+ faData.ftLastWriteTime.dwHighDateTime)
+ buf->st_mtime = filetime_to_time(&faData.ftLastWriteTime);
+
+ if (faData.ftLastAccessTime.dwLowDateTime ||
+ faData.ftLastAccessTime.dwHighDateTime)
+ buf->st_atime = filetime_to_time(&faData.ftLastAccessTime);
+ else
+ buf->st_atime = buf->st_mtime;
+
+ if (faData.ftCreationTime.dwLowDateTime ||
+ faData.ftCreationTime.dwHighDateTime)
+ buf->st_ctime = filetime_to_time(&faData.ftCreationTime);
+ else
+ buf->st_ctime = buf->st_mtime;
+
+ buf->st_mode = fileattr_to_unixmode(faData.dwFileAttributes, name);
+ buf->st_nlink = 1;
+
+ buf->st_size = ((__int64) faData.nFileSizeHigh) << 32 |
+ (__int64)(faData.nFileSizeLow);
+
+ return 0;
+}
+
+/*
+ * Wrapper for _pgstat64
+ * GetFinalPathNameByHandle minimum supported version: Windows Vista and
+ * Windows Server 2008
+ */
+int
+_pgfstat64(int fileno, struct stat *buf)
+{
+ const char filepath[_MAX_PATH];
+
+ if(GetFinalPathNameByHandle((HANDLE)_get_osfhandle(fileno),
+ filepath, _MAX_PATH, VOLUME_NAME_DOS))
+ return _pgstat64(filepath, buf);
+ return -1;
+}
+
+#endif /* WIN32 */
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index e07e459..4a14b24 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -101,7 +101,7 @@ sub mkvcbuild
pg_strong_random.c pgcheckdir.c pgmkdirp.c pgsleep.c pgstrcasecmp.c
pqsignal.c mkdtemp.c qsort.c qsort_arg.c quotes.c system.c
sprompt.c strerror.c tar.c thread.c
- win32env.c win32error.c win32security.c win32setlocale.c);
+ win32env.c win32error.c win32security.c win32setlocale.c win32_stat.c);
push(@pgportfiles, 'rint.c') if ($vsVersion < '12.00');
--
2.11.0
On Fri, Jun 28, 2019 at 11:34:38PM +0200, Juan José Santamaría Flecha wrote:
I wanted to make a quick test on the previous patch. So let me state
what have I tested and what I have not: it builds and pass tests in
Windows and Cygwin, but I have not setup a MinGW environment.
Thanks. Could you attach this patch to the next commit fest? We had
many complaints with the current limitations with large files (pg_dump
syncs its result files, so that breaks on Windows actually if the dump
is larger than 2GB..), and we are going to need to do something. I
find that stuff rather hard to backpatch, but let's see.
--
Michael
On Sat, Jun 29, 2019 at 4:30 AM Michael Paquier <michael@paquier.xyz> wrote:
Thanks. Could you attach this patch to the next commit fest? We had
many complaints with the current limitations with large files (pg_dump
syncs its result files, so that breaks on Windows actually if the dump
is larger than 2GB..), and we are going to need to do something. I
find that stuff rather hard to backpatch, but let's see.
Done. [1]https://commitfest.postgresql.org/23/2189/
Regards,
Juan José Santamaría Flecha
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo.santamaria@gmail.com> writes:
On Wed, Jun 26, 2019 at 4:23 AM Michael Paquier <michael@paquier.xyz> wrote:
It seems to me that you don't need the configure changes.
The changes in configuration are meant for gcc compilations in Windows
(Cygwin and Mingw).
Directly editing the configure script is Not Done ... or at least,
such changes wouldn't survive the next correctly-done configure
update. You have to edit configure.in (or one of the sub-files in
config/) and then regenerate configure using autoconf.
It seems likely that we *don't* need or want this for Cygwin;
that should be providing a reasonable stat() emulation already.
So probably you just want to add "AC_LIBOBJ(win32_stat)" to
the stanza beginning
# Win32 (really MinGW) support
if test "$PORTNAME" = "win32"; then
AC_CHECK_FUNCS(_configthreadlocale)
AC_REPLACE_FUNCS(gettimeofday)
AC_LIBOBJ(dirmod)
I'd also recommend that stat() fill all the fields in struct stat,
even if you don't have anything better to put there than zeroes.
Otherwise you're just opening things up for random misbehavior.
I'm not in a position to comment on the details of the conversion from
GetFileAttributesEx results to struct stat, but in general this
seems like a reasonable way to proceed.
regards, tom lane
Thanks for looking into this.
On Fri, Aug 23, 2019 at 11:49 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Directly editing the configure script is Not Done ... or at least,
such changes wouldn't survive the next correctly-done configure
update. You have to edit configure.in (or one of the sub-files in
config/) and then regenerate configure using autoconf.It seems likely that we *don't* need or want this for Cygwin;
that should be providing a reasonable stat() emulation already.
So probably you just want to add "AC_LIBOBJ(win32_stat)" to
the stanza beginningI'd also recommend that stat() fill all the fields in struct stat,
even if you don't have anything better to put there than zeroes.
Otherwise you're just opening things up for random misbehavior.
Fixed.
I'm not in a position to comment on the details of the conversion from
GetFileAttributesEx results to struct stat, but in general this
seems like a reasonable way to proceed.
Actually, due to the behaviour of GetFileAttributesEx with symbolic
links I think that using GetFileInformationByHandle instead can give a
more resilient solution. Also, by using a handle we get a good test
for ERROR_DELETE_PENDING. This is the approach for the attached patch.
Regards,
Juan José Santamaría Flecha
Attachments:
0001-WIP-support-for-large-files-on-Win32-v4.patchapplication/octet-stream; name=0001-WIP-support-for-large-files-on-Win32-v4.patchDownload
From af1625f676e21d2777aa4be70893bf01dc08328d Mon Sep 17 00:00:00 2001
From: Juan Jose Santamaria Flecha <juanjo.santamaria@gmail.com>
Date: Wed, 4 Sep 2019 17:17:37 -0400
Subject: [PATCH] WIP support for large files on Win32 . Workaround for Windows
<sys/stat.h> shortcomings, reimplement fstat()/stat() on top of
GetFileInformationByHandle() and use 64 bits struct stat.
---
configure | 6 +
configure.in | 1 +
src/include/port/win32_port.h | 40 +++++--
src/port/dirmod.c | 52 ---------
src/port/win32_stat.c | 258 ++++++++++++++++++++++++++++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 2 +-
6 files changed, 294 insertions(+), 65 deletions(-)
create mode 100644 src/port/win32_stat.c
diff --git a/configure b/configure
index f14709e..81271e5 100755
--- a/configure
+++ b/configure
@@ -16102,6 +16102,12 @@ fi
esac
case " $LIBOBJS " in
+ *" win32_stat.$ac_objext "* ) ;;
+ *) LIBOBJS="$LIBOBJS win32_stat.$ac_objext"
+ ;;
+esac
+
+ case " $LIBOBJS " in
*" kill.$ac_objext "* ) ;;
*) LIBOBJS="$LIBOBJS kill.$ac_objext"
;;
diff --git a/configure.in b/configure.in
index 805cf86..e260f2a 100644
--- a/configure.in
+++ b/configure.in
@@ -1770,6 +1770,7 @@ if test "$PORTNAME" = "win32"; then
AC_CHECK_FUNCS(_configthreadlocale)
AC_REPLACE_FUNCS(gettimeofday)
AC_LIBOBJ(dirmod)
+ AC_LIBOBJ(win32_stat)
AC_LIBOBJ(kill)
AC_LIBOBJ(open)
AC_LIBOBJ(system)
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index 1cf166a..2e317b7 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -52,7 +52,12 @@
#include <direct.h>
#include <sys/utime.h> /* for non-unicode version */
#undef near
-#include <sys/stat.h> /* needed before sys/stat hacking below */
+/* needed before sys/stat hacking below */
+#define fstat microsoft_native_fstat
+#define stat microsoft_native_stat
+#include <sys/stat.h>
+#undef fstat
+#undef stat
/* Must be here to avoid conflicting with prototype in windows.h */
#define mkdir(a,b) mkdir(a)
@@ -249,20 +254,31 @@ typedef int pid_t;
* Supplement to <sys/stat.h>.
*
* We must pull in sys/stat.h before this part, else our overrides lose.
- */
-#define lstat(path, sb) stat(path, sb)
-
-/*
* stat() is not guaranteed to set the st_size field on win32, so we
- * redefine it to our own implementation that is.
+ * redefine it to our own implementation. See src/port/win32_stat.c
*
- * Some frontends don't need the size from stat, so if UNSAFE_STAT_OK
- * is defined we don't bother with this.
+ * The struct stat is 32 bit in MSVC so we redefine it as a copy of struct
+ * _stat64. This also fixes the struct size for MINGW builds.
*/
-#ifndef UNSAFE_STAT_OK
-extern int pgwin32_safestat(const char *path, struct stat *buf);
-#define stat(a,b) pgwin32_safestat(a,b)
-#endif
+struct stat /* It is actually _stat64 */
+{
+ _dev_t st_dev;
+ _ino_t st_ino;
+ unsigned short st_mode;
+ short st_nlink;
+ short st_uid;
+ short st_gid;
+ _dev_t st_rdev;
+ __int64 st_size;
+ __time64_t st_atime;
+ __time64_t st_mtime;
+ __time64_t st_ctime;
+};
+extern int _pgfstat64(int fileno, struct stat *buf);
+extern int _pgstat64(char const *name, struct stat *buf);
+#define fstat(fileno, sb) _pgfstat64(fileno, sb)
+#define stat(path, sb) _pgstat64(path, sb)
+#define lstat(path, sb) _pgstat64(path, sb)
/* These macros are not provided by older MinGW, nor by MSVC */
#ifndef S_IRUSR
diff --git a/src/port/dirmod.c b/src/port/dirmod.c
index d793240..ec4b47f 100644
--- a/src/port/dirmod.c
+++ b/src/port/dirmod.c
@@ -353,55 +353,3 @@ pgwin32_is_junction(const char *path)
return ((attr & FILE_ATTRIBUTE_REPARSE_POINT) == FILE_ATTRIBUTE_REPARSE_POINT);
}
#endif /* defined(WIN32) && !defined(__CYGWIN__) */
-
-
-#if defined(WIN32) && !defined(__CYGWIN__)
-
-#undef stat
-
-/*
- * The stat() function in win32 is not guaranteed to update the st_size
- * field when run. So we define our own version that uses the Win32 API
- * to update this field.
- */
-int
-pgwin32_safestat(const char *path, struct stat *buf)
-{
- int r;
- WIN32_FILE_ATTRIBUTE_DATA attr;
-
- r = stat(path, buf);
- if (r < 0)
- {
- if (GetLastError() == ERROR_DELETE_PENDING)
- {
- /*
- * File has been deleted, but is not gone from the filesystem yet.
- * This can happen when some process with FILE_SHARE_DELETE has it
- * open and it will be fully removed once that handle is closed.
- * Meanwhile, we can't open it, so indicate that the file just
- * doesn't exist.
- */
- errno = ENOENT;
- return -1;
- }
-
- return r;
- }
-
- if (!GetFileAttributesEx(path, GetFileExInfoStandard, &attr))
- {
- _dosmaperr(GetLastError());
- return -1;
- }
-
- /*
- * XXX no support for large files here, but we don't do that in general on
- * Win32 yet.
- */
- buf->st_size = attr.nFileSizeLow;
-
- return 0;
-}
-
-#endif
diff --git a/src/port/win32_stat.c b/src/port/win32_stat.c
new file mode 100644
index 0000000..b2dcea4
--- /dev/null
+++ b/src/port/win32_stat.c
@@ -0,0 +1,258 @@
+/*-------------------------------------------------------------------------
+ *
+ * win32_stat.c
+ * Replacements for <sys/stat.h> functions using GetFileInformationByHandle
+ * on Win32.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/port/win32_stat.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifdef WIN32
+
+#include "c.h"
+#include <windows.h>
+/*
+ * NtQueryInformationFile is used when the Windows version is older than
+ * Vista / Server 2008. We load it from the ntdll library.
+ */
+#if _WIN32_WINNT < 0x0600
+#include <winternl.h>
+#endif
+
+static const unsigned __int64 EpochShift = UINT64CONST(116444736000000000);
+
+/*
+ * Converts a FILETIME struct into a 64 bit time_t.
+ */
+static __time64_t
+filetime_to_time(FILETIME const *ft)
+{
+ ULARGE_INTEGER unified_ft = { 0 };
+
+ unified_ft.LowPart = ft->dwLowDateTime;
+ unified_ft.HighPart = ft->dwHighDateTime;
+
+ if (unified_ft.QuadPart < EpochShift)
+ return -1;
+
+ unified_ft.QuadPart -= EpochShift;
+ unified_ft.QuadPart /= 10 * 1000 * 1000;
+
+ return unified_ft.QuadPart;
+}
+
+/*
+ * Converts WIN32 file attributes to unix mode.
+ * Only owner permissions are set.
+ */
+static unsigned short
+fileattr_to_unixmode(int attr)
+{
+ unsigned short uxmode = 0;
+
+ uxmode |= (unsigned short)((attr & FILE_ATTRIBUTE_DIRECTORY) ?
+ (_S_IFDIR) : (_S_IFREG));
+
+ uxmode |= (unsigned short)(attr & FILE_ATTRIBUTE_READONLY) ?
+ (_S_IREAD) : (_S_IREAD | _S_IWRITE);
+
+ /* We could simulate _S_IEXEC with CMD's PATHEXT extensions, unnecessary */
+ uxmode |= _S_IEXEC;
+
+ return uxmode;
+}
+
+/*
+ * Converts WIN32 file infomation to struct stat.
+ * GetFileInformationByHandle minimum supported version: Windows XP and
+ * Windows Server 2003.
+ */
+static int
+fileinfo_to_stat(HANDLE hFile, struct stat *buf)
+{
+ BY_HANDLE_FILE_INFORMATION fiData;
+
+ if (!GetFileInformationByHandle(hFile, &fiData))
+ {
+ errno = ENOENT;
+ return -1;
+ }
+
+ memset(buf, 0, sizeof(*buf));
+ if (fiData.ftLastWriteTime.dwLowDateTime || fiData.ftLastWriteTime.dwHighDateTime)
+ buf->st_mtime = filetime_to_time(&fiData.ftLastWriteTime);
+
+ if (fiData.ftLastAccessTime.dwLowDateTime || fiData.ftLastAccessTime.dwHighDateTime)
+ buf->st_atime = filetime_to_time(&fiData.ftLastAccessTime);
+ else
+ buf->st_atime = buf->st_mtime;
+
+ if (fiData.ftCreationTime.dwLowDateTime || fiData.ftCreationTime.dwHighDateTime)
+ buf->st_ctime = filetime_to_time(&fiData.ftCreationTime);
+ else
+ buf->st_ctime = buf->st_mtime;
+
+ buf->st_mode = fileattr_to_unixmode(fiData.dwFileAttributes);
+ buf->st_nlink = fiData.nNumberOfLinks;
+
+ buf->st_size = ((__int64) fiData.nFileSizeHigh) << 32 |
+ (__int64)(fiData.nFileSizeLow);
+ return 0;
+}
+
+#if _WIN32_WINNT < 0x0600
+#if !defined(__MINGW64_VERSION_MAJOR)
+/* MinGW includes this in <winternl.h>, but is missing in MSVC */
+typedef struct _FILE_STANDARD_INFORMATION
+{
+ LARGE_INTEGER AllocationSize;
+ LARGE_INTEGER EndOfFile;
+ ULONG NumberOfLinks;
+ BOOLEAN DeletePending;
+ BOOLEAN Directory;
+} FILE_STANDARD_INFORMATION, *PFILE_STANDARD_INFORMATION;
+#define FileStandardInformation 5
+#endif /* !defined(__MINGW64_VERSION_MAJOR) */
+
+typedef NTSTATUS (NTAPI *PFN_NTQUERYINFORMATIONFILE)
+(
+ IN HANDLE FileHandle,
+ OUT PIO_STATUS_BLOCK IoStatusBlock,
+ OUT PVOID FileInformation,
+ IN ULONG Length,
+ IN FILE_INFORMATION_CLASS FileInformationClass
+);
+static PFN_NTQUERYINFORMATIONFILE _NtQueryInformationFile = NULL;
+
+/*
+ * Load DLL file just once regardless of how many functions
+ * we load/call in it.
+ */
+static HMODULE ntdll = NULL;
+
+static void
+LoadNtdll()
+{
+ if (ntdll != NULL)
+ return;
+ ntdll = LoadLibraryEx("ntdll.dll", NULL, 0);
+}
+#endif /* _WIN32_WINNT < 0x0600 */
+
+/*
+ * We must use a handle so lstat() returns the information of the target file.
+ * As a reliable test for ERROR_DELETE_PENDING we use NtQueryInformationFile
+ * from Windows 2000 or GetFileInformationByHandleEx from Server 2008 / Vista
+ */
+int
+_pgstat64(char const *name, struct stat *buf)
+{
+ SECURITY_ATTRIBUTES sa;
+ HANDLE hFile;
+ int ret;
+#if _WIN32_WINNT < 0x0600
+ IO_STATUS_BLOCK ioStatus;
+ FILE_STANDARD_INFORMATION standardInfo;
+#else
+ FILE_STANDARD_INFO standardInfo;
+#endif
+
+ if (NULL == name || NULL == buf)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+
+ sa.nLength = sizeof(SECURITY_ATTRIBUTES);
+ sa.bInheritHandle = TRUE;
+ sa.lpSecurityDescriptor = NULL;
+ hFile = CreateFile(name, GENERIC_READ, (FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE),
+ &sa, OPEN_EXISTING,
+ FILE_FLAG_NO_BUFFERING | FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,
+ NULL);
+ if (hFile == INVALID_HANDLE_VALUE)
+ {
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
+
+#if _WIN32_WINNT < 0x0600
+ if (_NtQueryInformationFile == NULL)
+ {
+ LoadNtdll();
+ if(ntdll == NULL)
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ /* Do not return ENOENT */
+ errno = ENOMSG;
+ return -1;
+ }
+
+ _NtQueryInformationFile = (PFN_NTQUERYINFORMATIONFILE)
+ GetProcAddress(ntdll, "NtQueryInformationFile");
+ if (_NtQueryInformationFile == NULL)
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ errno = ENOMSG;
+ return -1;
+ }
+ }
+
+ memset(&standardInfo, 0, sizeof(standardInfo));
+ if (!NT_SUCCESS(_NtQueryInformationFile(hFile, &ioStatus, &standardInfo, sizeof(standardInfo),
+ FileStandardInformation)))
+#else
+ if (!GetFileInformationByHandleEx(hFile, FileStandardInfo, &standardInfo, sizeof(standardInfo)))
+#endif /* _WIN32_WINNT < 0x0600 */
+ {
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
+ if (standardInfo.DeletePending)
+ {
+ /*
+ * File has been deleted, but is not gone from the filesystem yet.
+ * This can happen when some process with FILE_SHARE_DELETE has it
+ * open and it will be fully removed once that handle is closed.
+ * Meanwhile, we can't open it, so indicate that the file just
+ * doesn't exist.
+ */
+ DWORD error = ERROR_DELETE_PENDING;
+ _dosmaperr(error);
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
+
+ ret = fileinfo_to_stat(hFile, buf);
+ CloseHandle(hFile);
+ return ret;
+}
+
+/* Shares _pgstat64() logic */
+int
+_pgfstat64(int fileno, struct stat *buf)
+{
+ HANDLE hFile = (HANDLE)_get_osfhandle(fileno);
+
+ if (INVALID_HANDLE_VALUE == hFile || NULL == buf)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+
+ return fileinfo_to_stat(hFile, buf);
+}
+
+#endif /* WIN32 */
\ No newline at end of file
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 2eab635..b2bf321 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -101,7 +101,7 @@ sub mkvcbuild
pg_strong_random.c pgcheckdir.c pgmkdirp.c pgsleep.c pgstrcasecmp.c
pqsignal.c mkdtemp.c qsort.c qsort_arg.c quotes.c system.c
sprompt.c strerror.c tar.c thread.c
- win32env.c win32error.c win32security.c win32setlocale.c);
+ win32env.c win32error.c win32security.c win32setlocale.c win32_stat.c);
push(@pgportfiles, 'rint.c') if ($vsVersion < '12.00');
--
2.11.0
Hi - is this likely to be applied to an upcoming release? / How does a novice apply a patch..?
Thanks
-----Original Message-----
From: Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>
Sent: 04 September 2019 22:48
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Michael Paquier <michael@paquier.xyz>; williamedwinallen@live.com; pgsql-bugs@lists.postgresql.org; Magnus Hagander <magnus@hagander.net>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: BUG #15858: could not stat file - over 4GB
Thanks for looking into this.
On Fri, Aug 23, 2019 at 11:49 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Directly editing the configure script is Not Done ... or at least,
such changes wouldn't survive the next correctly-done configure
update. You have to edit configure.in (or one of the sub-files in
config/) and then regenerate configure using autoconf.It seems likely that we *don't* need or want this for Cygwin; that
should be providing a reasonable stat() emulation already.
So probably you just want to add "AC_LIBOBJ(win32_stat)" to the stanza
beginningI'd also recommend that stat() fill all the fields in struct stat,
even if you don't have anything better to put there than zeroes.
Otherwise you're just opening things up for random misbehavior.
Fixed.
I'm not in a position to comment on the details of the conversion from
GetFileAttributesEx results to struct stat, but in general this seems
like a reasonable way to proceed.
Actually, due to the behaviour of GetFileAttributesEx with symbolic links I think that using GetFileInformationByHandle instead can give a more resilient solution. Also, by using a handle we get a good test for ERROR_DELETE_PENDING. This is the approach for the attached patch.
Regards,
Juan José Santamaría Flecha
On Mon, Oct 28, 2019 at 3:29 PM william allen <williamedwinallen@live.com>
wrote:
Hi - is this likely to be applied to an upcoming release? / How does a
novice apply a patch..?
At this moment is missing review, so it is probably far from being
commitable. Any attention is appreciated and might help pushing it forward.
As a personal note, I have to check that is still applies before the
upcoming commitfest.
As for applying this patch you would need a Windows development
environment. I would recommend Visual Studio as a starting point [1]https://www.postgresql.org/docs/current/install-windows.html. You
also have a very visual guide in the wiki [2]https://wiki.postgresql.org/wiki/Working_With_VisualStudio.
[1]: https://www.postgresql.org/docs/current/install-windows.html
[2]: https://wiki.postgresql.org/wiki/Working_With_VisualStudio
Regards,
Juan José Santamaría Flecha
The following review has been posted through the commitfest application:
make installcheck-world: not tested
Implements feature: tested, passed
Spec compliant: not tested
Documentation: not tested
I ran into this problem when using psql.exe and copy command.
I have checked out 11.6-release tarball and applied the patch.
The patch does not apply cleanly, but can be easily modified to apply. See Note 1.
After applying the patch I built using "build psql" and ran the new psql.exe binary.
In order to test I have done the following:
Against a PostgreSQL 11 server run two commands:
"COPY public.table FROM 'C:/file'" and "\copy public.table FROM 'C:/file'"
The first one runs in the context of the server, and does not work. It aborts with an error saying "cannot stat file", as expected.
The seconds on runs in the context of the new binary and does work. It copies data as expected.
Note 1:
src/tools/msvc/Mkvcbuild.pm should be
- sprompt.c strerror.c tar.c thread.c getopt.c getopt_long.c dirent.c
- win32env.c win32error.c win32security.c win32setlocale.c);
+ sprompt.c tar.c thread.c getopt.c getopt_long.c dirent.c
+ win32env.c win32error.c win32security.c win32setlocale.c win32_stat.c);
The new status of this patch is: Waiting on Author
On Wed, Feb 5, 2020 at 12:47 PM Emil Iggland <emil@iggland.com> wrote:
The following review has been posted through the commitfest application:
make installcheck-world: not tested
Implements feature: tested, passed
Spec compliant: not tested
Documentation: not tested
The latest version of this patch could benefit from an update. Please find
attached a new version.
Most changes are cosmetic, but they have been more extensive than a simple
rebase so I am changing the status back to 'needs review'.
To summarize those changes:
- Rename 'win32_stat.c' file to 'win32stat.c', as a better match of
project files.
- Improve indentation and comments.
- Remove cruft about old Windows versions.
Regards,
Juan José Santamaría Flecha
Attachments:
0001-Support-for-large-files-on-Win32-v5.patchapplication/octet-stream; name=0001-Support-for-large-files-on-Win32-v5.patchDownload
From 028000a85531190f2028985397a1f050a42e9a0b Mon Sep 17 00:00:00 2001
From: Juan Jose Santamaria Flecha <juanjo.santamaria@gmail.com>
Date: Fri, 28 Feb 2020 04:08:58 -0500
Subject: [PATCH] Support for large files on Win32. Workaround for Windows
<sys/stat.h> shortcomings, reimplement fstat()/stat() on top of
GetFileInformationByHandle() and use 64 bits struct stat.
---
configure | 6 +
configure.in | 1 +
src/include/port/win32_port.h | 41 +++++--
src/port/dirmod.c | 52 ---------
src/port/win32stat.c | 254 ++++++++++++++++++++++++++++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 2 +-
6 files changed, 291 insertions(+), 65 deletions(-)
create mode 100644 src/port/win32stat.c
diff --git a/configure b/configure
index d2c74e5..3acea3f 100755
--- a/configure
+++ b/configure
@@ -15822,6 +15822,12 @@ esac
;;
esac
+ case " $LIBOBJS " in
+ *" win32stat.$ac_objext "* ) ;;
+ *) LIBOBJS="$LIBOBJS win32stat.$ac_objext"
+ ;;
+esac
+
$as_echo "#define HAVE_SYMLINK 1" >>confdefs.h
diff --git a/configure.in b/configure.in
index 0b0a871..f9c73f6 100644
--- a/configure.in
+++ b/configure.in
@@ -1772,6 +1772,7 @@ if test "$PORTNAME" = "win32"; then
AC_LIBOBJ(win32error)
AC_LIBOBJ(win32security)
AC_LIBOBJ(win32setlocale)
+ AC_LIBOBJ(win32stat)
AC_DEFINE([HAVE_SYMLINK], 1,
[Define to 1 if you have the `symlink' function.])
AC_CHECK_TYPES(MINIDUMP_TYPE, [pgac_minidump_type=yes], [pgac_minidump_type=no], [
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index 8b6576b..cca7cdd 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -51,7 +51,12 @@
#include <signal.h>
#include <direct.h>
#undef near
-#include <sys/stat.h> /* needed before sys/stat hacking below */
+/* needed before sys/stat hacking below */
+#define fstat microsoft_native_fstat
+#define stat microsoft_native_stat
+#include <sys/stat.h>
+#undef fstat
+#undef stat
/* Must be here to avoid conflicting with prototype in windows.h */
#define mkdir(a,b) mkdir(a)
@@ -240,20 +245,32 @@ typedef int pid_t;
* Supplement to <sys/stat.h>.
*
* We must pull in sys/stat.h before this part, else our overrides lose.
- */
-#define lstat(path, sb) stat(path, sb)
-
-/*
+ *
* stat() is not guaranteed to set the st_size field on win32, so we
- * redefine it to our own implementation that is.
+ * redefine it to our own implementation. See src/port/win32stat.c.
*
- * Some frontends don't need the size from stat, so if UNSAFE_STAT_OK
- * is defined we don't bother with this.
+ * The struct stat is 32 bit in MSVC, so we redefine it as a copy of
+ * struct _stat64. This also fixes the struct size for MINGW builds.
*/
-#ifndef UNSAFE_STAT_OK
-extern int pgwin32_safestat(const char *path, struct stat *buf);
-#define stat(a,b) pgwin32_safestat(a,b)
-#endif
+struct stat /* It is actually _stat64 */
+{
+ _dev_t st_dev;
+ _ino_t st_ino;
+ unsigned short st_mode;
+ short st_nlink;
+ short st_uid;
+ short st_gid;
+ _dev_t st_rdev;
+ __int64 st_size;
+ __time64_t st_atime;
+ __time64_t st_mtime;
+ __time64_t st_ctime;
+};
+extern int _pgfstat64(int fileno, struct stat *buf);
+extern int _pgstat64(const char *name, struct stat *buf);
+#define fstat(fileno, sb) _pgfstat64(fileno, sb)
+#define stat(path, sb) _pgstat64(path, sb)
+#define lstat(path, sb) _pgstat64(path, sb)
/* These macros are not provided by older MinGW, nor by MSVC */
#ifndef S_IRUSR
diff --git a/src/port/dirmod.c b/src/port/dirmod.c
index e22a41c..8979f10 100644
--- a/src/port/dirmod.c
+++ b/src/port/dirmod.c
@@ -353,55 +353,3 @@ pgwin32_is_junction(const char *path)
return ((attr & FILE_ATTRIBUTE_REPARSE_POINT) == FILE_ATTRIBUTE_REPARSE_POINT);
}
#endif /* defined(WIN32) && !defined(__CYGWIN__) */
-
-
-#if defined(WIN32) && !defined(__CYGWIN__)
-
-#undef stat
-
-/*
- * The stat() function in win32 is not guaranteed to update the st_size
- * field when run. So we define our own version that uses the Win32 API
- * to update this field.
- */
-int
-pgwin32_safestat(const char *path, struct stat *buf)
-{
- int r;
- WIN32_FILE_ATTRIBUTE_DATA attr;
-
- r = stat(path, buf);
- if (r < 0)
- {
- if (GetLastError() == ERROR_DELETE_PENDING)
- {
- /*
- * File has been deleted, but is not gone from the filesystem yet.
- * This can happen when some process with FILE_SHARE_DELETE has it
- * open and it will be fully removed once that handle is closed.
- * Meanwhile, we can't open it, so indicate that the file just
- * doesn't exist.
- */
- errno = ENOENT;
- return -1;
- }
-
- return r;
- }
-
- if (!GetFileAttributesEx(path, GetFileExInfoStandard, &attr))
- {
- _dosmaperr(GetLastError());
- return -1;
- }
-
- /*
- * XXX no support for large files here, but we don't do that in general on
- * Win32 yet.
- */
- buf->st_size = attr.nFileSizeLow;
-
- return 0;
-}
-
-#endif
diff --git a/src/port/win32stat.c b/src/port/win32stat.c
new file mode 100644
index 0000000..a607797
--- /dev/null
+++ b/src/port/win32stat.c
@@ -0,0 +1,254 @@
+/*-------------------------------------------------------------------------
+ *
+ * win32stat.c
+ * Replacements for <sys/stat.h> functions using GetFileInformationByHandle
+ * on Win32.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/port/win32stat.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifdef WIN32
+
+#include "c.h"
+#include <windows.h>
+
+/*
+ * In order to support MinGW we use NtQueryInformationFile as an alternative
+ * for GetFileInformationByHandleEx. It is loaded from the ntdll library.
+ */
+#if _WIN32_WINNT < 0x0600
+#include <winternl.h>
+
+typedef NTSTATUS (NTAPI *PFN_NTQUERYINFORMATIONFILE)
+(
+ IN HANDLE FileHandle,
+ OUT PIO_STATUS_BLOCK IoStatusBlock,
+ OUT PVOID FileInformation,
+ IN ULONG Length,
+ IN FILE_INFORMATION_CLASS FileInformationClass
+);
+
+static PFN_NTQUERYINFORMATIONFILE _NtQueryInformationFile = NULL;
+
+/*
+ * Load DLL file just once regardless of how many functions we load/call in it.
+ */
+static HMODULE ntdll = NULL;
+
+static void
+LoadNtdll(void)
+{
+ if (ntdll != NULL)
+ return;
+ ntdll = LoadLibraryEx("ntdll.dll", NULL, 0);
+}
+#endif /* _WIN32_WINNT < 0x0600 */
+
+static const unsigned __int64 EpochShift = UINT64CONST(116444736000000000);
+
+/*
+ * Converts a FILETIME struct into a 64 bit time_t.
+ */
+static __time64_t
+filetime_to_time(FILETIME const *ft)
+{
+ ULARGE_INTEGER unified_ft = {0};
+
+ unified_ft.LowPart = ft->dwLowDateTime;
+ unified_ft.HighPart = ft->dwHighDateTime;
+
+ if (unified_ft.QuadPart < EpochShift)
+ return -1;
+
+ unified_ft.QuadPart -= EpochShift;
+ unified_ft.QuadPart /= 10 * 1000 * 1000;
+
+ return unified_ft.QuadPart;
+}
+
+/*
+ * Converts WIN32 file attributes to unix mode.
+ * Only owner permissions are set.
+ */
+static unsigned short
+fileattr_to_unixmode(int attr)
+{
+ unsigned short uxmode = 0;
+
+ uxmode |= (unsigned short)((attr & FILE_ATTRIBUTE_DIRECTORY) ?
+ (_S_IFDIR) : (_S_IFREG));
+
+ uxmode |= (unsigned short)(attr & FILE_ATTRIBUTE_READONLY) ?
+ (_S_IREAD) : (_S_IREAD | _S_IWRITE);
+
+ /* There is no need to simulate _S_IEXEC using CMD's PATHEXT extensions */
+ uxmode |= _S_IEXEC;
+
+ return uxmode;
+}
+
+/*
+ * Converts WIN32 file infomation to struct stat.
+ * GetFileInformationByHandle minimum supported version: Windows XP and
+ * Windows Server 2003.
+ */
+static int
+fileinfo_to_stat(HANDLE hFile, struct stat *buf)
+{
+ BY_HANDLE_FILE_INFORMATION fiData;
+
+ if (!GetFileInformationByHandle(hFile, &fiData))
+ {
+ _dosmaperr(GetLastError());
+ return -1;
+ }
+
+ memset(buf, 0, sizeof(*buf));
+ if (fiData.ftLastWriteTime.dwLowDateTime ||
+ fiData.ftLastWriteTime.dwHighDateTime)
+ buf->st_mtime = filetime_to_time(&fiData.ftLastWriteTime);
+
+ if (fiData.ftLastAccessTime.dwLowDateTime ||
+ fiData.ftLastAccessTime.dwHighDateTime)
+ buf->st_atime = filetime_to_time(&fiData.ftLastAccessTime);
+ else
+ buf->st_atime = buf->st_mtime;
+
+ if (fiData.ftCreationTime.dwLowDateTime ||
+ fiData.ftCreationTime.dwHighDateTime)
+ buf->st_ctime = filetime_to_time(&fiData.ftCreationTime);
+ else
+ buf->st_ctime = buf->st_mtime;
+
+ buf->st_mode = fileattr_to_unixmode(fiData.dwFileAttributes);
+ buf->st_nlink = fiData.nNumberOfLinks;
+
+ buf->st_size = ((__int64) fiData.nFileSizeHigh) << 32 |
+ (__int64)(fiData.nFileSizeLow);
+ return 0;
+}
+
+/*
+ * We must use a handle so lstat() returns the information of the target file.
+ * As a reliable test for ERROR_DELETE_PENDING we use NtQueryInformationFile
+ * from Windows 2000 or GetFileInformationByHandleEx from Server 2008 / Vista.
+ */
+int
+_pgstat64(const char *name, struct stat *buf)
+{
+ SECURITY_ATTRIBUTES sa;
+ HANDLE hFile;
+ int ret;
+#if _WIN32_WINNT < 0x0600
+ IO_STATUS_BLOCK ioStatus;
+ FILE_STANDARD_INFORMATION standardInfo;
+#else
+ FILE_STANDARD_INFO standardInfo;
+#endif
+
+ if (name == NULL || buf == NULL)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+ /* fast not exists */
+ if (GetFileAttributes(name) == INVALID_FILE_ATTRIBUTES)
+ {
+ errno = ENOENT;
+ return -1;
+ }
+
+ /* Get a file handle as lightweight as we can */
+ sa.nLength = sizeof(SECURITY_ATTRIBUTES);
+ sa.bInheritHandle = TRUE;
+ sa.lpSecurityDescriptor = NULL;
+ hFile = CreateFile(name, GENERIC_READ,
+ (FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE),
+ &sa, OPEN_EXISTING,
+ (FILE_FLAG_NO_BUFFERING | FILE_FLAG_BACKUP_SEMANTICS |
+ FILE_FLAG_OVERLAPPED), NULL);
+ if (hFile == INVALID_HANDLE_VALUE)
+ {
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
+
+ memset(&standardInfo, 0, sizeof(standardInfo));
+#if _WIN32_WINNT < 0x0600
+ if (_NtQueryInformationFile == NULL)
+ {
+ LoadNtdll();
+ if(ntdll == NULL)
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ return -1;
+ }
+
+ _NtQueryInformationFile = (PFN_NTQUERYINFORMATIONFILE)
+ GetProcAddress(ntdll, "NtQueryInformationFile");
+ if (_NtQueryInformationFile == NULL)
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ return -1;
+ }
+ }
+
+ if (!NT_SUCCESS(_NtQueryInformationFile(hFile, &ioStatus, &standardInfo,
+ sizeof(standardInfo), FileStandardInformation)))
+#else
+ if (!GetFileInformationByHandleEx(hFile, FileStandardInfo, &standardInfo,
+ sizeof(standardInfo)))
+#endif /* _WIN32_WINNT < 0x0600 */
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ return -1;
+ }
+ if (standardInfo.DeletePending)
+ {
+ /*
+ * File has been deleted, but is not gone from the filesystem yet.
+ * This can happen when some process with FILE_SHARE_DELETE has it
+ * open and it will be fully removed once that handle is closed.
+ * Meanwhile, we can't open it, so indicate that the file just
+ * doesn't exist.
+ */
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
+
+ ret = fileinfo_to_stat(hFile, buf);
+ CloseHandle(hFile);
+ return ret;
+}
+
+/*
+ * Since we already have a file handle there is no need to check for
+ * ERROR_DELETE_PENDING.
+ */
+int
+_pgfstat64(int fileno, struct stat *buf)
+{
+ HANDLE hFile = (HANDLE)_get_osfhandle(fileno);
+
+ if (hFile == INVALID_HANDLE_VALUE || buf == NULL)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+
+ return fileinfo_to_stat(hFile, buf);
+}
+
+#endif /* WIN32 */
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 834c2c3..6ff0783 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -103,7 +103,7 @@ sub mkvcbuild
pg_strong_random.c pgcheckdir.c pgmkdirp.c pgsleep.c pgstrcasecmp.c
pqsignal.c mkdtemp.c qsort.c qsort_arg.c quotes.c system.c
sprompt.c strerror.c tar.c thread.c
- win32env.c win32error.c win32security.c win32setlocale.c);
+ win32env.c win32error.c win32security.c win32setlocale.c win32stat.c);
push(@pgportfiles, 'strtof.c') if ($vsVersion < '14.00');
--
2.11.0
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo.santamaria@gmail.com> writes:
The latest version of this patch could benefit from an update. Please find
attached a new version.
The cfbot thinks this doesn't compile on Windows [1]https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.81541. Looks like perhaps
a missing-#include problem?
regards, tom lane
[1]: https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.81541
On Sat, Feb 29, 2020 at 12:44 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
The cfbot thinks this doesn't compile on Windows [1]. Looks like perhaps
a missing-#include problem?
The define logic for _WIN32_WINNT includes testing of _MSC_VER, and is not
a proper choice for MSVC 2013 as the cfbot is showing.
Please find attached a new version addressing this issue.
Regards,
Juan José Santamaría Flecha
Show quoted text
Attachments:
0001-Support-for-large-files-on-Win32-v6.patchapplication/octet-stream; name=0001-Support-for-large-files-on-Win32-v6.patchDownload
From ca3d2cc8d86b64b4863448ba45bd8386d4d49d04 Mon Sep 17 00:00:00 2001
From: Juan Jose Santamaria Flecha <juanjo.santamaria@gmail.com>
Date: Fri, 28 Feb 2020 19:22:35 -0500
Subject: [PATCH] Support for large files on Win32. Workaround for Windows
<sys/stat.h> shortcomings, reimplement fstat()/stat() on top of
GetFileInformationByHandle() and use 64 bits struct stat.
---
configure | 6 +
configure.in | 1 +
src/include/port/win32_port.h | 41 +++++--
src/port/dirmod.c | 52 ---------
src/port/win32stat.c | 254 ++++++++++++++++++++++++++++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 2 +-
6 files changed, 291 insertions(+), 65 deletions(-)
create mode 100644 src/port/win32stat.c
diff --git a/configure b/configure
index d2c74e5..3acea3f 100755
--- a/configure
+++ b/configure
@@ -15822,6 +15822,12 @@ esac
;;
esac
+ case " $LIBOBJS " in
+ *" win32stat.$ac_objext "* ) ;;
+ *) LIBOBJS="$LIBOBJS win32stat.$ac_objext"
+ ;;
+esac
+
$as_echo "#define HAVE_SYMLINK 1" >>confdefs.h
diff --git a/configure.in b/configure.in
index 0b0a871..f9c73f6 100644
--- a/configure.in
+++ b/configure.in
@@ -1772,6 +1772,7 @@ if test "$PORTNAME" = "win32"; then
AC_LIBOBJ(win32error)
AC_LIBOBJ(win32security)
AC_LIBOBJ(win32setlocale)
+ AC_LIBOBJ(win32stat)
AC_DEFINE([HAVE_SYMLINK], 1,
[Define to 1 if you have the `symlink' function.])
AC_CHECK_TYPES(MINIDUMP_TYPE, [pgac_minidump_type=yes], [pgac_minidump_type=no], [
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index 8b6576b..cca7cdd 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -51,7 +51,12 @@
#include <signal.h>
#include <direct.h>
#undef near
-#include <sys/stat.h> /* needed before sys/stat hacking below */
+/* needed before sys/stat hacking below */
+#define fstat microsoft_native_fstat
+#define stat microsoft_native_stat
+#include <sys/stat.h>
+#undef fstat
+#undef stat
/* Must be here to avoid conflicting with prototype in windows.h */
#define mkdir(a,b) mkdir(a)
@@ -240,20 +245,32 @@ typedef int pid_t;
* Supplement to <sys/stat.h>.
*
* We must pull in sys/stat.h before this part, else our overrides lose.
- */
-#define lstat(path, sb) stat(path, sb)
-
-/*
+ *
* stat() is not guaranteed to set the st_size field on win32, so we
- * redefine it to our own implementation that is.
+ * redefine it to our own implementation. See src/port/win32stat.c.
*
- * Some frontends don't need the size from stat, so if UNSAFE_STAT_OK
- * is defined we don't bother with this.
+ * The struct stat is 32 bit in MSVC, so we redefine it as a copy of
+ * struct _stat64. This also fixes the struct size for MINGW builds.
*/
-#ifndef UNSAFE_STAT_OK
-extern int pgwin32_safestat(const char *path, struct stat *buf);
-#define stat(a,b) pgwin32_safestat(a,b)
-#endif
+struct stat /* It is actually _stat64 */
+{
+ _dev_t st_dev;
+ _ino_t st_ino;
+ unsigned short st_mode;
+ short st_nlink;
+ short st_uid;
+ short st_gid;
+ _dev_t st_rdev;
+ __int64 st_size;
+ __time64_t st_atime;
+ __time64_t st_mtime;
+ __time64_t st_ctime;
+};
+extern int _pgfstat64(int fileno, struct stat *buf);
+extern int _pgstat64(const char *name, struct stat *buf);
+#define fstat(fileno, sb) _pgfstat64(fileno, sb)
+#define stat(path, sb) _pgstat64(path, sb)
+#define lstat(path, sb) _pgstat64(path, sb)
/* These macros are not provided by older MinGW, nor by MSVC */
#ifndef S_IRUSR
diff --git a/src/port/dirmod.c b/src/port/dirmod.c
index e22a41c..8979f10 100644
--- a/src/port/dirmod.c
+++ b/src/port/dirmod.c
@@ -353,55 +353,3 @@ pgwin32_is_junction(const char *path)
return ((attr & FILE_ATTRIBUTE_REPARSE_POINT) == FILE_ATTRIBUTE_REPARSE_POINT);
}
#endif /* defined(WIN32) && !defined(__CYGWIN__) */
-
-
-#if defined(WIN32) && !defined(__CYGWIN__)
-
-#undef stat
-
-/*
- * The stat() function in win32 is not guaranteed to update the st_size
- * field when run. So we define our own version that uses the Win32 API
- * to update this field.
- */
-int
-pgwin32_safestat(const char *path, struct stat *buf)
-{
- int r;
- WIN32_FILE_ATTRIBUTE_DATA attr;
-
- r = stat(path, buf);
- if (r < 0)
- {
- if (GetLastError() == ERROR_DELETE_PENDING)
- {
- /*
- * File has been deleted, but is not gone from the filesystem yet.
- * This can happen when some process with FILE_SHARE_DELETE has it
- * open and it will be fully removed once that handle is closed.
- * Meanwhile, we can't open it, so indicate that the file just
- * doesn't exist.
- */
- errno = ENOENT;
- return -1;
- }
-
- return r;
- }
-
- if (!GetFileAttributesEx(path, GetFileExInfoStandard, &attr))
- {
- _dosmaperr(GetLastError());
- return -1;
- }
-
- /*
- * XXX no support for large files here, but we don't do that in general on
- * Win32 yet.
- */
- buf->st_size = attr.nFileSizeLow;
-
- return 0;
-}
-
-#endif
diff --git a/src/port/win32stat.c b/src/port/win32stat.c
new file mode 100644
index 0000000..0dcbe03
--- /dev/null
+++ b/src/port/win32stat.c
@@ -0,0 +1,254 @@
+/*-------------------------------------------------------------------------
+ *
+ * win32stat.c
+ * Replacements for <sys/stat.h> functions using GetFileInformationByHandle
+ * on Win32.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/port/win32stat.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifdef WIN32
+
+#include "c.h"
+#include <windows.h>
+
+/*
+ * In order to support MinGW we use NtQueryInformationFile as an alternative
+ * for GetFileInformationByHandleEx. It is loaded from the ntdll library.
+ */
+#if defined(__MINGW32__) || defined(__MINGW64__)
+#include <winternl.h>
+
+typedef NTSTATUS (NTAPI *PFN_NTQUERYINFORMATIONFILE)
+(
+ IN HANDLE FileHandle,
+ OUT PIO_STATUS_BLOCK IoStatusBlock,
+ OUT PVOID FileInformation,
+ IN ULONG Length,
+ IN FILE_INFORMATION_CLASS FileInformationClass
+);
+
+static PFN_NTQUERYINFORMATIONFILE _NtQueryInformationFile = NULL;
+
+/*
+ * Load DLL file just once regardless of how many functions we load/call in it.
+ */
+static HMODULE ntdll = NULL;
+
+static void
+LoadNtdll(void)
+{
+ if (ntdll != NULL)
+ return;
+ ntdll = LoadLibraryEx("ntdll.dll", NULL, 0);
+}
+#endif /* defined(__MINGW32__) || defined(__MINGW64__) */
+
+static const unsigned __int64 EpochShift = UINT64CONST(116444736000000000);
+
+/*
+ * Converts a FILETIME struct into a 64 bit time_t.
+ */
+static __time64_t
+filetime_to_time(FILETIME const *ft)
+{
+ ULARGE_INTEGER unified_ft = {0};
+
+ unified_ft.LowPart = ft->dwLowDateTime;
+ unified_ft.HighPart = ft->dwHighDateTime;
+
+ if (unified_ft.QuadPart < EpochShift)
+ return -1;
+
+ unified_ft.QuadPart -= EpochShift;
+ unified_ft.QuadPart /= 10 * 1000 * 1000;
+
+ return unified_ft.QuadPart;
+}
+
+/*
+ * Converts WIN32 file attributes to unix mode.
+ * Only owner permissions are set.
+ */
+static unsigned short
+fileattr_to_unixmode(int attr)
+{
+ unsigned short uxmode = 0;
+
+ uxmode |= (unsigned short)((attr & FILE_ATTRIBUTE_DIRECTORY) ?
+ (_S_IFDIR) : (_S_IFREG));
+
+ uxmode |= (unsigned short)(attr & FILE_ATTRIBUTE_READONLY) ?
+ (_S_IREAD) : (_S_IREAD | _S_IWRITE);
+
+ /* There is no need to simulate _S_IEXEC using CMD's PATHEXT extensions */
+ uxmode |= _S_IEXEC;
+
+ return uxmode;
+}
+
+/*
+ * Converts WIN32 file infomation to struct stat.
+ * GetFileInformationByHandle minimum supported version: Windows XP and
+ * Windows Server 2003.
+ */
+static int
+fileinfo_to_stat(HANDLE hFile, struct stat *buf)
+{
+ BY_HANDLE_FILE_INFORMATION fiData;
+
+ if (!GetFileInformationByHandle(hFile, &fiData))
+ {
+ _dosmaperr(GetLastError());
+ return -1;
+ }
+
+ memset(buf, 0, sizeof(*buf));
+ if (fiData.ftLastWriteTime.dwLowDateTime ||
+ fiData.ftLastWriteTime.dwHighDateTime)
+ buf->st_mtime = filetime_to_time(&fiData.ftLastWriteTime);
+
+ if (fiData.ftLastAccessTime.dwLowDateTime ||
+ fiData.ftLastAccessTime.dwHighDateTime)
+ buf->st_atime = filetime_to_time(&fiData.ftLastAccessTime);
+ else
+ buf->st_atime = buf->st_mtime;
+
+ if (fiData.ftCreationTime.dwLowDateTime ||
+ fiData.ftCreationTime.dwHighDateTime)
+ buf->st_ctime = filetime_to_time(&fiData.ftCreationTime);
+ else
+ buf->st_ctime = buf->st_mtime;
+
+ buf->st_mode = fileattr_to_unixmode(fiData.dwFileAttributes);
+ buf->st_nlink = fiData.nNumberOfLinks;
+
+ buf->st_size = ((__int64) fiData.nFileSizeHigh) << 32 |
+ (__int64)(fiData.nFileSizeLow);
+ return 0;
+}
+
+/*
+ * We must use a handle so lstat() returns the information of the target file.
+ * As a reliable test for ERROR_DELETE_PENDING we use NtQueryInformationFile
+ * from Windows 2000 or GetFileInformationByHandleEx from Server 2008 / Vista.
+ */
+int
+_pgstat64(const char *name, struct stat *buf)
+{
+ SECURITY_ATTRIBUTES sa;
+ HANDLE hFile;
+ int ret;
+#if defined(__MINGW32__) || defined(__MINGW64__)
+ IO_STATUS_BLOCK ioStatus;
+ FILE_STANDARD_INFORMATION standardInfo;
+#else
+ FILE_STANDARD_INFO standardInfo;
+#endif
+
+ if (name == NULL || buf == NULL)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+ /* fast not exists */
+ if (GetFileAttributes(name) == INVALID_FILE_ATTRIBUTES)
+ {
+ errno = ENOENT;
+ return -1;
+ }
+
+ /* Get a file handle as lightweight as we can */
+ sa.nLength = sizeof(SECURITY_ATTRIBUTES);
+ sa.bInheritHandle = TRUE;
+ sa.lpSecurityDescriptor = NULL;
+ hFile = CreateFile(name, GENERIC_READ,
+ (FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE),
+ &sa, OPEN_EXISTING,
+ (FILE_FLAG_NO_BUFFERING | FILE_FLAG_BACKUP_SEMANTICS |
+ FILE_FLAG_OVERLAPPED), NULL);
+ if (hFile == INVALID_HANDLE_VALUE)
+ {
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
+
+ memset(&standardInfo, 0, sizeof(standardInfo));
+#if defined(__MINGW32__) || defined(__MINGW64__)
+ if (_NtQueryInformationFile == NULL)
+ {
+ LoadNtdll();
+ if(ntdll == NULL)
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ return -1;
+ }
+
+ _NtQueryInformationFile = (PFN_NTQUERYINFORMATIONFILE)
+ GetProcAddress(ntdll, "NtQueryInformationFile");
+ if (_NtQueryInformationFile == NULL)
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ return -1;
+ }
+ }
+
+ if (!NT_SUCCESS(_NtQueryInformationFile(hFile, &ioStatus, &standardInfo,
+ sizeof(standardInfo), FileStandardInformation)))
+#else
+ if (!GetFileInformationByHandleEx(hFile, FileStandardInfo, &standardInfo,
+ sizeof(standardInfo)))
+#endif /* defined(__MINGW32__) || defined(__MINGW64__) */
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ return -1;
+ }
+ if (standardInfo.DeletePending)
+ {
+ /*
+ * File has been deleted, but is not gone from the filesystem yet.
+ * This can happen when some process with FILE_SHARE_DELETE has it
+ * open and it will be fully removed once that handle is closed.
+ * Meanwhile, we can't open it, so indicate that the file just
+ * doesn't exist.
+ */
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
+
+ ret = fileinfo_to_stat(hFile, buf);
+ CloseHandle(hFile);
+ return ret;
+}
+
+/*
+ * Since we already have a file handle there is no need to check for
+ * ERROR_DELETE_PENDING.
+ */
+int
+_pgfstat64(int fileno, struct stat *buf)
+{
+ HANDLE hFile = (HANDLE)_get_osfhandle(fileno);
+
+ if (hFile == INVALID_HANDLE_VALUE || buf == NULL)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+
+ return fileinfo_to_stat(hFile, buf);
+}
+
+#endif /* WIN32 */
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 834c2c3..6ff0783 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -103,7 +103,7 @@ sub mkvcbuild
pg_strong_random.c pgcheckdir.c pgmkdirp.c pgsleep.c pgstrcasecmp.c
pqsignal.c mkdtemp.c qsort.c qsort_arg.c quotes.c system.c
sprompt.c strerror.c tar.c thread.c
- win32env.c win32error.c win32security.c win32setlocale.c);
+ win32env.c win32error.c win32security.c win32setlocale.c win32stat.c);
push(@pgportfiles, 'strtof.c') if ($vsVersion < '14.00');
--
2.11.0
On Sat, Feb 29, 2020 at 9:40 AM Juan José Santamaría Flecha <
juanjo.santamaria@gmail.com> wrote:
On Sat, Feb 29, 2020 at 12:44 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
The cfbot thinks this doesn't compile on Windows [1]. Looks like perhaps
a missing-#include problem?The define logic for _WIN32_WINNT includes testing of _MSC_VER, and is not
a proper choice for MSVC 2013 as the cfbot is showing.
The cfbot is not happy yet. I will backtrack a bit on the cruft cleanup.
Please find attached a new version addressing this issue.
Regards,
Juan José Santamaría Flecha
Show quoted text
Attachments:
0001-Support-for-large-files-on-Win32-v7.patchapplication/octet-stream; name=0001-Support-for-large-files-on-Win32-v7.patchDownload
From 30e074639076b30b4d10e7c62cf4603f23d5085b Mon Sep 17 00:00:00 2001
From: Juan Jose Santamaria Flecha <juanjo.santamaria@gmail.com>
Date: Sat, 29 Feb 2020 04:38:10 -0500
Subject: [PATCH] Support for large files on Win32. Workaround for Windows
<sys/stat.h> shortcomings, reimplement fstat()/stat() on top of
GetFileInformationByHandle() and use 64 bits struct stat.
---
configure | 6 +
configure.in | 1 +
src/include/port/win32_port.h | 41 +++++--
src/port/dirmod.c | 52 --------
src/port/win32stat.c | 267 ++++++++++++++++++++++++++++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 2 +-
6 files changed, 304 insertions(+), 65 deletions(-)
create mode 100644 src/port/win32stat.c
diff --git a/configure b/configure
index d2c74e5..3acea3f 100755
--- a/configure
+++ b/configure
@@ -15822,6 +15822,12 @@ esac
;;
esac
+ case " $LIBOBJS " in
+ *" win32stat.$ac_objext "* ) ;;
+ *) LIBOBJS="$LIBOBJS win32stat.$ac_objext"
+ ;;
+esac
+
$as_echo "#define HAVE_SYMLINK 1" >>confdefs.h
diff --git a/configure.in b/configure.in
index 0b0a871..f9c73f6 100644
--- a/configure.in
+++ b/configure.in
@@ -1772,6 +1772,7 @@ if test "$PORTNAME" = "win32"; then
AC_LIBOBJ(win32error)
AC_LIBOBJ(win32security)
AC_LIBOBJ(win32setlocale)
+ AC_LIBOBJ(win32stat)
AC_DEFINE([HAVE_SYMLINK], 1,
[Define to 1 if you have the `symlink' function.])
AC_CHECK_TYPES(MINIDUMP_TYPE, [pgac_minidump_type=yes], [pgac_minidump_type=no], [
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index 8b6576b..cca7cdd 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -51,7 +51,12 @@
#include <signal.h>
#include <direct.h>
#undef near
-#include <sys/stat.h> /* needed before sys/stat hacking below */
+/* needed before sys/stat hacking below */
+#define fstat microsoft_native_fstat
+#define stat microsoft_native_stat
+#include <sys/stat.h>
+#undef fstat
+#undef stat
/* Must be here to avoid conflicting with prototype in windows.h */
#define mkdir(a,b) mkdir(a)
@@ -240,20 +245,32 @@ typedef int pid_t;
* Supplement to <sys/stat.h>.
*
* We must pull in sys/stat.h before this part, else our overrides lose.
- */
-#define lstat(path, sb) stat(path, sb)
-
-/*
+ *
* stat() is not guaranteed to set the st_size field on win32, so we
- * redefine it to our own implementation that is.
+ * redefine it to our own implementation. See src/port/win32stat.c.
*
- * Some frontends don't need the size from stat, so if UNSAFE_STAT_OK
- * is defined we don't bother with this.
+ * The struct stat is 32 bit in MSVC, so we redefine it as a copy of
+ * struct _stat64. This also fixes the struct size for MINGW builds.
*/
-#ifndef UNSAFE_STAT_OK
-extern int pgwin32_safestat(const char *path, struct stat *buf);
-#define stat(a,b) pgwin32_safestat(a,b)
-#endif
+struct stat /* It is actually _stat64 */
+{
+ _dev_t st_dev;
+ _ino_t st_ino;
+ unsigned short st_mode;
+ short st_nlink;
+ short st_uid;
+ short st_gid;
+ _dev_t st_rdev;
+ __int64 st_size;
+ __time64_t st_atime;
+ __time64_t st_mtime;
+ __time64_t st_ctime;
+};
+extern int _pgfstat64(int fileno, struct stat *buf);
+extern int _pgstat64(const char *name, struct stat *buf);
+#define fstat(fileno, sb) _pgfstat64(fileno, sb)
+#define stat(path, sb) _pgstat64(path, sb)
+#define lstat(path, sb) _pgstat64(path, sb)
/* These macros are not provided by older MinGW, nor by MSVC */
#ifndef S_IRUSR
diff --git a/src/port/dirmod.c b/src/port/dirmod.c
index e22a41c..8979f10 100644
--- a/src/port/dirmod.c
+++ b/src/port/dirmod.c
@@ -353,55 +353,3 @@ pgwin32_is_junction(const char *path)
return ((attr & FILE_ATTRIBUTE_REPARSE_POINT) == FILE_ATTRIBUTE_REPARSE_POINT);
}
#endif /* defined(WIN32) && !defined(__CYGWIN__) */
-
-
-#if defined(WIN32) && !defined(__CYGWIN__)
-
-#undef stat
-
-/*
- * The stat() function in win32 is not guaranteed to update the st_size
- * field when run. So we define our own version that uses the Win32 API
- * to update this field.
- */
-int
-pgwin32_safestat(const char *path, struct stat *buf)
-{
- int r;
- WIN32_FILE_ATTRIBUTE_DATA attr;
-
- r = stat(path, buf);
- if (r < 0)
- {
- if (GetLastError() == ERROR_DELETE_PENDING)
- {
- /*
- * File has been deleted, but is not gone from the filesystem yet.
- * This can happen when some process with FILE_SHARE_DELETE has it
- * open and it will be fully removed once that handle is closed.
- * Meanwhile, we can't open it, so indicate that the file just
- * doesn't exist.
- */
- errno = ENOENT;
- return -1;
- }
-
- return r;
- }
-
- if (!GetFileAttributesEx(path, GetFileExInfoStandard, &attr))
- {
- _dosmaperr(GetLastError());
- return -1;
- }
-
- /*
- * XXX no support for large files here, but we don't do that in general on
- * Win32 yet.
- */
- buf->st_size = attr.nFileSizeLow;
-
- return 0;
-}
-
-#endif
diff --git a/src/port/win32stat.c b/src/port/win32stat.c
new file mode 100644
index 0000000..82c7351
--- /dev/null
+++ b/src/port/win32stat.c
@@ -0,0 +1,267 @@
+/*-------------------------------------------------------------------------
+ *
+ * win32stat.c
+ * Replacements for <sys/stat.h> functions using GetFileInformationByHandle
+ * on Win32.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/port/win32stat.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifdef WIN32
+
+#include "c.h"
+#include <windows.h>
+
+/*
+ * In order to support MinGW and MSVC2013 we use NtQueryInformationFile as an
+ * alternative for GetFileInformationByHandleEx. It is loaded from the ntdll
+ * library.
+ */
+#if _WIN32_WINNT < 0x0600
+#include <winternl.h>
+#if !defined(__MINGW32__) && !defined(__MINGW64__)
+/* MinGW includes this in <winternl.h>, but is missing in MSVC */
+typedef struct _FILE_STANDARD_INFORMATION
+{
+ LARGE_INTEGER AllocationSize;
+ LARGE_INTEGER EndOfFile;
+ ULONG NumberOfLinks;
+ BOOLEAN DeletePending;
+ BOOLEAN Directory;
+} FILE_STANDARD_INFORMATION, *PFILE_STANDARD_INFORMATION;
+#define FileStandardInformation 5
+#endif /* !defined(__MINGW32__) && !defined(__MINGW64__) */
+
+typedef NTSTATUS (NTAPI *PFN_NTQUERYINFORMATIONFILE)
+(
+ IN HANDLE FileHandle,
+ OUT PIO_STATUS_BLOCK IoStatusBlock,
+ OUT PVOID FileInformation,
+ IN ULONG Length,
+ IN FILE_INFORMATION_CLASS FileInformationClass
+);
+
+static PFN_NTQUERYINFORMATIONFILE _NtQueryInformationFile = NULL;
+
+/*
+ * Load DLL file just once regardless of how many functions we load/call in it.
+ */
+static HMODULE ntdll = NULL;
+
+static void
+LoadNtdll(void)
+{
+ if (ntdll != NULL)
+ return;
+ ntdll = LoadLibraryEx("ntdll.dll", NULL, 0);
+}
+#endif /* _WIN32_WINNT < 0x0600 */
+
+static const unsigned __int64 EpochShift = UINT64CONST(116444736000000000);
+
+/*
+ * Converts a FILETIME struct into a 64 bit time_t.
+ */
+static __time64_t
+filetime_to_time(FILETIME const *ft)
+{
+ ULARGE_INTEGER unified_ft = {0};
+
+ unified_ft.LowPart = ft->dwLowDateTime;
+ unified_ft.HighPart = ft->dwHighDateTime;
+
+ if (unified_ft.QuadPart < EpochShift)
+ return -1;
+
+ unified_ft.QuadPart -= EpochShift;
+ unified_ft.QuadPart /= 10 * 1000 * 1000;
+
+ return unified_ft.QuadPart;
+}
+
+/*
+ * Converts WIN32 file attributes to unix mode.
+ * Only owner permissions are set.
+ */
+static unsigned short
+fileattr_to_unixmode(int attr)
+{
+ unsigned short uxmode = 0;
+
+ uxmode |= (unsigned short)((attr & FILE_ATTRIBUTE_DIRECTORY) ?
+ (_S_IFDIR) : (_S_IFREG));
+
+ uxmode |= (unsigned short)(attr & FILE_ATTRIBUTE_READONLY) ?
+ (_S_IREAD) : (_S_IREAD | _S_IWRITE);
+
+ /* There is no need to simulate _S_IEXEC using CMD's PATHEXT extensions */
+ uxmode |= _S_IEXEC;
+
+ return uxmode;
+}
+
+/*
+ * Converts WIN32 file infomation to struct stat.
+ * GetFileInformationByHandle minimum supported version: Windows XP and
+ * Windows Server 2003.
+ */
+static int
+fileinfo_to_stat(HANDLE hFile, struct stat *buf)
+{
+ BY_HANDLE_FILE_INFORMATION fiData;
+
+ if (!GetFileInformationByHandle(hFile, &fiData))
+ {
+ _dosmaperr(GetLastError());
+ return -1;
+ }
+
+ memset(buf, 0, sizeof(*buf));
+ if (fiData.ftLastWriteTime.dwLowDateTime ||
+ fiData.ftLastWriteTime.dwHighDateTime)
+ buf->st_mtime = filetime_to_time(&fiData.ftLastWriteTime);
+
+ if (fiData.ftLastAccessTime.dwLowDateTime ||
+ fiData.ftLastAccessTime.dwHighDateTime)
+ buf->st_atime = filetime_to_time(&fiData.ftLastAccessTime);
+ else
+ buf->st_atime = buf->st_mtime;
+
+ if (fiData.ftCreationTime.dwLowDateTime ||
+ fiData.ftCreationTime.dwHighDateTime)
+ buf->st_ctime = filetime_to_time(&fiData.ftCreationTime);
+ else
+ buf->st_ctime = buf->st_mtime;
+
+ buf->st_mode = fileattr_to_unixmode(fiData.dwFileAttributes);
+ buf->st_nlink = fiData.nNumberOfLinks;
+
+ buf->st_size = ((__int64) fiData.nFileSizeHigh) << 32 |
+ (__int64)(fiData.nFileSizeLow);
+ return 0;
+}
+
+/*
+ * We must use a handle so lstat() returns the information of the target file.
+ * As a reliable test for ERROR_DELETE_PENDING we use NtQueryInformationFile
+ * from Windows 2000 or GetFileInformationByHandleEx from Server 2008 / Vista.
+ */
+int
+_pgstat64(const char *name, struct stat *buf)
+{
+ SECURITY_ATTRIBUTES sa;
+ HANDLE hFile;
+ int ret;
+#if _WIN32_WINNT < 0x0600
+ IO_STATUS_BLOCK ioStatus;
+ FILE_STANDARD_INFORMATION standardInfo;
+#else
+ FILE_STANDARD_INFO standardInfo;
+#endif
+
+ if (name == NULL || buf == NULL)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+ /* fast not exists */
+ if (GetFileAttributes(name) == INVALID_FILE_ATTRIBUTES)
+ {
+ errno = ENOENT;
+ return -1;
+ }
+
+ /* Get a file handle as lightweight as we can */
+ sa.nLength = sizeof(SECURITY_ATTRIBUTES);
+ sa.bInheritHandle = TRUE;
+ sa.lpSecurityDescriptor = NULL;
+ hFile = CreateFile(name, GENERIC_READ,
+ (FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE),
+ &sa, OPEN_EXISTING,
+ (FILE_FLAG_NO_BUFFERING | FILE_FLAG_BACKUP_SEMANTICS |
+ FILE_FLAG_OVERLAPPED), NULL);
+ if (hFile == INVALID_HANDLE_VALUE)
+ {
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
+
+ memset(&standardInfo, 0, sizeof(standardInfo));
+#if _WIN32_WINNT < 0x0600
+ if (_NtQueryInformationFile == NULL)
+ {
+ LoadNtdll();
+ if(ntdll == NULL)
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ return -1;
+ }
+
+ _NtQueryInformationFile = (PFN_NTQUERYINFORMATIONFILE)
+ GetProcAddress(ntdll, "NtQueryInformationFile");
+ if (_NtQueryInformationFile == NULL)
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ return -1;
+ }
+ }
+
+ if (!NT_SUCCESS(_NtQueryInformationFile(hFile, &ioStatus, &standardInfo,
+ sizeof(standardInfo), FileStandardInformation)))
+#else
+ if (!GetFileInformationByHandleEx(hFile, FileStandardInfo, &standardInfo,
+ sizeof(standardInfo)))
+#endif /* _WIN32_WINNT < 0x0600 */
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ return -1;
+ }
+ if (standardInfo.DeletePending)
+ {
+ /*
+ * File has been deleted, but is not gone from the filesystem yet.
+ * This can happen when some process with FILE_SHARE_DELETE has it
+ * open and it will be fully removed once that handle is closed.
+ * Meanwhile, we can't open it, so indicate that the file just
+ * doesn't exist.
+ */
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
+
+ ret = fileinfo_to_stat(hFile, buf);
+ CloseHandle(hFile);
+ return ret;
+}
+
+/*
+ * Since we already have a file handle there is no need to check for
+ * ERROR_DELETE_PENDING.
+ */
+int
+_pgfstat64(int fileno, struct stat *buf)
+{
+ HANDLE hFile = (HANDLE)_get_osfhandle(fileno);
+
+ if (hFile == INVALID_HANDLE_VALUE || buf == NULL)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+
+ return fileinfo_to_stat(hFile, buf);
+}
+
+#endif /* WIN32 */
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 834c2c3..6ff0783 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -103,7 +103,7 @@ sub mkvcbuild
pg_strong_random.c pgcheckdir.c pgmkdirp.c pgsleep.c pgstrcasecmp.c
pqsignal.c mkdtemp.c qsort.c qsort_arg.c quotes.c system.c
sprompt.c strerror.c tar.c thread.c
- win32env.c win32error.c win32security.c win32setlocale.c);
+ win32env.c win32error.c win32security.c win32setlocale.c win32stat.c);
push(@pgportfiles, 'strtof.c') if ($vsVersion < '14.00');
--
2.11.0
I assigned myself as a reviewer for this patch, as I hit this bug today and had to perform a workaround. I have never reviewed a patch before but will try to update within the next 5 days. I intend on performing "Implements Feature" reviewing.
On Mon, Oct 28, 2019 at 06:13:58PM +0100, Juan José Santamaría Flecha wrote:
At this moment is missing review, so it is probably far from being
commitable. Any attention is appreciated and might help pushing it forward.
As a personal note, I have to check that is still applies before the
upcoming commitfest.
Could you send a rebase of the patch? Thanks!
--
Michael
On Thu, Sep 17, 2020 at 9:46 AM Michael Paquier <michael@paquier.xyz> wrote:
Could you send a rebase of the patch? Thanks!
Thanks for the reminder. Please find attached a rebased version.
Regards,
Juan José Santamaría Flecha
Attachments:
0001-Support-for-large-files-on-Win32-v8.patchapplication/octet-stream; name=0001-Support-for-large-files-on-Win32-v8.patchDownload
From 95a62ddf6c71abd680170b6ea0c76c6a9205b871 Mon Sep 17 00:00:00 2001
From: Juan Jose Santamaria Flecha <juanjo.santamaria@gmail.com>
Date: Thu, 17 Sep 2020 11:08:41 -0400
Subject: [PATCH] Support for large files on Win32. Workaround for Windows
<sys/stat.h> shortcomings, reimplement fstat()/stat() on top of
GetFileInformationByHandle() and use 64 bits struct stat.
---
configure | 6 +
configure.ac | 1 +
src/include/port/win32_port.h | 41 +++++--
src/port/dirmod.c | 52 --------
src/port/win32stat.c | 268 ++++++++++++++++++++++++++++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 2 +-
6 files changed, 305 insertions(+), 65 deletions(-)
create mode 100644 src/port/win32stat.c
diff --git a/configure b/configure
index 19a3cd0..071d050 100755
--- a/configure
+++ b/configure
@@ -16137,6 +16137,12 @@ esac
;;
esac
+ case " $LIBOBJS " in
+ *" win32stat.$ac_objext "* ) ;;
+ *) LIBOBJS="$LIBOBJS win32stat.$ac_objext"
+ ;;
+esac
+
$as_echo "#define HAVE_SYMLINK 1" >>confdefs.h
diff --git a/configure.ac b/configure.ac
index 6b9d048..e9ce611 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1807,6 +1807,7 @@ if test "$PORTNAME" = "win32"; then
AC_LIBOBJ(win32error)
AC_LIBOBJ(win32security)
AC_LIBOBJ(win32setlocale)
+ AC_LIBOBJ(win32stat)
AC_DEFINE([HAVE_SYMLINK], 1,
[Define to 1 if you have the `symlink' function.])
AC_CHECK_TYPES(MINIDUMP_TYPE, [pgac_minidump_type=yes], [pgac_minidump_type=no], [
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index 8b6576b..1860bdd 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -51,7 +51,12 @@
#include <signal.h>
#include <direct.h>
#undef near
-#include <sys/stat.h> /* needed before sys/stat hacking below */
+/* needed before sys/stat hacking below */
+#define fstat microsoft_native_fstat
+#define stat microsoft_native_stat
+#include <sys/stat.h>
+#undef fstat
+#undef stat
/* Must be here to avoid conflicting with prototype in windows.h */
#define mkdir(a,b) mkdir(a)
@@ -240,20 +245,32 @@ typedef int pid_t;
* Supplement to <sys/stat.h>.
*
* We must pull in sys/stat.h before this part, else our overrides lose.
- */
-#define lstat(path, sb) stat(path, sb)
-
-/*
+ *
* stat() is not guaranteed to set the st_size field on win32, so we
- * redefine it to our own implementation that is.
+ * redefine it to our own implementation. See src/port/win32stat.c.
*
- * Some frontends don't need the size from stat, so if UNSAFE_STAT_OK
- * is defined we don't bother with this.
+ * The struct stat is 32 bit in MSVC, so we redefine it as a copy of
+ * struct _stat64. This also fixes the struct size for MINGW builds.
*/
-#ifndef UNSAFE_STAT_OK
-extern int pgwin32_safestat(const char *path, struct stat *buf);
-#define stat(a,b) pgwin32_safestat(a,b)
-#endif
+struct stat /* It is actually _stat64 */
+{
+ _dev_t st_dev;
+ _ino_t st_ino;
+ unsigned short st_mode;
+ short st_nlink;
+ short st_uid;
+ short st_gid;
+ _dev_t st_rdev;
+ __int64 st_size;
+ __time64_t st_atime;
+ __time64_t st_mtime;
+ __time64_t st_ctime;
+};
+extern int _pgfstat64(int fileno, struct stat *buf);
+extern int _pgstat64(const char *name, struct stat *buf);
+#define fstat(fileno, sb) _pgfstat64(fileno, sb)
+#define stat(path, sb) _pgstat64(path, sb)
+#define lstat(path, sb) _pgstat64(path, sb)
/* These macros are not provided by older MinGW, nor by MSVC */
#ifndef S_IRUSR
diff --git a/src/port/dirmod.c b/src/port/dirmod.c
index e22a41c..8979f10 100644
--- a/src/port/dirmod.c
+++ b/src/port/dirmod.c
@@ -353,55 +353,3 @@ pgwin32_is_junction(const char *path)
return ((attr & FILE_ATTRIBUTE_REPARSE_POINT) == FILE_ATTRIBUTE_REPARSE_POINT);
}
#endif /* defined(WIN32) && !defined(__CYGWIN__) */
-
-
-#if defined(WIN32) && !defined(__CYGWIN__)
-
-#undef stat
-
-/*
- * The stat() function in win32 is not guaranteed to update the st_size
- * field when run. So we define our own version that uses the Win32 API
- * to update this field.
- */
-int
-pgwin32_safestat(const char *path, struct stat *buf)
-{
- int r;
- WIN32_FILE_ATTRIBUTE_DATA attr;
-
- r = stat(path, buf);
- if (r < 0)
- {
- if (GetLastError() == ERROR_DELETE_PENDING)
- {
- /*
- * File has been deleted, but is not gone from the filesystem yet.
- * This can happen when some process with FILE_SHARE_DELETE has it
- * open and it will be fully removed once that handle is closed.
- * Meanwhile, we can't open it, so indicate that the file just
- * doesn't exist.
- */
- errno = ENOENT;
- return -1;
- }
-
- return r;
- }
-
- if (!GetFileAttributesEx(path, GetFileExInfoStandard, &attr))
- {
- _dosmaperr(GetLastError());
- return -1;
- }
-
- /*
- * XXX no support for large files here, but we don't do that in general on
- * Win32 yet.
- */
- buf->st_size = attr.nFileSizeLow;
-
- return 0;
-}
-
-#endif
diff --git a/src/port/win32stat.c b/src/port/win32stat.c
new file mode 100644
index 0000000..2367844
--- /dev/null
+++ b/src/port/win32stat.c
@@ -0,0 +1,268 @@
+/*-------------------------------------------------------------------------
+ *
+ * win32stat.c
+ * Replacements for <sys/stat.h> functions using GetFileInformationByHandle
+ * on Win32.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/port/win32stat.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifdef WIN32
+
+#include "c.h"
+#include <windows.h>
+
+/*
+ * In order to support MinGW and MSVC2013 we use NtQueryInformationFile as an
+ * alternative for GetFileInformationByHandleEx. It is loaded from the ntdll
+ * library.
+ */
+#if _WIN32_WINNT < 0x0600
+#include <winternl.h>
+#if !defined(__MINGW32__) && !defined(__MINGW64__)
+/* MinGW includes this in <winternl.h>, but is missing in MSVC */
+typedef struct _FILE_STANDARD_INFORMATION
+{
+ LARGE_INTEGER AllocationSize;
+ LARGE_INTEGER EndOfFile;
+ ULONG NumberOfLinks;
+ BOOLEAN DeletePending;
+ BOOLEAN Directory;
+} FILE_STANDARD_INFORMATION, *PFILE_STANDARD_INFORMATION;
+#define FileStandardInformation 5
+#endif /* !defined(__MINGW32__) && !defined(__MINGW64__) */
+
+typedef NTSTATUS (NTAPI *PFN_NTQUERYINFORMATIONFILE)
+(
+ IN HANDLE FileHandle,
+ OUT PIO_STATUS_BLOCK IoStatusBlock,
+ OUT PVOID FileInformation,
+ IN ULONG Length,
+ IN FILE_INFORMATION_CLASS FileInformationClass
+);
+
+static PFN_NTQUERYINFORMATIONFILE _NtQueryInformationFile = NULL;
+
+/*
+ * Load DLL file just once regardless of how many functions we load/call in it.
+ */
+static HMODULE ntdll = NULL;
+
+static void
+LoadNtdll(void)
+{
+ if (ntdll != NULL)
+ return;
+ ntdll = LoadLibraryEx("ntdll.dll", NULL, 0);
+}
+#endif /* _WIN32_WINNT < 0x0600 */
+
+static const unsigned __int64 EpochShift = UINT64CONST(116444736000000000);
+
+/*
+ * Converts a FILETIME struct into a 64 bit time_t.
+ */
+static __time64_t
+filetime_to_time(FILETIME const *ft)
+{
+ ULARGE_INTEGER unified_ft = {0};
+
+ unified_ft.LowPart = ft->dwLowDateTime;
+ unified_ft.HighPart = ft->dwHighDateTime;
+
+ if (unified_ft.QuadPart < EpochShift)
+ return -1;
+
+ unified_ft.QuadPart -= EpochShift;
+ unified_ft.QuadPart /= 10 * 1000 * 1000;
+
+ return unified_ft.QuadPart;
+}
+
+/*
+ * Converts WIN32 file attributes to unix mode.
+ * Only owner permissions are set.
+ */
+static unsigned short
+fileattr_to_unixmode(int attr)
+{
+ unsigned short uxmode = 0;
+
+ uxmode |= (unsigned short)((attr & FILE_ATTRIBUTE_DIRECTORY) ?
+ (_S_IFDIR) : (_S_IFREG));
+
+ uxmode |= (unsigned short)(attr & FILE_ATTRIBUTE_READONLY) ?
+ (_S_IREAD) : (_S_IREAD | _S_IWRITE);
+
+ /* there is no need to simulate _S_IEXEC using CMD's PATHEXT extensions */
+ uxmode |= _S_IEXEC;
+
+ return uxmode;
+}
+
+/*
+ * Converts WIN32 file infomation to struct stat.
+ * GetFileInformationByHandle minimum supported version: Windows XP and
+ * Windows Server 2003.
+ */
+static int
+fileinfo_to_stat(HANDLE hFile, struct stat *buf)
+{
+ BY_HANDLE_FILE_INFORMATION fiData;
+
+ if (!GetFileInformationByHandle(hFile, &fiData))
+ {
+ _dosmaperr(GetLastError());
+ return -1;
+ }
+
+ memset(buf, 0, sizeof(*buf));
+ if (fiData.ftLastWriteTime.dwLowDateTime ||
+ fiData.ftLastWriteTime.dwHighDateTime)
+ buf->st_mtime = filetime_to_time(&fiData.ftLastWriteTime);
+
+ if (fiData.ftLastAccessTime.dwLowDateTime ||
+ fiData.ftLastAccessTime.dwHighDateTime)
+ buf->st_atime = filetime_to_time(&fiData.ftLastAccessTime);
+ else
+ buf->st_atime = buf->st_mtime;
+
+ if (fiData.ftCreationTime.dwLowDateTime ||
+ fiData.ftCreationTime.dwHighDateTime)
+ buf->st_ctime = filetime_to_time(&fiData.ftCreationTime);
+ else
+ buf->st_ctime = buf->st_mtime;
+
+ buf->st_mode = fileattr_to_unixmode(fiData.dwFileAttributes);
+ buf->st_nlink = fiData.nNumberOfLinks;
+
+ buf->st_size = ((__int64) fiData.nFileSizeHigh) << 32 |
+ (__int64)(fiData.nFileSizeLow);
+ return 0;
+}
+
+/*
+ * We must use a handle so lstat() returns the information of the target file.
+ * As a reliable test for ERROR_DELETE_PENDING we use NtQueryInformationFile
+ * from Windows 2000 or GetFileInformationByHandleEx from Server 2008 / Vista.
+ */
+int
+_pgstat64(const char *name, struct stat *buf)
+{
+ SECURITY_ATTRIBUTES sa;
+ HANDLE hFile;
+ int ret;
+#if _WIN32_WINNT < 0x0600
+ IO_STATUS_BLOCK ioStatus;
+ FILE_STANDARD_INFORMATION standardInfo;
+#else
+ FILE_STANDARD_INFO standardInfo;
+#endif
+
+ if (name == NULL || buf == NULL)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+ /* fast not exists */
+ if (GetFileAttributes(name) == INVALID_FILE_ATTRIBUTES)
+ {
+ errno = ENOENT;
+ return -1;
+ }
+
+ /* get a file handle as lightweight as we can */
+ sa.nLength = sizeof(SECURITY_ATTRIBUTES);
+ sa.bInheritHandle = TRUE;
+ sa.lpSecurityDescriptor = NULL;
+ hFile = CreateFile(name, GENERIC_READ,
+ (FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE),
+ &sa, OPEN_EXISTING,
+ (FILE_FLAG_NO_BUFFERING | FILE_FLAG_BACKUP_SEMANTICS |
+ FILE_FLAG_OVERLAPPED), NULL);
+ if (hFile == INVALID_HANDLE_VALUE)
+ {
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
+
+ memset(&standardInfo, 0, sizeof(standardInfo));
+#if _WIN32_WINNT < 0x0600
+ if (_NtQueryInformationFile == NULL)
+ {
+ LoadNtdll();
+ if(ntdll == NULL)
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ return -1;
+ }
+
+ _NtQueryInformationFile = (PFN_NTQUERYINFORMATIONFILE)
+ GetProcAddress(ntdll, "NtQueryInformationFile");
+ if (_NtQueryInformationFile == NULL)
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ return -1;
+ }
+ }
+
+ if (!NT_SUCCESS(_NtQueryInformationFile(hFile, &ioStatus, &standardInfo,
+ sizeof(standardInfo), FileStandardInformation)))
+#else
+ if (!GetFileInformationByHandleEx(hFile, FileStandardInfo, &standardInfo,
+ sizeof(standardInfo)))
+#endif /* _WIN32_WINNT < 0x0600 */
+ {
+ _dosmaperr(GetLastError());
+ CloseHandle(hFile);
+ return -1;
+ }
+ if (standardInfo.DeletePending)
+ {
+ /*
+ * File has been deleted, but is not gone from the filesystem yet.
+ * This can happen when some process with FILE_SHARE_DELETE has it
+ * open and it will be fully removed once that handle is closed.
+ * Meanwhile, we can't open it, so indicate that the file just
+ * doesn't exist.
+ */
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
+
+ ret = fileinfo_to_stat(hFile, buf);
+ CloseHandle(hFile);
+ return ret;
+}
+
+/*
+ * Since we already have a file handle there is no need to check for
+ * ERROR_DELETE_PENDING.
+ */
+int
+_pgfstat64(int fileno, struct stat *buf)
+{
+ HANDLE hFile = (HANDLE)_get_osfhandle(fileno);
+
+ if (hFile == INVALID_HANDLE_VALUE || buf == NULL)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+
+ return fileinfo_to_stat(hFile, buf);
+}
+
+#endif /* WIN32 */
+
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 89e1b39..90594bd 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -103,7 +103,7 @@ sub mkvcbuild
pg_strong_random.c pgcheckdir.c pgmkdirp.c pgsleep.c pgstrcasecmp.c
pqsignal.c mkdtemp.c qsort.c qsort_arg.c quotes.c system.c
strerror.c tar.c thread.c
- win32env.c win32error.c win32security.c win32setlocale.c);
+ win32env.c win32error.c win32security.c win32setlocale.c win32stat.c);
push(@pgportfiles, 'strtof.c') if ($vsVersion < '14.00');
--
2.11.0
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo.santamaria@gmail.com> writes:
Thanks for the reminder. Please find attached a rebased version.
(This hasn't shown up on -hackers yet, maybe caught in moderation?)
I took a quick look through this. I'm not qualified to review the
actual Windows code in win32stat.c, but as far as the way you're
plugging it into the system goes, it looks good and seems to comport
with the discussion so far.
One thing I noticed, which is a pre-existing problem but maybe now
is a good time to consider it, is that we're mapping lstat() to be
exactly stat() on Windows. That made sense years ago when (we
believed that) Windows didn't have symlinks, but surely it no longer
makes sense.
Another more trivial point is that it'd be good to run the new code
through pgindent before committing.
regards, tom lane
Em qui., 17 de set. de 2020 às 14:37, Juan José Santamaría Flecha <
juanjo.santamaria@gmail.com> escreveu:
On Thu, Sep 17, 2020 at 9:46 AM Michael Paquier <michael@paquier.xyz>
wrote:Could you send a rebase of the patch? Thanks!
Thanks for the reminder. Please find attached a rebased version.
Sorry, I'm missing something?
What's wrong with _stat64?
Pasta de C:\tmp
18/08/2020 16:51 6.427.512.517 macOS_Catalina.7z
1 arquivo(s) 6.427.512.517 bytes
0 pasta(s) 149.691.797.504 bytes disponíveis
C:\usr\src\tests\stat>crt_stat
File size : 6427512517
Drive : C:
Time modified : Tue Aug 18 16:51:47 2020
regards,
Ranier Vilela
Attachments:
Ranier Vilela <ranier.vf@gmail.com> writes:
What's wrong with _stat64?
See upthread.
regards, tom lane
On Thu, Sep 17, 2020 at 6:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <
juanjo.santamaria@gmail.com> writes:Thanks for the reminder. Please find attached a rebased version.
(This hasn't shown up on -hackers yet, maybe caught in moderation?)
Thanks for looking into it. Finally, it went through. I will be removing
bug-list from now on.
I took a quick look through this. I'm not qualified to review the
actual Windows code in win32stat.c, but as far as the way you're
plugging it into the system goes, it looks good and seems to comport
with the discussion so far.One thing I noticed, which is a pre-existing problem but maybe now
is a good time to consider it, is that we're mapping lstat() to be
exactly stat() on Windows. That made sense years ago when (we
believed that) Windows didn't have symlinks, but surely it no longer
makes sense.
I will have to take a better look at it, but from a quick look it, all
lstat() calls seem to test just if the file exists, and that can be done
with a cheap call to GetFileAttributes(). Would a limited (but fast)
lstat(), where only st_mode could be informed, be acceptable?
Another more trivial point is that it'd be good to run the new code
through pgindent before committing.
I do not have pgindent in the WIN32 machine, but I will try to for the next
version.
Regards,
Juan José Santamaría Flecha
On Thu, Sep 17, 2020 at 8:47 PM Juan José Santamaría Flecha <
juanjo.santamaria@gmail.com> wrote:
On Thu, Sep 17, 2020 at 6:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
One thing I noticed, which is a pre-existing problem but maybe now
is a good time to consider it, is that we're mapping lstat() to be
exactly stat() on Windows. That made sense years ago when (we
believed that) Windows didn't have symlinks, but surely it no longer
makes sense.I will have to take a better look at it, but from a quick look it, all
lstat() calls seem to test just if the file exists, and that can be done
with a cheap call to GetFileAttributes(). Would a limited (but fast)
lstat(), where only st_mode could be informed, be acceptable?
After thinking more about this, that approach would be problematic for
DELETE_PENDING files. The proposed patch logic is meant to maintain current
behaviour, which is not broken for WIN32 symlinks AFAICT.
Regards,
Juan José Santamaría Flecha
The following review has been posted through the commitfest application:
make installcheck-world: tested, passed
Implements feature: tested, passed
Spec compliant: not tested
Documentation: not tested
I tested the patch at hand, and it performs as expected. Files larger than 4GB can be imported.
Steps:
0) create a csv-file that is sufficiently big (>4GB), and one that is small. Use these files to test.
1a) Attempt to import the small file using devel-version.
1b) EXPECTED: success, ACTUAL: success
2a) Attempt to import the big file using devel-version.
2b) EXPECTED: failure, ACTUAL: failure
3) Apply patch and build new version
4a) Attempt to import the small file using patched-version.
4b) EXPECTED: success, ACTUAL: success
4a) Attempt to import the big file using patched-version.
4b) EXPECTED: success, ACTUAL: success
The code looks sensible, it is easy to read and follow. The code uses appropriate win32 functions to perform the task.
Code calculates file size using the following method: buf->st_size = ((__int64) fiData.nFileSizeHigh) << 32 | (__int64)(fiData.nFileSizeLow);
The hard coded constant 32 is fine, nFileSizeHigh is defined as a DWORD in the Win32 API, which is a 32 bit unsigned integer. There is no need to a dynamic calculation.
There are minor "nit-picks" that I would change if it were my code, but do not change the functionality of the code.
1)
if (GetFileAttributes(name) == INVALID_FILE_ATTRIBUTES)
{
errno = ENOENT;
return -1;
}
Here I would call _dosmaperr(GetLastError()) instead, just to take account of the possibility that some other error occurred. Following this change there are slight inconsistency in the order of "CloseHandle(hFile), errno = ENOENT; return -1" and "_dosmaperr(GetLastError()); CloseHandle(hFile); return -1". I would prefer consistent ordering, but that is not important.
The new status of this patch is: Ready for Committer
Emil Iggland <emil@iggland.com> writes:
I tested the patch at hand, and it performs as expected. Files larger than 4GB can be imported.
Thanks for testing!
I'd been expecting one of our Windows-savvy committers to pick this up,
but since nothing has been happening, I took it on myself to push it.
I'll probably regret that :-(
I made a few cosmetic changes, mostly reorganizing comments in a way
that made more sense to me.
regards, tom lane
On Fri, Oct 9, 2020 at 10:22 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Emil Iggland <emil@iggland.com> writes:
I tested the patch at hand, and it performs as expected. Files larger
than 4GB can be imported.
Thanks for testing!
Thanks for testing! +1
I'd been expecting one of our Windows-savvy committers to pick this up,
but since nothing has been happening, I took it on myself to push it.
I'll probably regret that :-(
Thanks for taking care of this. I see no problems in the build farm, but
please reach me if I missed something.
I made a few cosmetic changes, mostly reorganizing comments in a way
that made more sense to me.I was working on a new version, which was pgindent-friendlier and clearer
about reporting an error when 'errno' is not informed. Please find attached
a patch with those changes.
Regards,
Juan José Santamaría Flecha
Attachments:
0001-win32stat-indent-and-errormapping-v1.patchapplication/octet-stream; name=0001-win32stat-indent-and-errormapping-v1.patchDownload
diff --git a/src/port/win32stat.c b/src/port/win32stat.c
index a70df3a..92c2e66 100644
--- a/src/port/win32stat.c
+++ b/src/port/win32stat.c
@@ -101,8 +101,8 @@ fileattr_to_unixmode(int attr)
uxmode |= (unsigned short) ((attr & FILE_ATTRIBUTE_DIRECTORY) ?
(_S_IFDIR) : (_S_IFREG));
- uxmode |= (unsigned short) (attr & FILE_ATTRIBUTE_READONLY) ?
- (_S_IREAD) : (_S_IREAD | _S_IWRITE);
+ uxmode |= (unsigned short) ((attr & FILE_ATTRIBUTE_READONLY) ?
+ (_S_IREAD) : (_S_IREAD | _S_IWRITE));
/* there is no need to simulate _S_IEXEC using CMD's PATHEXT extensions */
uxmode |= _S_IEXEC;
@@ -149,8 +149,8 @@ fileinfo_to_stat(HANDLE hFile, struct stat *buf)
buf->st_mode = fileattr_to_unixmode(fiData.dwFileAttributes);
buf->st_nlink = fiData.nNumberOfLinks;
- buf->st_size = (((uint64) fiData.nFileSizeHigh) << 32) |
- (uint64) fiData.nFileSizeLow;
+ buf->st_size = ((((uint64) fiData.nFileSizeHigh) << 32) |
+ fiData.nFileSizeLowi);
return 0;
}
@@ -188,7 +188,13 @@ _pgstat64(const char *name, struct stat *buf)
/* fast not-exists check */
if (GetFileAttributes(name) == INVALID_FILE_ATTRIBUTES)
{
- _dosmaperr(GetLastError());
+ DWORD err = GetLastError();
+
+ /* report when not ERROR_SUCCESS */
+ if (err == ERROR_FILE_NOT_FOUND || err == ERROR_PATH_NOT_FOUND)
+ errno = ENOENT;
+ else
+ _dosmaperr(err);
return -1;
}
@@ -220,8 +226,10 @@ _pgstat64(const char *name, struct stat *buf)
LoadNtdll();
if (ntdll == NULL)
{
- _dosmaperr(GetLastError());
+ DWORD err = GetLastError();
+
CloseHandle(hFile);
+ _dosmaperr(err);
return -1;
}
@@ -229,8 +237,10 @@ _pgstat64(const char *name, struct stat *buf)
GetProcAddress(ntdll, "NtQueryInformationFile");
if (_NtQueryInformationFile == NULL)
{
- _dosmaperr(GetLastError());
+ DWORD err = GetLastError();
+
CloseHandle(hFile);
+ _dosmaperr(err);
return -1;
}
}
@@ -239,16 +249,20 @@ _pgstat64(const char *name, struct stat *buf)
sizeof(standardInfo),
FileStandardInformation)))
{
- _dosmaperr(GetLastError());
+ DWORD err = GetLastError();
+
CloseHandle(hFile);
+ _dosmaperr(err);
return -1;
}
#else
if (!GetFileInformationByHandleEx(hFile, FileStandardInfo, &standardInfo,
sizeof(standardInfo)))
{
- _dosmaperr(GetLastError());
+ DWORD err = GetLastError();
+
CloseHandle(hFile);
+ _dosmaperr(err);
return -1;
}
#endif /* _WIN32_WINNT < 0x0600 */
On Sat, Oct 10, 2020 at 01:31:21PM +0200, Juan José Santamaría Flecha wrote:
Thanks for taking care of this. I see no problems in the build farm, but
please reach me if I missed something.
Thanks for continuing your work on this patch. I see no related
failures in the buildfarm.
- _dosmaperr(GetLastError());
+ DWORD err = GetLastError();
+
+ /* report when not ERROR_SUCCESS */
+ if (err == ERROR_FILE_NOT_FOUND || err == ERROR_PATH_NOT_FOUND)
+ errno = ENOENT;
+ else
+ _dosmaperr(err);
Why are you changing that? The original coding is fine, as
_dosmaperr() already maps ERROR_FILE_NOT_FOUND and
ERROR_PATH_NOT_FOUND to ENOENT.
- _dosmaperr(GetLastError());
+ DWORD err = GetLastError();
+
CloseHandle(hFile);
+ _dosmaperr(err);
These parts are indeed incorrect. CloseHandle() could overwrite
errno.
--
Michael
On Sat, Oct 10, 2020 at 2:24 PM Michael Paquier <michael@paquier.xyz> wrote:
- _dosmaperr(GetLastError()); + DWORD err = GetLastError(); + + /* report when not ERROR_SUCCESS */ + if (err == ERROR_FILE_NOT_FOUND || err == ERROR_PATH_NOT_FOUND) + errno = ENOENT; + else + _dosmaperr(err); Why are you changing that? The original coding is fine, as _dosmaperr() already maps ERROR_FILE_NOT_FOUND and ERROR_PATH_NOT_FOUND to ENOENT.
If the file does not exist there is no need to call _dosmaperr() and log
the error.
- _dosmaperr(GetLastError()); + DWORD err = GetLastError(); + CloseHandle(hFile); + _dosmaperr(err); These parts are indeed incorrect. CloseHandle() could overwrite errno.
The meaningful error should come from the previous call, and an error from
CloseHandle() could mask it. Not sure it makes a difference anyhow.
Regards,
Juan José Santamaría Flecha
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo.santamaria@gmail.com> writes:
If the file does not exist there is no need to call _dosmaperr() and log
the error.
I concur with Michael that it's inappropriate to make an end run around
_dosmaperr() here. If you think that the DEBUG5 logging inside that
is inappropriate, you should propose removing it outright.
Pushed the rest of this.
(pgindent behaved differently around PFN_NTQUERYINFORMATIONFILE today
than it did yesterday. No idea why.)
The meaningful error should come from the previous call, and an error from
CloseHandle() could mask it. Not sure it makes a difference anyhow.
Would CloseHandle() really touch errno at all? But this way is
certainly safer, so done.
regards, tom lane
On Sat, Oct 10, 2020 at 7:43 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
I concur with Michael that it's inappropriate to make an end run around
_dosmaperr() here. If you think that the DEBUG5 logging inside that
is inappropriate, you should propose removing it outright.Pushed the rest of this.
Great, thanks again to everyone who has taken some time to look into this.
Regards,
Juan José Santamaría Flecha
On Sat, Oct 10, 2020 at 09:00:27PM +0200, Juan José Santamaría Flecha wrote:
Great, thanks again to everyone who has taken some time to look into this.
We are visibly not completely out of the woods yet, jacana is
reporting a compilation error:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&dt=2020-10-10%2018%3A00%3A28
Oct 10 14:04:40
c:/mingw/msys/1.0/home/pgrunner/bf/root/HEAD/pgsql.build/../pgsql/src/port/win32stat.c:
In function 'fileinfo_to_stat':
Oct 10 14:04:40
c:/mingw/msys/1.0/home/pgrunner/bf/root/HEAD/pgsql.build/../pgsql/src/port/win32stat.c:151:13:
error: 'BY_HANDLE_FILE_INFORMATION {aka struct
_BY_HANDLE_FILE_INFORMATION}' has no member named 'nFileSizeLowi'; did
you mean 'nFileSizeLow'?
Oct 10 14:04:40 fiData.nFileSizeLowi);
Oct 10 14:04:40 ^~~~~~~~~~~~~
Oct 10 14:04:40 nFileSizeLow
I don't have the time to check MinGW and HEAD now, so that's just a
heads-up.
--
Michael
Michael Paquier <michael@paquier.xyz> writes:
We are visibly not completely out of the woods yet, jacana is
reporting a compilation error:
Nah, I fixed that hours ago (961e07b8c). jacana must not have run again
yet.
regards, tom lane
On Sat, Oct 10, 2020 at 08:34:48PM -0400, Tom Lane wrote:
Nah, I fixed that hours ago (961e07b8c). jacana must not have run again
yet.
Indeed, thanks. I have missed one sync here.
+ hFile = CreateFile(name,
+ GENERIC_READ,
+ (FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE),
+ &sa,
+ OPEN_EXISTING,
+ (FILE_FLAG_NO_BUFFERING | FILE_FLAG_BACKUP_SEMANTICS |
+ FILE_FLAG_OVERLAPPED),
+ NULL);
+ if (hFile == INVALID_HANDLE_VALUE)
+ {
+ CloseHandle(hFile);
+ errno = ENOENT;
+ return -1;
+ }
Why are we forcing errno=ENOENT here? Wouldn't it be correct to use
_dosmaperr(GetLastError()) to get the correct errno? This code would
for example consider as non-existing a file even if we fail getting it
because of ERROR_SHARING_VIOLATION, which should map to EACCES. This
case can happen with virus scanners taking a non-share handle on files
being looked at in parallel of this code path.
--
Michael
Michael Paquier <michael@paquier.xyz> writes:
Why are we forcing errno=ENOENT here? Wouldn't it be correct to use
_dosmaperr(GetLastError()) to get the correct errno?
Fair question. Juan, was there some good reason not to look at
GetLastError() in this step?
regards, tom lane
On Mon, Oct 12, 2020 at 5:27 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Michael Paquier <michael@paquier.xyz> writes:
Why are we forcing errno=ENOENT here? Wouldn't it be correct to use
_dosmaperr(GetLastError()) to get the correct errno?Fair question. Juan, was there some good reason not to look at
GetLastError() in this step?
Uhm, a good question indeed, forcing errno serves no purpose there.
Regards,
Juan José Santamaría Flecha
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo.santamaria@gmail.com> writes:
On Mon, Oct 12, 2020 at 5:27 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Michael Paquier <michael@paquier.xyz> writes:
Why are we forcing errno=ENOENT here? Wouldn't it be correct to use
_dosmaperr(GetLastError()) to get the correct errno?
Fair question. Juan, was there some good reason not to look at
GetLastError() in this step?
Uhm, a good question indeed, forcing errno serves no purpose there.
OK, changed.
regards, tom lane
On Mon, Oct 12, 2020 at 11:13:38AM -0400, Tom Lane wrote:
Juan José Santamaría Flecha wrote:
Uhm, a good question indeed, forcing errno serves no purpose there.
OK, changed.
Thanks!
--
Michael
Hi,
I noticed that this modification only commit into master branch,
there is still have a problem on 12.6 or 13.2 on Windows.
Do you have a plan to backpatch this commit into REL_12_STABLE or REL_13_STABLE ?
The commit:
https://github.com/postgres/postgres/commit/bed90759fcbcd72d4d06969eebab81e47326f9a2
https://github.com/postgres/postgres/commit/ed30b1a60dadf2b7cc58bce5009ad8676b8fe479
------
Best regards
Shenhao Wang
"wangsh.fnst@fujitsu.com" <wangsh.fnst@fujitsu.com> writes:
Do you have a plan to backpatch this commit into REL_12_STABLE or REL_13_STABLE ?
/messages/by-id/YCsZIX2A2Ilsvfnl@paquier.xyz
regards, tom lane
On Thu, Feb 25, 2021 at 06:07:06AM +0000, wangsh.fnst@fujitsu.com wrote:
I noticed that this modification only commit into master branch,
there is still have a problem on 12.6 or 13.2 on Windows.Do you have a plan to backpatch this commit into REL_12_STABLE or REL_13_STABLE ?
The change to be able to fix that stuff is invasive. So, while I
don't really object to a backpatch of this change in the future, I
think that it would be wiser to wait until we get more feedback with
the release of Postgres 14 before doing a backpatch to older
versions. So we are in a wait phase for the moment.
Thanks,
--
Michael
Thank you for sharing
Best regards
Shenhao Wang
-----Original Message-----
From: Michael Paquier <michael@paquier.xyz>
Sent: Thursday, February 25, 2021 2:22 PM
To: Wang, Shenhao/王 申豪 <wangsh.fnst@fujitsu.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>; Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>; Emil Iggland <emil@iggland.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: BUG #15858: could not stat file - over 4GB
On Thu, Feb 25, 2021 at 06:07:06AM +0000, wangsh.fnst@fujitsu.com wrote:
I noticed that this modification only commit into master branch,
there is still have a problem on 12.6 or 13.2 on Windows.Do you have a plan to backpatch this commit into REL_12_STABLE or REL_13_STABLE ?
The change to be able to fix that stuff is invasive. So, while I
don't really object to a backpatch of this change in the future, I
think that it would be wiser to wait until we get more feedback with
the release of Postgres 14 before doing a backpatch to older
versions. So we are in a wait phase for the moment.
Thanks,
--
Michael