pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi
Allow concurrent-safe open() and fopen() in frontend code for Windows
PostgreSQL uses a custom wrapper for open() and fopen() which is
concurrent-safe, allowing multiple processes to open and work on the
same file. This has a couple of advantages:
- pg_test_fsync does not handle O_DSYNC correctly otherwise, leading to
false claims that disks are unsafe.
- TAP tests can run into race conditions when a postmaster and pg_ctl
open postmaster.pid, fixing some random failures in the buildfam.
pg_upgrade is one frontend tool using workarounds to bypass file locking
issues with the log files it generates, however the interactions with
pg_ctl are proving to be tedious to get rid of, so this is left for
later.
Author: Laurenz Albe
Reviewed-by: Michael Paquier, Kuntal Ghosh
Discussion: /messages/by-id/1527846213.2475.31.camel@cybertec.at
Discussion: /messages/by-id/16922.1520722108@sss.pgh.pa.us
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/0ba06e0bfb8cfd24ff17aca92aa72245ddd6c4d7
Modified Files
--------------
src/bin/initdb/initdb.c | 8 ++++++++
src/bin/pg_basebackup/pg_receivewal.c | 2 +-
src/bin/pg_verify_checksums/pg_verify_checksums.c | 2 +-
src/common/file_utils.c | 4 ++--
src/include/port.h | 3 ---
5 files changed, 12 insertions(+), 7 deletions(-)
Michael Paquier <michael@paquier.xyz> writes:
Allow concurrent-safe open() and fopen() in frontend code for Windows
I'm surprised you didn't back-patch this --- isn't it a bug fix?
A compromise might be to push it to v11 but not further.
regards, tom lane
On Fri, Sep 14, 2018 at 11:22:55AM -0400, Tom Lane wrote:
I'm surprised you didn't back-patch this --- isn't it a bug fix?
A compromise might be to push it to v11 but not further.
Yes, that's clearly a bug fix from pg_ctl point of view with TAP tests.
However, I have rather cold feet about back-patching for two reasons as
this introduces a behavior change:
- The concurrency part disappears.
- Caller of open() needs to enforce text mode to strip correctly CRLF
characters.
I would be fine to get that into v11 though, as that is not released
yet. You will have to wait for a couple of days though, there is a long
week-end here away from laptops and electronic devices ;)
--
Michael
On Sat, Sep 15, 2018 at 07:49:39AM +0900, Michael Paquier wrote:
I would be fine to get that into v11 though, as that is not released
yet. You will have to wait for a couple of days though, there is a long
week-end here away from laptops and electronic devices ;)
OK, REL_11_STABLE has been patched as well, after doing a couple of
extra tests on Windows.
--
Michael
Michael Paquier <michael@paquier.xyz> writes:
OK, REL_11_STABLE has been patched as well, after doing a couple of
extra tests on Windows.
BTW, I'm a bit concerned by the fact that bowerbird has failed its
last couple of HEAD runs at the pgbench step. The first such
failure was here:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=bowerbird&dt=2018-09-15%2014%3A19%3A58
Looking at the set of commits between the prior run and that one,
it's hard to see anything that could have triggered the test failures
other than this patch --- but I also don't see how this patch would've
blown up pgbench without breaking earlier tests. Ideas?
regards, tom lane
On Mon, Sep 17, 2018 at 09:41:37AM -0400, Tom Lane wrote:
BTW, I'm a bit concerned by the fact that bowerbird has failed its
last couple of HEAD runs at the pgbench step. The first such
failure was here:https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=bowerbird&dt=2018-09-15%2014%3A19%3A58
Looking at the set of commits between the prior run and that one,
it's hard to see anything that could have triggered the test failures
other than this patch --- but I also don't see how this patch would've
blown up pgbench without breaking earlier tests. Ideas?
Thanks, I have been looking at the build farm but I missed this one.
dory, which uses VS 2015 is not complaining because it does not run
bincheck. At quick glance, it seems to be caused by process_file() in
pgbench.c which would need to open files in text mode, and the input
file parsing fails at the first '\' character found. I'll test that
stuff on tomorrow morning manually.
--
Michael
Michael Paquier <michael@paquier.xyz> writes:
On Mon, Sep 17, 2018 at 09:41:37AM -0400, Tom Lane wrote:
Looking at the set of commits between the prior run and that one,
it's hard to see anything that could have triggered the test failures
other than this patch --- but I also don't see how this patch would've
blown up pgbench without breaking earlier tests. Ideas?
Thanks, I have been looking at the build farm but I missed this one.
dory, which uses VS 2015 is not complaining because it does not run
bincheck. At quick glance, it seems to be caused by process_file() in
pgbench.c which would need to open files in text mode, and the input
file parsing fails at the first '\' character found.
Oh, you're thinking pgbench isn't robust against finding \r's visible
in its input? Could be.
I'll test that stuff on tomorrow morning manually.
We've got a bit of a timing problem because we want to wrap 11beta4/rc1
(still TBD) in a few hours. I'll take a look and see if I can push a
quick fix before that.
regards, tom lane
On 09/17/2018 10:48 AM, Tom Lane wrote:
Michael Paquier <michael@paquier.xyz> writes:
On Mon, Sep 17, 2018 at 09:41:37AM -0400, Tom Lane wrote:
Looking at the set of commits between the prior run and that one,
it's hard to see anything that could have triggered the test failures
other than this patch --- but I also don't see how this patch would've
blown up pgbench without breaking earlier tests. Ideas?Thanks, I have been looking at the build farm but I missed this one.
dory, which uses VS 2015 is not complaining because it does not run
bincheck. At quick glance, it seems to be caused by process_file() in
pgbench.c which would need to open files in text mode, and the input
file parsing fails at the first '\' character found.Oh, you're thinking pgbench isn't robust against finding \r's visible
in its input? Could be.I'll test that stuff on tomorrow morning manually.
We've got a bit of a timing problem because we want to wrap 11beta4/rc1
(still TBD) in a few hours. I'll take a look and see if I can push a
quick fix before that.
When you do I'll start a bowerbird run to check it.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
On 09/17/2018 10:48 AM, Tom Lane wrote:
We've got a bit of a timing problem because we want to wrap 11beta4/rc1
(still TBD) in a few hours. I'll take a look and see if I can push a
quick fix before that.
When you do I'll start a bowerbird run to check it.
Pushed, please test.
I think there's a general issue here of exactly how we want pgwin32_fopen
to behave, but the immediate problem is best fixed by making pgbench deal
with Windows newlines more thoroughly.
regards, tom lane
On 09/17/2018 12:13 PM, Tom Lane wrote:
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
On 09/17/2018 10:48 AM, Tom Lane wrote:
We've got a bit of a timing problem because we want to wrap 11beta4/rc1
(still TBD) in a few hours. I'll take a look and see if I can push a
quick fix before that.When you do I'll start a bowerbird run to check it.
Pushed, please test.
I think there's a general issue here of exactly how we want pgwin32_fopen
to behave, but the immediate problem is best fixed by making pgbench deal
with Windows newlines more thoroughly.
Tests are still running, but it's past the pgbench stage on HEAD, so I
think we're good.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
On 09/17/2018 12:13 PM, Tom Lane wrote:
Pushed, please test.
Tests are still running, but it's past the pgbench stage on HEAD, so I
think we're good.
I was more concerned about whether any of the post-pgbench steps would
show a failure; but it looks like HEAD's green, so probably v11 is too.
regards, tom lane
So we seem to be out of the woods in terms of 0ba06e0bf breaking the
regression tests, but I'm not very happy about the whole thing, because
that patch wasn't supposed to change the behavior of open/fopen in any
way other than allowing concurrent file access. Obviously, it did.
After looking at src/port/open.c a bit, it seems like the problem is
that our fopen() requires a nonstandard "t" option to select text mode.
I'd always supposed that you got binary mode with "b" and text mode
otherwise; is there some third possibility on Windows?
Anyway, I'm inclined to propose that we ought to do something like
the attached in HEAD and v11 in order to reduce the risk that there
are other unexpected behavioral changes. This should also let us
revert 0ba06e0bf's change in initdb.c.
I wonder whether pgwin32_open() also ought to enforce that either
O_BINARY or O_TEXT mode gets selected.
regards, tom lane
Attachments:
force-either-binary-or-text-open-mode.patchtext/x-diff; charset=us-ascii; name=force-either-binary-or-text-open-mode.patchDownload
diff --git a/src/port/open.c b/src/port/open.c
index a3ad946..85ab06e 100644
--- a/src/port/open.c
+++ b/src/port/open.c
@@ -154,7 +154,7 @@ pgwin32_fopen(const char *fileName, const char *mode)
if (strchr(mode, 'b'))
openmode |= O_BINARY;
- if (strchr(mode, 't'))
+ else
openmode |= O_TEXT;
fd = pgwin32_open(fileName, openmode);
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index b53d6eb..cb8c745 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -491,15 +491,7 @@ readfile(const char *path)
char *buffer;
int c;
-#ifdef WIN32
- /*
- * On Windows, we have to open the file in text mode so that carriage
- * returns are stripped.
- */
- if ((infile = fopen(path, "rt")) == NULL)
-#else
if ((infile = fopen(path, "r")) == NULL)
-#endif
{
fprintf(stderr, _("%s: could not open file \"%s\" for reading: %s\n"),
progname, path, strerror(errno));
On Mon, Sep 17, 2018 at 07:38:24PM -0400, Tom Lane wrote:
So we seem to be out of the woods in terms of 0ba06e0bf breaking the
regression tests, but I'm not very happy about the whole thing, because
that patch wasn't supposed to change the behavior of open/fopen in any
way other than allowing concurrent file access. Obviously, it did.
Thanks! I can see that as well.
After looking at src/port/open.c a bit, it seems like the problem is
that our fopen() requires a nonstandard "t" option to select text mode.
I'd always supposed that you got binary mode with "b" and text mode
otherwise; is there some third possibility on Windows?
There is a sort of automatic mode... Please see below.
Anyway, I'm inclined to propose that we ought to do something like
the attached in HEAD and v11 in order to reduce the risk that there
are other unexpected behavioral changes. This should also let us
revert 0ba06e0bf's change in initdb.c.I wonder whether pgwin32_open() also ought to enforce that either
O_BINARY or O_TEXT mode gets selected.
There is no need to go down that road I think, a lot of code paths
already call setmode to make sure that binary or text modes are used.
diff --git a/src/port/open.c b/src/port/open.c index a3ad946..85ab06e 100644 --- a/src/port/open.c +++ b/src/port/open.c @@ -154,7 +154,7 @@ pgwin32_fopen(const char *fileName, const char *mode)if (strchr(mode, 'b'))
openmode |= O_BINARY;
- if (strchr(mode, 't'))
+ else
openmode |= O_TEXT;
Hm. I don't think that this is correct either. The whole logic behind
the mode selected depends on how setmode() is being set to. Please see
here:
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setmode?view=vs-2017
And the point is that if neither 'b' nor 't' are set, then open() would
use the value set by the process, which is 't' by default if not set.
And initdb does not set that, so it would use 't'. miscinit.c sets it
to 'b', and your patch would likely cause some backend code to be
broken. So it seems to me that if 'b' or 't' are not defined by the
caller, then we ought to use what get_fmode() returns:
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/get-fmode?view=vs-2017
What I think I broke is that CreateFile ignores what _fmode uses, which
has caused the breakage, while calling directly open() or fopen() does
the work. There are also other things assuming that binary mode is
used, you can grep for "setmode" and see how miscinit.c or pg_basebackup
do the job.
--
Michael
On Tue, Sep 18, 2018 at 09:11:43AM +0900, Michael Paquier wrote:
What I think I broke is that CreateFile ignores what _fmode uses, which
has caused the breakage, while calling directly open() or fopen() does
the work. There are also other things assuming that binary mode is
used, you can grep for "setmode" and see how miscinit.c or pg_basebackup
do the job.
I have been playing with this stuff, and hacked the attached. Now, while
TAP tests of initdb and pgbench are happy (I can actually see the past
failure as well), pg_dump complains at authentication time when using
plain-text mode when using databases with all ASCII characters. That's
not something I expected first, but _get_fmode also influences
operations like pipe(), which is something that pg_dump uses, and
setmode is enforced to binary mode only when adapted.
I am getting to wonder if what's present on HEAD represents actually the
best deal we could get. Attached is the patch I used for reference.
Thoughts?
--
Michael
Attachments:
win32-open-mode-v1.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index b53d6eb9cc..cb8c7450d9 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -491,15 +491,7 @@ readfile(const char *path)
char *buffer;
int c;
-#ifdef WIN32
- /*
- * On Windows, we have to open the file in text mode so that carriage
- * returns are stripped.
- */
- if ((infile = fopen(path, "rt")) == NULL)
-#else
if ((infile = fopen(path, "r")) == NULL)
-#endif
{
fprintf(stderr, _("%s: could not open file \"%s\" for reading: %s\n"),
progname, path, strerror(errno));
diff --git a/src/port/open.c b/src/port/open.c
index a3ad946a60..f7a296ef28 100644
--- a/src/port/open.c
+++ b/src/port/open.c
@@ -71,6 +71,28 @@ pgwin32_open(const char *fileName, int fileFlags,...)
_O_SHORT_LIVED | O_DSYNC | O_DIRECT |
(O_CREAT | O_TRUNC | O_EXCL) | (O_TEXT | O_BINARY))) == fileFlags);
+ /*
+ * If caller has set neither O_BINARY nor O_TEXT, then look for what
+ * the default mode is for this process, then enforce it.
+ */
+ if ((fileFlags & (O_BINARY | O_TEXT)) == 0)
+ {
+ int pmode = 0;
+
+ /* only MSVC newer than 2015 support _get_fmode */
+#if (_MSC_VER >= 1900)
+ if (_get_fmode(&pmode) < 0)
+ {
+ /* get_fmode sets errno */
+ return -1;
+ }
+#else
+ pmode = _fmode;
+#endif
+
+ fileFlags |= pmode;
+ }
+
sa.nLength = sizeof(sa);
sa.bInheritHandle = TRUE;
sa.lpSecurityDescriptor = NULL;
Michael Paquier <michael@paquier.xyz> writes:
On Tue, Sep 18, 2018 at 09:11:43AM +0900, Michael Paquier wrote:
What I think I broke is that CreateFile ignores what _fmode uses, which
has caused the breakage, while calling directly open() or fopen() does
the work. There are also other things assuming that binary mode is
used, you can grep for "setmode" and see how miscinit.c or pg_basebackup
do the job.
I have been playing with this stuff, and hacked the attached. Now, while
TAP tests of initdb and pgbench are happy (I can actually see the past
failure as well), pg_dump complains at authentication time when using
plain-text mode when using databases with all ASCII characters. That's
not something I expected first, but _get_fmode also influences
operations like pipe(), which is something that pg_dump uses, and
setmode is enforced to binary mode only when adapted.
I am getting to wonder if what's present on HEAD represents actually the
best deal we could get. Attached is the patch I used for reference.
Thoughts?
Well, we have to do something. I have a report from EDB's packagers
that in 11beta4, "initdb --pwfile" is failing on Windows (ie, one can't
connect afterwards using the specified password). It seems nearly
certain to me that the reason is that the file is read with
FILE *pwf = fopen(pwfilename, "r");
and so the \r isn't getting stripped from what's used as the password.
So my opinion is that it's not negotiable that we have to restore
the previous behavior in this realm. I don't want to be running
around finding other cases one at a time, and I absolutely won't
hold still for putting an "#ifdef WIN32" around each such case.
(Even if we fixed all our own code, we'd still be breaking 3rd-party
code.)
What I don't understand yet is why your latest patch doesn't restore
the previous behavior. What exactly is still different?
In the meantime, we might be well advised to revert this patch in
v11 and just continue to work on the problem in HEAD. I see now
that this wasn't something to cram in during late beta ...
regards, tom lane
Tom Lane wrote:
Well, we have to do something. I have a report from EDB's packagers
that in 11beta4, "initdb --pwfile" is failing on Windows (ie, one can't
connect afterwards using the specified password). It seems nearly
certain to me that the reason is that the file is read withFILE *pwf = fopen(pwfilename, "r");
and so the \r isn't getting stripped from what's used as the password.
Perhaps there is something obvious that I'm missing, but it seems that
all the problems we observe are caused by frontend code suddenly defaulting
to binary mode when it was text mode before.
Would it be an option to have pgwin32_open default to text mode in
frontend code and to binary mode in backend code?
Yours,
Laurenz Albe
Laurenz Albe <laurenz.albe@cybertec.at> writes:
Would it be an option to have pgwin32_open default to text mode in
frontend code and to binary mode in backend code?
Well, the question is why Michael's latest proposed patch doesn't
accomplish that.
regards, tom lane
Tom Lane wrote:
Laurenz Albe <laurenz.albe@cybertec.at> writes:
Would it be an option to have pgwin32_open default to text mode in
frontend code and to binary mode in backend code?Well, the question is why Michael's latest proposed patch doesn't
accomplish that.
I was thinking of something trivial like this:
--- a/src/port/open.c
+++ b/src/port/open.c
@@ -71,6 +71,12 @@ pgwin32_open(const char *fileName, int fileFlags,...)
_O_SHORT_LIVED | O_DSYNC | O_DIRECT |
(O_CREAT | O_TRUNC | O_EXCL) | (O_TEXT | O_BINARY))) == fileFlags);+#ifdef FRONTEND
+ /* default to text mode in frontend code */
+ if (fileFlags & O_BINARY == 0)
+ fileFlags |= O_TEXT;
+#endif
+
sa.nLength = sizeof(sa);
sa.bInheritHandle = TRUE;
sa.lpSecurityDescriptor = NULL;That wouldn't influence pipes, which was what Michael said was a
problem for pg_dump.
I currently have no Windows system close, so I cannot test...
Yours,
Laurenz Albe
On Tue, Sep 18, 2018 at 10:45:09AM -0400, Tom Lane wrote:
In the meantime, we might be well advised to revert this patch in
v11 and just continue to work on the problem in HEAD. I see now
that this wasn't something to cram in during late beta ...
I can see that you have reverted the change just before I was able to
reply. Thanks I was going to do the same. Also, if we cannot find a
clear solution for HEAD rather soon, say by tomorrow, let's also remove
it there as well and go ack to the blackboard. I still want to test a
couple of other solutions first.
--
Michael
On Tue, Sep 18, 2018 at 06:04:58PM +0200, Laurenz Albe wrote:
That wouldn't influence pipes, which was what Michael said was a
problem for pg_dump.
Yeah, the authentication blows up badly on that.. You can see all the
tests using user and database names with all ASCII range turning red.
I currently have no Windows system close, so I cannot test...
I have spent a large portion of my morning trying to test all the
solutions proposed, and a winner shows up. Attached are three patches
which present all the solutions mentioned, attached here for visibility:
- win32-open-michael.patch, or the solution I was working on. This has
led me actually nowhere. Even trying to enforce _fmode to happen only
on the frontend has caused breakages of pg_dump for authentication.
Trying to be smarter than what other binaries do is nice and consistent
with the Windows experience I think, still it looks that this can lead
to breakages when a utility uses setmode() for some of the output
files. I noticed that pgwin32_open missed to enforce the mode used when
calling _fdmode, still that solves nothing.
- win32-open-tom.patch, which switches pgwin32_fopen() to use text mode
all the time if binary is not specified. While this looked promising,
I have noticed that this has been causing the same set of errors as what
my previous patch has been doing in pg_dump TAP tests. Anyway, a
solution needs also to happen for pgwin32_open() as we want direct calls
to it to get the right call.
- win32-open-laurenz.patch, which enforces to text mode only if binary
mode is not defined, which maps strictly to what pre-11 is doing when
calling the system _open or _fopen. And surprisingly, this is proving
to pass all the tests I ran: bincheck (including pgbench and pg_dump),
upgradecheck, recoverycheck, check, etc. initdb --pwfile is not
complaining to me either.
So the odds are that Laurenz's solution is what we are looking for.
Laurenz, Tom, any thoughts to share? I won't back-patch that ;)
--
Michael
Attachments:
win32-open-michael.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index b53d6eb9cc..cb8c7450d9 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -491,15 +491,7 @@ readfile(const char *path)
char *buffer;
int c;
-#ifdef WIN32
- /*
- * On Windows, we have to open the file in text mode so that carriage
- * returns are stripped.
- */
- if ((infile = fopen(path, "rt")) == NULL)
-#else
if ((infile = fopen(path, "r")) == NULL)
-#endif
{
fprintf(stderr, _("%s: could not open file \"%s\" for reading: %s\n"),
progname, path, strerror(errno));
diff --git a/src/port/open.c b/src/port/open.c
index a3ad946a60..9174cdce66 100644
--- a/src/port/open.c
+++ b/src/port/open.c
@@ -122,11 +122,35 @@ pgwin32_open(const char *fileName, int fileFlags,...)
return -1;
}
+ /*
+ * If caller has set neither O_BINARY nor O_TEXT, then look for what
+ * the default mode is for this process, then enforce it. This can
+ * be done only after opening the file and before switching the file
+ * mode.
+ */
+ if ((fileFlags & (O_BINARY | O_TEXT)) == 0)
+ {
+ int pmode = 0;
+
+ /* only MSVC newer than 2015 support _get_fmode */
+#if (_MSC_VER >= 1900)
+ if (_get_fmode(&pmode) < 0)
+ {
+ /* _get_fmode sets errno */
+ CloseHandle(h);
+ return -1;
+ }
+#else
+ pmode = _fmode;
+#endif
+
+ fileFlags |= pmode;
+ }
+
/* _open_osfhandle will, on error, set errno accordingly */
if ((fd = _open_osfhandle((intptr_t) h, fileFlags & O_APPEND)) < 0)
CloseHandle(h); /* will not affect errno */
- else if (fileFlags & (O_TEXT | O_BINARY) &&
- _setmode(fd, fileFlags & (O_TEXT | O_BINARY)) < 0)
+ else if (_setmode(fd, fileFlags & (O_TEXT | O_BINARY)) < 0)
{
_close(fd);
return -1;
@@ -138,6 +162,7 @@ pgwin32_open(const char *fileName, int fileFlags,...)
FILE *
pgwin32_fopen(const char *fileName, const char *mode)
{
+ char fdmode[32];
int openmode = 0;
int fd;
@@ -160,7 +185,36 @@ pgwin32_fopen(const char *fileName, const char *mode)
fd = pgwin32_open(fileName, openmode);
if (fd == -1)
return NULL;
- return _fdopen(fd, mode);
+
+ strcpy(fdmode, mode);
+
+ /*
+ * Like pgwin32_open, look for the default mode to be used for this
+ * file descriptor. Note that it is important to do that again here
+ * for _fdopen below.
+ */
+ if ((openmode & (O_BINARY | O_TEXT)) == 0)
+ {
+ int pmode = 0;
+
+ /* only MSVC newer than 2015 support _get_fmode */
+#if (_MSC_VER >= 1900)
+ if (_get_fmode(&pmode) < 0)
+ {
+ /* get_fmode sets errno */
+ _close(fd);
+ return NULL;
+ }
+#else
+ pmode = _fmode;
+#endif
+
+ snprintf(fdmode, sizeof(fdmode), "%s%s%s", fdmode,
+ (pmode & O_BINARY) != 0 ? "b" : "",
+ (pmode & O_TEXT) != 0 ? "t" : "");
+ }
+
+ return _fdopen(fd, fdmode);
}
#endif
win32-open-laurenz.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index b53d6eb9cc..cb8c7450d9 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -491,15 +491,7 @@ readfile(const char *path)
char *buffer;
int c;
-#ifdef WIN32
- /*
- * On Windows, we have to open the file in text mode so that carriage
- * returns are stripped.
- */
- if ((infile = fopen(path, "rt")) == NULL)
-#else
if ((infile = fopen(path, "r")) == NULL)
-#endif
{
fprintf(stderr, _("%s: could not open file \"%s\" for reading: %s\n"),
progname, path, strerror(errno));
diff --git a/src/port/open.c b/src/port/open.c
index a3ad946a60..63a7a1d2fb 100644
--- a/src/port/open.c
+++ b/src/port/open.c
@@ -71,6 +71,19 @@ pgwin32_open(const char *fileName, int fileFlags,...)
_O_SHORT_LIVED | O_DSYNC | O_DIRECT |
(O_CREAT | O_TRUNC | O_EXCL) | (O_TEXT | O_BINARY))) == fileFlags);
+#ifdef FRONTEND
+ /*
+ * Since PostgreSQL 12, those concurrent-safe versions of open() and
+ * fopen() can be used by frontends, having as side-effect to switch
+ * the file-transaction mode from O_TEXT to O_BINARY if none is
+ * specified. Caller may want to enforce the binary or text mode,
+ * but if nothing is defined make sure that the default mode maps
+ * with what versions older than 12 have been doing.
+ */
+ if ((fileFlags & O_BINARY) == 0)
+ fileFlags |= O_TEXT;
+#endif
+
sa.nLength = sizeof(sa);
sa.bInheritHandle = TRUE;
sa.lpSecurityDescriptor = NULL;
win32-open-tom.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index b53d6eb9cc..cb8c7450d9 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -491,15 +491,7 @@ readfile(const char *path)
char *buffer;
int c;
-#ifdef WIN32
- /*
- * On Windows, we have to open the file in text mode so that carriage
- * returns are stripped.
- */
- if ((infile = fopen(path, "rt")) == NULL)
-#else
if ((infile = fopen(path, "r")) == NULL)
-#endif
{
fprintf(stderr, _("%s: could not open file \"%s\" for reading: %s\n"),
progname, path, strerror(errno));
diff --git a/src/port/open.c b/src/port/open.c
index a3ad946a60..85ab06e518 100644
--- a/src/port/open.c
+++ b/src/port/open.c
@@ -154,7 +154,7 @@ pgwin32_fopen(const char *fileName, const char *mode)
if (strchr(mode, 'b'))
openmode |= O_BINARY;
- if (strchr(mode, 't'))
+ else
openmode |= O_TEXT;
fd = pgwin32_open(fileName, openmode);
Michael Paquier <michael@paquier.xyz> writes:
I have spent a large portion of my morning trying to test all the
solutions proposed, and a winner shows up. ...
- win32-open-laurenz.patch, which enforces to text mode only if binary
mode is not defined, which maps strictly to what pre-11 is doing when
calling the system _open or _fopen. And surprisingly, this is proving
to pass all the tests I ran: bincheck (including pgbench and pg_dump),
upgradecheck, recoverycheck, check, etc. initdb --pwfile is not
complaining to me either.
I'm OK with this approach. I wonder though what happens if you take
away the "#ifdef FRONTEND" and just enforce that one or the other mode
is selected always. That would seem like a sensible solution rather
than a wart to me ...
regards, tom lane
On Wed, Sep 19, 2018 at 11:55:01AM -0400, Tom Lane wrote:
I'm OK with this approach. I wonder though what happens if you take
away the "#ifdef FRONTEND" and just enforce that one or the other mode
is selected always. That would seem like a sensible solution rather
than a wart to me ...
Thanks, I have pushed the solution from Laurenz to maintain pure
compatibility. The origin of the failures reported by pg_dump actually
comes from a mismatch with pg_hba.conf in the way users and/or databases
are parsed. I am glad that we have tests for the full range of ASCII
characters in TAP so as it is easy to detect regressions at early
stages. We could try to manipulate more setmode calls like the one in
miscinit.c so as authentication gets the right call. I am not sure of
other side effects though...
--
Michael