"buffer too small" or "path too long"?
Recently we added the error messages "buffer for root directory too
small" and siblings to pg_upgrade. This means "<new_cluster's
pgdata>/pg_upgrade_output.d" was longer than MAXPGPATH.
I feel that the "root directory" is obscure here, and moreover "buffer
is too small" looks pointless since no user can do something on the
buffer length. At least I can't read out from the message concretely
what I should do next..
The root cause of the errors is that the user-provided directory path
of new cluster's root was too long. Anywhich one of the four buffers
is overflowed, it doesn't makes any difference for users and doesn't
offer any further detail to suppoerters/developers. I see "output
directory path of new cluster too long" clear enough.
Above all, this change reduces the number of messages that need
translation:)
# And the messages are missing trailing line breaks.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Attachments:
0001-Revise-some-error-messages-of-pg_upgrade.patchtext/x-patch; charset=us-asciiDownload
From 5fffa05aba3ae2aed36ba76e6edee5b465713be5 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Date: Mon, 13 Jun 2022 11:49:12 +0900
Subject: [PATCH] Revise some error messages of pg_upgrade
Some newly added error messages are too detailed and hardly give users
information on what was wrong on their side. And they are missing
trailing newlines. There's no point in differentiating the messages
each other so replace them with one message clear to users.
---
src/bin/pg_upgrade/pg_upgrade.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index ccb048ab2e..737d962658 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -228,7 +228,7 @@ make_outputdirs(char *pgdata)
log_opts.rootdir = (char *) pg_malloc0(MAXPGPATH);
len = snprintf(log_opts.rootdir, MAXPGPATH, "%s/%s", pgdata, BASE_OUTPUTDIR);
if (len >= MAXPGPATH)
- pg_fatal("buffer for root directory too small");
+ pg_fatal("directory path for new cluster too long\n");
/* BASE_OUTPUTDIR/$timestamp/ */
gettimeofday(&time, NULL);
@@ -241,21 +241,21 @@ make_outputdirs(char *pgdata)
len = snprintf(log_opts.basedir, MAXPGPATH, "%s/%s", log_opts.rootdir,
timebuf);
if (len >= MAXPGPATH)
- pg_fatal("buffer for base directory too small");
+ pg_fatal("directory path for new cluster too long\n");
/* BASE_OUTPUTDIR/$timestamp/dump/ */
log_opts.dumpdir = (char *) pg_malloc0(MAXPGPATH);
len = snprintf(log_opts.dumpdir, MAXPGPATH, "%s/%s/%s", log_opts.rootdir,
timebuf, DUMP_OUTPUTDIR);
if (len >= MAXPGPATH)
- pg_fatal("buffer for dump directory too small");
+ pg_fatal("directory path for new cluster too long\n");
/* BASE_OUTPUTDIR/$timestamp/log/ */
log_opts.logdir = (char *) pg_malloc0(MAXPGPATH);
len = snprintf(log_opts.logdir, MAXPGPATH, "%s/%s/%s", log_opts.rootdir,
timebuf, LOG_OUTPUTDIR);
if (len >= MAXPGPATH)
- pg_fatal("buffer for log directory too small");
+ pg_fatal("directory path for new cluster too long\n");
/*
* Ignore the error case where the root path exists, as it is kept the
--
2.31.1
Kyotaro Horiguchi <horikyota.ntt@gmail.com> writes:
The root cause of the errors is that the user-provided directory path
of new cluster's root was too long. Anywhich one of the four buffers
is overflowed, it doesn't makes any difference for users and doesn't
offer any further detail to suppoerters/developers. I see "output
directory path of new cluster too long" clear enough.
+1, but I'm inclined to make it read "... is too long".
# And the messages are missing trailing line breaks.
I was about to question that, but now I remember that pg_upgrade has
its own logging facility with a different idea about who provides
the trailing newline than common/logging.[hc] has. Undoubtedly
that's the source of this mistake. We really need to get pg_upgrade
out of the business of having its own logging conventions.
regards, tom lane
At Mon, 13 Jun 2022 13:25:01 -0400, Tom Lane <tgl@sss.pgh.pa.us> wrote in
Kyotaro Horiguchi <horikyota.ntt@gmail.com> writes:
The root cause of the errors is that the user-provided directory path
of new cluster's root was too long. Anywhich one of the four buffers
is overflowed, it doesn't makes any difference for users and doesn't
offer any further detail to suppoerters/developers. I see "output
directory path of new cluster too long" clear enough.+1, but I'm inclined to make it read "... is too long".
Yeah, I feel so and it is what I wondered about recently when I saw
some complete error messages. Is that because of the length of the
subject?
# And the messages are missing trailing line breaks.
I was about to question that, but now I remember that pg_upgrade has
its own logging facility with a different idea about who provides
the trailing newline than common/logging.[hc] has. Undoubtedly
that's the source of this mistake. We really need to get pg_upgrade
out of the business of having its own logging conventions.
Yes... I don't find a written reason excluding pg_upgrade in both the
commit 9a374b77fb and or the thread [1]/messages/by-id/941719.1645587865@sss.pgh.pa.us. But I guess that we decided
that we first provide the facility in the best style ignoring the
current impletent in pg_upgrade then let pg_upgrade use it. So I
think it should emerge in the next cycle? I'll give it a shot if no
one is willing to do that for now. (I believe it is straightforward..)
[1]: /messages/by-id/941719.1645587865@sss.pgh.pa.us
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
At Tue, 14 Jun 2022 09:48:26 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in
At Mon, 13 Jun 2022 13:25:01 -0400, Tom Lane <tgl@sss.pgh.pa.us> wrote in
+1, but I'm inclined to make it read "... is too long".
Yeah, I feel so and it is what I wondered about recently when I saw
some complete error messages. Is that because of the length of the
subject?
And I found that it is alrady done. Thanks!
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Tue, Jun 14, 2022 at 09:52:52AM +0900, Kyotaro Horiguchi wrote:
At Tue, 14 Jun 2022 09:48:26 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in
Yeah, I feel so and it is what I wondered about recently when I saw
some complete error messages. Is that because of the length of the
subject?And I found that it is alrady done. Thanks!
I have noticed this thread and 4e54d23 as a result this morning. If
you want to spread this style more, wouldn't it be better to do that
in all the places of pg_upgrade where we store paths to files? I can
see six code paths with log_opts.basedir that could do the same, as of
the attached. The hardcoded file names have various lengths, and some
of them are quite long making the generated paths more exposed to
being cut in the middle.
--
Michael
Attachments:
upgrade-paths.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index ace7387eda..e1dc031c24 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -704,12 +704,15 @@ check_proper_datallowconn(ClusterInfo *cluster)
FILE *script = NULL;
char output_path[MAXPGPATH];
bool found = false;
+ int len;
prep_status("Checking database connection settings");
- snprintf(output_path, sizeof(output_path), "%s/%s",
- log_opts.basedir,
- "databases_with_datallowconn_false.txt");
+ len = snprintf(output_path, sizeof(output_path), "%s/%s",
+ log_opts.basedir,
+ "databases_with_datallowconn_false.txt");
+ if (len >= sizeof(output_path))
+ pg_fatal("directory path for new cluster is too long\n");
conn_template1 = connectToServer(cluster, "template1");
@@ -823,6 +826,7 @@ check_for_isn_and_int8_passing_mismatch(ClusterInfo *cluster)
FILE *script = NULL;
bool found = false;
char output_path[MAXPGPATH];
+ int len;
prep_status("Checking for contrib/isn with bigint-passing mismatch");
@@ -834,9 +838,11 @@ check_for_isn_and_int8_passing_mismatch(ClusterInfo *cluster)
return;
}
- snprintf(output_path, sizeof(output_path), "%s/%s",
- log_opts.basedir,
- "contrib_isn_and_int8_pass_by_value.txt");
+ len = snprintf(output_path, sizeof(output_path), "%s/%s",
+ log_opts.basedir,
+ "contrib_isn_and_int8_pass_by_value.txt");
+ if (len >= sizeof(output_path))
+ pg_fatal("directory path for new cluster is too long\n");
for (dbnum = 0; dbnum < cluster->dbarr.ndbs; dbnum++)
{
@@ -909,12 +915,15 @@ check_for_user_defined_postfix_ops(ClusterInfo *cluster)
FILE *script = NULL;
bool found = false;
char output_path[MAXPGPATH];
+ int len;
prep_status("Checking for user-defined postfix operators");
- snprintf(output_path, sizeof(output_path), "%s/%s",
- log_opts.basedir,
- "postfix_ops.txt");
+ len = snprintf(output_path, sizeof(output_path), "%s/%s",
+ log_opts.basedir,
+ "postfix_ops.txt");
+ if (len >= sizeof(output_path))
+ pg_fatal("directory path for new cluster is too long\n");
/* Find any user defined postfix operators */
for (dbnum = 0; dbnum < cluster->dbarr.ndbs; dbnum++)
@@ -1009,12 +1018,15 @@ check_for_tables_with_oids(ClusterInfo *cluster)
FILE *script = NULL;
bool found = false;
char output_path[MAXPGPATH];
+ int len;
prep_status("Checking for tables WITH OIDS");
- snprintf(output_path, sizeof(output_path), "%s/%s",
- log_opts.basedir,
- "tables_with_oids.txt");
+ len = snprintf(output_path, sizeof(output_path), "%s/%s",
+ log_opts.basedir,
+ "tables_with_oids.txt");
+ if (len >= sizeof(output_path))
+ pg_fatal("directory path for new cluster is too long\n");
/* Find any tables declared WITH OIDS */
for (dbnum = 0; dbnum < cluster->dbarr.ndbs; dbnum++)
@@ -1265,12 +1277,15 @@ check_for_user_defined_encoding_conversions(ClusterInfo *cluster)
FILE *script = NULL;
bool found = false;
char output_path[MAXPGPATH];
+ int len;
prep_status("Checking for user-defined encoding conversions");
- snprintf(output_path, sizeof(output_path), "%s/%s",
- log_opts.basedir,
- "encoding_conversions.txt");
+ len = snprintf(output_path, sizeof(output_path), "%s/%s",
+ log_opts.basedir,
+ "encoding_conversions.txt");
+ if (len >= sizeof(output_path))
+ pg_fatal("directory path for new cluster is too long\n");
/* Find any user defined encoding conversions */
for (dbnum = 0; dbnum < cluster->dbarr.ndbs; dbnum++)
diff --git a/src/bin/pg_upgrade/function.c b/src/bin/pg_upgrade/function.c
index ea785df771..4f65726ed9 100644
--- a/src/bin/pg_upgrade/function.c
+++ b/src/bin/pg_upgrade/function.c
@@ -125,11 +125,14 @@ check_loadable_libraries(void)
FILE *script = NULL;
bool found = false;
char output_path[MAXPGPATH];
+ int len;
prep_status("Checking for presence of required libraries");
- snprintf(output_path, sizeof(output_path), "%s/%s",
- log_opts.basedir, "loadable_libraries.txt");
+ len = snprintf(output_path, sizeof(output_path), "%s/%s",
+ log_opts.basedir, "loadable_libraries.txt");
+ if (len >= sizeof(output_path))
+ pg_fatal("directory path for new cluster is too long\n");
/*
* Now we want to sort the library names into order. This avoids multiple
Michael Paquier <michael@paquier.xyz> writes:
I have noticed this thread and 4e54d23 as a result this morning. If
you want to spread this style more, wouldn't it be better to do that
in all the places of pg_upgrade where we store paths to files? I can
see six code paths with log_opts.basedir that could do the same, as of
the attached. The hardcoded file names have various lengths, and some
of them are quite long making the generated paths more exposed to
being cut in the middle.
Well, I just fixed the ones in make_outputdirs because it seemed weird
that that part of the function was not doing something the earlier parts
did. I didn't look around for more trouble.
I think that pg_fatal'ing on the grounds of path-too-long once we've
already started the upgrade isn't all that great. Really we want to
fail on that early on --- so coding make_outputdirs like this is
fine, but maybe we need a different plan for files made later.
regards, tom lane
Kyotaro Horiguchi <horikyota.ntt@gmail.com> writes:
At Mon, 13 Jun 2022 13:25:01 -0400, Tom Lane <tgl@sss.pgh.pa.us> wrote in
I was about to question that, but now I remember that pg_upgrade has
its own logging facility with a different idea about who provides
the trailing newline than common/logging.[hc] has. Undoubtedly
that's the source of this mistake. We really need to get pg_upgrade
out of the business of having its own logging conventions.
Yes... I don't find a written reason excluding pg_upgrade in both the
commit 9a374b77fb and or the thread [1].
Well, as far as 9a374b77fb went, Peter had left pg_upgrade out of the
mix in the original creation of common/logging.c, and I didn't think
that dealing with that was a reasonable part of my update patch.
But I guess that we decided
that we first provide the facility in the best style ignoring the
current impletent in pg_upgrade then let pg_upgrade use it. So I
think it should emerge in the next cycle? I'll give it a shot if no
one is willing to do that for now. (I believe it is straightforward..)
Actually, I spent some time earlier today looking into that, and I can
see why Peter stayed away from it :-(. There are a few issues:
* The inconsistency with the rest of the world about trailing newlines.
That aspect actually seems fixable fairly easily, and I have a patch
mostly done for it.
* logging.c believes it should prefix every line of output with the
program's name and so on. This doesn't seem terribly appropriate
for pg_upgrade's use --- at least, not unless we make pg_upgrade
WAY less chatty. Perhaps that'd be fine, I dunno.
* pg_upgrade's pg_log_v duplicates all (well, most) stdout messages
into the INTERNAL_LOG_FILE log file, something logging.c has no
provision for (and it'd not be too easy to do, because of the C
standard's restrictions on use of va_list). Personally I'd be okay
with nuking the INTERNAL_LOG_FILE log file from orbit, but I bet
somebody will fight to keep it.
* pg_log_v has also got a bunch of specialized rules around how
to format PG_STATUS message traffic. Again I wonder how useful
that whole behavior really is, but taking it out would be a big
user-visible change.
In short, it seems like pg_upgrade's logging habits are sufficiently
far out in left field that we couldn't rebase it on top of logging.c
without some seriously large user-visible behavioral changes.
I have better things to spend my time on than advocating for that.
However, the inconsistency in newline handling is a problem:
I found that there are already other bugs with missing or extra
newlines, and it will only get worse if we don't unify that
behavior. So my inclination for now is to fix that and let the
other issues go. Patch coming.
regards, tom lane
On Mon, Jun 13, 2022 at 10:41:41PM -0400, Tom Lane wrote:
* logging.c believes it should prefix every line of output with the
program's name and so on. This doesn't seem terribly appropriate
for pg_upgrade's use --- at least, not unless we make pg_upgrade
WAY less chatty. Perhaps that'd be fine, I dunno.
pg_upgrade was designed to be chatty because it felt it could fail under
unpredictable circumstances --- I am not sure how true that is today.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Indecision is a decision. Inaction is an action. Mark Batterson
On 14.06.22 03:55, Michael Paquier wrote:
On Tue, Jun 14, 2022 at 09:52:52AM +0900, Kyotaro Horiguchi wrote:
At Tue, 14 Jun 2022 09:48:26 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in
Yeah, I feel so and it is what I wondered about recently when I saw
some complete error messages. Is that because of the length of the
subject?And I found that it is alrady done. Thanks!
I have noticed this thread and 4e54d23 as a result this morning. If
you want to spread this style more, wouldn't it be better to do that
in all the places of pg_upgrade where we store paths to files? I can
see six code paths with log_opts.basedir that could do the same, as of
the attached. The hardcoded file names have various lengths, and some
of them are quite long making the generated paths more exposed to
being cut in the middle.
We have this problem of long file names being silently truncated all
over the source code. Instead of equipping each one of them with a
length check, why don't we get rid of the fixed-size buffers and
allocate dynamically, as in the attached patch.
Attachments:
0001-pg_upgrade-Use-psprintf-in-make_outputdirs.patchtext/plain; charset=UTF-8; name=0001-pg_upgrade-Use-psprintf-in-make_outputdirs.patchDownload
From 1a21434c584319b28fd483444aa24b7b54b4f949 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 15 Jun 2022 07:25:07 +0200
Subject: [PATCH] pg_upgrade: Use psprintf in make_outputdirs
---
src/bin/pg_upgrade/pg_upgrade.c | 42 ++++++++-------------------------
1 file changed, 10 insertions(+), 32 deletions(-)
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 265d829490..f7e3461cfe 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -216,19 +216,13 @@ main(int argc, char **argv)
static void
make_outputdirs(char *pgdata)
{
- FILE *fp;
- char **filename;
time_t run_time = time(NULL);
- char filename_path[MAXPGPATH];
+ char *filename_path;
char timebuf[128];
struct timeval time;
time_t tt;
- int len;
- log_opts.rootdir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(log_opts.rootdir, MAXPGPATH, "%s/%s", pgdata, BASE_OUTPUTDIR);
- if (len >= MAXPGPATH)
- pg_fatal("directory path for new cluster is too long\n");
+ log_opts.rootdir = psprintf("%s/%s", pgdata, BASE_OUTPUTDIR);
/* BASE_OUTPUTDIR/$timestamp/ */
gettimeofday(&time, NULL);
@@ -237,25 +231,13 @@ make_outputdirs(char *pgdata)
/* append milliseconds */
snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
".%03d", (int) (time.tv_usec / 1000));
- log_opts.basedir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(log_opts.basedir, MAXPGPATH, "%s/%s", log_opts.rootdir,
- timebuf);
- if (len >= MAXPGPATH)
- pg_fatal("directory path for new cluster is too long\n");
+ log_opts.basedir = psprintf("%s/%s", log_opts.rootdir, timebuf);
/* BASE_OUTPUTDIR/$timestamp/dump/ */
- log_opts.dumpdir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(log_opts.dumpdir, MAXPGPATH, "%s/%s/%s", log_opts.rootdir,
- timebuf, DUMP_OUTPUTDIR);
- if (len >= MAXPGPATH)
- pg_fatal("directory path for new cluster is too long\n");
+ log_opts.dumpdir = psprintf("%s/%s/%s", log_opts.rootdir, timebuf, DUMP_OUTPUTDIR);
/* BASE_OUTPUTDIR/$timestamp/log/ */
- log_opts.logdir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(log_opts.logdir, MAXPGPATH, "%s/%s/%s", log_opts.rootdir,
- timebuf, LOG_OUTPUTDIR);
- if (len >= MAXPGPATH)
- pg_fatal("directory path for new cluster is too long\n");
+ log_opts.logdir = psprintf("%s/%s/%s", log_opts.rootdir, timebuf, LOG_OUTPUTDIR);
/*
* Ignore the error case where the root path exists, as it is kept the
@@ -270,21 +252,17 @@ make_outputdirs(char *pgdata)
if (mkdir(log_opts.logdir, pg_dir_create_mode) < 0)
pg_fatal("could not create directory \"%s\": %m\n", log_opts.logdir);
- len = snprintf(filename_path, sizeof(filename_path), "%s/%s",
- log_opts.logdir, INTERNAL_LOG_FILE);
- if (len >= sizeof(filename_path))
- pg_fatal("directory path for new cluster is too long\n");
+ filename_path = psprintf("%s/%s", log_opts.logdir, INTERNAL_LOG_FILE);
if ((log_opts.internal = fopen_priv(filename_path, "a")) == NULL)
pg_fatal("could not open log file \"%s\": %m\n", filename_path);
/* label start of upgrade in logfiles */
- for (filename = output_files; *filename != NULL; filename++)
+ for (char **filename = output_files; *filename != NULL; filename++)
{
- len = snprintf(filename_path, sizeof(filename_path), "%s/%s",
- log_opts.logdir, *filename);
- if (len >= sizeof(filename_path))
- pg_fatal("directory path for new cluster is too long\n");
+ FILE *fp;
+
+ filename_path = psprintf("%s/%s", log_opts.logdir, *filename);
if ((fp = fopen_priv(filename_path, "a")) == NULL)
pg_fatal("could not write to log file \"%s\": %m\n", filename_path);
--
2.36.1
On Wed, Jun 15, 2022 at 2:51 AM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:
We have this problem of long file names being silently truncated all
over the source code. Instead of equipping each one of them with a
length check, why don't we get rid of the fixed-size buffers and
allocate dynamically, as in the attached patch.
I've always wondered why we rely on MAXPGPATH instead of dynamic
allocation. It seems pretty lame.
I don't know how much we gain by fixing one place and not all the
others, but maybe it would set a trend.
--
Robert Haas
EDB: http://www.enterprisedb.com
Robert Haas <robertmhaas@gmail.com> writes:
On Wed, Jun 15, 2022 at 2:51 AM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:We have this problem of long file names being silently truncated all
over the source code. Instead of equipping each one of them with a
length check, why don't we get rid of the fixed-size buffers and
allocate dynamically, as in the attached patch.
I don't know how much we gain by fixing one place and not all the
others, but maybe it would set a trend.
Yeah, that was what was bugging me about this proposal. Removing
one function's dependency on MAXPGPATH isn't much of a step forward.
I note also that the patch leaks quite a lot of memory (a kilobyte or
so per pathname, IIRC). That's probably negligible in this particular
context, but anyplace that was called more than once per program run
would need to be more tidy.
regards, tom lane
On Wed, Jun 15, 2022 at 02:02:03PM -0400, Tom Lane wrote:
Yeah, that was what was bugging me about this proposal. Removing
one function's dependency on MAXPGPATH isn't much of a step forward.
This comes down to out-of-memory vs path length at the end. Changing
only the paths of make_outputdirs() without touching all the paths in
check.c and the one in function.c does not sound good to me, as this
increases the risk of failing pg_upgrade in the middle, and that's
what we should avoid, as said upthread.
I note also that the patch leaks quite a lot of memory (a kilobyte or
so per pathname, IIRC). That's probably negligible in this particular
context, but anyplace that was called more than once per program run
would need to be more tidy.
Surely.
--
Michael
On 15.06.22 19:08, Robert Haas wrote:
On Wed, Jun 15, 2022 at 2:51 AM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:We have this problem of long file names being silently truncated all
over the source code. Instead of equipping each one of them with a
length check, why don't we get rid of the fixed-size buffers and
allocate dynamically, as in the attached patch.I've always wondered why we rely on MAXPGPATH instead of dynamic
allocation. It seems pretty lame.
I think it came in before we had extensible string buffers APIs.