Postmaster crashes on SIGHUP when oauth_validator_libraries holds only whitespace
Hi,
I ran into a postmaster crash while investigating a static analyzer
report against 18.6. I reproduced it on master (92aaf50e230,
--enable-cassert, Debian 13/aarch64).
To reproduce, set the following in postgresql.conf:
oauth_validator_libraries = ' '
and add an OAuth line without validator= to pg_hba.conf:
host all all 127.0.0.1/32 oauth issuer="https://example.com" scope="openid"
On reload the server log ends here:
LOG: received SIGHUP, reloading configuration files
LOG: parameter "oauth_validator_libraries" changed to " "
gdb then reports SIGSEGV in check_oauth_validator(), via load_hba().
The running instance goes down with every session on it. The same
configuration also prevents startup. pg_hba_file_rules() parses the
file from a regular backend, so the same NULL dereference there kills
the backend and the postmaster restarts the cluster.
check_oauth_validator() treats an unset list as an error by looking at
the raw GUC string, but SplitDirectoriesString() accepts a string of
spaces and returns NIL. The subsequent elemlist->length dereferences
that. This is a misconfiguration rather than a security issue -- the
GUC has PGC_SIGHUP context and is marked GUC_SUPERUSER_ONLY.
I've attached a patch against master that checks the parsed list
instead. It also adds a TAP test for reload. The patched master built
without warnings, and "make check" in src/test/modules/oauth_validator
passed with PG_TEST_EXTRA=oauth.
This appears to affect v18 onward, where OAuth support was added, so it
may need back-patching. The v18 error message has different wording
and would need a small adjustment.
Could someone take a look at the attached patch and let me know
if this is the right fix?
Thanks,
Yuriy
On 14 Sep 2026, at 13:22, Grigorev Jurij <ju.grigorev@ftdata.ru> wrote:
Could someone take a look at the attached patch and let me know
if this is the right fix?
Thanks for the report, that does indeed seem like the right fix. Added to my
list of patches to review and apply before the next minors
--
Daniel Gustafsson
On Mon, Sep 14, 2026 at 4:28 AM Daniel Gustafsson <daniel@yesql.se> wrote:
Thanks for the report, that does indeed seem like the right fix.
Yes, thanks! Couple thoughts on specific pieces:
+ /* + * An empty or all-whitespace setting is accepted by + * SplitDirectoriesString(), which returns an empty list for it, so the + * parse result has to be checked rather than the raw string. + */
Sometimes recording a historical bug in the comments can help prevent
future mistakes... but I don't think this is one of those cases,
especially since the new test prevents accidental regression.
+is($bgconn->query_safe('SELECT 1'), '1', + 'postmaster survives an empty OAuth validator list on reload');
query_safe() doesn't return on failure, so IMO we shouldn't wrap it in is().
--
This seems like a good time to mention that I have a checklist item to
fix the following (shouldn't block this patch):
if (!SplitDirectoriesString(rawstring, ',', &elemlist))
...
if (strcmp(allowed, hbaline->oauth_validator) == 0)
SplitDirectoriesString() canonicalizes its outputs, which we then
compare against the uncanonicalized hbaline->oauth_validator. That
could lead to annoying false negatives in more complicated setups.
Thanks,
--Jacob
On 14 Sep 2026, at 23:28, Jacob Champion <jacob.champion@enterprisedb.com> wrote:
+ /* + * An empty or all-whitespace setting is accepted by + * SplitDirectoriesString(), which returns an empty list for it, so the + * parse result has to be checked rather than the raw string. + */Sometimes recording a historical bug in the comments can help prevent
future mistakes... but I don't think this is one of those cases,
especially since the new test prevents accidental regression.
+1
This seems like a good time to mention that I have a checklist item to
fix the following (shouldn't block this patch):if (!SplitDirectoriesString(rawstring, ',', &elemlist))
...
if (strcmp(allowed, hbaline->oauth_validator) == 0)SplitDirectoriesString() canonicalizes its outputs, which we then
compare against the uncanonicalized hbaline->oauth_validator. That
could lead to annoying false negatives in more complicated setups.
Right, this patch wont move the needle in the wrong direction for future fixes
AFAICT.
--
Daniel Gustafsson
Hi Jacob, Daniel,
Thanks for the review! In the attached v2 I removed the explanatory
comment and call query_safe() directly. The functional change and the
reload test are otherwise unchanged.
Also left the validator-name canonicalization issue out of this patch.
Checked, the patch builds cleanly against master with --enable-cassert,
and PG_TEST_EXTRA=oauth make check in src/test/modules/oauth_validator
passes all 190 tests.
Regards,
Yuriy
________________________________________
От: Daniel Gustafsson <daniel@yesql.se>
Отправлено: 15 сентября 2026 г. 4:34:08
Кому: Jacob Champion
Копия: Григорьев Юрий; pgsql-bugs@lists.postgresql.org
Тема: Re: Postmaster crashes on SIGHUP when oauth_validator_libraries holds only whitespace
On 14 Sep 2026, at 23:28, Jacob Champion <jacob.champion@enterprisedb.com> wrote:
+ /* + * An empty or all-whitespace setting is accepted by + * SplitDirectoriesString(), which returns an empty list for it, so the + * parse result has to be checked rather than the raw string. + */Sometimes recording a historical bug in the comments can help prevent
future mistakes... but I don't think this is one of those cases,
especially since the new test prevents accidental regression.
+1
This seems like a good time to mention that I have a checklist item to
fix the following (shouldn't block this patch):if (!SplitDirectoriesString(rawstring, ',', &elemlist))
...
if (strcmp(allowed, hbaline->oauth_validator) == 0)SplitDirectoriesString() canonicalizes its outputs, which we then
compare against the uncanonicalized hbaline->oauth_validator. That
could lead to annoying false negatives in more complicated setups.
Right, this patch wont move the needle in the wrong direction for future fixes
AFAICT.
--
Daniel Gustafsson
- if (oauth_validator_libraries_string[0] == '\0')
- {
- ereport(elevel,
- errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("parameter \"%s\" must be set for authentication method \"%s\"",
- "oauth_validator_libraries", "oauth"),
- errcontext("line %d of configuration file \"%s\"",
- line_num, file_name));
- *err_msg = psprintf("parameter \"%s\" must be set for authentication method \"%s\"",
- "oauth_validator_libraries", "oauth");
- return false;
- }
-
/* SplitDirectoriesString needs a modifiable copy */
rawstring = pstrdup(oauth_validator_libraries_string);
pstrdup calls strlen which segfault on NULL. oauth_validator_libraries_string
has a default value of "" so it cannot be set to NULL by user action, but the
global variable backing the GUC is initialized as NULL so I wonder if it's
worth adding defensive programming like the below, or perhaps an Assert?
/* SplitDirectoriesString needs a modifiable copy */
- rawstring = pstrdup(oauth_validator_libraries_string);
+ rawstring = pstrdup(oauth_validator_libraries_string ?
+ oauth_validator_libraries_string : "");
--
Daniel Gustafsson
Hi Daniel,
You're right. I would rather not treat NULL as an empty setting, since
that would turn an internal programming error into the user-facing
"must be set" error. Attached v3 adds:
Assert(oauth_validator_libraries_string != NULL);
before the pstrdup().
I also moved the whitespace-only TAP case to after the
pg_hba_file_rules() check. Its previous position made the test racy:
wait_for_log() synchronized with the postmaster reload, but not
necessarily with the process-local GUC state of the existing bgconn.
As a result, pg_hba_file_rules() could run in a backend that still had
the whitespace value loaded and return unexpected empty fields.
Moving the case after that assertion avoids making the
pg_hba_file_rules() result depend on the timing of SIGHUP processing in
bgconn. After restoring the setting, the test runs SHOW
oauth_validator_libraries through bgconn, ensuring that the backend has
processed the second SIGHUP before the later tests continue.
Thanks,
Yuriy
On Tue, Sep 15, 2026 at 2:43 PM Daniel Gustafsson <daniel@yesql.se> wrote:
pstrdup calls strlen which segfault on NULL. oauth_validator_libraries_string
has a default value of "" so it cannot be set to NULL by user action, but the
global variable backing the GUC is initialized as NULL so I wonder if it's
worth adding defensive programming like the below, or perhaps an Assert?
It looks like there are no GUC_LIST_INPUT params with a NULL
.boot_val. I don't know if that's by design, but it's probably for the
best given the decision in ff4597acd4c.
There may be a bit of a tug-of-war going on between committers who
prefer the explicit Assert() crash vs committers who are fine with the
SEGV. (I prefer the assertion when it better documents intent, but at
time of authorship I'm not sure I would have wanted to argue about it.
:D)
Thanks,
--Jacob
On 17 Sep 2026, at 00:16, Jacob Champion <jacob.champion@enterprisedb.com> wrote:
There may be a bit of a tug-of-war going on between committers who
prefer the explicit Assert() crash vs committers who are fine with the
SEGV. (I prefer the assertion when it better documents intent, but at
time of authorship I'm not sure I would have wanted to argue about it.
:D)
I agree with the documenting aspect of the Assert.
--
Daniel Gustafsson
Pushed with tiny bits of tweaking and with a backpatch down to 18. Thanks!
./daniel
Hello Daniel,
18.09.2026 01:24, Daniel Gustafsson wrote:
Pushed with tiny bits of tweaking and with a backpatch down to 18. Thanks!
./daniel
The test addition coined with bca67e5a3 is not very robust,
unfortunately. Buildfarm animal serinus found a way to break it [1]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=serinus&dt=2026-09-29%2013%3A57%3A41
as below:
# Failed test 'oauth_validator_libraries restored'
# at /home/bf/bf-build/serinus/HEAD/pgsql/src/test/modules/oauth_validator/t/001_server.pl line 152.
# got: ' '
# expected: 'validator'
# Looks like you failed 1 test of 171.
I've managed to reproduce this locally with a sleep:
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -569,6 +569,7 @@ ProcessConfigFileInternal(GucContext context, bool applySettings, int elevel)
item->name, item->value)));
}
item->applied = true;
+pg_usleep(100000);
}
else if (scres == 0)
{
It makes oauth_validator/001_server fail deterministically for me.
Could you take a look, please?
[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=serinus&dt=2026-09-29%2013%3A57%3A41
Best regards,
Alexander
On Wed, Sep 30, 2026 at 11:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
The test addition coined with bca67e5a3 is not very robust,
unfortunately. Buildfarm animal serinus found a way to break it [1]
as below:
Bleh. These SIGHUP/log file races are really annoying, and I'm going
to need to trawl the list to see if anyone's proposed a more general
solution. (I keep thinking about assigning an xid as a generation
number to an in-memory configuration...)
For now, is the following test giving us much, or can we remove it?
We've already done the wait_for_log(). If that "sequence point" turns
out to be inadequate for some other reason, we'd have to fix it
anyway.
is($bgconn->query_safe('SHOW oauth_validator_libraries'),
'validator', 'oauth_validator_libraries restored');
Thanks,
--Jacob
On Wed, Sep 30, 2026 at 11:20 AM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
On Wed, Sep 30, 2026 at 11:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
The test addition coined with bca67e5a3 is not very robust,
unfortunately. Buildfarm animal serinus found a way to break it [1]
as below:Bleh. These SIGHUP/log file races are really annoying, and I'm going
to need to trawl the list to see if anyone's proposed a more general
solution. (I keep thinking about assigning an xid as a generation
number to an in-memory configuration...)
Moving over to -hackers in case anyone has thoughts. I notice [1]/messages/by-id/CAJTYsWWGXJRhefkYs6wZFUOHqE9k5VSoABH2uzke5FMBSmE0=w@mail.gmail.com
touched on a similar thing, only with pg_ident/hba.
The general issue, for anyone new to the problem, is that reloading
doesn't wait for the effects of the SIGHUP to complete. (And the
meaning of "complete" depends on the test case.) It feels like this
would pop up in more situations, but the buildfarm doesn't always show
it and it's an easy trap to fall into.
Outside of development, it really seems like a DBA would like to say
"I reloaded the server using <some command> and it succeeded, so now I
_know_ that my changes have been applied."
For now, is the following test giving us much, or can we remove it?
We've already done the wait_for_log(). If that "sequence point" turns
out to be inadequate for some other reason, we'd have to fix it
anyway.is($bgconn->query_safe('SHOW oauth_validator_libraries'),
'validator', 'oauth_validator_libraries restored');
I just pushed a commit removing this check. (We don't need the $bgconn
backend to reflect the current state. We only care that _new_ backends
will adopt the new setting.)
Thanks,
--Jacob
[1]: /messages/by-id/CAJTYsWWGXJRhefkYs6wZFUOHqE9k5VSoABH2uzke5FMBSmE0=w@mail.gmail.com
On 8 Oct 2026, at 23:28, Jacob Champion <jacob.champion@enterprisedb.com> wrote:
The general issue, for anyone new to the problem, is that reloading
doesn't wait for the effects of the SIGHUP to complete. (And the
meaning of "complete" depends on the test case.) It feels like this
would pop up in more situations, but the buildfarm doesn't always show
it and it's an easy trap to fall into.
Few days ago maintainers of pgconsul (HA tool) asked me: how to wait until
synchronous_stanby_names takes effect? For some reason they need to commit one
transaction with full quorum. The best solution we found was to restart the node
and insert\commit afterwards.
So yeah, +1 for having synchronous reload.
Best regards, Andrey Borodin.
On Thu, Oct 8, 2026 at 11:53 AM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
Few days ago maintainers of pgconsul (HA tool) asked me: how to wait until
synchronous_stanby_names takes effect? For some reason they need to commit one
transaction with full quorum. The best solution we found was to restart the node
and insert\commit afterwards.
Our tests tend to restart to get around this problem, too, and that's
really heavyweight.
The oauth_validator test suite waits for lines in the logs to avoid
the restart penalty. But strictly speaking, it's waiting for the
*wrong* log line ("received SIGHUP, reloading configuration files").
That only works because the postmaster isn't multithreaded, and
knowing that the postmaster has started processing the SIGHUP is
enough to know that the next connection will pick up the new setting.
But I'm not really confident that all the happens-before logic shakes
out correctly in every corner case.
So yeah, +1 for having synchronous reload.
What are the things we'd need to support for this to make sense as a
feature? I could see use cases for
- wait until new connections are guaranteed to see the new configuration
- wait until *all* backends are <ditto>
We would need to deal with concurrent changes/reloads correctly. Like,
I imagine no one's going to immediately demand "serializable
isolation", since the current architecture provides very few
guarantees, but it needs to make intuitive sense... If you make a
configuration change, and then sync-reload, you should see the result
of either your change or a change that came afterwards.
Thanks,
--Jacob
On 8 Oct 2026, Jacob Champion wrote:
- wait until *all* backends are <ditto>
A synchronous reload that guarantees the new configuration for
subsequent connections would already be useful. My
synchronous_standby_names example needs more: walsenders and the
checkpointer also update shared sync-rep state after
ProcessConfigFile(). Acknowledging completion inside that function
would be too early.
Building on your generation idea, could we number reload requests and
acknowledge the number captured before reading the files? A request
arriving during a reload must not be satisfied by completion of that
older reload. We'd also need to distinguish completion from successful
application.
For testing, I posted a patch for filesystem-controlled wait injection
points in pgsql-bugs [0]/messages/by-id/AE31E3B7-A21B-4C1C-B75B-8D542A9DC633@yandex-team.ru. A waiting process publishes a PID file; TAP
can observe it and release the process by removing that file, without
SQL. With points around a backend's reload, we could hold it back while
the postmaster finishes, then test the two guarantees separately. The
posted series uses this to coordinate startup during crash recovery.
Thank you!
Best regards, Andrey Borodin.
[0]: /messages/by-id/AE31E3B7-A21B-4C1C-B75B-8D542A9DC633@yandex-team.ru
Hello
There's still a leftover empty validator issue, even if not that
serious: '""'. It returns a list with a single empty string, so the
new check doesn't catch it, the hba line gets a single empty named
validator.
This hits an assertion in debug builds, but otherwise works mostly
fine in release builds with a 'could not access file ""' error.
The attached patch adds an additional check for this.