Emptying oauth_validator_libraries doesn't stop OAuth logins

Started by Zsolt Parragiabout 1 month ago14 messageshackers
Jump to latest
#1Zsolt Parragi
zsolt.parragi@percona.com

Hello

The docs for oauth_validator_libraries say "If set to an empty string
(the default), OAuth connections will be refused", but that's only
true for a freshly started server. On a running one, setting it to ''
and reloading leaves existing oauth HBA lines working.

LOG: parameter "oauth_validator_libraries" changed to ""
LOG: parameter "oauth_validator_libraries" must be set for
authentication method "oauth"
CONTEXT: line 2 of configuration file ".../pg_hba.conf"
LOG: pg_hba.conf was not reloaded

SHOW reports '', pg_hba_file_rules shows the error, but logins on the
oauth line still succeed.

The allowlist is only enforced by check_oauth_validator() while
pg_hba.conf is parsed. When the GUC change makes an oauth line
invalid, the whole HBA reload fails, and the postmaster keeps the
previously parsed lines, including the validator name that was
resolved against the old list. load_validator_library() never looks at
the current setting. The same happens if a line has an explicit
validator=foo and foo is removed from the list: the validator that is
"not permitted by oauth_validator_libraries" keeps authenticating
users.

The attached patch calls check_oauth_validator() again in
oauth_init(), before loading the library, and fails the authentication
if the validator isn't permitted anymore. Only logins on affected
oauth lines fail, everything else keeps working with the old HBA lines
as before.

The patch doesn't make the behavior stricter than what we already have
elsewhere:
* After a restart, the same configuration makes the postmaster refuse
to start (load_hba() fails at startup).
* With EXEC_BACKEND, every backend re-parses pg_hba.conf with the
current setting, so all new connections already fail with "could not
load pg_hba.conf", not only the oauth ones.

Attachments:

t254135_1
v1-0001-oauth-Recheck-oauth_validator_libraries-during-au.patchapplication/octet-stream; name=v1-0001-oauth-Recheck-oauth_validator_libraries-during-au.patchDownload+25-1
#2Daniel Gustafsson
daniel@yesql.se
In reply to: Zsolt Parragi (#1)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

On 10 Oct 2026, at 11:57, Zsolt Parragi <zsolt.parragi@percona.com> wrote:

Sorry for being slow to get to this, working through the backlog of bugs in the
inbox now.

The attached patch calls check_oauth_validator() again in
oauth_init(), before loading the library, and fails the authentication
if the validator isn't permitted anymore. Only logins on affected
oauth lines fail, everything else keeps working with the old HBA lines
as before.

Thanks for the patch, I agree that the current behaviour is incorrect wrt the
docs and we should fix this down to v18.

	+	if (!check_oauth_validator(port->hba, LOG, &err_msg))
	+		ereport(ERROR,

Since check_oauth_validator already performs error logging, why not let it
handle erroring out with a higher elevel than LOG? Also, we should be using
COMMERROR here to keep details from bleeding through during authentication.
The attached v2 implements this, thoughts on that?

Another option would be a larger refactoring to move this re-checking into
load_validator_library() to make sure it cannot bypass due to programmer error,
but for a backpatchable fix I think the current approach is better.

--
Daniel Gustafsson

Attachments:

t254135_2
v2-0001-oauth-Recheck-oauth_validator_libraries-during-au.patchapplication/octet-stream; name=v2-0001-oauth-Recheck-oauth_validator_libraries-during-au.patch; x-unix-mode=0644Download+18-1
#3Jacob Champion
jacob.champion@enterprisedb.com
In reply to: Zsolt Parragi (#1)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

On Sat, Oct 10, 2026 at 2:58 AM Zsolt Parragi <zsolt.parragi@percona.com> wrote:

Hello

Hi, thanks for the report! As a short detour from the patch itself, it
looks like I missed the group of emails sent on October 10, so
Daniel's responses today are the first I'm reading them. How many were
sent in total?

--Jacob

#4shihao zhong
zhong950419@gmail.com
In reply to: Jacob Champion (#3)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

Hi Daniel & Zsolt,

Thanks for working on this. This is my first time review oauth related
patches, and I hope my understanding is correct here, if not, please
correct me.

I tried v2 and the login still works. v1 checked the return value of
check_oauth_validator(), but v2 drops that.

COMMERROR only logs, and nothing checks the return value.
With the list set to '' and reloaded, I can still log in and run queries:

LOG: parameter "oauth_validator_libraries" must be set ...
LOG: connection authenticated: identity="test" method=oauth

The attached patch on top of v2 fails the login when the check fails.

Thanks,
Shihao

Attachments:

nocfbot-v2-0002-Reject-the-login-when-the-validator-is-no-longer-all.patchapplication/octet-stream; name=nocfbot-v2-0002-Reject-the-login-when-the-validator-is-no-longer-all.patchDownload+7-3
#5Zsolt Parragi
zsolt.parragi@percona.com
In reply to: Jacob Champion (#3)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

On Tue, 03 Nov 2026, Jacob Champion <jacob.champion@enterprisedb.com> wrote:

Hi, thanks for the report! As a short detour from the patch itself, it
looks like I missed the group of emails sent on October 10, so
Daniel's responses today are the first I'm reading them. How many were
sent in total?

There were 2 others.

* Custom oauth validator options:
/messages/by-id/CAN4CZFOZX7r4-i--ucw38mhb6z0WzC1U+VFt45mFwiw5qqqbXg@mail.gmail.com
* Another in the postmaster crash thread:
/messages/by-id/CAN4CZFP7ZMLB5Q1P4H__2d54om6fO+OEBZpWLpMwhFipobrEEw@mail.gmail.com

And there are also two earlier oauth-related threads:

* About the additional debug trace warning:
/messages/by-id/CAN4CZFOUG9csxguH_+cz=nQ8HQVKLcVvr=qJSbga+tj-3FZhLA@mail.gmail.com
* About documenting/improving not requiring SSL:
/messages/by-id/CAN4CZFNy23wWKF+78QG3cfunjJpQWFc7eNP7qcYPB6Sg-NhTQQ@mail.gmail.com

#6Daniel Gustafsson
daniel@yesql.se
In reply to: shihao zhong (#4)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

On 4 Nov 2026, at 03:48, shihao zhong <zhong950419@gmail.com> wrote:

I tried v2 and the login still works. v1 checked the return value of
check_oauth_validator(), but v2 drops that.

COMMERROR only logs, and nothing checks the return value.

Thanks, I was clearly undercaffeinated.

--
Daniel Gustafsson

#7Zsolt Parragi
zsolt.parragi@percona.com
In reply to: Daniel Gustafsson (#6)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

COMMERROR is a cosmetic change (during auth we send error or greater
to the client, not log), and the rest of the OAuth code uses ERROR,
which then gets promoted to FATAL.

The main question is how much information we should include in the
error message to the user, from that perspective Shihao's more generic
message might be better.

v3 is v1 + the COMMERROR and the error message change.

Attachments:

t254135_7
v3-0001-oauth-Recheck-oauth_validator_libraries-during-au.patchapplication/octet-stream; name=v3-0001-oauth-Recheck-oauth_validator_libraries-during-au.patchDownload+25-1
#8shihao zhong
zhong950419@gmail.com
In reply to: Zsolt Parragi (#7)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

Hi Zsolt,

Thanks for v3.

I would keep the generic message. It is the same string auth_failed()
uses for a bad token. So the client cannot tell a removed validator
from a normal failure, which I think is also what Daniel wanted from
COMMERROR.

Thanks,
Shihao

#9Daniel Gustafsson
daniel@yesql.se
In reply to: Zsolt Parragi (#7)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

On 4 Nov 2026, at 10:09, Zsolt Parragi <zsolt.parragi@percona.com> wrote:

v3 is v1 + the COMMERROR and the error message change.

I pushed this earlier today, but according the buildfarm the backpatch of the
test to 18 wasn't entirely correct. Will fix when back at the laptop.

--
Daniel Gustafsson

#10Jacob Champion
jacob.champion@enterprisedb.com
In reply to: Daniel Gustafsson (#9)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

On Thu, Nov 5, 2026 at 1:17 PM Daniel Gustafsson <daniel@yesql.se> wrote:

I pushed this earlier today, but according the buildfarm the backpatch of the
test to 18 wasn't entirely correct. Will fix when back at the laptop.

Thanks!

This falls into that same class of "atomic behavior when the
configuration is inconsistent" which I missed in the SNI set, too, so
add that to my list of blind spots. It'd be nice (with all our copious
free time?) to try to improve the whole system there... That
intersects with Zsolt's threads on GUC+HBA-et-al and/or Nico's
interest in moving config into the DB itself.

--Jacob

#11Daniel Gustafsson
daniel@yesql.se
In reply to: Zsolt Parragi (#5)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

On 4 Nov 2026, at 09:19, Zsolt Parragi <zsolt.parragi@percona.com> wrote:

On Tue, 03 Nov 2026, Jacob Champion <jacob.champion@enterprisedb.com> wrote:

Hi, thanks for the report! As a short detour from the patch itself, it
looks like I missed the group of emails sent on October 10, so
Daniel's responses today are the first I'm reading them. How many were
sent in total?

There were 2 others.

* Custom oauth validator options:
/messages/by-id/CAN4CZFOZX7r4-i--ucw38mhb6z0WzC1U+VFt45mFwiw5qqqbXg@mail.gmail.com

This will be deferred to v20.

* Another in the postmaster crash thread:
/messages/by-id/CAN4CZFP7ZMLB5Q1P4H__2d54om6fO+OEBZpWLpMwhFipobrEEw@mail.gmail.com

This one was pushed and backpatched yesterday.

* About the additional debug trace warning:
/messages/by-id/CAN4CZFOUG9csxguH_+cz=nQ8HQVKLcVvr=qJSbga+tj-3FZhLA@mail.gmail.com

This was on my radar but was forgotten about in the lead-up to events and other
things. I still think it's valuable but not something I want to have go in
just before a release freeze.

* About documenting/improving not requiring SSL:
/messages/by-id/CAN4CZFNy23wWKF+78QG3cfunjJpQWFc7eNP7qcYPB6Sg-NhTQQ@mail.gmail.com

I just posted an updated patch for the documentation changes.

--
Daniel Gustafsson

#12Alexander Lakhin
exclusion@gmail.com
In reply to: Daniel Gustafsson (#9)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

Hello Daniel and Zsolt,

05.11.2026 23:17, Daniel Gustafsson wrote:

I pushed this earlier today, but according the buildfarm the backpatch of the
test to 18 wasn't entirely correct. Will fix when back at the laptop.

Recent changes to oauth_validator/001_server.pl increased its duration so
it fails now on overloaded (due to concurrent activity?) animals, like in
[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=grassquit&dt=2026-11-07%2000%3A31%3A47
# die: IPC::Run: timeout on timer #1 at /usr/share/perl5/IPC/Run.pm line 3361.
# Looks like your test exited with 29 just after 151.

# die: IPC::Run: timeout on timer #1 at /usr/share/perl5/IPC/Run.pm line 3361.
# Looks like your test exited with 2 just after 133.

Logs of these animals show that other tests also ran for longer in these
runs, e.g. compare [2]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=olingo&dt=2026-11-07%2000%3A31%3A30:
218/401 recovery - postgresql:recovery/027_stream_regress OK             1048.99s   11 subtests passed
392/401 oauth_validator - postgresql:oauth_validator/001_server ERROR           191.48s   exit status 2

vs the next successful run [3]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=olingo&dt=2026-11-07%2002%3A52%3A57
183/401 recovery - postgresql:recovery/027_stream_regress OK               88.70s   11 subtests passed
400/401 oauth_validator - postgresql:oauth_validator/001_server OK               55.08s   173 subtests passed

Out of 200+ animals, only 25 have "oauth" in PG_TEST_EXTRA plus libcurl
enabled -- this can explain why definitely slower animals didn't fail the
test.

Could you please look if the test can be made more resilient against such slowdowns?

[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=grassquit&dt=2026-11-07%2000%3A31%3A47
[2]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=olingo&dt=2026-11-07%2000%3A31%3A30
[3]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=olingo&dt=2026-11-07%2002%3A52%3A57

Best regards,
Alexander

#13Zsolt Parragi
zsolt.parragi@percona.com
In reply to: Alexander Lakhin (#12)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

Could you please look if the test can be made more resilient against such slowdowns?

Attached a single line change that resets the timer for each query for the background connection. I don't see anything wrong with the tests otherwise, the problem is simply that we are using a single background connection for a long time, and on a heavily overloaded machine with asan, the test can sometimes exceed the timeout value.

Attachments:

t254135_13
v1-0001-oauth-Apply-bgconn-timeout-per-query-in-001_serve.patchtext/x-patchDownload+1-1
#14Daniel Gustafsson
daniel@yesql.se
In reply to: Zsolt Parragi (#13)
Re: Emptying oauth_validator_libraries doesn't stop OAuth logins

On 8 Nov 2026, at 11:49, Zsolt Parragi <zsolt.parragi@percona.com> wrote:

Could you please look if the test can be made more resilient against such slowdowns?

Attached a single line change that resets the timer for each query for
the background connection. I don't see anything wrong with the tests
otherwise, the problem is simply that we are using a single background
connection for a long time, and on a heavily overloaded machine with
asan, the test can sometimes exceed the timeout value.

Agreed, looking this over I cannot find any other issue. Will fix when the
backbranches are open again.

--
Daniel Gustafsson