007_pre_auth.pl fails if injection_points_wakeup hits unintended process

Started by Alexander Lakhin21 days ago5 messageshackers
Jump to latest
#1Alexander Lakhin
exclusion@gmail.com

Hello hackers,

A recent failure of auth, pre_auth test produced by culicidae:
[19:20:02.147](0.003s) ok 1 - authenticating connections are recorded in pg_stat_activity
[19:20:02.147](0.000s) # issuing query 4 via background psql: SELECT injection_points_wakeup('init-pre-auth');
[19:20:02.151](0.004s) # results query 4:
# {
#   'stderr' => 'background_psql: QUERY_SEPARATOR 4:
# ',
#   'stdout' => '
# background_psql: QUERY_SEPARATOR 4:
# '
# }
[19:23:02.313](180.162s) # die: IPC::Run: timeout on timer #2 at /usr/share/perl5/IPC/Run.pm line 3361.
[19:23:02.319](0.006s) 1..1
IPC::Run: timeout on timer #2 at /usr/share/perl5/IPC/Run.pm line 3361.

indicates that the test is not bulletproof. I reproduced this locally when
running multiple tests in parallel, and with the following diagnostic
added:
@@ -753,3 +753,5 @@ InitPostgres(const char *in_dbname, Oid dboid,
                 pgstat_bestart_initial();
+elog(LOG, "!!!before injection_point init-pre-auth");
                 INJECTION_POINT("init-pre-auth", NULL);
+elog(LOG, "!!!after injection_point init-pre-auth");
         }
observed:
23      t/007_pre_auth.pl (Wstat: 7424 (exited 29) Tests: 1 Failed: 0)
23        Non-zero exit status: 29
23      Files=1, Tests=1, 236 wallclock secs ( 0.07 usr  0.02 sys + 0.55 cusr  2.25 csys =  2.89 CPU)
23      Result: FAIL
23      # die: IPC::Run: timeout on timer #2 at /usr/share/perl5/IPC/Run.pm line 3361.
23      # Looks like your test exited with 29 just after 1.
23      make: *** [Makefile:21: check] Error 1

2026-11-01 11:20:41.510 UTC [478942][client backend][2/0:0] LOG: !!!before injection_point init-pre-auth
2026-11-01 11:20:41.510 UTC [478942][client backend][2/0:0] LOG: !!!after injection_point init-pre-auth
2026-11-01 11:20:44.597 UTC [478942][client backend][2/2:0] LOG: statement: SELECT
injection_points_attach('init-pre-auth', 'wait')
2026-11-01 11:20:45.539 UTC [479057][autovacuum worker][101/0:0] LOG:  !!!before injection_point init-pre-auth
2026-11-01 11:20:47.346 UTC [479096][client backend][3/0:0] LOG: !!!before injection_point init-pre-auth
2026-11-01 11:20:50.028 UTC [478942][client backend][2/11:0] LOG: statement: SELECT
injection_points_wakeup('init-pre-auth');
2026-11-01 11:20:50.029 UTC [479057][autovacuum worker][101/0:0] LOG:  !!!after injection_point init-pre-auth
2026-11-01 11:21:13.777 UTC [479226][autovacuum worker][102/0:0] LOG:  !!!before injection_point init-pre-auth

or:
2026-11-01 07:19:21.187 UTC [426308][autovacuum launcher][100/0:0] LOG:  !!!before injection_point init-pre-auth
2026-11-01 07:19:21.187 UTC [426308][autovacuum launcher][100/0:0] LOG:  !!!after injection_point init-pre-auth
2026-11-01 07:19:21.244 UTC [426365][client backend][2/2:0] LOG: statement: SELECT
injection_points_attach('init-pre-auth', 'wait')
2026-11-01 07:19:21.612 UTC [426309][logical replication launcher][118/0:0] LOG:  !!!before injection_point init-pre-auth
2026-11-01 07:19:23.750 UTC [426479][client backend][3/0:0] LOG: !!!before injection_point init-pre-auth
2026-11-01 07:19:25.194 UTC [426365][client backend][2/5:0] LOG: statement: SELECT injection_points_wakeup('init-pre-auth');
2026-11-01 07:19:25.232 UTC [426309][logical replication launcher][118/0:0] LOG:  !!!after injection_point init-pre-auth
2026-11-01 07:19:52.568 UTC [427022][autovacuum worker][101/0:0] LOG:  !!!before injection_point init-pre-auth

That is, the wakeup can hit autovacuum worker or logical replication
launcher, not the intended client backend. This test failure can be easily
reproduced with:
@@ -753,2 +753,3 @@ InitPostgres(const char *in_dbname, Oid dboid,
          pgstat_bestart_initial();
+if (AmRegularBackendProcess()) pg_usleep(1000000);
          INJECTION_POINT("init-pre-auth", NULL);

and
echo "autovacuum_naptime = 1s" >/tmp/temp.config
TEMP_CONFIG=/tmp/temp.config PROVE_TESTS="t/007*" make -s check -C src/test/authentication

Reproduced starting from e2080261c.

[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=culicidae&dt=2026-10-30%2018%3A10%3A12

Best regards,
Alexander

#2Andrey Borodin
amborodin@acm.org
In reply to: Alexander Lakhin (#1)
Re: 007_pre_auth.pl fails if injection_points_wakeup hits unintended process

Hi Alexander,

On 1 Nov 2026, Alexander Lakhin wrote:

the wakeup can hit autovacuum worker or logical replication
launcher, not the intended client backend.

For a small fix, I'd propose restricting "init-pre-auth" to
AmRegularBackendProcess().

/* Advertising of a related patch starts here */

I also prepared a variant using the filesystem-controlled wait points
from [0]/messages/by-id/AE31E3B7-A21B-4C1C-B75B-8D542A9DC633@yandex-team.ru. Attached are two patches: 0001 rebases that interface onto
current master, and 0002 uses it in 007_pre_auth.pl.

The test still attaches the point through SQL and finds the client's
PID in pg_stat_activity. It then waits for
pg_injection_points/init-pre-auth/<pid> and removes that file to release
only the intended client. Waiting for the file matters because the
wait event is reported before the marker is created.

The test also parks another session at the same point before starting
the client, and checks that it remains blocked after the client reaches
idle. This makes the multiple-waiter case deterministic.

Thank you!

Best regards, Andrey Borodin.

[0]: /messages/by-id/AE31E3B7-A21B-4C1C-B75B-8D542A9DC633@yandex-team.ru

Attachments:

t254505_2
v1-0001-Allow-filesystem-control-of-wait-injection-points.patchapplication/octet-stream; name=v1-0001-Allow-filesystem-control-of-wait-injection-points.patch; x-unix-mode=0644Download+441-7
v1-0002-Wake-the-pre-auth-test-s-intended-backend-by-PID.patchapplication/octet-stream; name=v1-0002-Wake-the-pre-auth-test-s-intended-backend-by-PID.patch; x-unix-mode=0644Download+47-13
#3Michael Paquier
michael@paquier.xyz
In reply to: Alexander Lakhin (#1)
Re: 007_pre_auth.pl fails if injection_points_wakeup hits unintended process

On Sun, Nov 01, 2026 at 03:00:00PM +0200, Alexander Lakhin wrote:

That is, the wakeup can hit autovacuum worker or logical replication
launcher, not the intended client backend. This test failure can be easily
reproduced with:
@@ -753,2 +753,3 @@ InitPostgres(const char *in_dbname, Oid dboid,
         pgstat_bestart_initial();
+if (AmRegularBackendProcess()) pg_usleep(1000000);
         INJECTION_POINT("init-pre-auth", NULL);

and
echo "autovacuum_naptime = 1s" >/tmp/temp.config
TEMP_CONFIG=/tmp/temp.config PROVE_TESTS="t/007*" make -s check -C src/test/authentication

Reproduced starting from e2080261c.

Thanks for the report. I am planning to tackle this one with the
following tweak to the test's conf:
autovacuum = off
max_logical_replication_workers = 0

Does that work for you?
--
Michael

Attachments:

t254505_3
preauth-test.patchtext/plain; charset=us-asciiDownload+5-0
#4Andrey Borodin
amborodin@acm.org
In reply to: Michael Paquier (#3)
Re: 007_pre_auth.pl fails if injection_points_wakeup hits unintended process

On 2 Nov 2026, at 11:08, Michael Paquier <michael@paquier.xyz> wrote:

Thanks for the report. I am planning to tackle this one with the
following tweak to the test's conf:
autovacuum = off
max_logical_replication_workers = 0

Does that work for you?

I think it's fine. But when we will have new bgworker hitting this point we might
need to change this test again.

Best regards, Andrey Borodin.

#5Michael Paquier
michael@paquier.xyz
In reply to: Alexander Lakhin (#1)
Re: 007_pre_auth.pl fails if injection_points_wakeup hits unintended process

On Sun, Nov 01, 2026 at 03:00:00PM +0200, Alexander Lakhin wrote:

Reproduced starting from e2080261c.

And this one should be fixed as of 590788ad0bff.
--
Michael