Stabilize and shorten test_checksums/013_rewind test
Hi,
I encountered two problems with the test_checksums/013_rewind test:
1. It takes ~45 seconds to finish on my machine and it seems timing
doesn't depend the machine it runs on. It is always last finishing
tests on my machine and I need to wait ~30 seconds after all other
tests are done. Some numbers:
Local: 013_rewind -> 45s | 027_stream_regress -> 24s
Windows CI: 013_rewind -> 33s | 027_stream_regress -> 150s
2. The "8 - last common checkpoint is a shutdown checkpoint" test is
flaky. This test fails ~1/10 of the runs on my machine without any
external effort. Error message:
[11:31:29.021](0.007s) not ok 8 - last common checkpoint is a shutdown
checkpoint
[11:31:29.022](0.001s) # Failed test 'last common checkpoint is a
shutdown checkpoint'
# at /home/nbyavuz/Desktop/projects/postgres/src/test/modules/test_checksums/t/013_rewind.pl
line 161.
[11:31:29.022](0.000s) # ''
# doesn't match '(?^:CHECKPOINT_SHUTDOWN)'
I also saw same error on the CI [1]https://github.com/postgres/postgres/actions/runs/35096253076/job/104794579050#step:13:499.
----------------------------------------
I spent some time fixing these problems with the help of an LLM.
Problem #1:
We wait for the primary's insert LSN in three places, but the primary
might be idle and not have sent it yet:
1.1:
$node_a->backup('backup');
my $node_b = PostgreSQL::Test::Cluster->new('node_b');
$node_b->init_from_backup($node_a, 'backup', has_streaming => 1);
$node_b->start;
$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert'));
test_checksum_state($node_a, 'off');
test_checksum_state($node_b, 'off');
backup() already flushes the LSN, we can wait for the flush LSN here.
1.2
$node_a->safe_psql('postgres', "CHECKPOINT;");
$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert'));
Since there is already a CHECKPOINT, the changes should be flushed. We
can wait for the flush LSN.
1.3
# Start the rewound node as a standby of the new primary. Replay runs
# through the pre-enable WAL stretch and the online enable.
#
# The rewind replaced the configuration files with those of the new
# primary, so put the port back.
...
$node_a->set_standby_mode;
$node_a->start;
$node_b->wait_for_catchup($node_a, 'replay', $node_b->lsn('insert'));
test_checksum_state($node_a, 'on');
We can use pg_switch_wal() function to make sure changes are flushed
and then we can wait for the flush LSN.
These 3 changes reduces test time from ~45s to ~3s on my local environment.
----------------------------------------
Problem #2
($stdout, $stderr) = run_command(
[
'pg_waldump',
'-p' => $node_a->data_dir . '/pg_wal',
'-t' => 1,
'-s' => $shutdown_ckpt,
'-n' => 1,
]);
like($stdout, qr/CHECKPOINT_SHUTDOWN/,
'last common checkpoint is a shutdown checkpoint');
We don't specifiy which WAL file that pg_waldump() will use, then
pg_waldump select first WAL data available in the directory. Then, it
might select a WAL file whose header is not initialized yet (a
preallocated WAL file). So, when pg_waldump tries to get segment_size
from this file it reads 0 and fails. I run pg_waldump on the failed
test artifacts and I get this error; which I think confirms the
problem:
$ pg_waldump -p
testrun/test_checksums/013_rewind/data/t_013_rewind_node_a_data/pgdata/pg_wal/
pg_waldump: error: invalid WAL segment size in WAL file
"000000020000000000000005" (0 bytes)
pg_waldump: detail: The WAL segment size must be a power of two
between 1 MB and 1 GB.
This problem is solved by specifying the WAL file.
----------------------------------------
Two patchs are attached, 0001 for the problem #1 and 0002 for the problem #2.
[1]: https://github.com/postgres/postgres/actions/runs/35096253076/job/104794579050#step:13:499
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
t253838_1v1-0001-Avoid-idle-WAL-waits-in-the-checksum-rewind-test.patchtext/x-patch; charset=US-ASCII; name=v1-0001-Avoid-idle-WAL-waits-in-the-checksum-rewind-test.patchDownload+9-4
v1-0002-Fix-WAL-file-selection-in-checksum-rewind-test.patchtext/x-patch; charset=US-ASCII; name=v1-0002-Fix-WAL-file-selection-in-checksum-rewind-test.patchDownload+10-5
On 18 Sep 2026, at 12:26, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
I encountered two problems with the test_checksums/013_rewind test:
...
I spent some time fixing these problems with the help of an LLM.
Thanks!
Problem #1:
We wait for the primary's insert LSN in three places, but the primary
might be idle and not have sent it yet:
Makes sense.
-$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert'));
+# Backup completion has flushed the required WAL.
+$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('flush'));
In these cases, couldn't we just wait for the replay LSN, ie the default for
wait_for_catchup?
Problem #2
($stdout, $stderr) = run_command(
[
'pg_waldump',
'-p' => $node_a->data_dir . '/pg_wal',
'-t' => 1,
'-s' => $shutdown_ckpt,
'-n' => 1,
]);
like($stdout, qr/CHECKPOINT_SHUTDOWN/,
'last common checkpoint is a shutdown checkpoint');We don't specifiy which WAL file that pg_waldump() will use, then
pg_waldump select first WAL data available in the directory. Then, it
might select a WAL file whose header is not initialized yet (a
preallocated WAL file).
Nice find, this patch seems quite straightforward.
--
Daniel Gustafsson
Hi,
Thank you for looking into this!
On Fri, 18 Sept 2026 at 15:51, Daniel Gustafsson <daniel@yesql.se> wrote:
On 18 Sep 2026, at 12:26, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
Problem #1:We wait for the primary's insert LSN in three places, but the primary
might be idle and not have sent it yet:Makes sense.
-$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert')); +# Backup completion has flushed the required WAL. +$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('flush'));In these cases, couldn't we just wait for the replay LSN, ie the default for
wait_for_catchup?
Sorry, I am a bit confused. AFAIU, you meant:
$node_a->wait_for_catchup($node_b);
which is
$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('write'));
If that is the case, could we still need to wait for an additional
primary-side flush, since written WAL might not have been flushed yet?
I understand that this might not cause the same long delay as waiting
for the insert LSN. On my local, ->lsn('write') and ->lsn('flush') are
basically same, they both take ~3 seconds.
My idea was that since the WAL needed by the tests has already been
flushed, I thought that would be enough for the tests. Also, is using
->lsn('write') considered better for these cases or do you recommend
it because of the simplicity?
--
Regards,
Nazir Bilal Yavuz
Microsoft
On 18 Sep 2026, at 15:47, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
On Fri, 18 Sept 2026 at 15:51, Daniel Gustafsson <daniel@yesql.se> wrote:
In these cases, couldn't we just wait for the replay LSN, ie the default for
wait_for_catchup?Sorry, I am a bit confused. AFAIU, you meant:
$node_a->wait_for_catchup($node_b);
which is
$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('write'));
If that is the case, could we still need to wait for an additional
primary-side flush, since written WAL might not have been flushed yet?
I understand that this might not cause the same long delay as waiting
for the insert LSN. On my local, ->lsn('write') and ->lsn('flush') are
basically same, they both take ~3 seconds.
I don't see how the test could do that given the sequence of commands in the
TAP file. At the end of the day, it's not super important though.
My idea was that since the WAL needed by the tests has already been
flushed, I thought that would be enough for the tests. Also, is using
->lsn('write') considered better for these cases or do you recommend
it because of the simplicity?
I prefer when tests have the least amount of special needs setup outside of the
code under test. This isn't really making assumptions which make the test
synthetic though.
Another option for going with your patch is to use wait_for_replay_catchup
which should accomplish the same.
--
Daniel Gustafsson
Hi,
On Mon, 21 Sept 2026 at 14:05, Daniel Gustafsson <daniel@yesql.se> wrote:
On 18 Sep 2026, at 15:47, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
My idea was that since the WAL needed by the tests has already been
flushed, I thought that would be enough for the tests. Also, is using
->lsn('write') considered better for these cases or do you recommend
it because of the simplicity?I prefer when tests have the least amount of special needs setup outside of the
code under test. This isn't really making assumptions which make the test
synthetic though.
That makes sense.
Another option for going with your patch is to use wait_for_replay_catchup
which should accomplish the same.
Nice, I updated them with wait_for_replay_catchup().
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
t253838_5v2-0002-Fix-WAL-file-selection-in-checksum-rewind-test.patchapplication/octet-stream; name=v2-0002-Fix-WAL-file-selection-in-checksum-rewind-test.patchDownload+10-5
v2-0001-Avoid-idle-WAL-waits-in-the-checksum-rewind-test.patchapplication/octet-stream; name=v2-0001-Avoid-idle-WAL-waits-in-the-checksum-rewind-test.patchDownload+9-4
Hello!
Shouldn't 021 get the same fix (0001)? That's only executed with checksum_extended so its not that visible on CI, but it has the same issue.
Hi,
On Tue, 22 Sept 2026 at 00:38, Zsolt Parragi <zsolt.parragi@percona.com> wrote:
Hello!
Shouldn't 021 get the same fix (0001)? That's only executed with
checksum_extended so its not that visible on CI, but it has the same
issue.
Yes, you are right. I realized that we can do the same thing for all
test_checksum tests, so I applied similar change to all of the related
checksum tests in 0001.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
t253838_7v3-0001-Avoid-idle-WAL-waits-in-checksum-tests.patchtext/x-patch; charset=US-ASCII; name=v3-0001-Avoid-idle-WAL-waits-in-checksum-tests.patchDownload+22-26
v3-0002-Fix-WAL-file-selection-in-013_rewind-test.patchtext/x-patch; charset=US-ASCII; name=v3-0002-Fix-WAL-file-selection-in-013_rewind-test.patchDownload+10-5
On 23 Sep 2026, at 12:15, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
On Tue, 22 Sept 2026 at 00:38, Zsolt Parragi <zsolt.parragi@percona.com> wrote:
Shouldn't 021 get the same fix (0001)? That's only executed with
checksum_extended so its not that visible on CI, but it has the same
issue.Yes, you are right. I realized that we can do the same thing for all
test_checksum tests, so I applied similar change to all of the related
checksum tests in 0001.
I've been looking at this patch and have tested it extensively both on local
hardware and CI instances. Copilot review pointed out one thing which I think
is accurate (though unlikely to happen), if the START WAL LOCATION and
CHECKPOINT LOCATION straddle segments then the test would fail. Extracting the
WAL filename from $shutdown_ckpt is the safer option. That fix and some commit
message tweaking are the only changes in the attached v4.
--
Daniel Gustafsson
Attachments:
t253838_8v4-0001-Avoid-idle-WAL-waits-in-checksum-tests.patchapplication/octet-stream; name=v4-0001-Avoid-idle-WAL-waits-in-checksum-tests.patch; x-unix-mode=0644Download+22-26
v4-0002-Fix-WAL-file-selection-in-013_rewind-test.patchapplication/octet-stream; name=v4-0002-Fix-WAL-file-selection-in-013_rewind-test.patch; x-unix-mode=0644Download+13-5
Hi,
On Mon, 28 Sept 2026 at 12:46, Daniel Gustafsson <daniel@yesql.se> wrote:
I've been looking at this patch and have tested it extensively both on local
hardware and CI instances. Copilot review pointed out one thing which I think
is accurate (though unlikely to happen), if the START WAL LOCATION and
CHECKPOINT LOCATION straddle segments then the test would fail. Extracting the
WAL filename from $shutdown_ckpt is the safer option. That fix and some commit
message tweaking are the only changes in the attached v4.
Thanks, v4 LGTM.
There is one more issue found by an LLM: the checkpoint record itself
can straddle two WAL segments. However, I think this is a very rare
corner case and might not be worth fixing.
--
Regards,
Nazir Bilal Yavuz
Microsoft
On 28 Sep 2026, at 14:49, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
Hi,
On Mon, 28 Sept 2026 at 12:46, Daniel Gustafsson <daniel@yesql.se> wrote:
I've been looking at this patch and have tested it extensively both on local
hardware and CI instances. Copilot review pointed out one thing which I think
is accurate (though unlikely to happen), if the START WAL LOCATION and
CHECKPOINT LOCATION straddle segments then the test would fail. Extracting the
WAL filename from $shutdown_ckpt is the safer option. That fix and some commit
message tweaking are the only changes in the attached v4.Thanks, v4 LGTM.
There is one more issue found by an LLM: the checkpoint record itself
can straddle two WAL segments. However, I think this is a very rare
corner case and might not be worth fixing.
Thanks. I'll add a comment explaining this in the test to make sure we keep it
in mind should we need it.
--
Daniel Gustafsson
This patchset was committed yesterday. Thanks for the investigation and fixes!
--
Daniel Gustafsson
Hello Daniel,
29.09.2026 10:26, Daniel Gustafsson wrote:
This patchset was committed yesterday. Thanks for the investigation and fixes!
Could you please look at two more stability issues with
003_standby_restarts.pl? One was unveiled today by hippopotamus [1]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hippopotamus&dt=2026-10-18%2022%3A15%3A04:
# Failed test 'ensure checksums are set to inprogress-on on standby_restarts_standby'
# at /home/buildfarm/hippopotamus/buildroot/HEAD/pgsql.build/src/test/modules/test_checksums/t/DataChecksums/Utils.pm
line 69.
# got: 'off'
# expected: 'inprogress-on'
# Looks like you failed 1 test of 29.
t/003_standby_restarts.pl ..............
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/29 subtests
I've managed to reproduce it locally with:
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -111,2 +111,3 @@
#include "utils/wait_event.h"
+#include "common/pg_prng.h"
@@ -2991,2 +2992,3 @@ XLogFlush(XLogRecPtr record)
+if (pg_prng_double(&pg_global_prng_state) < 0.1) pg_usleep(100000);
XLogWrite(WriteRqst, insertTLI, false);
(and then running `make check` in a loop).
Seems to be the same issue that was fixed by 5f76ed5d6.
The other one hasn't been seen on buildfarm yet, but maybe you could find it
worth fixing in passing. This delay (proposed at [2] to reproduce other
test's failure):
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -660,2 +660,3 @@ pgstat_shutdown_hook(int code, Datum arg)
+pg_usleep(100000);
pgstat_report_stat(true);
makes this test fail as below:
# +++ tap check in src/test/modules/test_checksums +++
t/003_standby_restarts.pl .. 6/? # poll_query_until timed out executing this query:
# SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';
# expecting this output:
# on
# last actual query output:
# off
# with stderr:
t/003_standby_restarts.pl .. 14/?
# Failed test 'ensure data checksums are transitioned to on on standby_restarts_primary'
# at .../src/test/modules/test_checksums/t/DataChecksums/Utils.pm line 94.
# got: '0'
# expected: '1'
# poll_query_until timed out executing this query:
# SELECT setting FROM pg_catalog.pg_settings WHERE name = 'data_checksums';
# expecting this output:
# on
# last actual query output:
# off
# with stderr:
t/003_standby_restarts.pl .. 15/?
# Failed test 'ensure data checksums are transitioned to on on standby_restarts_standby'
# at .../src/test/modules/test_checksums/t/DataChecksums/Utils.pm line 94.
# got: '0'
# expected: '1'
t/003_standby_restarts.pl .. 26/? # Looks like you failed 2 tests of 29.
t/003_standby_restarts.pl .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/29 subtests
[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hippopotamus&dt=2026-10-18%2022%3A15%3A04
Best regards,
Alexander
On 19 Oct 2026, at 09:00, Alexander Lakhin <exclusion@gmail.com> wrote:
29.09.2026 10:26, Daniel Gustafsson wrote:
This patchset was committed yesterday. Thanks for the investigation and fixes!
Could you please look at two more stability issues with
003_standby_restarts.pl?
Thanks for the report, I am currently travelling attending PGConf.EU but I will
look at them as soon as I can.
--
Daniel Gustafsson