walsummarizer can get stuck when switching timelines

On 14 Jul 2026, at 01:39, Robert Haas <robertmhaas@gmail.com> wrote:

Here is a patch, with a test case.

Hi Robert!

I read the patch and I think it's correct. Two small things on the new
fallback. First, the test covers a single switch; the tricky case is several
switches within one segment, where the fallback loop and the switch_lsn bound
interact per timeline. It seems correct to me: we stop at switch_lsn, so we
only read the pre-switch bytes both timelines share. But it might be worth a
test. Second, in the fallback path WALRead reopens the segment on every page,
because the requested tli never matches seg.ws_tli once
summarizer_wal_segment_open() has moved to a descendant. These open/close
might be an exatra overhead?

I think, there's a symmetric bug on the recovery side with the same root cause.
Your patch makes the summarizer fall forward to a descendant timeline, which
looks safe. XLogFileReadAnyTLI() does the opposite: when the switch-point segment
is missing on the target timeline, it falls back to an ancestor and reads the
same segno. But that segment holds divergent data continuing old timeline!
Recovery applies it silently and then can't reach the intended timeline.

The comment you added to t/003_tli_switch.pl states the assumption that breaks
here:

# ... recovery will read them from there and work just fine.

That holds only when the new timeline's segment is already there. Concretely,
with just two timelines: TL2 forks from TL1 in segment 67, which is archived
for TL1 but not yet for TL2. Recovery targets TL2 and asks for segment 67;
TL2's copy isn't in the archive, so XLogFileReadAnyTLI() falls back to TL1 and
hands back TL1's segment 67. Past the switch point that is the old primary's
divergent WAL, recovery applies it, and now it can never get onto TL2. The
beginseg check doesn't help here - it only skips a timeline for segments older
than that timeline's start, so nothing stops the fallback to the ancestor TL1
for the very segment where the two diverge. I reported it with a fix [0]/messages/by-id/85386EF6-16B7-4D62-86BE-526A10F93825@yandex-team.ru (CF #6525).

Regards, Andrey Borodin.

[0]: /messages/by-id/85386EF6-16B7-4D62-86BE-526A10F93825@yandex-team.ru (CF #6525)

On Tue, Jul 14, 2026 at 2:09 AM Robert Haas <robertmhaas@gmail.com> wrote:

Here is a patch, with a test case.

I tried the patch on Windows and got following failure:

# executing test in
C:\....\postgresql\build/testrun/pg_walsummary/003_tli_switch group
pg_walsummary test 003_tli_switch
# initializing database system by copying initdb template
not ok 1 - WAL summarization on node3 advanced past timeline switch
ok 2 - no summaries from before LSN 0/04016090
not ok 3 - at least one summary from LSN 0/04016090 or later
1..3
# test failed
----------------------------------- stderr -----------------------------------
# poll_query_until timed out executing this query:
# SELECT EXISTS (SELECT * FROM pg_available_wal_summaries() WHERE tli = 2)
#
# expecting this output:
# t
# last actual query output:
# f
# with stderr:
# Failed test 'WAL summarization on node3 advanced past timeline switch'
# at C:/Workspace/code/postgresql/src/bin/pg_walsummary/t/003_tli_switch.pl
line 92.
# Failed test 'at least one summary from LSN 0/04016090 or later'
# at C:/Workspace/code/postgresql/src/bin/pg_walsummary/t/003_tli_switch.pl
line 109.
# Looks like you failed 2 tests of 3.

(test program exited with status code 2)
==============================================================================

The failure log of node-3 is attached. I'll investigate this further
and share my findings but in the meantime if you have any inputs based
on the LOG then do let me know.

--
With Regards,
Amit Kapila.

On 14 Jul 2026, at 17:49, Amit Kapila <amit.kapila16@gmail.com> wrote:

I tried the patch on Windows and got following failure:

Hi Amit,

Thanks for testing! That output is exactly what an unpatched summarizer produces
on my machine. I suspect the fix didn't make it into your build.
Also the test passes for me on Windows CI with the patch applied [0]https://github.com/x4m/postgres_g/actions/runs/29330084058/job/87076116050.

Could you double-check that you observe the failure?

Best regards, Andrey Borodin.

[0]: https://github.com/x4m/postgres_g/actions/runs/29330084058/job/87076116050

On Tue, Jul 14, 2026 at 6:35 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:

On 14 Jul 2026, at 17:49, Amit Kapila <amit.kapila16@gmail.com> wrote:

I tried the patch on Windows and got following failure:

Hi Amit,

Thanks for testing! That output is exactly what an unpatched summarizer produces
on my machine. I suspect the fix didn't make it into your build.
Also the test passes for me on Windows CI with the patch applied [0].

Could you double-check that you observe the failure?

Yes. This time I tried adding some basic log in walsummarizer.c to
ensure that the test picks up new code. See
extra_debuglog_walsummary_1.txt patch atop the main patch. You can see
the node-3 LOG file (003_tli_switch_node3) attached which has this
following new LOG:
2026-07-15 10:46:56.439 IST walsummarizer[14680] DEBUG: modified WAL
summarizer started
2026-07-15 10:46:56.440 IST walsummarizer[14680] LOG: inside
summarizer_wal_segment_open

Is this a sufficient proof or do you think there could be something
else I am missing or messing up?

It passed on my mac but the test took a bit longer than other tests
and the additional observation that the Windows CI passed raises a
question whether this could be some sort of timing issue in the test.
Test timing on mac:
t/001_basic.pl ....... ok 349 ms ( 0.00 usr 0.00 sys + 0.04
cusr 0.03 csys = 0.07 CPU)
t/002_blocks.pl ...... ok 1464 ms ( 0.00 usr 0.00 sys + 0.09
cusr 0.31 csys = 0.40 CPU)
t/003_tli_switch.pl .. ok 13117 ms ( 0.00 usr 0.00 sys + 0.64
cusr 1.85 csys = 2.49 CPU)

Can we first rule out that this can't be a timing issue in the test case itself?

--
With Regards,
Amit Kapila.

On 15 Jul 2026, at 11:28, Amit Kapila <amit.kapila16@gmail.com> wrote:

Can we first rule out that this can't be a timing issue in the test case itself?

Thanks! That log points at the test, not the summarizer. On node3
it's "waiting for WAL to become available at 0/03000018", which
is before the switch, so recovery (and the summarizer) never gets there.
The pre-switch segment node3 needs never reaches node2's archive:
with archive_mode=on, segments node2 received only by
streaming aren't archived, and it's already gone by the time node2 is
promoted. Whether this bites depends on how much WAL precedes the
promotion, which can easily differ on Windows.

I can reproduce it locally by adding an extra WAL switch on node1
before promoting node2. It can probably be made deterministic either by
polling pg_stat_archiver before relying on the segment (as
025_stuck_on_old_timeline.pl does) or by setting archive_mode=always on
node2 (I would expect it to be a bit faster).

Best regards, Andrey Borodin.

On Wed, Jul 15, 2026 at 4:52 AM Andrey Borodin <x4mmm@yandex-team.ru> wrote:

I can reproduce it locally by adding an extra WAL switch on node1
before promoting node2. It can probably be made deterministic either by
polling pg_stat_archiver before relying on the segment (as
025_stuck_on_old_timeline.pl does) or by setting archive_mode=always on
node2 (I would expect it to be a bit faster).

archive_mode=always seems like the correct fix. Here's v2.

--
Robert Haas
EDB: http://www.enterprisedb.com

Hi,

i have looked into the patch and it LGTM , will also look into the test.

a nitpick in the 003_tli_switch.pl file the comment says
"....giving it a ".partial" suffix, and is not archived," but AFAIK,
We do archive the .partial files right? CleanupAfterArchiveRecovery
renames and notifies the archiver, or am I reading it wrong?

On Tue, Jul 14, 2026 at 6:11 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:

I think, there's a symmetric bug on the recovery side with the same root
cause.
Your patch makes the summarizer fall forward to a descendant timeline,
which
looks safe. XLogFileReadAnyTLI() does the opposite: when the switch-point
segment
is missing on the target timeline, it falls back to an ancestor and reads
the
same segno. But that segment holds divergent data continuing old timeline!
Recovery applies it silently and then can't reach the intended timeline.

The comment you added to t/003_tli_switch.pl states the assumption that
breaks
here:

# ... recovery will read them from there and work just fine.

That holds only when the new timeline's segment is already there.
Concretely,
with just two timelines: TL2 forks from TL1 in segment 67, which is
archived
for TL1 but not yet for TL2. Recovery targets TL2 and asks for segment 67;
TL2's copy isn't in the archive, so XLogFileReadAnyTLI() falls back to TL1
and
hands back TL1's segment 67. Past the switch point that is the old
primary's
divergent WAL, recovery applies it, and now it can never get onto TL2. The
beginseg check doesn't help here - it only skips a timeline for segments
older
than that timeline's start, so nothing stops the fallback to the ancestor
TL1
for the very segment where the two diverge.

+1

--
Thanks :)
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/

On Tue, Jul 14, 2026 at 8:41 AM Andrey Borodin <x4mmm@yandex-team.ru> wrote:

It seems correct to me: we stop at switch_lsn, so we
only read the pre-switch bytes both timelines share. But it might be worth a
test.

I don't think I'm sufficiently motivated to want to write a TAP test
for two timeline switches in the same segment. If you want to try it,
feel free. We can add that in a later commit; it doesn't need to be
part of this patch.

Second, in the fallback path WALRead reopens the segment on every page,
because the requested tli never matches seg.ws_tli once
summarizer_wal_segment_open() has moved to a descendant. These open/close
might be an exatra overhead?

I can believe this might be the case, and I'll look into fixing it. I
don't think it makes any practical difference because segments
containing timeline switches are rare and only contain so many blocks,
and opening and closing files is not that expensive. But it makes
sense to get this right.

I think, there's a symmetric bug on the recovery side with the same root cause.
Your patch makes the summarizer fall forward to a descendant timeline, which
looks safe. XLogFileReadAnyTLI() does the opposite: when the switch-point segment
is missing on the target timeline, it falls back to an ancestor and reads the
same segno. But that segment holds divergent data continuing old timeline!
Recovery applies it silently and then can't reach the intended timeline.

I think this is only a problem if somebody reads past the switchpoint.
In any case, this thread is not about fixing bugs in
XLogFileReadAnyTLI() or its callers.

--
Robert Haas
EDB: http://www.enterprisedb.com

On Wed, Jul 15, 2026 at 1:43 PM Srinath Reddy Sadipiralla
<srinath2133@gmail.com> wrote:

a nitpick in the 003_tli_switch.pl file the comment says
"....giving it a ".partial" suffix, and is not archived," but AFAIK,
We do archive the .partial files right? CleanupAfterArchiveRecovery
renames and notifies the archiver, or am I reading it wrong?

You're right. I'll reword the comment.

+1

I think this is all a digression from the problem at hand. I
understand Andrey's probable desire to have me review his patch on
that thread, but I think whatever is going on there is only
tangentially related to this problem.

--
Robert Haas
EDB: http://www.enterprisedb.com

On Wed, Jul 15, 2026 at 7:46 PM Robert Haas <robertmhaas@gmail.com> wrote:

On Wed, Jul 15, 2026 at 4:52 AM Andrey Borodin <x4mmm@yandex-team.ru> wrote:

I can reproduce it locally by adding an extra WAL switch on node1
before promoting node2. It can probably be made deterministic either by
polling pg_stat_archiver before relying on the segment (as
025_stuck_on_old_timeline.pl does) or by setting archive_mode=always on
node2 (I would expect it to be a bit faster).

archive_mode=always seems like the correct fix. Here's v2.

Thanks for the updated patch, it works for me.

--
With Regards,
Amit Kapila.

On 16 Jul 2026, at 01:18, Robert Haas <robertmhaas@gmail.com> wrote:

understand Andrey's probable desire to have me review his patch on
that thread, but I think whatever is going on there is only
tangentially related to this problem.

Of course I'd like that fix committed too :) But that's not why I'm raising it here.

+# The reason for the problem is that when a new primary is promoted, the
+# partial file that ends the old timeline is renamed, giving it a ".partial"
+# suffix, and is not archived. That's not a problem for recovery, because
+# the bytes that appear at the start of that segment will be copied into the
+# first segment on the new timeline, and recovery will read them from there
+# and work just fine.

That only holds when the new timeline's segment is already available. If it isn't,
XLogFileReadAnyTLI() falls back to the ancestor's segment and, past the switch point,
hands recovery the old timeline's divergent WAL. It's the mirror, on the recovery
side, of what your patch fixes for the summarizer.

I'm not asking you to fix XLogFileReadAnyTLI() here. I just don't think the new
comment should claim recovery is unaffected.

Best regards, Andrey Borodin.

On Wednesday, July 15, 2026 10:16 PM Robert Haas <robertmhaas@gmail.com> wrote:

On Wed, Jul 15, 2026 at 4:52 AM Andrey Borodin <x4mmm@yandex-
team.ru> wrote:

I can reproduce it locally by adding an extra WAL switch on node1
before promoting node2. It can probably be made deterministic either
by polling pg_stat_archiver before relying on the segment (as
025_stuck_on_old_timeline.pl does) or by setting archive_mode=always
on
node2 (I would expect it to be a bit faster).

archive_mode=always seems like the correct fix. Here's v2.

While reading the WAL summarization code, I ran the new 003_tli_switch test and
noticed it takes about 14 seconds to finish on my machine, which prompted me to
debug it a bit. The delay comes from the 10-second wait in the error recovery
path (WAIT_EVENT_WAL_SUMMARIZER_ERROR) in WalSummarizerMain, where it retries
after failing to read an unarchived WAL file.

I was wondering whether it's worth improving this. We could enable summarize_wal
on node3 only after the WALs on the new timeline become readable on node3, which
would eliminate the retry and reduce the test time by about 10 seconds on my
machine. However, if the intention is to exercise the retry logic in
WalSummarizerMain as well, the current test is fine as is. Just sharing this in
case the test time wasn't intentional.

Apart from the test, the fix looks good to me.

Best Regards,
Hou zj