Assertion failure in WAL summarizer

Hi Fujii,

On Thu, Oct 22, 2026 at 10:38 AM Fujii Masao <masao.fujii@gmail.com> wrote:

I haven't investigated this yet, but wanted to report it in advance.

I can reproduce it with your steps on master, every time once the
three conditions hold, and also on REL_19, REL_18 and REL_17 with
cassert, so it goes back to when the summarizer was added.

Before promotion the summarizer has already gone past the recovery
target. In recovery GetLatestLSN() returns the walreceiver flush
position, and the walreceiver keeps streaming after replay pauses.
With your steps on master, replay pauses at the recovery target
0/03001B08, the walreceiver has flushed to 0/03001D98, and the
summarizer has summarized up to 0/03001CB0 and is waiting for WAL at
0/03001D98.

After promotion TLI 1 ends at 0/03001B08 ("1 0/03001B08 before LSN
0/03001B08" in 00000002.history), behind the summarizer. With gdb
attached to the summarizer:

1. summarizer_read_local_xlog_page() notices TLI 2 while waiting and
sets read_upto = switchpoint: 0/03001D98 -> 0/03001B08. The comment
there expects this ("this will cause read_upto to move backwards,
because we might have been promoted before reaching the end of the
previous timeline").

2. The next read is past read_upto on a timeline that is now historic,
so end_of_wal is set and SummarizeWAL() returns
summary_end_lsn = private_data->read_upto = 0/03001B08, although it
started at 0/03001CB0.

3. WalSummarizerMain(): current_lsn = 0/03001CB0,
end_of_summary_lsn = 0/03001B08 -> the assertion at line 460.

The main loop already handles "we passed the switch point": at the top
of the loop, if current_lsn >= switch_lsn it backs up to the switch
point on the new timeline. But that path only runs when the timeline
change is seen there. When it is seen inside SummarizeWAL(), the
backwards LSN comes back as the return value instead.

Without assertions nothing bad seems to happen. On a non-cassert build
of master with the same steps, the summarizer takes 0/03001B08 as its
position, sees TLI 2 on the next iteration, switches at 0/03001B08 and
goes on. It is the same process, and you end up with the same
summaries and state as with the patch below. So as far as I can tell
this only affects assert-enabled builds, but there the summarizer
crash makes the postmaster restart the whole cluster right after the
promotion.

The attached patch makes SummarizeWAL() report no progress when
read_upto ended up before the start of the summary, so the existing
code in the main loop does the rewind on the next iteration. With it,
your steps no longer trip the assertion, and summarization resumes on
TLI 2 exactly at the target, on master, 19, 18 and 17 (the patch
applies to all four). It also adds
src/bin/pg_walsummary/t/004_promote_past_target.pl, which follows your
steps: it fails with the TRAP on unpatched master and passes with the
fix. make check, pg_walsummary and pg_combinebackup pass.

Another option is to relax the assertion and keep the current
behavior, since that is already what a non-assert build does. I went
with the clamp because then SummarizeWAL() never returns an LSN before
its start, and summarized_lsn doesn't go backwards on the old timeline
for an iteration before the rewind. No strong opinion, though.

Regards,
Manu

Hi,

I hit the same assertion independently earlier and can confirm Manu's
analysis.

Before promotion, with Fujii's steps on master:

replay 0/03001B08
receive 0/03001D98
summarized 0/03001CB0

which matches the numbers in Manu's mail. The size of the gap between
replay and receive doesn't matter: with ~20MB of streamed WAL past an
arbitrary recovery_target_lsn (recovery_target_inclusive = true) the
summarizer is ~20MB ahead and the crash is the same.

With cassert disabled, I don't see any adverse effects either: summaries
on TLI 2 start at the switch point, and an incremental backup across the
promotion works.

With v1 applied on master, the same scenarios pass with cassert and the
new test passes; with the walsummarizer.c hunk reverted, the test fails
with the TRAP. pg_walsummary and pg_combinebackup TAP suites pass.

A couple of minor comments on v1:

1. The comment in summarizer_read_local_xlog_page() about read_upto
moving backwards says the next iteration will "likely conclude that
we've reached end of WAL". It might be worth mentioning there that
SummarizeWAL() then reports no progress, so the link between the two
places is visible from both sides.

2. In the test, the summarizer PID check and the log check both detect
the same thing, a restart after the crash. Without the fix neither
is reached anyway: pg_ctl promote fails once the postmaster
reinitializes, and the test dies without a plan. I think one of the
two checks would be sufficient.

Separately, while looking at this I found a related but different
problem: if the switch point falls exactly on a page or segment
boundary, the first summary on the new timeline starts after the page
header, and incremental backups on top of any pre-promotion backup fail
permanently with "WAL summaries are required on timeline 2 from
0/04000000 ...". v1 doesn't change that, and it isn't expected to. I'll
look into it
more closely and then maybe start a separate thread with a reproducer.

--
Daria Lepikhova

Hi Daria,

Thanks for confirming the analysis independently, and for the review
and testing.

On Thu, Oct 22, 2026 at 10:30 AM Daria Lepikhova
<daria.n.lepikhova@gmail.com> wrote:

1. The comment in summarizer_read_local_xlog_page() about read_upto
moving backwards says the next iteration will "likely conclude that
we've reached end of WAL". It might be worth mentioning there that
SummarizeWAL() then reports no progress, so the link between the two
places is visible from both sides.

Done in the attached v2. The comment now ends with "If the switch
point precedes the start of the current summary, SummarizeWAL() then
reports no progress." The comment on the clamp in SummarizeWAL()
already points back to summarizer_read_local_xlog_page().

2. In the test, the summarizer PID check and the log check both detect
the same thing, a restart after the crash. Without the fix neither
is reached anyway: pg_ctl promote fails once the postmaster
reinitializes, and the test dies without a plan. I think one of the
two checks would be sufficient.

Agreed. v2 keeps only the log check, since log_contains() doesn't need
a running server.

TAP nodes run with restart_after_crash = off, so the crash shuts the
standby down instead of reinitializing it, 73-105 ms after it reaches
production in my runs. pg_ctl promote polls every 100 ms, so whether
it sees the promotion first is a race. Here promote won in 13 of 13
runs without the fix, and the v1 test then failed "not ok 1" and died
on the PID query against the stopped server. With v2 and the fix
reverted, it now ends with "failed 2 tests of 2" instead (4 of 4).
When promote loses the race, as in your run, the test still bails out
at promote. I left that as it is.

v2 applies to master, 19, 18 and 17 like v1, and passes there.

Separately, while looking at this I found a related but different
problem: if the switch point falls exactly on a page or segment
boundary, [...]

Looking forward to that thread. I'm happy to run your reproducer
against v2 once you post it.

Regards,
Manu