Fix XLogFileReadAnyTLI silently applying divergent WAL from wrong timeline

On Fri, 20 Feb 2026 at 15:09, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

We hit an interesting bug(?) in archive recovery during a planned HA
switchover. One of standbys followed wrong timeline.
I think I saw this problem before in pgsql-hackers but googling didn't help.

I saw something very similar some time ago. [1]/messages/by-id/CANwKhkMN3QwAcvuDZHb6wsvLRtkweBiYso-KLFykkQVWuQLcOw@mail.gmail.com

[1]: /messages/by-id/CANwKhkMN3QwAcvuDZHb6wsvLRtkweBiYso-KLFykkQVWuQLcOw@mail.gmail.com

Regards,
Ants Aasma

On 20 Feb 2026, at 22:53, Ants Aasma <ants.aasma@cybertec.at> wrote:

I saw something very similar some time ago. [1]

Thanks for the link. Yes, this thread is about the same bug and your
patch is functionally equivalent to what I just proposed.

Basically, it all boils down to simple invariant: "When restoring to
specific timeline we should not take turns to other timelines."

Best regards, Andrey Borodin.

On 20 Feb 2026, at 23:36, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

Basically, it all boils down to simple invariant: "When restoring to
specific timeline we should not take turns to other timelines."

Here is patch 0002 -- an optimization, independent of 0001.

After walrcv_endstreaming() returns, walreceiver fetches the new
timeline's history file (WalRcvFetchTimeLineHistoryFiles) before
transitioning to WALRCV_WAITING. During this window walreceiver
remains in WALRCV_STREAMING.

Startup sleeps in WAIT_EVENT_RECOVERY_WAL_STREAM receiving no new
data. When it wakes and finds WalRcvStreaming()==true in the
XLOG_FROM_STREAM handler, it kills walreceiver. The new walreceiver
must reconnect and re-request the same switch -- wasteful but
harmless. (In the original report this appears as "terminating
walreceiver process due to administrator command" at 11:52:12.)

Fix: add WALRCV_SWITCHING_TIMELINE. Walreceiver enters it just
before WalRcvFetchTimeLineHistoryFiles(). WalRcvStreaming() returns
false for this state, so startup backs off instead of killing
walreceiver. WakeupRecovery() is called immediately after the
transition so startup exits its indefinite RECOVERY_WAL_STREAM sleep
without waiting for WalRcvWaitForStartPosition().

A guard in RequestXLogStreaming() is also required: because
WALRCV_SWITCHING_TIMELINE is not "streaming", the XLOG_FROM_STREAM
failure path no longer calls XLogShutdownWalRcv() before retrying
archive. When startup cycles back to RequestXLogStreaming(),
walreceiver may still be in WALRCV_SWITCHING_TIMELINE, which would
Assert-fail the STOPPED||WAITING check. The guard returns early in
that case.

One concern: WALRCV_SWITCHING_TIMELINE is not protected by
wal_receiver_timeout, which only runs inside the streaming loop.
Before this patch startup's kill provided an implicit bound on the
history fetch; now only TCP-level timeouts apply. I think this
warrants a follow-up, but it is out of scope here.

Test 054 uses an injection point to freeze walreceiver in
WALRCV_SWITCHING_TIMELINE and verifies startup enters
RecoveryRetrieveRetryInterval rather than killing walreceiver.

WDYT?

Best regards, Andrey Borodin.

Hi Andrey,

I finally got some time to review this patch. +1 it's clearly the same bug
as the 2022 thread Ants linked, and 0001 lands where that discussion ended
up.
Since it's silent corruption on an ordinary setup and reproduces back to
v12, I think it needs a back-patch to all supported branches.

I iterated through the cases that seemed interesting, like a mid-segment
switch, a switch that lands exactly on a segment boundary, and
a three-timeline chain and the patch works as expected.
In particular the beginseg check stays a "continue", so a too-new timeline
doesn't consume the single attempt and the real owner is still tried.
The 053 test is a good catch: a fix that only guarded the target timeline's
own switch point would walk straight past that intermediate case.

A couple of comments:
1. Removing the fallback also removes the one case where it did the right
thing: a cascading failover where the child timeline promoted and archived
its
history file but died before archiving the switch-point segment. There,
only the parent's copy of that segment is available, and everything in it
before the switch point is still valid WAL and today we replay it, with
this patch we stall.
I think stalling is the correct default (a visible wait is far better than
silently applying divergent data), maybe we call this out in the commit
message.
The complete fix, stopping at the exact switch LSN instead of at segment
granularity, is a bigger change and fair to leave for later.

2. The comment "the parent's WAL is no longer valid" overstates it; the
pre-switch bytes are valid, we just can't stop mid-segment.
Worth rewording, plus a line on how found_eligible relates to the existing
"tli < curFileTLI" guard (they're not redundant).

3. The 052/053 tests only assert the negative, so they'd also pass if
recovery never reached the segment.
Could you add a positive check where the divergent rows never appear, and
in one test that recovery finishes on the right
timeline once the correct segment shows up?

4. With the fallback gone, a missing child segment now looks like a hang
with only the DEBUG2 "could not open file" trace. A
LOG/DEBUG1 line where we decline to fall back ("waiting for the owning
timeline's segment N") would make the intended wait self-explaining.

I haven't dug into 0002/0003 yet, but from a high level they look
independent of 001 patch. I'll review those soon.

Thanks for chasing this down.

Regards,
Surya Poondla

Hi Andrey,

First of all, I would like to ask on which version you obtained the log
with the bug (unjustified death of the val receiver). I have looked
carefully and noticed that you are fixing a comment in your patch that
appeared only in the version where the bug was already absent (I will
explain what I mean later).

I tried to reproduce the behavior you showed in your logs on the master
branch, but couldn't, walreceiver did not die in any way. It is also worth
noting that the original problem appeared during the transition from 14.17
to 14.18. The discussion regarding this can be found here[1]/messages/by-id/19093-c4fff49a608f82a0@postgresql.org:

The discussion in that thread ultimately led to patch [2]https://github.com/postgres/postgres/commit/17b2d5ec759c0d26b29def7e57f51d0515ddca1f#diff-12d3e48c566898484e306c9662afb37a426c4601a1efa2a09869d25f2659dd02; note the added
check in src/backend/access/transam/xlogrecovery.c.

Previously we’d unconditionally kill the walreceiver in the
XLOG_FROM_STREAM handler; now we check whether it’s actually streaming, and
if it’s in WALRCV_WAITING state we leave it alone.

On 20 Feb 2026, at 13:19:17, Andrey Borodin <x4mmm(at)yandex-team(dot)ru>

wrote:

After walrcv_endstreaming() returns, walreceiver fetches the new
timeline's history file (WalRcvFetchTimeLineHistoryFiles) before
transitioning to WALRCV_WAITING. During this window
walreceiver remains in WALRCV_STREAMING.

I think behavior here is different. Both before this patch and after it,
when the startup process wakes up during a timeline change, it stays inside
the same case in loop iteration and hits this check:

if (!WalRcvStreaming())
{
lastSourceFailed = true;
break;
}

If the receiver is still in streaming state, lastSourceFailed remains
false, so the loop continues (sleeping on the latch but not leaving the
current case). It’s only once the state eventually transitions to
WALRCV_WAITING that the process wakes up, !WalRcvStreaming() returns true,
lastSourceFailed gets set, and we break out of the current case. Because of
it we go in the XLOG_FROM_STREAM handler and, before the patch, the
receiver was killed there without any additional guard. The patch simply
adds a check that says "if it's already in WAITING state (e.g., due to
timeline divergence), we only need to reset the install".

Besides, lastSourceFailed is set to true only in
WaitForWALToBecomeAvailable/XLogPageRead/ReadRecord.

I am concerned that an unjustified death still happens, but only in the
case where XLogPageRead/ReadRecord sets the lastSourceFailed = true at the
very moment when the walreceiver is switching timelines and hasn't yet
changed its state to WAITING, while the startup process is making its way
to the call of WaitForWALToBecomeAvailable and the check on the
walreceiver's state there. Then the walreceiver would be killed
unnecessarily. However, fixing such rare scenario would likely introduce
bugs.

So, I don't quite understand the point of patch 0002. Since we already have
a general WALRCV_WAITING state, why do we need to introduce another
specific state just to clarify what exactly we are waiting for?

Best wishes,
Stepan Tyagushev

[1]: /messages/by-id/19093-c4fff49a608f82a0@postgresql.org
/messages/by-id/19093-c4fff49a608f82a0@postgresql.org
[2]: https://github.com/postgres/postgres/commit/17b2d5ec759c0d26b29def7e57f51d0515ddca1f#diff-12d3e48c566898484e306c9662afb37a426c4601a1efa2a09869d25f2659dd02
https://github.com/postgres/postgres/commit/17b2d5ec759c0d26b29def7e57f51d0515ddca1f#diff-12d3e48c566898484e306c9662afb37a426c4601a1efa2a09869d25f2659dd02

On Mon, Aug 17, 2026 at 10:53 AM Andrey Borodin <x4mmm@yandex-team.ru>
wrote:

Hi Stepan,

On 17 Aug 2026, Stepan Tyagushev wrote:

The discussion in that thread ultimately led to patch [2]; note the
added
check in src/backend/access/transam/xlogrecovery.c.

Previously we'd unconditionally kill the walreceiver in the
XLOG_FROM_STREAM handler; now we check whether it's actually
streaming,
and
if it's in WALRCV_WAITING state we leave it alone.

Thank you for pointing this out. The production cluster was running the
current minor release at the time of the report. In any case, I
rechecked the test on a tree that already contains 17b2d5ec759, and
there is still a smaller window. After
walrcv_endstreaming() returns and while
WalRcvFetchTimeLineHistoryFiles() runs, the shared state is still
WALRCV_STREAMING. WALRCV_WAITING is set only later by
WalRcvWaitForStartPosition(). If startup wakes in that interval, the
check added by 17b2d5ec759 still calls XLogShutdownWalRcv().

I also checked the negative direction of test 054. I kept its
deterministic WakeupRecovery() and injection point, but removed
WALRCV_SWITCHING_TIMELINE. Startup then enters XLogShutdownWalRcv() and
waits for the walreceiver to exit instead of reaching
RecoveryRetrieveRetryInterval. The test fails as expected. With the new
state it passes. So failure to reproduce this without the injection
point seems to be a matter of the window being short.

No need to manually reproduce the problem, it is simulated in the test.
And test is failing if we do not apply the fix.

Since we already have a general WALRCV_WAITING state, why do we need
to
introduce another specific state just to clarify what exactly we are
waiting for?

WALRCV_WAITING means that walreceiver is ready for new instructions.
That is not yet true while it is fetching the history file, so setting
WAITING earlier would allow startup to request another stream before the
fetch finishes. The separate state distinguishes those two conditions.

Nevertheless, this is independent from the main problem in this thread:
XLogFileReadAnyTLI applying a divergent segment from an older timeline.
The extra walreceiver restart is much less serious, and the additional
state deserves its own discussion. Maybe I should move patches 0002/0003
to a separate thread and keep the next version here focused on 0001.

I also saw Surya's review of 0001. The comments about the conservative
switch-point behavior, diagnostics, and positive test coverage are
valuable, and I agree with them. I am not ready to post a v3 addressing
them yet, but I hope to do that soon.

Thank you!

Best regards, Andrey Borodin.

On 2 Jul 2026, at 20:59, surya poondla <suryapoondla4@gmail.com> wrote:

I finally got some time to review this patch.

Hi Surya,

Thank you for the review! PFA v3.

The code now stops after trying the newest timeline eligible for the
requested segment. The comment explains why a parent copy of the
switch-point segment cannot be used as a whole even though its prefix is
valid. A DEBUG1 message makes the resulting wait explicit.

Both tests now first prove that recovery reaches the missing child
segment and refuses the available parent copy. They then publish the
correct child segment, let recovery reach the requested timeline, and
check the exact rows, including the absence of divergent parent rows.

I left the separate walreceiver state change out of v3. Maybe, I will
discuss it in a separate thread. Restart of the walreceiver itself is not
such a big problem as not reaching target timeline.

Thank you!

Best regards, Andrey Borodin.