BUG #19640: Standby permanently stuck re-requesting old timeline after promotion, never switches to new timeline
The following bug has been logged on the website:
Bug reference: 19640
Logged by: Harshit Singh
Email address: harshit.singh817775@gmail.com
PostgreSQL version: 18.0
Operating system: rhel 10
Description:
PostgreSQL version: Reproduced on both 17.6 and 18.0 — present in the
current release, not something already fixed upstream.
Operating system: Linux x86_64 (RHEL-family, built with Red Hat gcc 14.3.1)
Description:
A standby configured with recovery_target_timeline = 'latest' (the default
under streaming-replication HA managers like Patroni) can get permanently
stuck after a timeline promotion elsewhere in the cluster: it endlessly
re-requests WAL on its own old, now-superseded timeline, is told "end of WAL
reached" by the primary each time, disconnects, and immediately reconnects
requesting the same old timeline again — never advancing to the new one.
This repeats indefinitely (observed for over an hour in one case, tight
~10-20ms reconnect loop), consuming CPU, with no error surfaced to indicate
the process needs manual intervention — patronictl/monitoring tooling on top
of it just reports the node as "starting" forever.
Steps to reproduce:
1. 3-node streaming replication cluster (repro used Patroni-managed
Postgres, but the core issue appears to be in core recovery logic, not
Patroni).
2. Node A is primary; other nodes stream from it.
3. Node A is stopped/killed (or even just a plain systemctl restart of the
current leader — no exotic failure needed). Another node is promoted
(timeline N → N+1).
4. Node A is later restarted and attempts to rejoin as a replica of the new
leader.
5. Node A's local timeline is N; the new leader is on N+1.
6. Node A's log shows, repeating forever:
LOG: started streaming WAL from primary at <LSN> on timeline N
LOG: replication terminated by primary server
DETAIL: End of WAL reached on timeline N at <LSN>.
FATAL: terminating walreceiver process due to administrator command
LOG: waiting for WAL to become available at <LSN>
— new walreceiver PID each cycle, always requesting timeline N, never
N+1.
Reproduced 5 times across different sessions/timelines (N=1 through N=4) and
both PostgreSQL 17.6 and 18.0, including once under active write load (real
WAL divergence existed, not just an idle-DB edge case), and via completely
ordinary systemctl stop/start of the current leader — not an exotic
scenario. The fact that it reproduces identically on 18.0 indicates this
isn't a regression already fixed in the latest release.
Expected behavior:
The standby should detect, via rescanLatestTimeLine(), that a newer timeline
(N+1) now exists and switch its target to follow it, per
recovery_target_timeline = 'latest' semantics.
Suspected root cause / prior art:
This looks closely related to the mechanism described by Dilip Kumar on
-hackers in "Race condition in recovery?"
(https://postgrespro.com/list/thread-id/2526828):
WaitForWALToBecomeAvailable() initializes expectedTLEs from receiveTLI
rather than recoveryTargetTLI. When rescanLatestTimeLine() finds the newest
TLE already matches recoveryTargetTLI, it concludes "nothing to change" —
but expectedTLEs is left referencing the old timeline regardless, so every
subsequent WAL request keeps using it. A patch was proposed there
(initializing from recoveryTargetTLI instead) but the thread doesn't show it
as committed, and given this is still reproducible on 18.0, it appears that
patch — or an equivalent fix — never landed.
Additional context:
A near-identical symptom was reported independently against CloudNativePG
(https://github.com/cloudnative-pg/cloudnative-pg/issues/10419), consistent
with this being a core recovery-logic issue rather than something specific
to any one HA orchestration layer.
Hi Harshit,
On Wed, Aug 26, 2026 at 11:56 AM PG Bug reporting form wrote:
Node A's local timeline is N; the new leader is on N+1.
Node A's log shows, repeating forever:
LOG: started streaming WAL from primary at <LSN> on timeline N
DETAIL: End of WAL reached on timeline N at <LSN>.
There are several cases that can produce a symptom like this, and the
actual LSNs are important for distinguishing them.
First, Node A was the former primary. If the new primary forked from
timeline N at a position behind Node A's local WAL, Node A is already
past the switchpoint and cannot continue on N+1. It has to be rewound
or rebuilt. Could you check whether Patroni attempted pg_rewind,
whether it succeeded, and whether data checksums or wal_log_hints make
pg_rewind possible in this cluster? If Patroni did not run pg_rewind,
or if it failed, that part is probably better investigated with the
Patroni project.
For this thread, please include the complete PostgreSQL log around one
cycle, including the startup process and walreceiver messages, rather
than only the repeated lines.
The proposed receiveTLI -> recoveryTargetTLI fix from the 2021 thread
was committed as 6eb5b9ae391 [0]https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=6eb5b9ae39176a7d40003b4c2e9ca22e6b205def. It was released in PostgreSQL
9.6.23, 10.18, 11.13, 12.8, and 13.4, and has been present since 14.0.
In particular, both 17.6 and 18.0 include it. So that particular
explanation should not apply here.
There are two nearby core recovery issues, but I would consider them
only after ruling out the cases above. Recovery can currently read an
old timeline's copy of a switchpoint segment when the new timeline's
copy is not available yet [1]/messages/by-id/85386EF6-16B7-4D62-86BE-526A10F93825@yandex-team.ru. That can move replay past the
switchpoint on the wrong timeline. There is also a smaller window where
startup can stop a walreceiver while it is fetching the new timeline
history file, before it enters WALRCV_WAITING.
Could you provide:
* the switchpoint from the N+1 history file;
* the real start and end LSNs from the repeated messages;
* Node A's latest checkpoint and redo positions from pg_controldata;
* the complete log from the first attempt to follow N+1;
* whether pg_rewind was attempted, and its final result.
Those values should tell us whether Node A was already divergent,
replayed past the switchpoint during this startup, or reached the
switchpoint but failed to change timelines.
Thank you!
Best regards, Andrey Borodin.
[0]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=6eb5b9ae39176a7d40003b4c2e9ca22e6b205def
[1]: /messages/by-id/85386EF6-16B7-4D62-86BE-526A10F93825@yandex-team.ru