RE: Recovery at replica stuck because recovery incorrectly trusts an old high-water mark
v1 rewound walrcv->flushedUpto in RequestXLogStreaming() whenever the
(segment-rounded) start pointer was behind it. That was too broad:
* RequestXLogStreaming() rounds recptr down to a segment boundary
before the comparison, so the rewind fired on ordinary reconnects,
not only after a failed read.
* pg_last_wal_receive_lsn() jumped backward.
* Cascading walsenders can hit Assert(sentPtr <= SendRqstPtr) when
GetStandbyFlushRecPtr() drops below a position they already sent.
* Kyotaro Horiguchi already argued against a blanket flushedUpto
rewind in 2021 [1]/messages/by-id/20210329.105441.1978082841561262877.horikyota.ntt@gmail.com.
v2 keeps flushedUpto monotonic. Startup uses a separate
applyFlushedUpto for "is this streamed WAL readable?". That pointer is
reset only after lastSourceFailed on archive/pg_wal, when the unrounded
restart LSN is still behind flushedUpto on the same timeline. Healthy
reconnects (including pendingWalRcvRestart after SIGHUP /
primary_conninfo change) keep the old apply pointer, so replay is not
stalled on WAL that is already on disk. After lastSourceFailed,
WaitForWALToBecomeAvailable() still tries archive/pg_wal first; the
reset only happens if those sources also fail.
The TAP test is now t/058_stream_repair.pl (053 and later numbers were
taken on current master). It checks both:
1. primary_conninfo reload with apply lag does not move
pg_last_wal_receive_lsn() backward.
2. A corrupt unreplayed record after receive has moved into the next
segment is replaced via streaming, instead of looping on the local
copy.
I have not re-run the recovery TAP suite against this tree (no local
install of current master).
[1]: /messages/by-id/20210329.105441.1978082841561262877.horikyota.ntt@gmail.com
/messages/by-id/20210329.105441.1978082841561262877.horikyota.ntt@gmail.com
Konstantin Knizhnik (1):
Keep flushedUpto monotonic; reset applyFlushedUpto after a failed
read.
src/backend/access/transam/xlogrecovery.c | 38 ++++-
src/backend/replication/walreceiver.c | 2 +
src/backend/replication/walreceiverfuncs.c | 45 +++++-
src/include/replication/walreceiver.h | 16 ++-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/058_stream_repair.pl | 158 +++++++++++++++++++++
6 files changed, 255 insertions(+), 5 deletions(-)
create mode 100644 src/test/recovery/t/058_stream_repair.pl
On Thu, 2026-09-24 at 21:30 +0300, Konstantin Knizhnik wrote:
v2 keeps flushedUpto monotonic. Startup uses a separate
applyFlushedUpto for "is this streamed WAL readable?"
The reread-from-primary behavior seems to trace to 6da07cd80d, which
looks more like an opportunistic retry than a policy.
We could make it a policy to try to get readable WAL from the primary,
and then keep the extra state you propose. But I think we need more
explanation about why we'd expect this situation (corrupt and flushed
on standby and valid on primary) to occur.
For instance:
/* ...Only after a failed read of
* WAL that flushedUpto still reports as present do we
* rewind the apply pointer, so startup waits for
* replacement bytes.
raises questions for anyone reading that code about why it might be
flushed and unreadable.
Regards,
Jeff Davis
Hi,
Konstantin Knizhnik <knizhnik@garret.ru> wrote:
I have not re-run the recovery TAP suite against this tree (no local
install of current master).
I ran it on master 2c10c2ce4d7 with v2 (cassert, injection points):
src/test/recovery passes, 802 tests against 799 on master, the new
ones being 058. Without the C changes, 058 fails as intended, with
startup rereading the corrupt local copy until the timeout.
One ordering issue: the reset of applyFlushedUpto runs after
RequestXLogStreaming(). If the new walreceiver flushes the
replacement bytes before the reset, the pointer goes back below them
and startup waits for the next flush. The window is tiny, but with a
3 s sleep there, replay sat for about 18 s until an unrelated
running-xacts record arrived.
Doing the reset before RequestXLogStreaming() closes it: with the
same sleep there is no stall, and with only that change (attached, on
top of v2) the recovery suite passes and 058 passed 20 of 20 runs.
Logs and the review-only diffs are in the second attachment.
Regards,
Manu
On 25/09/2026 4:21 AM, Jeff Davis wrote:
On Thu, 2026-09-24 at 21:30 +0300, Konstantin Knizhnik wrote:
v2 keeps flushedUpto monotonic. Startup uses a separate
applyFlushedUpto for "is this streamed WAL readable?"The reread-from-primary behavior seems to trace to 6da07cd80d, which
looks more like an opportunistic retry than a policy.We could make it a policy to try to get readable WAL from the primary,
and then keep the extra state you propose. But I think we need more
explanation about why we'd expect this situation (corrupt and flushed
on standby and valid on primary) to occur.For instance:
/* ...Only after a failed read of
* WAL that flushedUpto still reports as present do we
* rewind the apply pointer, so startup waits for
* replacement bytes.raises questions for anyone reading that code about why it might be
flushed and unreadable.Regards,
Jeff Davis
Paul has found and fixed original problem at primary side:
/messages/by-id/178956158235.97809.18200289969141907831@mail.gmail.com
But the proposed patch is still worth keeping as a standby self-heal for
any other invalid bytes under a high |flushedUpto| (other producer bugs,
CF 5199-style missing file, on-disk junk) still hang without it. The
included test demonstrates it.