Recovery at replica stuck because recovery incorrectly trusts an old high-water mark
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t253778psql -h localhost -U postgresBuilt from patchset v1 (message #1), September 13, 2026 at 01:44 PM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t253778_1 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t253778_1 && git checkout t253778_1Patchset v1 (message #1) is on t253778_1
Hi,
A physical standby can get stuck forever if local pg_wal contains a
corrupt record that walreceiver has already reported as flushed.
What happens:
1. walreceiver streams past the bad record and leaves flushedUpto
ahead of it.
2. Startup reads the corrupt bytes, fails ValidXLogRecordHeader,
and treats streaming as a failed source.
3. It shuts walreceiver down and calls RequestXLogStreaming() from
the start of that record (rounded down to the segment boundary).
4. RequestXLogStreaming() only resets flushedUpto on first start or
a timeline change, so the old high-water mark remains.
5. WaitForWALToBecomeAvailable() sees RecPtr < flushedUpto, decides
the WAL is already on disk, and rereads the same bytes.
6. Startup kills walreceiver again ("terminating walreceiver process
due to administrator command") before START_REPLICATION.
The result is a tight loop: invalid record, launch walreceiver, reread
local WAL, kill walreceiver. Replay never advances.
This still happens on current master. I reproduced it with the TAP
test in 0001: without the C change, 053_stream_repair.pl times out
waiting for replay to pass the injected record; with the change it
passes.
The fix resets flushedUpto (and latestChunkStart) when the requested
start is behind the previous flush pointer. That is the same
RequestXLogStreaming() site that already rewinds on first start and on
timeline change. recptr has already been rounded down to a segment
boundary. After lastSourceFailed the current WAL file is closed, so
startup re-reads the record from its start once the replacement bytes
have been flushed.
This is intentionally not "always rewind flushedUpto". Cai Mengjuan
proposed that in 2021, and Kyotaro Horiguchi pointed out that a
blanket rewind can mix the already-read prefix of a spanning record
with later replacement data [1]/messages/by-id/20210329.105441.1978082841561262877.horikyota.ntt@gmail.com. We only rewind when streaming is
explicitly restarted from an earlier LSN.
A related patch was CF 5199 (pixian shi, withdrawn 2026-09-07) [2]https://commitfest.postgresql.org/patch/5199/[3]/messages/by-id/CAAccyYKrRojjO-weeXFs1EqLFHWSBfjBzObwVDv4u-ZxmU=7Rg@mail.gmail.com Konstantin Knizhnik (1): Rewind walreceiver flushedUpto when restarting from an earlier LSN. src/backend/replication/walreceiverfuncs.c | 15 ++- src/include/replication/walreceiver.h | 3 +- src/test/recovery/meson.build | 1 + src/test/recovery/t/053_stream_repair.pl | 109 +++++++++++++++++++++ 4 files changed, 124 insertions(+), 4 deletions(-) create mode 100644 src/test/recovery/t/053_stream_repair.pl.
That compared recptr against receiveStart. receiveStart is the start
of the current walreceiver session and is not updated as flush
advances, so restarting from the same segment does not rewind even
when flushedUpto is already in a later segment. That is exactly the
hang this TAP test covers. Comparing against flushedUpto also covers
the switchover PANIC from CF 5199, where startup tried to open a WAL
file that existed only according to a stale flush pointer.
Side effect: pg_last_wal_receive_lsn() can move backward across such a
restart. I think that is preferable to reporting WAL that recovery
has just rejected.
I have not changed the XLogData handler to reject a stream that skips
forward inside a segment. That would be a separate invariant.
Tested with:
make -C src/test/recovery check PROVE_TESTS='t/053_stream_repair.pl'
[1]: /messages/by-id/20210329.105441.1978082841561262877.horikyota.ntt@gmail.com
/messages/by-id/20210329.105441.1978082841561262877.horikyota.ntt@gmail.com
[2]: https://commitfest.postgresql.org/patch/5199/
[3]: /messages/by-id/CAAccyYKrRojjO-weeXFs1EqLFHWSBfjBzObwVDv4u-ZxmU=7Rg@mail.gmail.com Konstantin Knizhnik (1): Rewind walreceiver flushedUpto when restarting from an earlier LSN. src/backend/replication/walreceiverfuncs.c | 15 ++- src/include/replication/walreceiver.h | 3 +- src/test/recovery/meson.build | 1 + src/test/recovery/t/053_stream_repair.pl | 109 +++++++++++++++++++++ 4 files changed, 124 insertions(+), 4 deletions(-) create mode 100644 src/test/recovery/t/053_stream_repair.pl
/messages/by-id/CAAccyYKrRojjO-weeXFs1EqLFHWSBfjBzObwVDv4u-ZxmU=7Rg@mail.gmail.com
Konstantin Knizhnik (1):
Rewind walreceiver flushedUpto when restarting from an earlier LSN.
src/backend/replication/walreceiverfuncs.c | 15 ++-
src/include/replication/walreceiver.h | 3 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/053_stream_repair.pl | 109 +++++++++++++++++++++
4 files changed, 124 insertions(+), 4 deletions(-)
create mode 100644 src/test/recovery/t/053_stream_repair.pl