pgsql: Forget the prefetcher's current record in XLogPrefetcherBeginRea

Started by Tomas Vondra3 days ago4 messagescommitters
Jump to latest
#1Tomas Vondra
tomas.vondra@postgresql.org

Forget the prefetcher's current record in XLogPrefetcherBeginRead()

XLogPrefetcherBeginRead() reset the decoder's record queue, but not
prefetcher->record, whose block references XLogPrefetcherNextBlock()
examines. The next read then continued with the remaining block
references of that stale record, before decoding the record at the new
read position.

FinishWalRecovery() re-reads the last replayed record to find the end
of WAL. When recovery stops at a recovery target, the prefetcher may
be part-way through a later record, and if its remaining block
references fill up the I/O queue, no record is decoded. This fails an
assertion in XLogPrefetcherReadRecord(). Without assertions, the start
of the last replayed record silently becomes the end of WAL, the new
timeline branches off at the wrong point, and the end-of-recovery
checkpoint fails with "xlog flush request ... is not satisfied". The
recovery signal file is gone by then, so a restart performs crash
recovery past the recovery target.

This mostly happens with maintenance_io_concurrency set to 7 or less,
and a recovery target shortly before an XLOG_FPI record written by an
index build, CLUSTER or similar operation, as such records carry up to
XLR_MAX_BLOCK_ID block references that need no I/O.

Fix by forgetting the current record in XLogPrefetcherBeginRead(). The
other callers run before the redo loop, when there is no current record.

Add a TAP test that stops recovery shortly before a multi-block XLOG_FPI
record of an SP-GiST index build, with maintenance_io_concurrency = 1.
The test uses SP-GiST rather than B-tree, so that it works unchanged in
the back branches. B-tree builds before 17 WAL-log each page in a
separate XLOG_FPI record with a single block reference, so the
prefetcher is never part-way through such a record.

Author: Tomas Vondra <tomas@vondra.me>
Reviewed-by: surya poondla <suryapoondla4@gmail.com>
Reviewed-by: shihao zhong <zhong950419@gmail.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Discussion: /messages/by-id/a797f236-286a-43e6-9b82-5f3d70bd5e99@vondra.me
Backpatch-through: 15

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/736ecc413b38555637802ccd8dfd35f034235796

Modified Files
--------------
src/backend/access/transam/xlogprefetcher.c | 3 +
src/test/recovery/meson.build | 1 +
.../recovery/t/061_recovery_target_prefetch.pl | 94 ++++++++++++++++++++++
3 files changed, 98 insertions(+)

#2Michael Paquier
michael@paquier.xyz
In reply to: Tomas Vondra (#1)
Re: pgsql: Forget the prefetcher's current record in XLogPrefetcherBeginRea

On Thu, Nov 19, 2026 at 06:23:46PM +0000, Tomas Vondra wrote:

Forget the prefetcher's current record in XLogPrefetcherBeginRead()

XLogPrefetcherBeginRead() reset the decoder's record queue, but not
prefetcher->record, whose block references XLogPrefetcherNextBlock()
examines. The next read then continued with the remaining block
references of that stale record, before decoding the record at the new
read position.

Since this commit, the buildfarm is unstable on various branches for
various hosts, for branches up to 17. Two examples:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=longfin&dt=2026-11-20%2005%3A13%3A08
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=indri&dt=2026-11-20%2005%3A00%3A27

And here is your failure, from the logs of the standby host in test
061:
2026-11-20 00:07:37.476 EST [33350:1] LOG: invalid value for
parameter "maintenance_io_concurrency": 1
2026-11-20 00:07:37.476 EST [33350:2] DETAIL:
"maintenance_io_concurrency" must be set to 0 on platforms that lack
posix_fadvise().
--
Michael

#3Tomas Vondra
tomas@vondra.me
In reply to: Michael Paquier (#2)
Re: pgsql: Forget the prefetcher's current record in XLogPrefetcherBeginRea

On 11/20/26 06:44, Michael Paquier wrote:

On Thu, Nov 19, 2026 at 06:23:46PM +0000, Tomas Vondra wrote:

Forget the prefetcher's current record in XLogPrefetcherBeginRead()

XLogPrefetcherBeginRead() reset the decoder's record queue, but not
prefetcher->record, whose block references XLogPrefetcherNextBlock()
examines. The next read then continued with the remaining block
references of that stale record, before decoding the record at the new
read position.

Since this commit, the buildfarm is unstable on various branches for
various hosts, for branches up to 17. Two examples:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=longfin&dt=2026-11-20%2005%3A13%3A08
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=indri&dt=2026-11-20%2005%3A00%3A27

And here is your failure, from the logs of the standby host in test
061:
2026-11-20 00:07:37.476 EST [33350:1] LOG: invalid value for
parameter "maintenance_io_concurrency": 1
2026-11-20 00:07:37.476 EST [33350:2] DETAIL:
"maintenance_io_concurrency" must be set to 0 on platforms that lack
posix_fadvise().
--

Yes, this was discussed in [1]/messages/by-id/CAHGQGwHQkzvPCd9_RdsBM_Y7iNkzDQAMm8CBgzkVAxcSp60Xow@mail.gmail.com. I just pushed a fix, so that the test is
skipped on platforms without posix_fadvise (on 15-17).

thanks

[1]: /messages/by-id/CAHGQGwHQkzvPCd9_RdsBM_Y7iNkzDQAMm8CBgzkVAxcSp60Xow@mail.gmail.com
/messages/by-id/CAHGQGwHQkzvPCd9_RdsBM_Y7iNkzDQAMm8CBgzkVAxcSp60Xow@mail.gmail.com

--
Tomas Vondra

#4Michael Paquier
michael@paquier.xyz
In reply to: Tomas Vondra (#3)
Re: pgsql: Forget the prefetcher's current record in XLogPrefetcherBeginRea

On Fri, Nov 20, 2026 at 12:17:32PM +0100, Tomas Vondra wrote:

Yes, this was discussed in [1]. I just pushed a fix, so that the test is
skipped on platforms without posix_fadvise (on 15-17).

Thanks for the fix.
--
Michael