Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

Started by Greg Burd24 days ago34 messageshackers
Jump to latest
#1Greg Burd
greg@burd.me

Hi again,

This is a follow-up to the hash-index WAL CRC thread [1]/messages/by-id/055bb33c-bccc-bc1d-c2f8-8598534448ac@gmail.com. Michael asked
that the larger hash rework not ride on that thread, since it is a
refactor that needs its own review attention rather than being buried
under a bug report [2]/messages/by-id/auKvDKa3umaD7Hwi@paquier.xyz:

"Could you start a new thread about all that, with all the authors of
this original code in CC at least? Hiding such a large refactoring on
a thread for the CRC failure would not help in terms of attracting the
attention this requires in terms of reviews."

So here it is, on its own thread, with the hash WAL-logging authors in
CC.

Background:

The CRC failure in [1]/messages/by-id/055bb33c-bccc-bc1d-c2f8-8598534448ac@gmail.com was that, under wal_consistency_checking='hash',
a standby could stop with "incorrect resource manager data checksum in
record". The cause: hashbucketcleanup(), _hash_squeezebucket() and
_hash_freeovflpage() register the bucket's primary page with
REGBUF_NO_CHANGE while holding only a pin on it (no content lock),
solely so that replay takes a cleanup lock on that page.
XLR_CHECK_CONSISTENCY then forces a full-page image of a page a
concurrent inserter is still free to modify, and the record CRC is
computed over shifting data.

The immediate CRC fix is committed and back-patched to v17 as
27caf4ca5ae5 ("Do not take a full-page image for REGBUF_NO_CHANGE
buffers"): XLogRecordAssemble() no longer forces an image for a
REGBUF_NO_CHANGE buffer, and verifyBackupPageConsistency() skips a
registered block that has no image.

That stops the bleeding but leaves the real problem in place: hash
registers a buffer it does not hold locked or dirty, which breaks the
invariant every other caller of XLogRegisterBuffer() keeps. The plan
proposed (agreed on?) in [1]/messages/by-id/055bb33c-bccc-bc1d-c2f8-8598534448ac@gmail.com was three steps:

1. Avoid the CRC problem down to v17, including HEAD. [done: 27caf4ca5ae5]
2. Fix hash so it stops breaking the rule. [0001 here]
3. Remove REGBUF_NO_CHANGE and restore the asserts. [0002 here]

This thread is steps 2 and 3.

0001: Don't register pin-only primary bucket pages in hash WAL records

Replay of XLOG_HASH_DELETE, XLOG_HASH_MOVE_PAGE_CONTENTS and
XLOG_HASH_SQUEEZE_PAGE must take a cleanup lock on the primary bucket
page so that a hot-standby scan is not in progress in the bucket, and
cannot start in it, while tuples are removed or moved. Today that lock
is obtained by registering the page (pin-only) with REGBUF_NO_CHANGE.
Redo never reads data from that block; the registration exists only for
the lock.

0001 instead stores the primary bucket page's block number in the main
data of these three records (a new bucket_blkno field) and has replay
read that block and call LockBufferForCleanup() on it directly, at the
same point and in the same order as today. The page is registered only
when the record actually changes it. For the squeeze record this also
removes the pin-only registration of the write page (the case where the
write page is the primary bucket page and nothing is moved onto it).

After 0001, hash registers only buffers it holds exclusively locked and
dirty, and there are no REGBUF_NO_CHANGE users left anywhere in the
tree.

Reading an unregistered block in redo has precedent: nbtree's "pin scan"
did exactly that (XLogReadBufferExtended + LockBufferForCleanup) until
9f83468b353 removed it as no longer needed, not as unsafe. Two
consequences, both benign:

- Recovery prefetch no longer sees the bucket page for these records,
so redo may do one synchronous read of it when it is not already in
shared buffers.
- Block-change consumers (pg_rewind, the WAL summarizer) stop counting
the bucket page as changed by these records. It never was changed,
so this is a correction, not a loss.

The three record formats change, so 0001 bumps XLOG_PAGE_MAGIC; master
only.

Why not just content-lock the primary page, as Jeff asked in [1]/messages/by-id/055bb33c-bccc-bc1d-c2f8-8598534448ac@gmail.com?

I looked for a safe placement and it deadlocks where the buffer is
registered. In _hash_freeovflpage() that point is after the metapage is
locked exclusive. A scan of a bucket still being populated by an
incomplete split holds that bucket's primary page share-locked and waits
for the metapage in _hash_get_oldblock_from_newbucket(); VACUUM holds
the metapage and waits for the primary page. Both are buffer content
locks, so the deadlock detector does not see the cycle and a cancel does
not break it. I reproduced it deterministically with an injection point
(3 of 3): pg_cancel_backend() on the VACUUM does nothing and every
insert queues behind the metapage. Taking the bucket lock before the
metapage is read avoids the deadlock, but that variant still costs a
full-page image of the bucket page per checkpoint and holds inserts and
scans on the bucket across the bitmap/metapage work. 0001 as proposed
has neither cost and takes no new lock on the primary page.

0002: Remove REGBUF_NO_CHANGE

With 0001 there are no users of the flag left. 0002 removes it,
restores the unconditional assert in XLogRegisterBuffer() that every
registered buffer is content-locked and dirty, and reverts the two
defensive measures from 27caf4ca5ae5 on master: the XLogRecordAssemble()
special case and the verifyBackupPageConsistency() skip. 27caf4ca5ae5
stays in the back branches, where this hash rework is not applied.

Testing (16-vCPU Debian, cassert, master incl. 27caf4ca5ae5)

- meson test regress, isolation, and recovery (including
recovery/027_stream_regress with
PG_TEST_EXTRA=wal_consistency_checking): all pass with 0001+0002
applied.

- CRC reproduction with 0001+0002, i.e. after 0002 has removed the
step-1 escape and re-added the asserts: primary + streaming standby,
wal_consistency_checking='hash', 8 sessions inserting and deleting on
a single-bucket index with aggressive autovacuum. The standby caught
up with no "incorrect resource manager data checksum", no inconsistent
page, and no assertion failure, 3 of 3. This shows the fix comes from
the hash rework itself, not from the step-1 safety net.

(Earlier, more detailed A/B/C/D runs on the older posting are in [3]/messages/by-id/auHpsdSdOXspzuqW@floki,
including a hot-standby cursor control showing the cleanup lock still
serializes replay against an in-progress scan.)

Open item from review:

Amit suggested [4]/messages/by-id/CAA4eK1+JpYJnUWwrSO7=gs=P3qHkYTrV+gMMPKY-ERv3pHdt-Q@mail.gmail.com registering bucket_blkno only for the overflow-page
cases, since for a primary-bucket-page record the block is already
known, saving 4 bytes on the common records. I have not made that
change yet; I can in the next version if we agree it is worth the extra
branch in each redo path.

Vignesh contributed a CRC injection-point test on the old thread that
I'd like to review and see how that might fit in and help.

best.

-greg

[1]: /messages/by-id/055bb33c-bccc-bc1d-c2f8-8598534448ac@gmail.com
[2]: /messages/by-id/auKvDKa3umaD7Hwi@paquier.xyz
[3]: /messages/by-id/auHpsdSdOXspzuqW@floki
[4]: /messages/by-id/CAA4eK1+JpYJnUWwrSO7=gs=P3qHkYTrV+gMMPKY-ERv3pHdt-Q@mail.gmail.com

Attachments:

t254446_1
v2-0002-Remove-REGBUF_NO_CHANGE.patchtext/plain; charset=us-ascii; name=v2-0002-Remove-REGBUF_NO_CHANGE.patchDownload+8-36
v2-0001-Don-t-register-pin-only-primary-bucket-pages-in-h.patchtext/plain; charset=us-ascii; name=v2-0001-Don-t-register-pin-only-primary-bucket-pages-in-h.patchDownload+93-78
#2vignesh C
vignesh21@gmail.com
In reply to: Greg Burd (#1)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

On Thu, 29 Oct 2026 at 21:42, Greg Burd <greg@burd.me> wrote:

Open item from review:

Amit suggested [4] registering bucket_blkno only for the overflow-page
cases, since for a primary-bucket-page record the block is already
known, saving 4 bytes on the common records. I have not made that
change yet; I can in the next version if we agree it is worth the extra
branch in each redo path.

Yes, that would be better. The
v3-0002-Append-the-hash-primary-bucket-block-number-to-WA.patch patch
has the changes for the same. The v3-0001 is the same as v2-0001
posted by Greg. Please merge it if you feel the changes are ok.
I did not include the v2-0002-Remove-REGBUF_NO_CHANGE patch as we can
get to it once we bring the first patch into committable shape.

Regards,
Vignesh

Attachments:

t254446_2
v3-0002-Append-the-hash-primary-bucket-block-number-to-WA.patchapplication/octet-stream; name=v3-0002-Append-the-hash-primary-bucket-block-number-to-WA.patchDownload+43-32
v3-0001-Don-t-register-pin-only-primary-bucket-pages-in-h.patchapplication/octet-stream; name=v3-0001-Don-t-register-pin-only-primary-bucket-pages-in-h.patchDownload+93-78
#3vignesh C
vignesh21@gmail.com
In reply to: Greg Burd (#1)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

On Thu, 29 Oct 2026 at 21:42, Greg Burd <greg@burd.me> wrote:

Hi again,

This is a follow-up to the hash-index WAL CRC thread [1]. Michael asked
that the larger hash rework not ride on that thread, since it is a
refactor that needs its own review attention rather than being buried
under a bug report [2]:

"Could you start a new thread about all that, with all the authors of
this original code in CC at least? Hiding such a large refactoring on
a thread for the CRC failure would not help in terms of attracting the
attention this requires in terms of reviews."

So here it is, on its own thread, with the hash WAL-logging authors in
CC.

Background:

The CRC failure in [1] was that, under wal_consistency_checking='hash',
a standby could stop with "incorrect resource manager data checksum in
record". The cause: hashbucketcleanup(), _hash_squeezebucket() and
_hash_freeovflpage() register the bucket's primary page with
REGBUF_NO_CHANGE while holding only a pin on it (no content lock),
solely so that replay takes a cleanup lock on that page.
XLR_CHECK_CONSISTENCY then forces a full-page image of a page a
concurrent inserter is still free to modify, and the record CRC is
computed over shifting data.

The immediate CRC fix is committed and back-patched to v17 as
27caf4ca5ae5 ("Do not take a full-page image for REGBUF_NO_CHANGE
buffers"): XLogRecordAssemble() no longer forces an image for a
REGBUF_NO_CHANGE buffer, and verifyBackupPageConsistency() skips a
registered block that has no image.

That stops the bleeding but leaves the real problem in place: hash
registers a buffer it does not hold locked or dirty, which breaks the
invariant every other caller of XLogRegisterBuffer() keeps. The plan
proposed (agreed on?) in [1] was three steps:

1. Avoid the CRC problem down to v17, including HEAD. [done: 27caf4ca5ae5]
2. Fix hash so it stops breaking the rule. [0001 here]
3. Remove REGBUF_NO_CHANGE and restore the asserts. [0002 here]

This thread is steps 2 and 3.

0001: Don't register pin-only primary bucket pages in hash WAL records

Replay of XLOG_HASH_DELETE, XLOG_HASH_MOVE_PAGE_CONTENTS and
XLOG_HASH_SQUEEZE_PAGE must take a cleanup lock on the primary bucket
page so that a hot-standby scan is not in progress in the bucket, and
cannot start in it, while tuples are removed or moved. Today that lock
is obtained by registering the page (pin-only) with REGBUF_NO_CHANGE.
Redo never reads data from that block; the registration exists only for
the lock.

The patches change the WAL records, and XLOG_PAGE_MAGIC cannot be
bumped in a minor release, so I looked at what happens if they are
backpatched. I built the base REL_18_STABLE, REL_18 + 0001, + 0002,
and + 0003 (attached, adapted for REL_18: the squeeze code there lacks
the locals present on master, so 0001 computes mod_wbuf itself) and
ran them against each other with streaming replication and crash
recovery, with wal_consistency_checking = 'hash' and VACUUM of hash
indexes in which every shape of the three records occurs.

Results (primary -> standby; all rows and index/heap scans checked):
p3 -> base works
base -> p3 works
p1 -> base fails: failed to locate backup block with ID 0
p2 -> base fails: same
base -> p1 fails: replay reads bucket_blkno that old
records do not have
base -> p2 fails: same

So 0001 and 0002 on their own are not safe in a minor release, in
either direction: a) a new primary writes records without block 0,
which an old standby cannot replay b) a new standby, or crash recovery
after upgrade, replays records written by an old server, which have
block 0 and no appended block number.

I'm not sure if we should back patch this patch. If we plan to back
patch, here is a patch for the same. The 0001 and 0002 are the same
patches as the v3 version posted in previous mail (rebased for PG18)
and 0003 has the backward compatible patch to fix them.

Thoughts?

Regards,
Vignesh

Attachments:

v3_REL18-0002-Append-the-hash-primary-bucket-block-number.patchapplication/octet-stream; name=v3_REL18-0002-Append-the-hash-primary-bucket-block-number.patchDownload+43-32
v3_REL18-0001-Don-t-register-pin-only-primary-bucket-page.patchapplication/octet-stream; name=v3_REL18-0001-Don-t-register-pin-only-primary-bucket-page.patchDownload+94-85
v3_REL18-0003-Hash-keep-the-WAL-of-the-previous-changes-r.patchapplication/octet-stream; name=v3_REL18-0003-Hash-keep-the-WAL-of-the-previous-changes-r.patchDownload+111-26
#4Greg Burd
greg@burd.me
In reply to: vignesh C (#3)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

On Fri, 30 Oct 2026 17:18:34 +0530, vignesh C <vignesh21@gmail.com> wrote:

On Thu, 29 Oct 2026 at 21:42, Greg Burd <greg@burd.me> wrote:

Hi again,

This is a follow-up to the hash-index WAL CRC thread [1]. Michael asked
that the larger hash rework not ride on that thread, since it is a
refactor that needs its own review attention rather than being buried
under a bug report [2]:

"Could you start a new thread about all that, with all the authors of
this original code in CC at least? Hiding such a large refactoring on
a thread for the CRC failure would not help in terms of attracting the
attention this requires in terms of reviews."

So here it is, on its own thread, with the hash WAL-logging authors in
CC.

Background:

The CRC failure in [1] was that, under wal_consistency_checking='hash',
a standby could stop with "incorrect resource manager data checksum in
record". The cause: hashbucketcleanup(), _hash_squeezebucket() and
_hash_freeovflpage() register the bucket's primary page with
REGBUF_NO_CHANGE while holding only a pin on it (no content lock),
solely so that replay takes a cleanup lock on that page.
XLR_CHECK_CONSISTENCY then forces a full-page image of a page a
concurrent inserter is still free to modify, and the record CRC is
computed over shifting data.

The immediate CRC fix is committed and back-patched to v17 as
27caf4ca5ae5 ("Do not take a full-page image for REGBUF_NO_CHANGE
buffers"): XLogRecordAssemble() no longer forces an image for a
REGBUF_NO_CHANGE buffer, and verifyBackupPageConsistency() skips a
registered block that has no image.

That stops the bleeding but leaves the real problem in place: hash
registers a buffer it does not hold locked or dirty, which breaks the
invariant every other caller of XLogRegisterBuffer() keeps. The plan
proposed (agreed on?) in [1] was three steps:

1. Avoid the CRC problem down to v17, including HEAD. [done: 27caf4ca5ae5]
2. Fix hash so it stops breaking the rule. [0001 here]
3. Remove REGBUF_NO_CHANGE and restore the asserts. [0002 here]

This thread is steps 2 and 3.

0001: Don't register pin-only primary bucket pages in hash WAL records

Replay of XLOG_HASH_DELETE, XLOG_HASH_MOVE_PAGE_CONTENTS and
XLOG_HASH_SQUEEZE_PAGE must take a cleanup lock on the primary bucket
page so that a hot-standby scan is not in progress in the bucket, and
cannot start in it, while tuples are removed or moved. Today that lock
is obtained by registering the page (pin-only) with REGBUF_NO_CHANGE.
Redo never reads data from that block; the registration exists only for
the lock.

Hi Vignesh,

Thanks for picking this up and for the two things you added: splitting
the four-byte saving out as its own patch, and the back-branch analysis.

On the structure: I've taken your v3-0001 (same as my v2-0001) and your
v3-0002 (append the block number only when the primary page is not
registered) as the base, and built the rest of the series on top. The
attached v3 is:

0001 Don't register pin-only primary bucket pages in hash WAL records
(unchanged from your v3-0001 / my v2-0001)
0002 Append the hash primary bucket block number to WAL records only
when needed (your v3-0002, Amit's suggestion)
0003 Add a recovery test for the hash WAL pin-only bucket page fix
0004 Remove REGBUF_NO_CHANGE

0002 keeps the records the same size as today in the common case, so I
think it belongs in the series rather than as a follow-up; thanks for
writing it.

0003 is the recovery test. It turns the CRC window into a deterministic
one with a general, test-only injection point in XLogRecordAssemble(),
right after the record CRC is computed and before the record is copied
into the WAL buffers. The injection point's argument names the record
as "rmid/info/mainlen", so the test waits on exactly the overflow-page
XLOG_HASH_DELETE (whose main data carries the appended bucket block
number, so it is distinguishable from the primary-page delete), lets a
concurrent INSERT change the pin-only primary bucket page, then crashes
and checks that pg_waldump and crash recovery both accept the WAL. This
is your test from the CRC thread, carried forward: the record selector
moved from the REGBUF_NO_CHANGE buffer count (which the rework zeroes)
to the main-data length so it still picks the right record after the
rework. I kept you as a co-author.

0004 removes REGBUF_NO_CHANGE and restores the unconditional assert in
XLogRegisterBuffer(), and reverts the two defensive measures from
27caf4ca5ae5 on master.

The patches change the WAL records, and XLOG_PAGE_MAGIC cannot be
bumped in a minor release, so I looked at what happens if they are
backpatched. I built the base REL_18_STABLE, REL_18 + 0001, + 0002,
and + 0003 (attached, adapted for REL_18: the squeeze code there lacks
the locals present on master, so 0001 computes mod_wbuf itself) and
ran them against each other with streaming replication and crash
recovery, with wal_consistency_checking = 'hash' and VACUUM of hash
indexes in which every shape of the three records occurs.

On your back-branch question: I don't think we should back-patch this.
27caf4ca5ae5 already fixes the actual CRC failure down to v17, so the
back branches are correct as they stand. The rework here is a cleanup
that changes three WAL record formats and bumps XLOG_PAGE_MAGIC, which,
as your p1/p2-vs-base runs show, is not compatible across a minor
release in either direction. That is exactly the kind of change we keep
to master. So I'd leave the back branches on 27caf4ca5ae5 and not carry
your v3_REL18-0003 compatibility shim; it is good to have proven the
incompatibility, but the conclusion is "master only", which 0001 already
says in its XLOG_PAGE_MAGIC bump.

One fix went into 0002 after a first CI round: pulling the primary
bucket block number out of the record struct into its own data segment
needs one more slot than XLogEnsureRecordSpace() reserved in
_hash_freeovflpage() and the move-page-contents path, so a 32-bit build
hit "too much WAL data" during a hash VACUUM that packs more tuples per
WAL record than a 64-bit build does. 0002 now reserves the extra
segment.

Patches 0001-0004 attached, against current master.

best.

-greg

Attachments:

t254446_4
v3-0002-Append-the-hash-primary-bucket-block-number-to-WA.patchtext/plain; charset=us-ascii; name=v3-0002-Append-the-hash-primary-bucket-block-number-to-WA.patchDownload+53-36
v3-0003-Add-a-recovery-test-for-the-hash-WAL-pin-only-buc.patchtext/plain; charset=us-ascii; name=v3-0003-Add-a-recovery-test-for-the-hash-WAL-pin-only-buc.patchDownload+199-1
v3-0004-Remove-REGBUF_NO_CHANGE.patchtext/plain; charset=us-ascii; name=v3-0004-Remove-REGBUF_NO_CHANGE.patchDownload+8-36
v3-0001-Don-t-register-pin-only-primary-bucket-pages-in-h.patchtext/plain; charset=us-ascii; name=v3-0001-Don-t-register-pin-only-primary-bucket-pages-in-h.patchDownload+93-78
#5Amit Kapila
amit.kapila16@gmail.com
In reply to: Greg Burd (#4)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

On Fri, Oct 30, 2026 at 10:51 PM Greg Burd <greg@burd.me> wrote:

On Fri, 30 Oct 2026 17:18:34 +0530, vignesh C <vignesh21@gmail.com> wrote:

The patches change the WAL records, and XLOG_PAGE_MAGIC cannot be
bumped in a minor release, so I looked at what happens if they are
backpatched. I built the base REL_18_STABLE, REL_18 + 0001, + 0002,
and + 0003 (attached, adapted for REL_18: the squeeze code there lacks
the locals present on master, so 0001 computes mod_wbuf itself) and
ran them against each other with streaming replication and crash
recovery, with wal_consistency_checking = 'hash' and VACUUM of hash
indexes in which every shape of the three records occurs.

On your back-branch question: I don't think we should back-patch this.

+1. I also think this should be a master-only change helping us to
remove the use of REGBUF_NO_CHANGE from hash indexes. Also, the below
change to make it work on back-branches appears too hacky to me.

+#if HASH_WAL_REGISTER_BUCKET_PAGE
+ /* for standbys that predate the block number, see hash_xlog.h */
+ if (!xlrec.is_primary_bucket_page)
+ {
+ uint8 flags = REGBUF_STANDARD | REGBUF_NO_IMAGE | REGBUF_NO_CHANGE;
+
+ XLogRegisterBuffer(0, bucket_buf, flags);
+ }
+#else

--
With Regards,
Amit Kapila.

#6Greg Burd
greg@burd.me
In reply to: Amit Kapila (#5)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

On Sat, 31 Oct 2026 17:02:34 +0530, Amit Kapila <amit.kapila16@gmail.com> wrote:

+1. I also think this should be a master-only change helping us to
remove the use of REGBUF_NO_CHANGE from hash indexes. Also, the below
change to make it work on back-branches appears too hacky to me.

Thanks. Then the back branches stay on 27caf4ca5ae, and this series is
master only. The #if HASH_WAL_REGISTER_BUCKET_PAGE shim you quoted was
in Vignesh's REL_18 compatibility patch, and it is not part of this
series.

v4 is attached, rebased on current master (5681f0cfa5a). The only
change from v3 is formatting: pgperltidy rewraps a few lines in the new
TAP test. The C code is the same as in v3.

0001 Don't register pin-only primary bucket pages in hash WAL records
0002 Append the hash primary bucket block number to WAL records only
when needed (Vignesh; Amit's suggestion)
0003 Add a recovery test for the hash WAL pin-only bucket page fix
(co-authored with Vignesh)
0004 Remove REGBUF_NO_CHANGE

One point I'd like a reviewer to look at in 0001:

redo now reads a block the record does not register. The block number
is appended to the record's main data only when the primary bucket page
is not one of the pages the record changes. Redo takes the relation
from a block that is always registered (block 1 for DELETE and
MOVE_PAGE_CONTENTS, block 2 for SQUEEZE_PAGE), and reads the page with
RBM_NORMAL, so a missing page goes through the same invalid-page
tracking as when the page was registered.

The page is not modified, so it needs no full-page image and no LSN
update; the old code registered it with REGBUF_NO_IMAGE, so redo already
worked from the on-disk copy. Two consumers of block references stop
seeing it. Recovery prefetch is one, so redo may do one synchronous
read of the bucket page per record when the page isn't cached. The
other is tools that collect changed blocks from WAL, such as pg_rewind
and the WAL summarizer, and losing it there is harmless because the page
is not changed. FSM redo reads unregistered blocks the same way, and
nbtree's pin scan did until 9f83468b353.

Testing:

- meson test setup, regress, isolation, recovery, pg_waldump and the
hash module, with PG_TEST_EXTRA=wal_consistency_checking, all four
patches applied: 68 of 68 suites pass on both builds, including
061_hash_vacuum_insert_crc and 027_stream_regress. With 0001 applied
alone, the same suites minus isolation also pass (64 of 64), so 0001
can be committed on its own.

- Streaming-standby CRC reproducer: 8 sessions inserting and deleting on
a single-bucket index under aggressive autovacuum,
wal_consistency_checking = 'hash'. As a control I ran master with
27caf4ca5ae reverted: the standby stops on "incorrect resource manager
data checksum". With v4, which also has 0004's REGBUF_NO_CHANGE
removal and restored asserts, the standby caught up with no errors.

- Cleanup lock on the standby: a cursor is parked in the bucket's
overflow chain, then the primary vacuums so that every DELETE,
MOVE_PAGE_CONTENTS and SQUEEZE_PAGE record has the primary bucket page
unmodified (is_primary F). On both builds replay waits on
BufferCleanup until the cursor finishes, and the cursor sees all 2250
rows, the same as unpatched master.

- I also reran the deadlock check from my first message on this thread
against current master. Taking the primary bucket page lock in
_hash_freeovflpage() after the metapage is locked exclusive still
hangs VACUUM and a scan of a bucket being populated by an incomplete
split. Taking it before the metapage read does not. 0001 takes
neither lock; this is only why the series doesn't lock the page.

I think 0001-0003 are ready for a committer. 0004 can go in with them
or right after.

best.

-greg

Attachments:

t254446_6
v4-0001-Don-t-register-pin-only-primary-bucket-pages-in-h.patchtext/plain; charset=us-ascii; name=v4-0001-Don-t-register-pin-only-primary-bucket-pages-in-h.patchDownload+93-78
v4-0004-Remove-REGBUF_NO_CHANGE.patchtext/plain; charset=us-ascii; name=v4-0004-Remove-REGBUF_NO_CHANGE.patchDownload+8-36
v4-0003-Add-a-recovery-test-for-the-hash-WAL-pin-only-buc.patchtext/plain; charset=us-ascii; name=v4-0003-Add-a-recovery-test-for-the-hash-WAL-pin-only-buc.patchDownload+204-1
v4-0002-Append-the-hash-primary-bucket-block-number-to-WA.patchtext/plain; charset=us-ascii; name=v4-0002-Append-the-hash-primary-bucket-block-number-to-WA.patchDownload+53-36
#7Amit Kapila
amit.kapila16@gmail.com
In reply to: Greg Burd (#6)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

On Tue, Nov 3, 2026 at 9:41 PM Greg Burd <greg@burd.me> wrote:

On Sat, 31 Oct 2026 17:02:34 +0530, Amit Kapila <amit.kapila16@gmail.com> wrote:

+1. I also think this should be a master-only change helping us to
remove the use of REGBUF_NO_CHANGE from hash indexes. Also, the below
change to make it work on back-branches appears too hacky to me.

Thanks. Then the back branches stay on 27caf4ca5ae, and this series is
master only. The #if HASH_WAL_REGISTER_BUCKET_PAGE shim you quoted was
in Vignesh's REL_18 compatibility patch, and it is not part of this
series.

v4 is attached, rebased on current master (5681f0cfa5a). The only
change from v3 is formatting: pgperltidy rewraps a few lines in the new
TAP test. The C code is the same as in v3.

0001 Don't register pin-only primary bucket pages in hash WAL records
0002 Append the hash primary bucket block number to WAL records only
when needed (Vignesh; Amit's suggestion)

I think we can merge 0001 and 0002 now. I don't see the need for
additional four bytes in common cases for each WAL record we modify as
part of this patch.

One additional comment:
*
- * Backup Blk 0: primary bucket page
  * Backup Blk 1: page from which tuples are deleted
+ *
+ * If block 1 is not the primary bucket page, replay takes a cleanup lock on
+ * bucket_blkno, which is not registered.
  */
 typedef struct xl_hash_delete
 {
+ BlockNumber bucket_blkno; /* primary bucket page */

After this change, shouldn't Backup Blk 1 be numbered as 0? Because
now 0 doesn't have anything. The same is true for other records that
are modified.

--
With Regards,
Amit Kapila.

#8shihao zhong
zhong950419@gmail.com
In reply to: Amit Kapila (#7)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

Hi Greg,

One point I'd like a reviewer to look at in 0001:

redo now reads a block the record does not register.

Could that be a torn page?

The page is not modified, so it needs no full-page image and no LSN
update; the old code registered it with REGBUF_NO_IMAGE, so redo already
worked from the on-disk copy.

I think if we have a torn bucket page and an XLOG_HASH_DELETE for one
of its overflow pages before the FPI, we may hit an issue?

We do not register an FPI for it in hashbucketcleanup().

if (!xlrec.is_primary_bucket_page)
{
uint8 flags = REGBUF_STANDARD | REGBUF_NO_IMAGE |
REGBUF_NO_CHANGE;

XLogRegisterBuffer(0, bucket_buf, flags);
}

XLogRegisterBuffer(1, buf, REGBUF_STANDARD);

Then hash_xlog_delete() reads it from disk in the else branch, and
that read fails.

if (xldata->is_primary_bucket_page)
action = XLogReadBufferForRedoExtended(record, 1, RBM_NORMAL,
true, &deletebuf);
else
{
(void) XLogReadBufferForRedoExtended(record, 0, RBM_NORMAL,
true, &bucketbuf);

action = XLogReadBufferForRedo(record, 1, &deletebuf);
}

I don't think this is caused by your patch. This looks like an
existing bug in master.

I am afraid that adding an FPI will not work either, because VACUUM
only holds a pin on the page and not an exclusive lock. But getting an
exclusive lock may hit the deadlock you mentioned. Maybe the right way
is to fix it on the replay side?

Thanks,
Shihao

#9vignesh C
vignesh21@gmail.com
In reply to: shihao zhong (#8)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

On Wed, 11 Nov 2026 at 08:33, shihao zhong <zhong950419@gmail.com> wrote:

Hi Greg,

One point I'd like a reviewer to look at in 0001:

redo now reads a block the record does not register.

Could that be a torn page?

The page is not modified, so it needs no full-page image and no LSN
update; the old code registered it with REGBUF_NO_IMAGE, so redo already
worked from the on-disk copy.

I think if we have a torn bucket page and an XLOG_HASH_DELETE for one
of its overflow pages before the FPI, we may hit an issue?

We do not register an FPI for it in hashbucketcleanup().

if (!xlrec.is_primary_bucket_page)
{
uint8 flags = REGBUF_STANDARD | REGBUF_NO_IMAGE |
REGBUF_NO_CHANGE;

XLogRegisterBuffer(0, bucket_buf, flags);
}

XLogRegisterBuffer(1, buf, REGBUF_STANDARD);

Then hash_xlog_delete() reads it from disk in the else branch, and
that read fails.

if (xldata->is_primary_bucket_page)
action = XLogReadBufferForRedoExtended(record, 1, RBM_NORMAL,
true, &deletebuf);
else
{
(void) XLogReadBufferForRedoExtended(record, 0, RBM_NORMAL,
true, &bucketbuf);

action = XLogReadBufferForRedo(record, 1, &deletebuf);
}

I don't think this is caused by your patch. This looks like an
existing bug in master.

Thanks for the report. I was able to reproduce this on HEAD.
The attached test
(src/test/recovery/t/068a_hash_torn_primary_page_delete_first.pl) does
the following, with data checksums enabled: a) Builds a hash index
with all keys in one bucket, consisting of a primary page P and an
overflow page, and takes a checkpoint. b) Runs VACUUM to remove dead
entries from the overflow page only. This writes an XLOG_HASH_DELETE
record for the overflow page without modifying P. c) Runs a second
VACUUM to remove dead entries from P. This is the first modification
to P since the checkpoint, so the corresponding WAL record includes a
full-page image (FPI) of P. d) Crashes the server, overwrites the
second half of P in the data file with random bytes to simulate a torn
write, and restarts the server.

Crash recovery then fails while replaying the first record (the DELETE
for the overflow page), before reaching the record containing the FPI
of P:
FATAL: invalid page in block 7 of relation "base/5/16387"
CONTEXT: WAL redo at 0/017745E0 for Hash/DELETE: ... is_primary F;
blkref #1: ... blk 10 FPW

During redo, we read P only to acquire the cleanup lock, using
RBM_NORMAL. The read therefore fails checksum verification. The test
fails in the same way without the patch, where P is accessed through
block 0, and with the patch, where it is accessed through
bucket_blkno.
Since we only need P to acquire the cleanup lock and don't actually
need its contents, I think we should read it using RBM_ZERO_ON_ERROR
in hash_xlog_lock_bucket(). This would zero the buffer instead of
raising an error when a torn page is encountered, allowing the FPI
later in the WAL to restore the correct contents.

Do you think RBM_ZERO_ON_ERROR is the right approach here, or is there
a better solution to fix this?

Regards,
Vignesh

Attachments:

068a_hash_torn_primary_page_delete_first.pltext/x-perl-script; charset=US-ASCII; name=068a_hash_torn_primary_page_delete_first.plDownload
#10Amit Kapila
amit.kapila16@gmail.com
In reply to: vignesh C (#9)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

On Wed, Nov 11, 2026 at 2:59 PM vignesh C <vignesh21@gmail.com> wrote:

On Wed, 11 Nov 2026 at 08:33, shihao zhong <zhong950419@gmail.com> wrote:

Thanks for the report. I was able to reproduce this on HEAD.
The attached test
(src/test/recovery/t/068a_hash_torn_primary_page_delete_first.pl) does
the following, with data checksums enabled: a) Builds a hash index
with all keys in one bucket, consisting of a primary page P and an
overflow page, and takes a checkpoint. b) Runs VACUUM to remove dead
entries from the overflow page only. This writes an XLOG_HASH_DELETE
record for the overflow page without modifying P. c) Runs a second
VACUUM to remove dead entries from P. This is the first modification
to P since the checkpoint, so the corresponding WAL record includes a
full-page image (FPI) of P. d) Crashes the server, overwrites the
second half of P in the data file with random bytes to simulate a torn
write, and restarts the server.

Crash recovery then fails while replaying the first record (the DELETE
for the overflow page), before reaching the record containing the FPI
of P:
FATAL: invalid page in block 7 of relation "base/5/16387"
CONTEXT: WAL redo at 0/017745E0 for Hash/DELETE: ... is_primary F;
blkref #1: ... blk 10 FPW

During redo, we read P only to acquire the cleanup lock, using
RBM_NORMAL. The read therefore fails checksum verification. The test
fails in the same way without the patch, where P is accessed through
block 0, and with the patch, where it is accessed through
bucket_blkno.
Since we only need P to acquire the cleanup lock and don't actually
need its contents, I think we should read it using RBM_ZERO_ON_ERROR
in hash_xlog_lock_bucket(). This would zero the buffer instead of
raising an error when a torn page is encountered, allowing the FPI
later in the WAL to restore the correct contents.

Do you think RBM_ZERO_ON_ERROR is the right approach here, or is there
a better solution to fix this?

The code comments related to RBM_ZERO_ON_ERROR states following:
"RBM_ZERO_ON_ERROR is like the normal mode, but if the page header is
not
valid, the page is zeroed instead of throwing an error. This is
intended for non-critical data, where the caller is prepared to repair
errors.". And, I see RBM_ZERO_ON_ERROR mode is only used for VM and
FSM pages which don't contain user-data. So, it doesn't sound like a
good idea to use RBM_ZERO_ON_ERROR.

Can we explore guarding the read/lock of bucket_page during replay by
HotStandbyActiveInReplay()? IIRC, the cleanup lock is required during
replay to prevent concurrent scans from reading incorrect data. So,
once the consistency is reached, we should be okay to read/lock the
primary page.

I feel we should also explore the idea of locking the bucket page even
when we are registering without it being changed to avoid this
problem. I'll also think about it.

--
With Regards,
Amit Kapila.

#11Greg Burd
greg@burd.me
In reply to: Amit Kapila (#10)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

Hello,

Thanks Amit for the review, and thanks Shihao for finding the torn-page
case.

Per Amit's suggestion I merged the old 0001 and 0002 into a single
patch. The primary bucket block number is now appended to the record
only when the primary bucket page is not one of the pages the record
changes, so there is no common-case size increase at any point in the
series. The structs never carry the block number.

I also renumbered the registered blocks as Amit noted. Since the
primary bucket page is no longer registered as block 0 in DELETE,
MOVE_PAGE_CONTENTS or SQUEEZE_PAGE, I shifted the remaining blocks down
by one, so each record's registered blocks start at 0 again:

DELETE block 0: page tuples are deleted from
MOVE_PAGE_CONTENTS block 0: page tuples are moved to
block 1: page tuples are moved from
SQUEEZE_PAGE block 0: write page (if modified)
block 1: freed overflow page
block 2: page before the freed overflow page
block 3: page after the freed overflow page
block 4: bitmap page
block 5: meta page

The record descriptions in hash_xlog.h are updated to match, and the
recovery test still selects the overflow-page DELETE by its main-data
length, which does not change.

0001 Don't register pin-only primary bucket pages in hash WAL records
(now includes the append-only-when-needed change; Vignesh,
Amit's suggestion)
0002 Add a recovery test for the hash WAL pin-only bucket page fix
(co-authored with Vignesh)
0003 Remove REGBUF_NO_CHANGE

The case of the torn-page

I reproduced and traced Shihao's case and I agree it is a real,
pre-existing bug on master, not something this series introduces. The
sequence is: with data checksums on, a VACUUM writes an XLOG_HASH_DELETE
for an overflow page while holding only a pin on the primary bucket
page, so the primary page is not in that record. A later record carries
the first full-page image of the primary page.

If the primary page is torn on disk and recovery replays the DELETE
before that image, redo reads the primary page only to take the cleanup
lock, with RBM_NORMAL, and the read fails the checksum and raises
"invalid page", before the image that would repair it is reached.

On master today the same page is read through block 0 with the same
RBM_NORMAL, so the failure is identical with and without the series.

0001 just reaches the page through the appended block number instead of
block 0. So this is worth fixing on its own, and the fix is orthogonal
to whether we register the page or carry its block number.

On the approaches raised so far:

- RBM_ZERO_ON_ERROR (Vignesh): it does make the read tolerate the torn
page, and a later FPI repairs it. But as Amit noted it is documented
for non-critical forks (VM, FSM) where the caller repairs, and we'd be
pointing it at a main-fork index page, which reads wrong even if it
happens to work here. I share the hesitation.

- HotStandbyActiveInReplay() guard (Amit): the cleanup lock on the
primary page exists only to keep a hot-standby scan from running in
the bucket during replay. During crash recovery and before
consistency, no such scan can exist, so there is nothing to serialize
against and the read/lock is not needed then. That lines up with why
the lock is taken at all, and it avoids reading the page in exactly
the window where the torn copy has not yet been repaired. I think
this is the most promising direction.

- Lock the bucket page even when it is registered unchanged (Amit): I'll
look at this too, together with the deadlock I described earlier in
the thread when taking the primary-page lock in _hash_freeovflpage()
after the metapage is locked.

I'd rather not pick one and bake it into this series until we've
actually tested them against Shihao's and Vignesh's reproducers (and
confirmed the HotStandbyActiveInReplay path is safe on a standby that
reaches consistency mid-bucket). I'll do that and come back with a
separate patch for the torn-page fix, since it stands on its own against
master.

Does treating it as its own fix, rather than folding it into 0001, sound
right to you? Does anyone think this should be a separate patch against
master?

Patches 0001-0003 attached, against current master.

best.

-greg

Attachments:

t254446_11
v5-0003-Remove-REGBUF_NO_CHANGE.patchtext/plain; charset=us-ascii; name=v5-0003-Remove-REGBUF_NO_CHANGE.patchDownload+8-36
v5-0001-Don-t-register-pin-only-primary-bucket-pages-in-h.patchtext/plain; charset=us-ascii; name=v5-0001-Don-t-register-pin-only-primary-bucket-pages-in-h.patchDownload+149-116
v5-0002-Add-a-recovery-test-for-the-hash-WAL-pin-only-buc.patchtext/plain; charset=us-ascii; name=v5-0002-Add-a-recovery-test-for-the-hash-WAL-pin-only-buc.patchDownload+204-1
#12Greg Burd
greg@burd.me
In reply to: Amit Kapila (#10)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

Hello,

Following up on the torn-page problem Shihao found and Vignesh
reproduced, and on Amit's comments. I reproduced it, tested the fixes,
and have reworked the series so the torn-page fix comes first.

The case of the torn-page: The bug is on master today, independent of
the pin-only rework. I ran Vignesh's reproducer (adapted so it finds
the primary bucket page from a record that registers it, rather than
from the removed bucket_blkno field) against three trees:

- plain master: recovery fails replaying the overflow DELETE, "invalid
page in block 7", before the later full-page image of block 7.
- the pin-only series: same failure, reached through the appended
block number instead of block 0.
- master + the fix: recovery succeeds and the index returns the right
rows.

The hazard was spotted during the original hash WAL work. In 2017
Robert raised it on the write-ahead-logging thread [1]/messages/by-id/CAA4eK1LwZk0MaK0LncZz6hsqcHnamfJB47nA8xuwRp4CyLFKzw@mail.gmail.com:

"the page is registered and therefore an FPI will be taken if this is
the first reference to this page since the checkpoint, but
hash_xlog_delete won't restore the FPI. I think that exposes us to a
torn-page hazard. _hash_freeovflpage/hash_xlog_squeeze_page seems to
have the same disease, as does
_hash_squeezebucket/hash_xlog_move_page_contents."

Redo was then switched to XLogReadBufferForRedoExtended() so it restores
an image when the record has one, but these records register the primary
bucket page pin-only with REGBUF_NO_IMAGE, so there is nothing to
restore and the RBM_NORMAL read still reaches the torn page. So the
hazard stayed open; Shihao and Vignesh have now hit it.

On the fix: RBM_ZERO_ON_ERROR (Vignesh) does fix the reproducer, but as
Amit noted it is documented for non-critical forks where the caller
repairs, and this is a main-fork index page. Guarding the read with
InHotStandby (Amit's suggestion) also fixes it and is the right one:

- The read exists only to take the cleanup lock, which only matters to
serialize against a scan on a hot standby. When hot standby is not
active no such scan can exist, so neither the read nor the lock is
needed.
- That is exactly the window where the torn-page hazard lives. The
repairing image always precedes the point where hot standby becomes
active, so once InHotStandby is true any earlier image has already
been replayed and the page is no longer torn.
- It matches existing practice: nbtree and gist already gate their
redo cleanup-lock work on InHotStandby.

Since the bug is pre-existing, the fix is patch 0001: it stands on its
own against master, applies with or without the rework, and (the
back-branch question aside) is separate from the refactor. The rest of
the series then builds on it; in 0002 the same guard lives in the one
helper the refactor introduces, hash_xlog_lock_bucket().

0001 Fix hash index redo reading a torn primary bucket page
0002 Don't register pin-only primary bucket pages in hash WAL records
(merges the old 0001 and the four-byte optimization, and
renumbers the registered blocks per Amit's comment)
0003 Add a recovery test for the hash WAL pin-only bucket page fix
0004 Remove REGBUF_NO_CHANGE

For 0002 I folded the append-only-when-needed change into the main patch
so there is no common-case size increase at any point, and renumbered
the registered blocks so each record starts at 0 again now that the
primary bucket page is no longer block 0.

Testing (on current master): the new 068a recovery test fails on master
and on the pin-only series, and passes with 0001; the full recovery,
isolation and regress suites pass with PG_TEST_EXTRA=
wal_consistency_checking across the series.

best.

-greg

[1]: /messages/by-id/CAA4eK1LwZk0MaK0LncZz6hsqcHnamfJB47nA8xuwRp4CyLFKzw@mail.gmail.com

Attachments:

t254446_12
v6-0004-Remove-REGBUF_NO_CHANGE.patchtext/x-patch; charset=us-ascii; name=v6-0004-Remove-REGBUF_NO_CHANGE.patchDownload+8-36
v6-0002-Don-t-register-pin-only-primary-bucket-pages-in-h.patchtext/x-patch; charset=us-ascii; name=v6-0002-Don-t-register-pin-only-primary-bucket-pages-in-h.patchDownload+165-140
v6-0001-Fix-hash-index-redo-reading-a-torn-primary-bucket.patchtext/x-patch; charset=us-ascii; name=v6-0001-Fix-hash-index-redo-reading-a-torn-primary-bucket.patchDownload+176-10
v6-0003-Add-a-recovery-test-for-the-hash-WAL-pin-only-buc.patchtext/x-patch; charset=us-ascii; name=v6-0003-Add-a-recovery-test-for-the-hash-WAL-pin-only-buc.patchDownload+204-1
#13vignesh C
vignesh21@gmail.com
In reply to: Greg Burd (#12)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

On Thu, 12 Nov 2026 at 02:41, Greg Burd <greg@burd.me> wrote:

Hello,

Following up on the torn-page problem Shihao found and Vignesh
reproduced, and on Amit's comments. I reproduced it, tested the fixes,
and have reworked the series so the torn-page fix comes first.

The case of the torn-page: The bug is on master today, independent of
the pin-only rework. I ran Vignesh's reproducer (adapted so it finds
the primary bucket page from a record that registers it, rather than
from the removed bucket_blkno field) against three trees:

- plain master: recovery fails replaying the overflow DELETE, "invalid
page in block 7", before the later full-page image of block 7.
- the pin-only series: same failure, reached through the appended
block number instead of block 0.
- master + the fix: recovery succeeds and the index returns the right
rows.

The hazard was spotted during the original hash WAL work. In 2017
Robert raised it on the write-ahead-logging thread [1]:

"the page is registered and therefore an FPI will be taken if this is
the first reference to this page since the checkpoint, but
hash_xlog_delete won't restore the FPI. I think that exposes us to a
torn-page hazard. _hash_freeovflpage/hash_xlog_squeeze_page seems to
have the same disease, as does
_hash_squeezebucket/hash_xlog_move_page_contents."

Redo was then switched to XLogReadBufferForRedoExtended() so it restores
an image when the record has one, but these records register the primary
bucket page pin-only with REGBUF_NO_IMAGE, so there is nothing to
restore and the RBM_NORMAL read still reaches the torn page. So the
hazard stayed open; Shihao and Vignesh have now hit it.

On the fix: RBM_ZERO_ON_ERROR (Vignesh) does fix the reproducer, but as
Amit noted it is documented for non-critical forks where the caller
repairs, and this is a main-fork index page. Guarding the read with
InHotStandby (Amit's suggestion) also fixes it and is the right one:

- The read exists only to take the cleanup lock, which only matters to
serialize against a scan on a hot standby. When hot standby is not
active no such scan can exist, so neither the read nor the lock is
needed.
- That is exactly the window where the torn-page hazard lives. The
repairing image always precedes the point where hot standby becomes
active, so once InHotStandby is true any earlier image has already
been replayed and the page is no longer torn.
- It matches existing practice: nbtree and gist already gate their
redo cleanup-lock work on InHotStandby.

Since the bug is pre-existing, the fix is patch 0001: it stands on its
own against master, applies with or without the rework, and (the
back-branch question aside) is separate from the refactor.

The fix skips reading and acquiring the cleanup lock on the primary
bucket page unless 'InHotStandby' is true. This works for crash
recovery, and the first test (068a) passes. However, 'InHotStandby'
does not mean that hot standby is ready to accept queries. It becomes
true as soon as the first 'XLOG_RUNNING_XACTS' record is replayed,
which happens shortly after the redo pointer is reached, well before
recovery reaches consistency. With 'hot_standby = on', redo can
therefore still read the page during this window.

A base backup can contain a torn copy of the page if it is written
while the backup files are being copied. The full-page image for that
page is then present in WAL generated after the backup starts. The
DELETE record for the overflow page can precede that image, while the
standby reaches consistency only after the backup ends, by which time
the image should have been replayed.

The attached test (068b) reproduces this scenario: a) Takes a
low-level backup using 'pg_backup_start()'. b) While the backup is in
progress, runs VACUUM to generate a DELETE record for the overflow
page, followed by another VACUUM that modifies the primary page for
the first time since the backup's checkpoint. The latter record
therefore includes a full-page image of the primary page. c) Tears the
backup copy of the primary page. d) Starts a standby with 'hot_standby
= on' using that backup.

With the fix, the standby fails to start:
FATAL: invalid page in block 7 of relation "base/5/16387"
CONTEXT: WAL redo at 0/020099A0 for Hash/DELETE: ... is_primary F;
blkref #0: ... blk 7; blkref #1: ... blk 10 FPW

The attached patch (0001) uses 'HotStandbyActiveInReplay()' instead.
This becomes true only after recovery has reached consistency and the
postmaster has been notified that connections can be accepted. Since
no scans can be running before that point, there is no need to read or
lock the primary bucket page earlier. By then, any full-page image
needed to repair a torn page should also have been replayed.
How about changing InHotStandby to HotStandbyActiveInReplay to fix this issue?

Regards,
Vignesh

Attachments:

068b_hash_torn_primary_page_standby.pltext/x-perl-script; charset=US-ASCII; name=068b_hash_torn_primary_page_standby.plDownload
Use_HotStandbyActiveInReplay_to_skip_primary_bucket.patchapplication/octet-stream; name=Use_HotStandbyActiveInReplay_to_skip_primary_bucket.patchDownload+27-26
#14Hayato Kuroda (Fujitsu)
kuroda.hayato@fujitsu.com
In reply to: Greg Burd (#12)
RE: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

Hi,

I started to study the thread. Here are minor comments for 0002.
I understood the refactoring must be done master-only.

01.
```
+static Buffer
+hash_xlog_lock_bucket(XLogReaderState *record, uint8 block_id, Size rec_size)
```

I think the function name is too general. IIUC the function intends to lock a
primary page, but it's not described in the function name. How about renaming to
"hash_xlog_lock_primary_bucket" or similar?

02.
```
+ /* block 0 is absent if the primary bucket page is not modified */
```

I felt this comment can be more friendly like:

```
/*
* If the primary bucket page is modified, it is registered as block 0
* and read for REDO. Otherwise, acquire its cleanup lock separately.
*/
```

Best regards,
Hayato Kuroda
FUJITSU LIMITED

#15vignesh C
vignesh21@gmail.com
In reply to: Greg Burd (#12)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

On Thu, 12 Nov 2026 at 02:41, Greg Burd <greg@burd.me> wrote:

Since the bug is pre-existing, the fix is patch 0001: it stands on its
own against master, applies with or without the rework, and (the
back-branch question aside)

We will have to back patch till the PG15 version. I could reproduce
the issue in all the supported branches and the
HotStandbyActiveInReplay changes at [1]/messages/by-id/CALDaNm3ZudVo+NT6HKA+kPbmXmCuKv+Zw95Do5G5+EUE9GGn=g@mail.gmail.com fixes the issue.
One small improvement could be done to the patches,
hash_xlog_move_page_contents, hash_xlog_squeeze_page and
hash_xlog_delete all have the below same code, can we use a helper
static function for this:
/*
* The cleanup lock on the primary bucket page serializes replay
* against scans on a hot standby, which pin the page for their whole
* duration. Scans can run only once hot standby accepts queries, so
* until then we need not read or lock the page. Skipping the read
* also sidesteps a torn copy of the page whose repairing full-page
* image comes later in the WAL: that hazard exists only before
* consistency is reached, and queries are accepted only after that,
* so once HotStandbyActiveInReplay() is true any earlier image has
* already been replayed.
*/
if (HotStandbyActiveInReplay())
(void) XLogReadBufferForRedoExtended(record, 0, RBM_NORMAL, true, &bucketbuf)

@Gregory Burd Could you post the 0001 patches for the torn-page issue,
covering branches up to PG15? We can make the patch ready for commit.

[1]: /messages/by-id/CALDaNm3ZudVo+NT6HKA+kPbmXmCuKv+Zw95Do5G5+EUE9GGn=g@mail.gmail.com

Regards,
Vignesh

#16Greg Burd
greg@burd.me
In reply to: vignesh C (#13)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

Hello,

Thanks Vignesh for catching that, and the 068b test.
Also, thanks Hayato-san for the review and comments.

Agreed, InHotStandby becomes true as soon as the first
XLOG_RUNNING_XACTS is replayed, well before recovery reaches
consistency, so with hot_standby = on redo can still read the torn page
in that window. Thanks for the 068b reproducer.

v7 uses HotStandbyActiveInReplay() instead, as you suggested. That is
true only after recovery has reached consistency and the postmaster has
been told connections can start, so there is no scan to serialize
against before then, and by then any full-page image that repairs the
page has been replayed. I made HotStandbyActiveInReplay() extern for
this. 068b now passes, and so does 068a.

I folded your 068b into 0001 next to 068a, with one change: the primary
bucket page is found from blkref #0 or #1, since after 0002 the page is
registered as block 0 rather than 1 when it is the modified page, and
the test runs against the whole series as well as master.

I've applied both of Hayato-san's comments in 0002:
hash_xlog_lock_bucket() is renamed to hash_xlog_lock_primary_bucket(),
and the "block 0 is absent" comment now reads:

/*
* If the primary bucket page is modified, it is registered as block 0
* and read for redo. Otherwise, acquire its cleanup lock separately.
*/

v7:
0001 Fix hash index redo reading a torn primary bucket page
(now with 068a and 068b, and HotStandbyActiveInReplay)
0002 Don't register pin-only primary bucket pages in hash WAL records
(hash_xlog_lock_primary_bucket, carries the same guard)
0003 Add a recovery test for the hash WAL pin-only bucket page fix
0004 Remove REGBUF_NO_CHANGE

0001 stands on its own against master; the rest build on it.

best.

-greg

Attachments:

t254446_16
v7-0002-Don-t-register-pin-only-primary-bucket-pages-in-h.patchtext/x-patch; charset=us-ascii; name=v7-0002-Don-t-register-pin-only-primary-bucket-pages-in-h.patchDownload+166-139
v7-0003-Add-a-recovery-test-for-the-hash-WAL-pin-only-buc.patchtext/x-patch; charset=us-ascii; name=v7-0003-Add-a-recovery-test-for-the-hash-WAL-pin-only-buc.patchDownload+204-1
v7-0001-Fix-hash-index-redo-reading-a-torn-primary-bucket.patchtext/x-patch; charset=us-ascii; name=v7-0001-Fix-hash-index-redo-reading-a-torn-primary-bucket.patchDownload+351-12
v7-0004-Remove-REGBUF_NO_CHANGE.patchtext/x-patch; charset=us-ascii; name=v7-0004-Remove-REGBUF_NO_CHANGE.patchDownload+8-36
#17Joao Rito
joaovictor.rito@gmail.com
In reply to: Greg Burd (#16)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

Hi Greg,

Thank you for the v7 patch and the new recovery test.

I tested v7 on a local build. While running 061_hash_vacuum_insert_crc on
Linux (ext4, meson build), subtest 2 failed for me with:

pg_waldump: error: invalid WAL segment size in WAL file
"000000010000000000000002" (0 bytes)

Both LSNs are in segment ...0001, and segment ...0002 is a zero-filled
16 MB file. When --path is a directory, pg_waldump seems to read the
segment size from the first WAL file readdir() returns, so the result
depends on directory order (here 0002 came first).

Passing the segment file for $start_lsn (for example from pg_walfile_name)
worked for me by hand. Subtest 3 passed and I saw no CRC problem. I did
not apply the change to the test and rerun it under meson.

Thanks again!

Regards,
Joao Rito

#18Hayato Kuroda (Fujitsu)
kuroda.hayato@fujitsu.com
In reply to: Greg Burd (#16)
RE: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

Hi,

Thanks for updating the patch. Few more comments. Note that I did not review 0003,
because I was not sure it's intended to be pushed.

01.
```
+                                * If buf is an overflow page, the primary bucket page is not
+                                * changed and we hold only a pin on it, so it is not
+                                * registered.  Append its block number instead, and replay
+                                * takes a cleanup lock on it.  It is not needed otherwise
```

I think we should clarify why the primary bucket itself or block number should be
registered. Idea:

```
/*
* Replay must take a cleanup lock on the primary bucket page to
* serialize with scans on a hot standby. If buf is an overflow
* page, the unchanged primary page is held only by a pin, so append
* its block number instead of registering it. Otherwise block 0
* identifies the primary page.
*/
```

02.
```
+/*
+ * Take a cleanup lock on the primary bucket page of the bucket that a
+ * squeeze or delete record operates on, when that page is not one of the
+ * pages the record changes and so is not registered in it.  Its block number
+ * is stored in the main data of the record, right after the fixed-size struct
+ * of rec_size bytes.
+ *
+ * This serializes replay against scans on a hot standby, which pin the
+ * primary bucket page for their whole duration; see the comments in the
+ * callers.  The page is not modified, so nothing is done with it besides
+ * locking it.  block_id must be a block of the same index registered in
+ * the record.
+ */
+static Buffer
+hash_xlog_lock_primary_bucket(XLogReaderState *record, uint8 block_id, Size rec_size)
```

In contrast this might be too detail, the user of the function might not need
such an info. How about:

```
/*
* Take a cleanup lock on an unregistered primary bucket page, whose
* block number is stored in the record's main data. This serializes
* replay against scans on a hot standby. block_id must identify a
* registered block of the same index.
*/
```

03.
```
+                * Register only the pages this record changes.  The primary bucket
+                * page is registered only if it is the write page and is modified;
+                * otherwise we may hold only a pin on it, so append its block number
+                * and replay takes a cleanup lock on it.
```

Same as 01, we can clarify the reason why the primary bucket is needed. Idea:

```
/*
* Replay needs the primary bucket block number to acquire a cleanup
* lock. Register it as block 0 when it is modified; otherwise append
* its block number to the record.
*/
```

Best regards,
Hayato Kuroda
FUJITSU LIMITED

#19vignesh C
vignesh21@gmail.com
In reply to: Joao Rito (#17)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

On Thu, 12 Nov 2026 at 21:43, Joao Rito <joaovictor.rito@gmail.com> wrote:

Hi Greg,

Thank you for the v7 patch and the new recovery test.

I tested v7 on a local build. While running 061_hash_vacuum_insert_crc on
Linux (ext4, meson build), subtest 2 failed for me with:y

pg_waldump: error: invalid WAL segment size in WAL file
"000000010000000000000002" (0 bytes)

Both LSNs are in segment ...0001, and segment ...0002 is a zero-filled
16 MB file. When --path is a directory, pg_waldump seems to read the
segment size from the first WAL file readdir() returns, so the result
depends on directory order (here 0002 came first).

Passing the segment file for $start_lsn (for example from pg_walfile_name)
worked for me by hand. Subtest 3 passed and I saw no CRC problem. I did
not apply the change to the test and rerun it under meson.

Thanks for the comment, this is addressed in the v8 version patch attached.
Apart from that, the following changes were done:
1) prepared back branch patches for the 0001 patch.
2) The test 068 was renamed to the next available 061 test file.
3) Most of the test code was common in
hash_torn_primary_page_delete_first and
hash_torn_primary_page_standby, merged them to a single test
4) the test added along with 0001 was not passing with the first patch
alone, fixed it so that the first patch can be committed separately.

Regards,
Vignesh

Attachments:

v8-0001-Fix-hash-index-redo-reading-a-torn-primary-bucket.patchapplication/octet-stream; name=v8-0001-Fix-hash-index-redo-reading-a-torn-primary-bucket.patchDownload+263-12
v8-0003-Add-a-recovery-test-for-the-hash-WAL-pin-only-buc.patchapplication/octet-stream; name=v8-0003-Add-a-recovery-test-for-the-hash-WAL-pin-only-buc.patchDownload+214-1
v8-0002-Don-t-register-pin-only-primary-bucket-pages-in-h.patchapplication/octet-stream; name=v8-0002-Don-t-register-pin-only-primary-bucket-pages-in-h.patchDownload+169-142
v8-0004-Remove-REGBUF_NO_CHANGE.patchapplication/octet-stream; name=v8-0004-Remove-REGBUF_NO_CHANGE.patchDownload+8-36
v8_REL15-0001-Fix-hash-index-redo-reading-a-torn-primary-.patchapplication/octet-stream; name=v8_REL15-0001-Fix-hash-index-redo-reading-a-torn-primary-.patchDownload+262-12
v8_REL17-0001-Fix-hash-index-redo-reading-a-torn-primary-.patchapplication/octet-stream; name=v8_REL17-0001-Fix-hash-index-redo-reading-a-torn-primary-.patchDownload+263-12
v8_REL19-0001-Fix-hash-index-redo-reading-a-torn-primary-.patchapplication/octet-stream; name=v8_REL19-0001-Fix-hash-index-redo-reading-a-torn-primary-.patchDownload+263-12
v8_REL16-0001-Fix-hash-index-redo-reading-a-torn-primary-.patchapplication/octet-stream; name=v8_REL16-0001-Fix-hash-index-redo-reading-a-torn-primary-.patchDownload+263-12
v8_REL18-0001-Fix-hash-index-redo-reading-a-torn-primary-.patchapplication/octet-stream; name=v8_REL18-0001-Fix-hash-index-redo-reading-a-torn-primary-.patchDownload+263-12
#20Joao Rito
joaovictor.rito@gmail.com
In reply to: vignesh C (#19)
Re: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE)

Hi Vignesh, Greg,

Thank you for v8, and for fixing the pg_waldump segment size problem I
hit with v7.

I tested v8-0001..0004 applied in order on current master
(77bba5454f), with cassert and injection points enabled (meson, Debian
12 in a container, ext4):

- recovery/061_hash_torn_primary_page: 9 subtests passed
- recovery/062_hash_vacuum_insert_crc: 4 subtests passed, none skipped

Passing the segment files from pg_walfile_name() works as expected,
and I did not see the "invalid WAL segment size" error again. No CRC
problem was reported either.

I only ran these two tests, once, on Linux ext4. I did not run
check-world or other platforms.

Thanks again for the work on this.

Best regards,
Joao Rito

Em sex., 13 de nov. de 2026 às 07:42, vignesh C <vignesh21@gmail.com> escreveu:

Show quoted text

On Thu, 12 Nov 2026 at 21:43, Joao Rito <joaovictor.rito@gmail.com> wrote:

Hi Greg,

Thank you for the v7 patch and the new recovery test.

I tested v7 on a local build. While running 061_hash_vacuum_insert_crc on
Linux (ext4, meson build), subtest 2 failed for me with:y

pg_waldump: error: invalid WAL segment size in WAL file
"000000010000000000000002" (0 bytes)

Both LSNs are in segment ...0001, and segment ...0002 is a zero-filled
16 MB file. When --path is a directory, pg_waldump seems to read the
segment size from the first WAL file readdir() returns, so the result
depends on directory order (here 0002 came first).

Passing the segment file for $start_lsn (for example from pg_walfile_name)
worked for me by hand. Subtest 3 passed and I saw no CRC problem. I did
not apply the change to the test and rerun it under meson.

Thanks for the comment, this is addressed in the v8 version patch attached.
Apart from that, the following changes were done:
1) prepared back branch patches for the 0001 patch.
2) The test 068 was renamed to the next available 061 test file.
3) Most of the test code was common in
hash_torn_primary_page_delete_first and
hash_torn_primary_page_standby, merged them to a single test
4) the test added along with 0001 was not passing with the first patch
alone, fixed it so that the first patch can be committed separately.

Regards,
Vignesh

#21Greg Burd
greg@burd.me
In reply to: Hayato Kuroda (Fujitsu) (#18)
#22Greg Burd
greg@burd.me
In reply to: Hayato Kuroda (Fujitsu) (#18)
#23Greg Burd
greg@burd.me
In reply to: Greg Burd (#22)
#24Amit Kapila
amit.kapila16@gmail.com
In reply to: Greg Burd (#23)
#25Greg Burd
greg@burd.me
In reply to: Amit Kapila (#24)
#26vignesh C
vignesh21@gmail.com
In reply to: Amit Kapila (#24)
#27Greg Burd
greg@burd.me
In reply to: vignesh C (#26)
#28Amit Kapila
amit.kapila16@gmail.com
In reply to: Greg Burd (#27)
#29vignesh C
vignesh21@gmail.com
In reply to: Amit Kapila (#28)
#30Greg Burd
greg@burd.me
In reply to: Amit Kapila (#28)
#31Hayato Kuroda (Fujitsu)
kuroda.hayato@fujitsu.com
In reply to: vignesh C (#29)
#32Greg Burd
greg@burd.me
In reply to: Hayato Kuroda (Fujitsu) (#31)
#33Greg Burd
greg@burd.me
In reply to: Greg Burd (#32)
#34shihao zhong
zhong950419@gmail.com
In reply to: Greg Burd (#33)