Hash index corruption and CLOG lookup errors from checkpoint races
Subject: Index corruption: two more checkpoint races
Hi,
Two more races like [1]/messages/by-id/CAGRkXqRhpq=fd8Qf_-Ki1OW7b1iL+o3YuHiHQhLw5RUJy2JGXQ@mail.gmail.com and [2]/messages/by-id/CAGRkXqSuiJ_X4+Z6rsD42pcuACnD9PhoQ-Os+mUJNmJarBWkNg@mail.gmail.com. A checkpoint can start between a WAL
record and the file change.
1. vac_truncate_clog() truncates CLOG before it advances oldestXid.
After a crash, pg_xact_status() on a removed XID fails with "could not
open file pg_xact/0000" instead of returning NULL. A client that
checks an old transaction after a reconnect gets an error.
The next vacuum does not fix it. It stays until another CLOG segment
is removed.
0001 swaps the two steps, like commit_ts. 0002 is the repro.
2. _hash_alloc_buckets() WAL-logs the new splitpoint before it extends
the file. If a base backup starts in between, the hash index on the new
standby or restored server is broken. It looks fine at first. Once the
index needs an overflow page, inserts fail with "access to noncontiguous
page" until a REINDEX.
A power loss at the same moment can do the same on the primary.
0003 sets DELAY_CHKPT_START around the two steps,
like bulk_write.c. 0004 is the repro.
The fixes apply to 14 and up. 14 needs the old delayChkpt flag. The
tests need 18 and up.
[1]: /messages/by-id/CAGRkXqRhpq=fd8Qf_-Ki1OW7b1iL+o3YuHiHQhLw5RUJy2JGXQ@mail.gmail.com
/messages/by-id/CAGRkXqRhpq=fd8Qf_-Ki1OW7b1iL+o3YuHiHQhLw5RUJy2JGXQ@mail.gmail.com
[2]: /messages/by-id/CAGRkXqSuiJ_X4+Z6rsD42pcuACnD9PhoQ-Os+mUJNmJarBWkNg@mail.gmail.com
/messages/by-id/CAGRkXqSuiJ_X4+Z6rsD42pcuACnD9PhoQ-Os+mUJNmJarBWkNg@mail.gmail.com
Thanks,
Shihao
Attachments:
t254503_1v1-0002-Add-test-for-a-checkpoint-that-starts-during-CLOG.patchapplication/octet-stream; name=v1-0002-Add-test-for-a-checkpoint-that-starts-during-CLOG.patchDownload+72-2
v1-0003-Delay-checkpoint-start-while-allocating-hash-inde.patchapplication/octet-stream; name=v1-0003-Delay-checkpoint-start-while-allocating-hash-inde.patchDownload+16-1
v1-0001-Advance-oldestXid-before-truncating-CLOG.patchapplication/octet-stream; name=v1-0001-Advance-oldestXid-before-truncating-CLOG.patchDownload+16-7
v1-0004-Add-test-for-a-checkpoint-that-starts-during-hash.patchapplication/octet-stream; name=v1-0004-Add-test-for-a-checkpoint-that-starts-during-hash.patchDownload+158-1
On Sun, 1 Nov 2026 at 06:35, shihao zhong <zhong950419@gmail.com> wrote:
Subject: Index corruption: two more checkpoint races
Hi,
Two more races like [1] and [2]. A checkpoint can start between a WAL
record and the file change.1. vac_truncate_clog() truncates CLOG before it advances oldestXid.
After a crash, pg_xact_status() on a removed XID fails with "could not
open file pg_xact/0000" instead of returning NULL. A client that
checks an old transaction after a reconnect gets an error.The next vacuum does not fix it. It stays until another CLOG segment
is removed.
Good catch.
The CLOG fix made me look at the other limits recorded in a checkpoint
and restored during crash recovery: oldestXid, oldestCommitTsXid and
oldestMulti. oldestCommitTsXid is already advanced before
TruncateCommitTs(), but oldestMulti has the same race that oldestXid
had. I was able to reproduce it.
Having a similar test for oldestMultiXact results in an oldest
checkpointed MultiXactId refers to a segment that no longer exists.
After a crash, this produces:
LOG: MultiXact member truncation is disabled because oldest checkpointed
MultiXact 1 does not exist on disk
Attached a one-line injection point in TruncateMultiXact(), after
WriteMTruncateXlogRec() and before the in-memory update (loaded before
START_CRIT_SECTION, since it
is hit inside it), and a TAP test for src/test/recovery
Thoughts?
Regards,
Vignesh
On Sun, 1 Nov 2026 at 06:35, shihao zhong <zhong950419@gmail.com> wrote:
2. _hash_alloc_buckets() WAL-logs the new splitpoint before it extends
the file. If a base backup starts in between, the hash index on the new
standby or restored server is broken. It looks fine at first. Once the
index needs an overflow page, inserts fail with "access to noncontiguous
page" until a REINDEX.A power loss at the same moment can do the same on the primary.
0003 sets DELAY_CHKPT_START around the two steps,
like bulk_write.c. 0004 is the repro.
Thanks for identifying this issue.
I noticed one issue:
After DELAY_CHKPT_START is set, itcan fail with an ordinary ERROR, for
example, due to lack of disk space. If this happens, the flag is left
set. A top-level abort clears it through ProcArrayEndTransaction(),
but AbortSubTransaction() does not. So if the error is caught by a
savepoint or a PL/pgSQL EXCEPTION block, the flag remains set for the
rest of the transaction. Every checkpoint then waits in
CheckpointDelayStart until that transaction ends. Even an
idle-in-transaction session can block checkpoints. The attached test
uses the error action of the new hash-alloc-buckets-after-wal
injection point to reproduce this issue.
Regards,
Vignesh
Attachments:
063_hash_alloc_buckets_error.pltext/x-perl-script; charset=US-ASCII; name=063_hash_alloc_buckets_error.plDownload
Hi Vignesh,
Thanks for catching this, and for the test. It fails on v1 as you
describe.
v2 drops the flag. _hash_alloc_buckets() now extends the file before
writing the WAL record, so a checkpoint either replays the record or
syncs the new block.
^^^^ is aligned with current heap extend behavior. Another way is to
use PG_TRY and FINALLY to reset the flag. I prefer to use
same mechanism because it is easy to understand in the future.
0001 and 0002 are unchanged.
Thanks,
Shihao.
Attachments:
t254503_4v2-0004-Add-test-for-a-checkpoint-that-starts-during-hash.patchapplication/x-patch; name=v2-0004-Add-test-for-a-checkpoint-that-starts-during-hash.patchDownload+210-1
v2-0002-Add-test-for-a-checkpoint-that-starts-during-CLOG.patchapplication/x-patch; name=v2-0002-Add-test-for-a-checkpoint-that-starts-during-CLOG.patchDownload+72-2
v2-0001-Advance-oldestXid-before-truncating-CLOG.patchapplication/x-patch; name=v2-0001-Advance-oldestXid-before-truncating-CLOG.patchDownload+16-7
v2-0003-Extend-hash-index-before-WAL-logging-new-bucket-p.patchapplication/x-patch; name=v2-0003-Extend-hash-index-before-WAL-logging-new-bucket-p.patchDownload+16-5
On Tue, 3 Nov 2026 at 09:38, shihao zhong <zhong950419@gmail.com> wrote:
Hi Vignesh,
Thanks for catching this, and for the test. It fails on v1 as you
describe.v2 drops the flag. _hash_alloc_buckets() now extends the file before
writing the WAL record, so a checkpoint either replays the record or
syncs the new block.^^^^ is aligned with current heap extend behavior. Another way is to
use PG_TRY and FINALLY to reset the flag. I prefer to use
same mechanism because it is easy to understand in the future.0001 and 0002 are unchanged.
The issue reported at [1]/messages/by-id/CALDaNm1+SiZ9SGt=X9KpOAx4vG8ZjrsXSq1o4nEiP7C_=NDGnA@mail.gmail.com has not yet been fixed right? Just checking
to confirm it is not lost in the discussion.
[1]: /messages/by-id/CALDaNm1+SiZ9SGt=X9KpOAx4vG8ZjrsXSq1o4nEiP7C_=NDGnA@mail.gmail.com
Regards,
Vignesh
On Tue, 3 Nov 2026 at 09:38, shihao zhong <zhong950419@gmail.com> wrote:
Hi Vignesh,
Thanks for catching this, and for the test. It fails on v1 as you
describe.v2 drops the flag. _hash_alloc_buckets() now extends the file before
writing the WAL record, so a checkpoint either replays the record or
syncs the new block.^^^^ is aligned with current heap extend behavior. Another way is to
use PG_TRY and FINALLY to reset the flag. I prefer to use
same mechanism because it is easy to understand in the future.
Thanks for providing the updated version, the patch does not apply to
back branches as back branches use PageSetChecksumInplace instead of
PageSetChecksum. We will need a separate patch for the back branch:
git am v2-0003-Extend-hash-index-before-WAL-logging-new-bucket-p.patch
Applying: Extend hash index before WAL-logging new bucket pages
error: patch failed: src/backend/access/hash/hashpage.c:1032
error: src/backend/access/hash/hashpage.c: patch does not apply
Regards,
Vignesh
Hi Vignesh,
Thanks for the test and for trying the backport.
How did I miss your MultiXact case..... 0005 take care of it.
For the back branches I attached two versions of 0003. The one for 16
to 18 uses PageSetChecksumInplace(). The one for 14 and 15 also uses
rd_node.
I can send back-branch versions of 0005 once the master version looks
fine to people. 0001 to 0004 have no code changes.
Thanks,
Shihao