Hash index corruption and CLOG lookup errors from checkpoint races
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t254503psql -h localhost -U postgresBuilt from patchset v2 (message #2), November 02, 2026 at 12:37 PM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t254503_2 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t254503_2 && git checkout t254503_2Patchset v2 (message #2) is on t254503_2
Subject: Index corruption: two more checkpoint races
Hi,
Two more races like [1]/messages/by-id/CAGRkXqRhpq=fd8Qf_-Ki1OW7b1iL+o3YuHiHQhLw5RUJy2JGXQ@mail.gmail.com and [2]/messages/by-id/CAGRkXqSuiJ_X4+Z6rsD42pcuACnD9PhoQ-Os+mUJNmJarBWkNg@mail.gmail.com. A checkpoint can start between a WAL
record and the file change.
1. vac_truncate_clog() truncates CLOG before it advances oldestXid.
After a crash, pg_xact_status() on a removed XID fails with "could not
open file pg_xact/0000" instead of returning NULL. A client that
checks an old transaction after a reconnect gets an error.
The next vacuum does not fix it. It stays until another CLOG segment
is removed.
0001 swaps the two steps, like commit_ts. 0002 is the repro.
2. _hash_alloc_buckets() WAL-logs the new splitpoint before it extends
the file. If a base backup starts in between, the hash index on the new
standby or restored server is broken. It looks fine at first. Once the
index needs an overflow page, inserts fail with "access to noncontiguous
page" until a REINDEX.
A power loss at the same moment can do the same on the primary.
0003 sets DELAY_CHKPT_START around the two steps,
like bulk_write.c. 0004 is the repro.
The fixes apply to 14 and up. 14 needs the old delayChkpt flag. The
tests need 18 and up.
[1]: /messages/by-id/CAGRkXqRhpq=fd8Qf_-Ki1OW7b1iL+o3YuHiHQhLw5RUJy2JGXQ@mail.gmail.com
/messages/by-id/CAGRkXqRhpq=fd8Qf_-Ki1OW7b1iL+o3YuHiHQhLw5RUJy2JGXQ@mail.gmail.com
[2]: /messages/by-id/CAGRkXqSuiJ_X4+Z6rsD42pcuACnD9PhoQ-Os+mUJNmJarBWkNg@mail.gmail.com
/messages/by-id/CAGRkXqSuiJ_X4+Z6rsD42pcuACnD9PhoQ-Os+mUJNmJarBWkNg@mail.gmail.com
Thanks,
Shihao
Attachments:
t254503_1v1-0002-Add-test-for-a-checkpoint-that-starts-during-CLOG.patchapplication/octet-stream; name=v1-0002-Add-test-for-a-checkpoint-that-starts-during-CLOG.patchDownload+72-2
v1-0003-Delay-checkpoint-start-while-allocating-hash-inde.patchapplication/octet-stream; name=v1-0003-Delay-checkpoint-start-while-allocating-hash-inde.patchDownload+16-1
v1-0001-Advance-oldestXid-before-truncating-CLOG.patchapplication/octet-stream; name=v1-0001-Advance-oldestXid-before-truncating-CLOG.patchDownload+16-7
v1-0004-Add-test-for-a-checkpoint-that-starts-during-hash.patchapplication/octet-stream; name=v1-0004-Add-test-for-a-checkpoint-that-starts-during-hash.patchDownload+158-1
On Sun, 1 Nov 2026 at 06:35, shihao zhong <zhong950419@gmail.com> wrote:
Subject: Index corruption: two more checkpoint races
Hi,
Two more races like [1] and [2]. A checkpoint can start between a WAL
record and the file change.1. vac_truncate_clog() truncates CLOG before it advances oldestXid.
After a crash, pg_xact_status() on a removed XID fails with "could not
open file pg_xact/0000" instead of returning NULL. A client that
checks an old transaction after a reconnect gets an error.The next vacuum does not fix it. It stays until another CLOG segment
is removed.
Good catch.
The CLOG fix made me look at the other limits recorded in a checkpoint
and restored during crash recovery: oldestXid, oldestCommitTsXid and
oldestMulti. oldestCommitTsXid is already advanced before
TruncateCommitTs(), but oldestMulti has the same race that oldestXid
had. I was able to reproduce it.
Having a similar test for oldestMultiXact results in an oldest
checkpointed MultiXactId refers to a segment that no longer exists.
After a crash, this produces:
LOG: MultiXact member truncation is disabled because oldest checkpointed
MultiXact 1 does not exist on disk
Attached a one-line injection point in TruncateMultiXact(), after
WriteMTruncateXlogRec() and before the in-memory update (loaded before
START_CRIT_SECTION, since it
is hit inside it), and a TAP test for src/test/recovery
Thoughts?
Regards,
Vignesh
On Sun, 1 Nov 2026 at 06:35, shihao zhong <zhong950419@gmail.com> wrote:
2. _hash_alloc_buckets() WAL-logs the new splitpoint before it extends
the file. If a base backup starts in between, the hash index on the new
standby or restored server is broken. It looks fine at first. Once the
index needs an overflow page, inserts fail with "access to noncontiguous
page" until a REINDEX.A power loss at the same moment can do the same on the primary.
0003 sets DELAY_CHKPT_START around the two steps,
like bulk_write.c. 0004 is the repro.
Thanks for identifying this issue.
I noticed one issue:
After DELAY_CHKPT_START is set, itcan fail with an ordinary ERROR, for
example, due to lack of disk space. If this happens, the flag is left
set. A top-level abort clears it through ProcArrayEndTransaction(),
but AbortSubTransaction() does not. So if the error is caught by a
savepoint or a PL/pgSQL EXCEPTION block, the flag remains set for the
rest of the transaction. Every checkpoint then waits in
CheckpointDelayStart until that transaction ends. Even an
idle-in-transaction session can block checkpoints. The attached test
uses the error action of the new hash-alloc-buckets-after-wal
injection point to reproduce this issue.
Regards,
Vignesh