BUG #19620: pg_class index corruption caused by statement_timeout during VACUUM FULL

The following bug has been logged on the website:

Bug reference: 19620
Logged by: Zheng Hacker
Email address: hackerzheng666@gmail.com
PostgreSQL version: 19beta3
Operating system: Linux x86_64
Description:

SUMMARY

On PostgreSQL 18devel (git master cc1ea71), a single backend that
repeatedly runs catalog-churning DDL/maintenance under a short
statement_timeout drives the system-catalog index
pg_class_relname_nsp_index into a corrupt state: one btree leaf page
acquires two index tuples pointing to the same pg_class heap TID.

A subsequent CREATE TABLE touches that leaf page, a btree
simple-delete/dedup pass collects the two identical TIDs, and the
cassert assertion in index_delete_sort_cmp (heapam.c:8668) fires.

On a production (non-cassert) build, the corrupt catalog index is
silent and can produce wrong catalog lookups (we separately observe
the Assert(relid == targetRelId) in relcache.c:1138 from the same
corruption when a pg_class OID-index scan returns the wrong row).

KEY DISTINCTION from known reports (#18490, #17386, #17255): this
reproducer is purely single-session, single-backend -- all parallelism
disabled (max_worker_processes=0), no concurrent connections. The
trigger is statement cancellation (CHECK_FOR_INTERRUPTS) during
catalog-index maintenance, not a concurrency race.

VERSION / BUILD

PostgreSQL master (ahead of 19beta3), git commit cc1ea71
("Remove unnecessary list_free() calls in OpenTableList()")
Build: clang -O1 -g --enable-cassert (no sanitizers, no sancov),
x86_64 Linux.
First observed on an ASan+sancov build; confirmed on this clean
build to rule out instrumentation artifacts.

REPRODUCTION

1) Server setup:

initdb -D $PGDATA --no-locale -A trust
cat >> $PGDATA/postgresql.conf << 'EOF'
fsync = off
debug_parallel_query = off
max_parallel_workers = 0
max_parallel_maintenance_workers = 0
max_worker_processes = 0
EOF
pg_ctl -D $PGDATA -l $PGDATA/server.log start
createdb testdb

2) Save as repro.sql:

CREATE TABLE t1 (i int, t text);
INSERT INTO t1 SELECT g, md5(g::text) FROM generate_series(1,100) g;
CREATE INDEX t1_i ON t1(i);
CREATE INDEX t1_t ON t1(t);
CREATE TABLE t2 (i int PRIMARY KEY, a int[], p point);
INSERT INTO t2 SELECT g, array[g,g+1], point(g,g) FROM
generate_series(1,100) g;
CREATE INDEX t2_hash ON t2 USING hash(i);
CREATE INDEX t2_gin ON t2 USING gin(a);
CREATE INDEX t2_gist ON t2 USING gist(p);
CREATE TABLE t3 (k int, v text);
INSERT INTO t3 SELECT g, repeat('x',200) FROM generate_series(1,100) g;
CREATE INDEX t3_k ON t3(k);
CREATE TABLE t4 (a int) PARTITION BY LIST(a);
CREATE TABLE t4p1 PARTITION OF t4 FOR VALUES IN (1);
CREATE TABLE t4p2 PARTITION OF t4 FOR VALUES IN (2);
CREATE TABLE t4p3 PARTITION OF t4 FOR VALUES IN (3);
INSERT INTO t4 SELECT (g%3)+1 FROM generate_series(1,60) g;
CREATE TABLE t5 (i int PRIMARY KEY, t text);
ALTER TABLE t5 ALTER COLUMN t SET STORAGE EXTERNAL;
INSERT INTO t5(i,t) VALUES (generate_series(1,20),
repeat('1234567890',269));
VACUUM FULL t1;
VACUUM FULL t2;
VACUUM FULL t3;
VACUUM FULL t5;
DELETE FROM t1 WHERE i < 50;
DELETE FROM t3 WHERE k < 50;
VACUUM t1;
VACUUM t3;
DROP TABLE t1;
DROP TABLE t2;
DROP TABLE t3;
DROP TABLE t4 CASCADE;
DROP TABLE t5;
VACUUM FULL pg_class;
VACUUM FULL pg_am;
VACUUM FULL pg_database;

3) Save as repro.sh and run:

#!/bin/bash
Q="psql -d testdb -q"
for round in $(seq 1 200); do
to=$(( (round % 12) + 1 ))
{ echo "SET statement_timeout='${to}ms';"
cat repro.sql
echo "SET statement_timeout=0;"
echo "CREATE TABLE probe_${round}(a int, t text);"
echo "DROP TABLE IF EXISTS probe_${round};"
} | $Q 2>/dev/null
if grep -q "TRAP:" "$PGDATA/server.log" 2>/dev/null; then
echo "CRASH at round $round"
grep -m1 "TRAP:" "$PGDATA/server.log"
break
fi
done

RESULT: crashes 5/5 runs within 37 rounds (rounds 13, 25, 37, 37, 37).
Using the first 187 lines of src/test/regress/sql/vacuum.sql instead
(heavier pg_class churn) accelerates the crash to 3/3 within round 24.
Without statement_timeout (timeout = 0), the same workload runs for
hundreds of rounds without corruption -- query cancellation is the
essential trigger.

ASSERTION + BACKTRACE

TRAP: failed Assert("false"), File: "heapam.c", Line: 8668
#5 ExceptionalCondition assert.c:65
#6 index_delete_sort_cmp heapam.c:8668
#7 index_delete_sort heapam.c:8708
#8 heap_index_delete_tuples heapam.c:8366
#10 _bt_delitems_delete_check nbtpage.c:1536
#11 _bt_simpledel_pass nbtinsert.c:2959
#12 _bt_delete_or_dedup_one_page nbtinsert.c:2770
#14 _bt_doinsert nbtinsert.c:261
#15 btinsert nbtree.c:219
#16 index_insert indexam.c:231
#17 CatalogIndexInsert indexing.c:170
#18 CatalogTupleInsert indexing.c:243
#19 InsertPgClassTuple heap.c:989
#20 AddNewRelationTuple heap.c:1049
#21 heap_create_with_catalog heap.c:1447
#22 create_toast_table toasting.c:254
#25 ProcessUtilitySlow utility.c:1197
#31 exec_simple_query
"CREATE TABLE covering_index_heap (f1 int, f2 int, f3 text);"

DIRECT CORRUPTION EVIDENCE (core dump)

frame _bt_delitems_delete_check:
rel (index) = OID 2663 "pg_class_relname_nsp_index"
heapRel (heap) = OID 1259 "pg_class"
delstate->ndeltids = 119
delstate->deltids:
entry[3] = { tid=(block 0, offset 5), id=45 }
entry[6] = { tid=(block 0, offset 5), id=45 } <-- DUPLICATE

Two entries carry the identical heap TID (0,5). This can only happen
if pg_class_relname_nsp_index contains two live index pointers to the
same pg_class heap tuple -- the catalog index is corrupt.

SERIAL CONFIRMATION

Re-run with all parallelism disabled (debug_parallel_query=off,
max_parallel_workers=0, max_parallel_maintenance_workers=0,
max_worker_processes=0): still crashes at rounds 13-24 across
multiple runs. The crashing backend has no parallel/bgworker frames
anywhere in the stack. This rules out any parallel-index-build or
concurrent-reindex race condition.

A second core dump from a serial run shows a different crashing
statement (REINDEX TABLE CONCURRENTLY testcomment) hitting the same
corrupt pg_class_relname_nsp_index leaf via CatalogTupleUpdate ->
_bt_simpledel_pass. The corruption is latent in the index; any
subsequent catalog write that hits the corrupt page triggers the
assert.

ROOT CAUSE ANALYSIS

The corruption is a pg_class_relname_nsp_index btree leaf page
holding two pointers to one heap TID. Bisection of the workload
identifies three necessary ingredients:

1. VACUUM FULL pg_class -- rewrites the entire pg_class heap and
rebuilds all pg_class btree indexes from scratch. This is the
essential catalog operation; removing it eliminates the crash.

2. Heavy DDL churn (CREATE TABLE/INDEX, DROP TABLE, VACUUM FULL on
user tables) -- generates many pg_class inserts, updates, and
deletes, filling btree pages and creating conditions for
dedup/simple-delete passes during subsequent inserts.

3. statement_timeout cancellation -- interrupts (1) or (2) mid-flight
via CHECK_FOR_INTERRUPTS(). Without cancellation, no corruption
occurs even after hundreds of rounds.

The most likely mechanism: VACUUM FULL pg_class calls cluster_rel()
which rewrites the heap and rebuilds all indexes via index_build().
If statement_timeout fires during this rebuild at a
CHECK_FOR_INTERRUPTS() site, the transaction is aborted, but a
catalog-index btree page may be left in an inconsistent state with a
duplicate pointer to the same heap TID. The abort/rollback path
does not fully undo the partial btree page modification.

PRIOR ART

The "duplicate heap TID in a pg_class index" corruption class is
known:

- BUG #18490 (2024) -- same symptom, index
pg_class_tblspc_relfilenode_index, trigger = concurrent REINDEX
during table creation; Peter Geoghegan: "a known issue", no fix
committed.
- BUG #17386 (2022) -- btree corruption after REINDEX CONCURRENTLY.
- BUG #17255 (2021) -- index_delete_sort_cmp via parallel-vacuum
race; fixed 2022.

What is new: all known reports require concurrency (parallel workers,
concurrent connections, concurrent REINDEX). This reproducer triggers
the same corruption in a single session with all parallelism disabled,
via statement cancellation alone. This is an interrupt-safety bug in
catalog index maintenance, distinct from the concurrency races in the
known reports.

DISCOVERY

Credit: Zheng Wang, Yanjie Zhao, Yiyang Liu.