Stop masking visibility information during WAL consistency checking
Hi,
Today wal_consistency_checking compares neither the bitmap on VM pages
nor the PD_ALL_VISIBLE flag on heap pages. The attached patches make
it compare both. We discussed checking the VM bitmap in the WAL_LOG
CREATE DATABASE thread [1]/messages/by-id/2889116c-7da1-4ab0-b10a-3fadb99c24e7@gmail.com, and I proposed checking PD_ALL_VISIBLE in
the thread about registering VM buffers for clears [2]/messages/by-id/CAAKRu_aRdQ6RjneKeQh9+RVMFgMrtjTYFX=zDgBdSiQ8JDs6Jg@mail.gmail.com. I think we
should do both on master.
- Melanie
[1]: /messages/by-id/2889116c-7da1-4ab0-b10a-3fadb99c24e7@gmail.com
[2]: /messages/by-id/CAAKRu_aRdQ6RjneKeQh9+RVMFgMrtjTYFX=zDgBdSiQ8JDs6Jg@mail.gmail.com
Attachments:
t259594_1v1-0001-Stop-masking-visibility-map-bits-during-WAL-consi.patchtext/x-patch; charset=US-ASCII; name=v1-0001-Stop-masking-visibility-map-bits-during-WAL-consi.patchDownload+80-27
v1-0002-Stop-masking-PD_ALL_VISIBLE-when-checking-WAL-con.patchtext/x-patch; charset=US-ASCII; name=v1-0002-Stop-masking-PD_ALL_VISIBLE-when-checking-WAL-con.patchDownload+0-7
Hi Melanie,
Thank you for starting this thread and the patches.
I tested v1 0001+0002 on master (65ebed1f5e2) and did a couple of tests:
1. make check passes, including the new test_vm_mask().
2. Primary with wal_consistency_checking = all plus a streaming standby,
running the regression suite against the primary (similar to
027_stream_regress with PG_TEST_EXTRA=wal_consistency_checking, huge
wal was generated), the standby replayed everything with no
inconsistent-page failures.
3. Crash recovery with wal_consistency_checking = all over a workload aimed
at VM/PD_ALL_VISIBLE transitions (VACUUM and VACUUM FREEZE, VM clears from
UPDATE/DELETE/row locks, on-access pruning, COPY FREEZE, VM truncation)
they all look clean.
I didn't see any false positives from unmasking either the VM bitmap or
PD_ALL_VISIBLE.
Otherwise both patches look good to me.
Regards,
Surya Poondla
Hello
Stop masking PD_ALL_VISIBLE, as doing so can only conceal real bugs.
Claude pointed out that wal_consistency_checking can trigger an error with 0002. This uses very specific flags, but both full_page_writes=off and disabled data checksums are supported, so it seems like a plausible scenario.
initdb --no-data-checksums, wal_consistency_checking = 'heap,heap2', full_page_writes = off, autovacuum = off.
CREATE EXTENSION pg_buffercache;
CREATE TABLE t(a int);
INSERT INTO t SELECT generate_series(1, 10);
CHECKPOINT;
INSERT INTO t VALUES (11);
VACUUM t;
SELECT pg_buffercache_evict_relation('t');
-- pg_ctl stop -m immediate; pg_ctl start
FATAL: inconsistent page found, rel 1663/5/16460, forknum 0, blkno 0
CONTEXT: WAL redo at 0/01733368 for Heap/INSERT: off: 11 ...
+ /* On VM pages the apparent free space contains the visibility bitmap. */
+ if (forknum == VISIBILITYMAP_FORKNUM)
+ return;
+
And also for this change, with the same config:
CREATE TABLE u(a int) WITH (fillfactor = 10);
INSERT INTO u SELECT generate_series(1, 250);
VACUUM u;
CHECKPOINT;
DELETE FROM u WHERE a > 125;
VACUUM u; -- truncates to 6 blocks
SELECT pg_buffercache_evict_relation('u');
-- immediate stop / start
FATAL: inconsistent page found, rel 1663/5/16469, forknum 2, blkno 0
On Fri, Nov 20, 2026 at 6:17 PM Zsolt Parragi <zsolt.parragi@percona.com> wrote:
Stop masking PD_ALL_VISIBLE, as doing so can only conceal real bugs.
Claude pointed out that wal_consistency_checking can trigger an error
with 0002. This uses very specific flags, but both
full_page_writes=off and disabled data checksums are supported, so it
seems like a plausible scenario.
Ah, right, I forgot that when we only sets the VM bits and both
checksums/wal_log_hints are off, we can set PD_ALL_VISIBLE without
advancing the heap page LSN. So, I suppose we can't do 0002. I'll
replace it with a replacement of the comment explaining why we have to
mask PD_ALL_VISIBLE. Thanks!
- Melanie
Hi,
On Sat, Nov21, 2026 at 07:30 AM Melanie Plageman wrote:
On Fri, Nov 20, 2026 at 6:17 PM Zsolt Parragi <zsolt.parragi@percona.com> wrote:
Stop masking PD_ALL_VISIBLE, as doing so can only conceal real bugs.
Claude pointed out that wal_consistency_checking can trigger an
error with 0002. This uses very specific flags, but both
full_page_writes=off and disabled data checksums are supported, so
it seems like a plausible scenario.Ah, right, I forgot that when we only sets the VM bits and both
checksums/wal_log_hints are off, we can set PD_ALL_VISIBLE without
advancing the heap page LSN. So, I suppose we can't do 0002. I'll
replace it with a replacement of the comment explaining why we have to
mask PD_ALL_VISIBLE. Thanks!
I analyzed this bit more, and think the same issue can also cause a
false positive in 0001. visibilitymap_prepare_truncate() also modifies
the VM page without advancing pd_lsn when checksums and wal_log_hints
are off (as in Zsolt's second test case). If the server crashes
immediately after that VM page is flushed to disk, recovery sees the
post-truncation VM page contents with the pre-truncation pd_lsn. When
replaying the last VM-modifying record before the truncation,
wal_consistency_checking fails because the on-disk page does not match
the image logged before the truncation. We could fix this by
unconditionally WAL-logging the VM page there, which advances pd_lsn as
well. (But the fix looks like separate change worth some discussions and
some consensus)
And I also noticed two other cases (when full_page_writes is off) that
cause similar failures:
1. If the server crashes after dropping a table, the VM file has already
been unlinked. During recovery, replaying an earlier XLOG_HEAP2_PRUNE_*
record passes RBM_ZERO_ON_ERROR when reading the VM page, which creates
a new zero-filled page if the file does not exist. After replaying that
record's VM change onto the new page, its contents still do not match
the logged image because any other bits previously set on that VM page
were lost.
2. If the VM fork is truncated and then re-extended, vm_extend() writes
an all-zero page on disk while the page header is initialized only in
shared_buffers, leaving an all-zero page on disk. If the server crashes
immediately after extending the fork, recovery runs the consistency
check against that all-zero page and fails (note that this issue also
exists on master for zero-extended heap pages, where mask_unused_space()
fails on pd_lower == 0). To fix this, we could skip the consistency
check when the page is new (PageIsNew()) and the current WAL record does
not initialize it.
--
Best Regards,
Zhijie Hou
Hi,
On 21 Nov 2026, Zhijie Hou wrote:
We could fix this by unconditionally WAL-logging the VM page there,
which advances pd_lsn as well.
Could we work out all the conditions under which redo must reproduce
each field, and keep the consistency check enabled whenever they hold?
I'd analyze PD_ALL_VISIBLE and the VM bitmap separately. Besides
checksums, wal_log_hints and full_page_writes, we need to consider LSN
ordering and the page lifecycle: truncation, unlink/recreation and
zero-extension, including whether the current record initializes the
page.
Decisions based on configuration should use the WAL writer's settings,
including changes recorded in the replayed WAL. We could then mask only
the fields whose prerequisites are not satisfied, or skip a comparison
when a missing or new page cannot be reconstructed.
Thank you!
Best regards, Andrey Borodin.