Reduce WAL volume for heap tuple hint bits

Hi Andrey,

On Wed, Aug 5, 2026 at 5:18 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:

Hi hackers,

Heap tuple visibility hint bits avoid repeated transaction status
lookups. With wal_log_hints enabled, PostgreSQL currently logs a
full-page image for the first hint bit change to a page after each
checkpoint. This can generate considerably more WAL than the hint bits
themselves require.

My main motivation comes from our PostgreSQL fork with compute/storage
separation, where this WAL volume is costly. The same overhead also
occurs in vanilla clusters with wal_log_hints enabled.
The concept is mostly viable when torn pages are not a problem.

The attached patch records heap tuple offsets and visibility hint bits
in a compact WAL record when checksums are disabled. WAL replay applies
the hints on standbys. The record does not advance the page LSN, so a
later ordinary change still generates the FPI required by
full_page_writes. Checksummed pages continue to use FPI_FOR_HINT.

For the first scan of one million rows, WAL fell from 36.8 MB to 4.2 MB
for an int-only table and from 134.7 MB to 4.9 MB for a
pgbench_accounts-shaped table. With wal_compression=lz4, WAL fell from
13.9 MB to 4.2 MB and from 16.3 MB to 4.9 MB, respectively. About 3.3x in
both cases.

PFA.

Thanks for working on this — the WAL overhead of wal_log_hints is a
real pain point, and the
compact record for the checksums-off case is a nice win.

I read v1 and tested it, and the correctness-critical parts hold up:
not advancing the page LSN correctly
preserves the FPI for the next ordinary change and keeps replay
idempotent, and the DataChecksumsNeedWrite()
gating leaves no torn-page window. A crash-recovery run with
wal_consistency_checking = 'all' showed no inconsistencies,
the 056 test passes, and pg_waldump confirms the ~655-byte records vs
an 8 KB FPI.

One design question before going further: why a new resource manager
(RM_HEAP_HINT) rather than a new info opcode
under the existing heap rmgr? A dedicated rmgr spends one of the
limited RM_*_ID slots, and folding it into heap would keep
heap WAL together. Is there a reason I'm missing?

Thanks again!

Best regards, Andrey Borodin.

--
Regards,
Ewan Young

On Wed, 5 Aug 2026 at 11:18, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

Hi hackers,

Heap tuple visibility hint bits avoid repeated transaction status
lookups. With wal_log_hints enabled, PostgreSQL currently logs a
full-page image for the first hint bit change to a page after each
checkpoint. This can generate considerably more WAL than the hint bits
themselves require.

My main motivation comes from our PostgreSQL fork with compute/storage
separation, where this WAL volume is costly. The same overhead also
occurs in vanilla clusters with wal_log_hints enabled.
The concept is mostly viable when torn pages are not a problem.

I'd say it's _only_ viable when torn writes are not a problem. Torn
writes may [^1] leave subsequent reads with actual garbage on the
page, not just an older version of the page, and without an FPI you
can't recover from that. I don't think there's a more accurate GUC
for "I need torn page protection" than the "full_page_writes" GUC, so
this patch seems like a non-starter to me.

Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)

[^1] Torn write behaviour depends on the FS and possibly the disk
used, and I've seen reports of actual garbage on pages after power
loss. Local testing with anything less than that is probably not
representative of the whole problem space.

On 17 Aug 2026, at 16:53, Matthias van de Meent <boekewurm+postgres@gmail.com> wrote:

Hi Matthias, Surya, Ewan,

Surya's reply appears to have been threaded under "Compression of bigger
WAL records", probably because of a stale References header. I am
bringing that part of the discussion back to the hint-bit thread here.

Thank you for the reviews. I think they expose two separate questions
that I had mixed together in v1: protection against page damage and
propagation of hint bits to standbys.

First, I agree with Matthias that the failure mode is broader than a
write containing some sectors from the old page and some from the new
page. An interrupted page write may leave bytes that belong to neither
version. A record containing only tuple offsets and hint bits cannot
repair such damage.

PostgreSQL already permits a clean page to be dirtied by a hint-only
change without an FPI when full_page_writes is on but wal_log_hints and
checksums are both off. That does not make it a good model to extend.
We should not use an existing hole in the protection as a reason to
design a better replicated /dev/null.

Surya wrote that the compact record should advance the page LSN so that
FlushBuffer() flushes the WAL before the page. I deliberately avoided
that, although the reason was not explained clearly enough in v1.

For a non-critical hint change, writing the page before the WAL record
is harmless by itself:

* if the page reaches disk first, the hint is already present;
* if the WAL record reaches disk first, redo can apply the hint;
* if neither reaches disk, the hint is lost and can be recomputed.

PostgreSQL already allows the first case when it sets hints without WAL.
Thus, an on-disk hint without a corresponding durable WAL record is not
by itself a violation of the WAL rule for critical data.

Advancing the page LSN would create a different problem. A later
ordinary modification could see an LSN newer than the checkpoint redo
pointer and conclude that the page was already protected by an FPI.
The compact hint record is not such protection, so that could leave the
ordinary modification exposed to page damage. I do not think we can
use PageSetLSN() for this record without separating the WAL-before-data
LSN from the LSN used to decide whether an FPI is required.

There is, however, a potentially useful and much narrower purpose for
the record. With full_page_writes off, PostgreSQL still emits an
XLOG_FPI_FOR_HINT record, but it contains only a block reference and its
redo is a no-op. It does not propagate the hint bits to a standby.

As I understand the standby side, it may compute the hints itself while
reading the page. However, when wal_log_hints or checksums require hint
WAL, recovery cannot write a new WAL record and therefore does not dirty
an otherwise clean page. The locally computed hints then remain only
in memory and can disappear on eviction. Please correct me if I am
missing another path that persists them.

This suggests reclassifying the proposal. Instead of reducing WAL by
replacing an FPI, it could deliver heap visibility hints with
wal_log_hints on, full_page_writes off, and checksums disabled.
In that configuration the compact record will usually be larger than
today's block-reference-only FPI_FOR_HINT record, so WAL reduction is no
longer present. The benefit would be that the standby receives and
can persist the hints selected by the primary.

Matthias, Surya, Ewan, do you think that is a useful and sound scope for
the patch? It seems very narrow, so we can just rejection the patch.

Ewan, thank you for suggesting that the new record use an existing
resource manager. I have prepared a version that moves it to Heap2 as
XLOG_HEAP2_HINT_BITS. I am not attaching it yet because I think we
should agree on the design and the actual purpose of the patch first.

Best regards, Andrey Borodin.