Offline data checksum changes can cause incorrect checksum state on standbys
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t253388psql -h localhost -U postgresBuilt from patchset v1 (message #1), August 18, 2026 at 08:16 AM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t253388_1 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t253388_1 && git checkout t253388_1Patchset v1 (message #1) is on t253388_1
Hi hackers,
while working on [1]/messages/by-id/ajAAwSFy0WVMroyk@bdtpg, I hit 2 issues involving offline checksum changes.
First one, is a case where a standby can enable checksum verification without its
own pages having been checksummed, making the standby unreadable.
The issue is due to f19c0eccae96 as the standby can enable checksum verification
from primary WAL.
Repro 1:
1/ create a primary and a standby both with checksum set to false
2/ stop the standby and the primary
3/ enable checksums only on the primary
4/ restart the primary and run checkpoint: this checkpoint XLOG_CHECKPOINT_REDO
record carries data_checksum_version=on. It gives the standby a WAL record that
makes it enable checksum verification.
5/ restart the standby: that will replay the primary’s new checksum state, despite
never having its own pages checksummed.
6/ try to connect to the standby: FATAL: invalid page in block 0 of relation "global/1260"
The second issue occurs when combining online and offline checksum transitions.
Repro 2:
1/ create a primary and a standby both with checksum set to false
2/ enable checksums online on the primary and wait that data_checksums is on
on the primary and on the standby
3/ stop only the standby
4/ disable checksums offline only on the standby
5/ restart the standby and check data_checksums. You'll see that it's still on
despites that we disabled it in step 4/
I initially considered detecting the mismatch during WAL replay and reporting an
error. Although this produces a clear error message, it does not help much in
practice because the standby cannot recover and has to be recreated.
Therefore, I think a simpler fix is to preserve the pre-f19c0eccae96 behavior
for offline checksum changes: they are not propagated through WAL. In the second
repro, the standby therefore remains off, honoring its local offline change.
This is what the attached patch proposes: it marks offline changes as local and
tracks the latest WAL-logged transition, so recovery ignores remote offline
states while still applying newer online transitions.
If this looks like too much code changes so close to the v19 release, another
option could be to remove pg_checksums --enable and --disable while keeping --check
and require checksum state changes to be done online.
As this is a 19 regression, I think it should be added as an open item.
Thoughts?
[1]: /messages/by-id/ajAAwSFy0WVMroyk@bdtpg
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On 12 Aug 2026, at 09:55, Bertrand Drouvot <bertranddrouvot.pg@gmail.com> wrote:
First one, is a case where a standby can enable checksum verification without its
own pages having been checksummed, making the standby unreadable.
Thanks for the report. While I don't have a proposal ready at this time of
writing, I wanted to ACK having seen this and make it known (to RMT) that it is
being worked on.
Therefore, I think a simpler fix is to preserve the pre-f19c0eccae96 behavior
for offline checksum changes: they are not propagated through WAL. In the second
repro, the standby therefore remains off, honoring its local offline change.
I wholeheartedly disagree, running a cluster with mismatched data_checksums
settings across the nodes is not a supported mode of operation, and is already
documented to not work (albeit it way too vague wording IMO). This doesn't
work as it is right now (in any version of postgres), pg_rewind or other file
based tools can break it, and we should not attempt to make it work.
Detecting a cluster with mismatched settings and safely erroring out as well as
improving the documentation is what I think we should do.
If this looks like too much code changes so close to the v19 release, another
option could be to remove pg_checksums --enable and --disable while keeping --check
and require checksum state changes to be done online.
That's also not a good option, I think we need to make sure offline enabling of
checksums *if done correctly* works as intended, and if done incorrectly errors
out safely.
I have a patch proposal brewing, and I know Zsolt has been looking into it as
well. Hopefully there will be something to share very soon.
--
Daniel Gustafsson
Hi,
On Fri, Aug 14, 2026 at 03:59:06PM +0200, Daniel Gustafsson wrote:
running a cluster with mismatched data_checksums
settings across the nodes is not a supported mode of operation, and is already
documented to not work
Thanks for feedback!
The pre f19c0eccae96 pg_checksums documentation said:
"
When using a replication setup with tools which perform direct copies
of relation file blocks (for example pg_rewind), enabling or disabling
checksums can lead to page corruptions in the shape of incorrect
checksums if the operation is not done consistently across all nodes.
"
I read that as a recommendation to stop and switch all nodes consistently, and
as a warning about direct block-copy tools. That interpretation, together with
the pre f19c0eccae96 behavior, is why v1 proposed keeping offline pg_checksums
changes local. The intention was to preserve the previous behavior, not to
introduce a new supported mode.
I just realized that f19c0eccae96 explicitly changed the "Off-line Enabling of
Checksums" documentation:
"
Data checksums are enabled or disabled at the full cluster level, and cannot
be specified individually for databases or tables.
"
by:
"
Data checksums are enabled or disabled at the full cluster level, and cannot
be specified individually for databases, tables or replicated cluster members.
"
while leaving the pg_checksums documentation quoted above unchanged.
Depending on how the issue will be addressed, it might be worth changing this
pg_checksums wording too?
Looking forward to seeing your and Zsolt's proposals.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com