Snapshot export on a standby corrupts hint bits on subxact overflow
I have heard periodic reports that indicate that pg_export_snapshot
can cause a standby to incorrectly set hint bits, which I suspected
was tied to subxact overflow. For example, Francisco Reinolds 2023 bug
report [1]/messages/by-id/17846-1a0e5ce976f4c01a@postgresql.org describes symptoms that match what I'd heard elsewhere. I
was reminded of this by a recent bug report from Scott Ray [2]/messages/by-id/QpAansP4iVg_ttSs9x81PFAptL2sqR3AS06u8Jksm3_bHJvUwQjHOocRajbxBc3iiLdf9ZMC6gtXjZsxdxvOmqp98hLZcZuWyBDuQxS6uZc=@scottray.io -- Peter Geoghegan that
also involves snapshot import/export. That turned out to be an
unrelated issue, though it looks legit.
I decided to reinvestigate the problem today, with help from Claude
code. I found a bug that exactly matches the known symptoms. Attached
patch 0001 has a reproducer + draft bug fix. This is likely a bug in
2017 commit 6c2003f8. There's also a second patch 0002 that fixes
another bug found along the way (though that's much less serious than
the one that 0001 deals with).
The test case in 0001 shows a scenario where pg_export_snapshot on a
standby hands out a snapshot that claims no transaction is running,
which is wrong. A backend that imports it writes wrong hint bits,
which are then seen by every other session on that standby --
including sessions that never touched the exported snapshot.
User-visible symptoms include rows reappearing after deletion, rows
vanishing after insertion, and duplicate entries in unique indexes
(all symptoms that I've personally seen in the wild). pg_dump -j can
hit this when run on a replica.
Recall that HeapTupleSatisfiesMVCC treats an XID that the snapshot
says isn't running, and that clog doesn't show as committed, as
aborted. That's how it's possible for a snapshot that shows no running
xacts to corrupt hint bits instead of just giving temporary wrong
answers (of course an FPI is also likely to mask the problem in
practice, I saw that myself when I investigated the problem a few
years back).
0002 is a separate bug of the same general nature.
pg_current_snapshot() copies the active snapshot's xip array, so on a
standby it returns a pg_snapshot claiming nothing is in progress, and
pg_visible_in_snapshot answers "visible" for transactions that are
still running:
primary pg_current_snapshot() = 696:698:696 visible(696) = f
standby pg_current_snapshot() = 696:698: visible(696) = t
This is what we see for transaction 696, which is still running (and
whose deleted row is correctly still visible on both nodes).
Both bug fixes were entirely authored by Claude, so treat them
skeptically. But I'm sure that at least the first one is a real bug.
[1]: /messages/by-id/17846-1a0e5ce976f4c01a@postgresql.org
[2]: /messages/by-id/QpAansP4iVg_ttSs9x81PFAptL2sqR3AS06u8Jksm3_bHJvUwQjHOocRajbxBc3iiLdf9ZMC6gtXjZsxdxvOmqp98hLZcZuWyBDuQxS6uZc=@scottray.io -- Peter Geoghegan
--
Peter Geoghegan
Attachments:
v1-0002-Read-the-in-progress-set-from-subxip-in-pg_curren.patchapplication/x-patch; name=v1-0002-Read-the-in-progress-set-from-subxip-in-pg_curren.patchDownload+30-3
v1-0001-Don-t-discard-subxip-when-exporting-a-snapshot-ta.patchapplication/x-patch; name=v1-0001-Don-t-discard-subxip-when-exporting-a-snapshot-ta.patchDownload+250-5
Hi,
On Sun, Jul 26, 2026 at 10:17:21PM -0400, Peter Geoghegan wrote:
I decided to reinvestigate the problem today, with help from Claude
code. I found a bug that exactly matches the known symptoms. Attached
patch 0001 has a reproducer + draft bug fix. This is likely a bug in
2017 commit 6c2003f8. There's also a second patch 0002 that fixes
another bug found along the way (though that's much less serious than
the one that 0001 deals with).
Thanks for having looked at it! The first problem is also something I worked
on in the past without success.
The test case in 0001 shows a scenario where pg_export_snapshot on a
standby hands out a snapshot that claims no transaction is running,
which is wrong. A backend that imports it writes wrong hint bits,
which are then seen by every other session on that standby --
including sessions that never touched the exported snapshot.
User-visible symptoms include rows reappearing after deletion, rows
vanishing after insertion, and duplicate entries in unique indexes
(all symptoms that I've personally seen in the wild).
The fix in 0001 makes sense to me.
Some comments:
=== 1
+ if (snapshot->subxcnt + nchildren > GetMaxSnapshotSubxidCount())
+ ereport(ERROR,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
Yes, this check is needed here.
+ for (int32 i = 0; i < snapshot->subxcnt; i++)
+ appendStringInfo(&buf, "sxp:%u\n", snapshot->subxip[i]);
+ for (int32 i = 0; i < nchildren; i++)
+ appendStringInfo(&buf, "sxp:%u\n", children[i]);
We count every existing subxip entry and every committed child without filtering
against snapshot->xmax. A committed child created after the imported snapshot
was taken can have an XID >= snapshot->xmax. I think that such an entry is
unnecessary for visibility, since XidInMVCCSnapshot() classifies every XID >= xmax
as still in progress before looking at subxip.
It can be observed by applying the attached post-xmax.txt on top of 0001, which
produces:
# BDT snapshot saved for post-promotion re-export 00000007-00000002-1: sof=1
# BDT re-exported snapshot 00000000-00000004-1: sof=1; xmax=781; sxp=[698, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 782]; sxp >= xmax=[782]
We can see that xmax=781, and that 782 has been exported.
That's not a visibility issue, however, those entries count toward GetMaxSnapshotSubxidCount()
and could therefore cause an avoidable PROGRAM_LIMIT_EXCEEDED.
=== 2
0002 is a separate bug of the same general nature.
+ if (cur->takenDuringRecovery)
+ {
+ nxip = cur->subxcnt;
+ xip = cur->subxip;
+ }
+ else
SnapshotData.subxip permits entries at or above xmax, whereas pg_snapshot.xip
requires every entry to satisfy xmin <= xip[i] < xmax.
So, I think filtering is appropriate in both places:
1/ In ExportSnapshot(), do not include recovery subxip entries and committed
child XIDs at or above xmax when counting and serializing them, so unnecessary
entries do not consume the limited recovery subxip capacity.
2/ In pg_current_snapshot(), do not include source XIDs outside [xmin, xmax),
so that it enforces the rule regardless of how the source snapshot was produced.
=== 3
0002 can now expose subtransaction IDs, so I think some comments:
"Note that only top-level transaction IDs are exposed to user sessions"
"Note that only top-transaction XIDs are included in the snapshot"
and the docs for pg_current_snapshot() and xip_list need updates to mention the
recovery specific exception.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com