toast table corrupted by vacuum - missing chunk number 0 for toast value

On Sun, Jul 12, 2026 at 08:48:28PM +0200, Pavel Stehule wrote:

Hi,

My customer reported corrupted toast tables on PostgreSQL 14.20. He uses wal
archiving - so it was possible to read data from broken pages. The corrupted
rows are detected by SELECT command. 

I would first look to see if it was fixed from 14.20 to 14.24.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Do not let urgent matters crowd out time for investment in the future.

Hi

čt 20. 8. 2026 v 18:52 odesílatel Bruce Momjian <bruce@momjian.us> napsal:

On Sun, Jul 12, 2026 at 08:48:28PM +0200, Pavel Stehule wrote:

Hi,

My customer reported corrupted toast tables on PostgreSQL 14.20. He uses

wal

archiving - so it was possible to read data from broken pages. The

corrupted

rows are detected by SELECT command.

I would first look to see if it was fixed from 14.20 to 14.24.

Unfortunately I am not able to check this. The customer reported this issue
only once. Now, he did upgrade, and he didn't report this issue again.
We didn't find any way how this issue can be reproduced. We manually
executed VACUUM on the database that was restored to the time close before
this issue - and the data was not broken - but now, I am not sure what was
the minor version that we used for this test (we did these tests a few
months after this issue).

I am almost sure this bug has ultra low frequency - but not zero - and
probably there are more different reasons with the same error message at
the end.

Regards

Pavel

Hi Pavel,

On Thu, Aug 20, 2026 at 9:56 PM Pavel Stehule wrote:

We didn't find any way how this issue can be reproduced. We manually
executed VACUUM on the database that was restored to the time close
before this issue - and the data was not broken - but now, I am not
sure what was the minor version that we used for this test.

I wonder whether the missing TOAST data could be a secondary symptom,
rather than evidence that VACUUM removed TOAST belonging to the current
row version.

I have seen incidents where an old row version became visible again.
Its TOAST data had already been removed legitimately, while the actual
current row version, including its TOAST data, could still be found by
primary key. Reading the resurrected version then failed with the same
missing-chunk error. As far as I recall, in at least one case the old
tuple was considered visible because of its hint bits even though the
transaction status did not agree.

The investigation attached to your original report selects the damaged
tuple directly by ctid. Could you check whether another version of the
same logical row exists, in particular one with the same primary key?
It would be useful to select only ctid, xmin, xmax, and the primary key,
so that inspecting the rows does not itself try to fetch TOAST data. I
would also try it with index and bitmap scans disabled.

The HEAP_UPDATED and HEAP_XMIN_COMMITTED flags on the reported tuple
make this possibility seem worth checking. If another version exists,
the pruning of the TOAST page may have been correct, and the actual
corruption may be that an old heap tuple became visible again. If the
relevant pg_xact data is still available in the restored instance, it
would also be interesting to compare the inserting transaction's status
with the hint bits on this tuple.

Thank you!

Best regards, Andrey Borodin.

On 7/12/26 20:48, Pavel Stehule wrote:

Hi,

My customer reported corrupted toast tables on PostgreSQL 14.20. He uses
wal archiving - so it was possible to read data from broken pages. The
corrupted rows are detected by SELECT command. 

My customer prepared two instances:  before corruption, after
corruption. There were more corrupted pages. All checks that I found
from extensions amcheck, pg_visibility are ok on instance before
corruption. Corrupted instances have problems with toast and toast
tables have problems with visibility map. Index on the toast table is ok.

The corrupted toast page contains four tuples. Related main table was
not updated. There was only one operation over the toast page - pruning
(forced by vacuum executed by cron).

WAL: 0000000100000D6E0000008A
rmgr: Heap2       len (rec/tot):     59/    99, tx:          0, lsn:
D6E/8A8A6AD0, prev D6E/8A8A6A68, desc: PRUNE latestRemovedXid 0
nredirected 0 ndead 4, blkref #0: rel 1663/4267629/16352341 blk 17880358 FPW

These vacuum doesn't reports any errors

2026-04-18 05:03:13 CEST [1688]: [46307-1] user=,db=,app=,client= LOG:
 checkpoint starting: time
2026-04-18 05:03:24 CEST [3760026]: [10-1]
user=postgres,db=jira_prod,app=vacuumdb,client=[local] LOG:  duration:
15037.899 ms  statement: VACUUM (VERBOSE, ANALYZE)
public."AO_544E33_AUDIT_LOG_ENTRY";
2026-04-18 05:03:30 CEST [3760026]: [11-1]
user=postgres,db=jira_prod,app=vacuumdb,client=[local] LOG:  duration:
5511.161 ms  statement: VACUUM (VERBOSE, ANALYZE) public.jiraaction;
2026-04-18 05:03:40 CEST [3760026]: [12-1]
user=postgres,db=jira_prod,app=vacuumdb,client=[local] LOG:  duration:
10052.323 ms  statement: VACUUM (VERBOSE, ANALYZE)
public."AO_C5D949_ISSUE_SLA";

After this operation, the toast page had only dead lp pointers and next
select fails on reported error.

Attached some data collected before and after corruption.

Can I collect some other data?

If you still have the backup + WAL, I'd start by doing two things:

(1) running pg_waldump on older WAL segments, to figure out when / how
was the page modified before VACUUM kicked in

(2) restoring the cluster to LSN shortly before D6E/8A8A6AD0

(3) dumping the page somehow (copying the 8K block, looking at it using
pageinspect, etc. ...)

(4) promote the instance, run a manual vacuum and see if it gets
corrupted again

If there are more corrupted pages, then do this for multiple of them,
and see if there's some similarity (e.g. all the corruption may be tied
to the same XID, or maybe it's multixacts, ...).

It'd also be good to get some additional info:

1) history of the cluster - How old is it? Did it start as PG14, or is
it an older cluster that went through pg_upgrade?

2) are there any extensions installed - especially third-party ones

regards

--
Tomas Vondra

Hi

pá 21. 8. 2026 v 12:31 odesílatel Andrey Borodin <x4mmm@yandex-team.ru>
napsal:

Hi Pavel,

On Thu, Aug 20, 2026 at 9:56 PM Pavel Stehule wrote:

We didn't find any way how this issue can be reproduced. We manually
executed VACUUM on the database that was restored to the time close
before this issue - and the data was not broken - but now, I am not
sure what was the minor version that we used for this test.

I wonder whether the missing TOAST data could be a secondary symptom,
rather than evidence that VACUUM removed TOAST belonging to the current
row version.

I have dumped pages

/messages/by-id/attachment/199738/investigation.txt

xmin, xmax was same on main table, and toast table. These rows was not
updated.

I have seen incidents where an old row version became visible again.
Its TOAST data had already been removed legitimately, while the actual
current row version, including its TOAST data, could still be found by
primary key. Reading the resurrected version then failed with the same
missing-chunk error. As far as I recall, in at least one case the old
tuple was considered visible because of its hint bits even though the
transaction status did not agree.

I don't think this is the same issue. rows in main table was visible by seq
scan

The investigation attached to your original report selects the damaged
tuple directly by ctid. Could you check whether another version of the
same logical row exists, in particular one with the same primary key?
It would be useful to select only ctid, xmin, xmax, and the primary key,
so that inspecting the rows does not itself try to fetch TOAST data. I
would also try it with index and bitmap scans disabled.

I didn't check this - but we used amcheck, and we didn't find any broken
indexes

The HEAP_UPDATED and HEAP_XMIN_COMMITTED flags on the reported tuple
make this possibility seem worth checking. If another version exists,
the pruning of the TOAST page may have been correct, and the actual
corruption may be that an old heap tuple became visible again. If the
relevant pg_xact data is still available in the restored instance, it
would also be interesting to compare the inserting transaction's status
with the hint bits on this tuple.

Unfortunately now it is too late and backups are rewritten :-/. So all
what I have is attached

/messages/by-id/CAFj8pRDcenNE3qUf=6YsrhMLki7BbbM8fLMi9DUzLn_jbLtpGw@mail.gmail.com

Regards

Pavel

Hi

pá 21. 8. 2026 v 13:01 odesílatel Tomas Vondra <tomas@vondra.me> napsal:

On 7/12/26 20:48, Pavel Stehule wrote:

Hi,

My customer reported corrupted toast tables on PostgreSQL 14.20. He uses
wal archiving - so it was possible to read data from broken pages. The
corrupted rows are detected by SELECT command.

My customer prepared two instances: before corruption, after
corruption. There were more corrupted pages. All checks that I found
from extensions amcheck, pg_visibility are ok on instance before
corruption. Corrupted instances have problems with toast and toast
tables have problems with visibility map. Index on the toast table is ok.

The corrupted toast page contains four tuples. Related main table was
not updated. There was only one operation over the toast page - pruning
(forced by vacuum executed by cron).

WAL: 0000000100000D6E0000008A
rmgr: Heap2 len (rec/tot): 59/ 99, tx: 0, lsn:
D6E/8A8A6AD0, prev D6E/8A8A6A68, desc: PRUNE latestRemovedXid 0
nredirected 0 ndead 4, blkref #0: rel 1663/4267629/16352341 blk 17880358

FPW

These vacuum doesn't reports any errors

2026-04-18 05:03:13 CEST [1688]: [46307-1] user=,db=,app=,client= LOG:
checkpoint starting: time
2026-04-18 05:03:24 CEST [3760026]: [10-1]
user=postgres,db=jira_prod,app=vacuumdb,client=[local] LOG: duration:
15037.899 ms statement: VACUUM (VERBOSE, ANALYZE)
public."AO_544E33_AUDIT_LOG_ENTRY";
2026-04-18 05:03:30 CEST [3760026]: [11-1]
user=postgres,db=jira_prod,app=vacuumdb,client=[local] LOG: duration:
5511.161 ms statement: VACUUM (VERBOSE, ANALYZE) public.jiraaction;
2026-04-18 05:03:40 CEST [3760026]: [12-1]
user=postgres,db=jira_prod,app=vacuumdb,client=[local] LOG: duration:
10052.323 ms statement: VACUUM (VERBOSE, ANALYZE)
public."AO_C5D949_ISSUE_SLA";

After this operation, the toast page had only dead lp pointers and next
select fails on reported error.

Attached some data collected before and after corruption.

Can I collect some other data?

If you still have the backup + WAL, I'd start by doing two things:

Unfortunately we do not have it - it is too late now.

(1) running pg_waldump on older WAL segments, to figure out when / how
was the page modified before VACUUM kicked in

We did it - and there was not any change.

This is only one record related to this page

WAL: 0000000100000D6E0000008A
rmgr: Heap2 len (rec/tot): 59/ 99, tx: 0, lsn:
D6E/8A8A6AD0, prev D6E/8A8A6A68, desc: PRUNE latestRemovedXid 0
nredirected 0 ndead 4, blkref #0: rel 1663/4267629/16352341 blk
17880358 FPW

looks so broken rows was not updated - xmin is same for main rows, and
toast rows

(2) restoring the cluster to LSN shortly before D6E/8A8A6AD0

(3) dumping the page somehow (copying the 8K block, looking at it using
pageinspect, etc. ...)

please, see
/messages/by-id/attachment/199738/investigation.txt

(4) promote the instance, run a manual vacuum and see if it gets
corrupted again

we did it - "without success" - table was not corrupted

If there are more corrupted pages, then do this for multiple of them,
and see if there's some similarity (e.g. all the corruption may be tied
to the same XID, or maybe it's multixacts, ...).

there was more corrupted rows, unfortunately we did check just one page

I have to ask Roman if amcheck reported other issues. I think so not, but I
am not sure

It'd also be good to get some additional info:

1) history of the cluster - How old is it? Did it start as PG14, or is
it an older cluster that went through pg_upgrade?

I'll forward these questions to Roman Sindelar - maintainer of the
corrupted database

Regards

Pavel

On 8/21/26 17:47, Pavel Stehule wrote:

Hi

pá 21. 8. 2026 v 12:31 odesílatel Andrey Borodin <x4mmm@yandex-team.ru
<mailto:x4mmm@yandex-team.ru>> napsal:

Hi Pavel,

On Thu, Aug 20, 2026 at 9:56 PM Pavel Stehule wrote:

We didn't find any way how this issue can be reproduced. We manually
executed VACUUM on the database that was restored to the time close
before this issue - and the data was not broken - but now, I am not
sure what was the minor version that we used for this test.

I wonder whether the missing TOAST data could be a secondary symptom,
rather than evidence that VACUUM removed TOAST belonging to the current
row version.

I have dumped pages

/messages/by-id/attachment/199738/
investigation.txt <https://www.postgresql.org/message-id/
attachment/199738/investigation.txt>

xmin, xmax was same on main table, and toast table. These rows was not
updated.

Thanks. Interesting. It's weird the TOAST rows have XID 2779428784 and
flags {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}, yet the
VACUUM apparently decided all of them are dead. Judging by:

rmgr: Heap2 len (rec/tot): 59/ 99, tx: 0, lsn:
D6E/8A8A6AD0, prev D6E/8A8A6A68, desc: PRUNE latestRemovedXid 0
nredirected 0 ndead 4, blkref #0: rel 1663/4267629/16352341 blk

17880358 FPW

I mean, ndead=4 seems to suggest that. But how could that be with
HEAP_XMIN_COMMITTED?

I wonder if the FPW from the WAL matches what you saw as corrupted, or
if it's correct (it should, given the ndead=4). But maybe try dumping it
using pg_waldump - although, 14 doesn't have that :-(.

What happened with the XID 2779428784? Did it commit sometime right
before the vacuum, or what? Judging by the page LSN, it must have been
pretty far back.

What does the clog say about this transaction? I think it should be:
pg_xact/0A5A, page 21, byte offset 175596, bits 0-1, so

$ od -An -tx1 -j 175596 -N 1 $PGDATA/pg_xact/0A5A

Also, what does the RUNNING_XACTS right before/after vacuum say?

--
Tomas Vondra

pá 21. 8. 2026 v 19:13 odesílatel Tomas Vondra <tomas@vondra.me> napsal:

On 8/21/26 17:47, Pavel Stehule wrote:

Hi

pá 21. 8. 2026 v 12:31 odesílatel Andrey Borodin <x4mmm@yandex-team.ru
<mailto:x4mmm@yandex-team.ru>> napsal:

Hi Pavel,

On Thu, Aug 20, 2026 at 9:56 PM Pavel Stehule wrote:

We didn't find any way how this issue can be reproduced. We

manually

executed VACUUM on the database that was restored to the time close
before this issue - and the data was not broken - but now, I am not
sure what was the minor version that we used for this test.

I wonder whether the missing TOAST data could be a secondary symptom,
rather than evidence that VACUUM removed TOAST belonging to the

current

row version.

I have dumped pages

/messages/by-id/attachment/199738/
investigation.txt <https://www.postgresql.org/message-id/
attachment/199738/investigation.txt>

xmin, xmax was same on main table, and toast table. These rows was not
updated.

Thanks. Interesting. It's weird the TOAST rows have XID 2779428784 and
flags {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}, yet the
VACUUM apparently decided all of them are dead. Judging by:

rmgr: Heap2 len (rec/tot): 59/ 99, tx: 0, lsn:
D6E/8A8A6AD0, prev D6E/8A8A6A68, desc: PRUNE latestRemovedXid 0
nredirected 0 ndead 4, blkref #0: rel 1663/4267629/16352341 blk

17880358 FPW

I mean, ndead=4 seems to suggest that. But how could that be with
HEAP_XMIN_COMMITTED?

I wonder if the FPW from the WAL matches what you saw as corrupted, or
if it's correct (it should, given the ndead=4). But maybe try dumping it
using pg_waldump - although, 14 doesn't have that :-(.

What happened with the XID 2779428784? Did it commit sometime right
before the vacuum, or what? Judging by the page LSN, it must have been
pretty far back.

What does the clog say about this transaction? I think it should be:
pg_xact/0A5A, page 21, byte offset 175596, bits 0-1, so

$ od -An -tx1 -j 175596 -N 1 $PGDATA/pg_xact/0A5A

Also, what does the RUNNING_XACTS right before/after vacuum say?

Unfortunately these backups was overwritten month ago

Regards

Pavel

On Sat, Aug 22, 2026 at 10:25 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:

Unfortunately these backups was overwritten month ago

I thought that your complaint might be related to a recently reported
issue with subxact corruption:

/messages/by-id/447185db-3f06-4a38-8518-ecbd42d8b7d5@gmail.com

That case involves a subtransaction that has already subcommitted
being forced to abort. This reminded me of what Tomas said about it
being weird that the TOAST table's entry appears to be for a
committing transaction, yet other evidence suggests that the same XID
aborted.

I asked Opus 5 to build a test case (based on one from Andrey Borodin
on that other thread) that produces symptoms consistent with Pavel's
report. It succeeded in doing so -- see the attached patch adding
these tests.

These tests fail, demonstrating the same symptoms (you might have to
comment-out an assertion in clog.c to allow the script to get as far
as executing the pageinspect steps that show those specific details).
With Bryan Green's bug fix patch from the other thread applied, the
tests all pass.

Of course, I'm not claiming that this is incontrovertible proof that
Pavel's case was an example of this bug causing corruption in the
field; other bugs have had similar symptoms. For example, the snapshot
export bug I'm working on [1]/messages/by-id/CAH2-WzmHVeYY=pjz9x8DhhxVjXHX0pvoQ-MdiB1Tt6=o2GTiKg@mail.gmail.com -- Peter Geoghegan can make committed subxacts temporarily
appear to have aborted, albeit on a standby (that one doesn't fit
Pavel's reported symptoms but isn't too far off). I still think this
explanation has a decent chance of being the correct one.

With that said, Pavel should look at this test case and try to find
any factor that might rule this explanation out. It would be worth
working out which ERROR actually happened. We simulate an OOM or other
error using the test's subxact-after-childxids-transfer injection
point; we don't know what ERROR actually occurred in the field (if we
assume this really was the bug involved, then some specific error must
have triggered it).

Question for Pavel: When I run the failing test, I see "WARNING:
AbortSubTransaction while in COMMIT state" in the test server logs. Do
any of the logs that you collected show that? If they do, then this
diagnosis is almost certainly correct.

[1]: /messages/by-id/CAH2-WzmHVeYY=pjz9x8DhhxVjXHX0pvoQ-MdiB1Tt6=o2GTiKg@mail.gmail.com -- Peter Geoghegan
--
Peter Geoghegan

ne 23. 8. 2026 v 1:59 odesílatel Peter Geoghegan <pg@bowt.ie> napsal:

On Sat, Aug 22, 2026 at 10:25 AM Pavel Stehule <pavel.stehule@gmail.com>
wrote:

Unfortunately these backups was overwritten month ago

I thought that your complaint might be related to a recently reported
issue with subxact corruption:

/messages/by-id/447185db-3f06-4a38-8518-ecbd42d8b7d5@gmail.com

That case involves a subtransaction that has already subcommitted
being forced to abort. This reminded me of what Tomas said about it
being weird that the TOAST table's entry appears to be for a
committing transaction, yet other evidence suggests that the same XID
aborted.

I asked Opus 5 to build a test case (based on one from Andrey Borodin
on that other thread) that produces symptoms consistent with Pavel's
report. It succeeded in doing so -- see the attached patch adding
these tests.

These tests fail, demonstrating the same symptoms (you might have to
comment-out an assertion in clog.c to allow the script to get as far
as executing the pageinspect steps that show those specific details).
With Bryan Green's bug fix patch from the other thread applied, the
tests all pass.

Of course, I'm not claiming that this is incontrovertible proof that
Pavel's case was an example of this bug causing corruption in the
field; other bugs have had similar symptoms. For example, the snapshot
export bug I'm working on [1] can make committed subxacts temporarily
appear to have aborted, albeit on a standby (that one doesn't fit
Pavel's reported symptoms but isn't too far off). I still think this
explanation has a decent chance of being the correct one.

With that said, Pavel should look at this test case and try to find
any factor that might rule this explanation out. It would be worth
working out which ERROR actually happened. We simulate an OOM or other
error using the test's subxact-after-childxids-transfer injection
point; we don't know what ERROR actually occurred in the field (if we
assume this really was the bug involved, then some specific error must
have triggered it).

Question for Pavel: When I run the failing test, I see "WARNING:
AbortSubTransaction while in COMMIT state" in the test server logs. Do
any of the logs that you collected show that? If they do, then this
diagnosis is almost certainly correct.

I'll ask the customer - now he it out of office, but I afraid so logs from
this time are already thrown

but I remember so on same server we found a vaccum errors "cannot to freeze
commtted xmax" - I don't know if this is related to this bug, but it was
few weeks after problem with broken TOAST

regards

Pavel

Regards

Pavel