PostgreSQL 18.6/17.11: standby PANIC on restart after VM truncation

On Sun, 27 Sept 2026 at 11:08, Jacky Nguyen <nktpro@gmail.com> wrote:

Hi PostgreSQL team,

We encountered a standby startup PANIC after a switchover on PostgreSQL 18.6. A standalone reproducer and recovery TAP test reproduce it on 18.6, 17.11, REL_18_STABLE, and master; the same test passes on 18.4 and 17.10.

Expected: a standby that has replayed a heap/visibility-map truncation restarts recovery successfully.
Actual on restart:

LOG: redo starts at 0/3059738
WARNING: page 0 of relation base/5/16396_vm does not exist
PANIC: WAL contains references to invalid pages
LOG: startup process (PID 55286) was terminated by signal 6: Abort trap: 6

Reproduction: Build PostgreSQL with --enable-tap-tests --enable-injection-points and install injection_points. Copy the attached t/058_vm_truncate_invalid_pages.pl into src/test/recovery/t/, then run:

make -C src/test/recovery check PROVE_TESTS=t/058_vm_truncate_invalid_pages.pl

The test holds a standby restartpoint, promotes that standby, deletes and vacuums an all-visible table (truncating its heap and VM forks), rejoins the old primary as a standby, and restarts it. On 18.6, 20/20 TAP runs failed with this PANIC; on 18.4, 20/20 passed. A shell reproducer, exact steps, WAL excerpts, version matrix, and two candidate patches are included in the attachment.

The affected 18.6 build is REL_18_6 (724edf9); 17.11 is REL_17_11 (083ac03). The runs were on macOS 15 / Darwin 24.6.0, arm64, clang 21.1.8, with debug and injection points enabled and without cassert. The original symptom was on Linux in a three-node streaming setup. The analysis in REPORT.md points to VM block reads without an FPI during redo after an already replayed VM truncation; that diagnosis and the candidate fixes are provided for review.

Regards,
Jacky Nguyen

Hi!
Thanks for report. Looks like this bug existed since first VM commit,
which was already missing invalid page guard from [0]https://github.com/postgres/postgres/commit/defe93463c69f8e0bb717294a34d67c34ac0b03f.
So, case from [0]https://github.com/postgres/postgres/commit/defe93463c69f8e0bb717294a34d67c34ac0b03f reintroduces one we started to register VM pages.

Also I prefer candidate fix b from your email

[0]: https://github.com/postgres/postgres/commit/defe93463c69f8e0bb717294a34d67c34ac0b03f

--
Best regards,
Kirill Reshke

Hi Jacky and Kirill,

Melanie's v2-0002 in the VM clear thread [1]/messages/by-id/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0=+idehHw@mail.gmail.com is the same change as
candidate (a), and it fixes this report too. On master the attached test
fails without it and passes with it. The same change also passes on 17,
18 and 19.

This does not need a switchover. With full_page_writes = off, a plain
standby restart hits it if the restartpoint is before a DELETE or UPDATE
that cleared VM bits and a later VACUUM truncated the VM. So it would be
good to get v2-0002 in before the next minor release.

I asked Fable to create a test for me.

The attached clears VM bits by delete, HOT update
and cross-page update, and fails if any one of the three reads still uses
RBM_NORMAL.

Candidate (b) would also work, but once the read no longer uses
RBM_NORMAL, no invalid page is logged for the VM, so it is not needed.

Until the fix ships, ignore_invalid_pages = on lets the standby start. It
can be turned off after the next restartpoint.

[1]: /messages/by-id/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0=+idehHw@mail.gmail.com
/messages/by-id/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0=+idehHw@mail.gmail.com

Thanks,
Shihao

Hi,

On Mon, 28 Sep 2026, shihao zhong wrote:

Melanie's v2-0002 in the VM clear thread [1] is the same change as
candidate (a), and it fixes this report too.

I tested Melanie's v2 series as posted (0001-0003, which apply to
REL_19_STABLE) and candidate (a) on REL_18_STABLE, using the steps
from Shihao's test: full_page_writes = off and a plain standby
restart.  Both were debug builds with assertions, and the fix works:

- Unpatched, 18 and 19 fail to restart with "WAL contains references
  to invalid pages" after "page 0 of relation ..._vm does not exist"
  for each table.
- Patched, the standby restarts and its VM forks end up truncated
  again.
- On 18, reverting any one of the three RBM_ZERO_ON_ERROR changes
  brings the PANIC back, so the test covers each of them.

However, with wal_consistency_checking = all on the primary, the
patched standby still fails to restart:

  FATAL:  invalid page pd_lower 0 pd_upper 0 pd_special 0
  CONTEXT:  WAL redo at 0/03A80068 for Heap/DELETE: ...;
  blkref #1: rel 1663/5/16384, fork 2, blk 0 FPW

RBM_ZERO_ON_ERROR recreates the truncated VM page as all zeros.
verifyBackupPageConsistency() then masks it with heap_mask(), and
mask_unused_space() rejects a page with pd_lower 0.
heap_xlog_prune_freeze() and heap_xlog_multi_insert() already
initialize a VM page that was read as zeros, and doing the same at
the three VM clear sites fixes it.  The attached diff applies on top
of v2 on REL_19_STABLE; with it, the standby restarts with and
without wal_consistency_checking.  On 18, candidate (b) alone also
passes with wal_consistency_checking, since it never recreates the
page.

This only matters with wal_consistency_checking, but buildfarm
animals that use it could trip over Shihao's test once it's in.

Also, v2-0002 applies only on top of v2-0001 on REL_19_STABLE.  On
master it doesn't apply after master-v2-0001, 18 needs its own
version (candidate (a) applies there as is), and in 17 the redo code
is in heapam.c.

[1]: /messages/by-id/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0=+idehHw@mail.gmail.com

Regards,
Rahul Yadav

Hi Rahul,

On Thu, Oct 1, 2026, rahul@rhyadav.dev wrote:

However, with wal_consistency_checking = all on the primary, the
patched standby still fails to restart:

I can reproduce this on master, and your change fixes it. It matches
heap_xlog_prune_freeze(), and vm_readbuf() on the primary.

No posted version of the fix applies to master on its own, so here
is one as v3. I will add a CommitFest entry so this does not get lost
before the November releases.

0001 is Melanie's v2-0002 from [1]/messages/by-id/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0=+idehHw@mail.gmail.com. The changed lines and the commit
message are hers, only the context is adjusted for master. I left out
her v2-0001 and v2-0003, this report does not need them.

0002 is your diff. It had no commit message, so I put one together
from your mail. Please correct it if it is wrong.

0003 is my test. It now sets wal_consistency_checking = all, and it
takes a row lock first. Without the lock, the VM_NEW read in
heap_xlog_update() is not covered once 0002 is in.

The test fails without 0001. It also fails if any one of the three
RBM_ZERO_ON_ERROR or the three PageInit() calls is removed. That
holds on master, 19, 18 and 17. The nocfbot files hold the same three
patches for the back branches.

Melanie, if you would rather handle this in your thread, tell me and
I will close the entry.

[1]: /messages/by-id/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0=+idehHw@mail.gmail.com
/messages/by-id/CAAKRu_bApoksLDb-HX0GYciU3uLWqA1JagntaV8GP0=+idehHw@mail.gmail.com

Thanks,
Shihao

Hi Shihao,

On Fri, 2 Oct 2026, shihao zhong wrote:

0002 is your diff. It had no commit message, so I put one together
from your mail. Please correct it if it is wrong.

Thanks for putting v3 together.  The message is right for master and
19.  On 17 and 18 the function that already does this is
heap_xlog_visible(), so in those versions the sentence would be:

  heap_xlog_visible() already initializes a VM page that was read as
  zeros.

0002 could also carry the same "Backpatch-through: 17" as 0001.

I ran my reproducer, which follows the same steps as 0003, against v3
on master and the nocfbot versions on 19, 18 and 17.  Unpatched
master and 17 fail to restart with the invalid pages PANIC.  With v3
the standby restarts cleanly on all four branches, in each of these
setups:

  - full_page_writes = off
  - full_page_writes = off, wal_consistency_checking = all
  - full_page_writes = on, wal_consistency_checking = all

0003 itself passes here too on all four branches, and fails on
master without 0001 and 0002.

Regards,
Rahul Yadav

Hi Rahul,

Thanks for testing v3 on all four branches.

You are right about 17 and 18. I will change that sentence to
heap_xlog_visible() in those two versions, and add
"Backpatch-through: 17" to 0002. Both are commit message changes
only, so I will fold them into the next version and not post a v4
just for that.

I added a CommitFest entry so this is tracked for the November
releases.

https://commitfest.postgresql.org/patch/7386/

Thanks,
Shihao

Hi Shihao,

cfbot says v3 needs a rebase.  0001 and 0002 still apply to master,
but the test's number is now taken: 12f866cd5f and cc33773482 added
058_shutdown_crash_restart.pl and 059_remote_apply_status_interval.pl
to src/test/recovery.  With the test renamed to
060_vm_clear_truncate.pl, the series applies to master (16b94c4012)
and the test passes.

As you were going to fold the heap_xlog_visible() wording for 17 and
18 into the next version, this may be a good time for v4.

Regards,
Rahul Yadav

Oct 3, 2026, 05:13 by zhong950419@gmail.com:

Hi Rahul,

Thanks for checking. Here is v4, rebased on the current master.

The test is now 060_vm_clear_truncate.pl, on the back branches too.
No code changes.

0002 now has "Backpatch-through: 17". In the 17 and 18 files its
commit message names heap_xlog_visible(), as you pointed out.

Thanks,
Shihao

Jacky's original fix with XLogTruncateRelation() is the one I prefer.
I'm leaning toward sticking with RBM_NORMAL for heap_xlog_vm_clear()
as I don't see it halting recovery except for the truncation case, so
the natural solution to the truncation issue is to do what we do for
the main fork.

On Thu, Oct 8, 2026 at 11:13 PM shihao zhong <zhong950419@gmail.com> wrote:

The test is now 060_vm_clear_truncate.pl, on the back branches too.
No code changes.

Attached patch uses pg_visibility to truncate and then piggy-backs on
one of its existing tests.

- Melanie

Hi Melanie,

Jacky's original fix with XLogTruncateRelation() is the one I prefer.

Fine with me. I tested v5 on master, 19, 18 and 17. With the fix the
standby restarts.

One small thing. On 17 the test part does not apply, because
001_concurrent_transaction.pl has no standby there.

Thanks,
Shihao