64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

Started by Noah Mischabout 2 months ago14 messageshackers
Beta feature

Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.

won't retrysuccessCI history

This thread has been committed, so CI has stopped here. Anything below is the last result it produced.

You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:

docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t253584
psql -h localhost -U postgres

Built from patchset v11 (message #11), September 23, 2026 at 06:55 PM.

Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:

git clone --branch t253584_11 https://github.com/hackorum-dev/postgres.git

In a checkout you already have, add the fork once:

git remote add hackorum https://github.com/hackorum-dev/postgres.git

then, for this patchset and every later one:

git fetch hackorum t253584_11 && git checkout t253584_11

Patchset v11 (message #11) is on t253584_11

Jump to latest
#1Noah Misch
noah@leadboat.com

commit bd8d9c9 wrote:

Commit: Heikki Linnakangas <heikki.linnakangas@iki.fi>
CommitDate: Tue Dec 9 13:53:03 2025 +0200

Widen MultiXactOffset to 64 bits

--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
+		nxtmulti = old_cluster.controldata.chkpnt_nxtmulti;
+		if (old_cluster.controldata.cat_ver >= MULTIXACT_FORMATCHANGE_CAT_VER)
+		{
+			/* Versions 9.3 - 18: convert all multixids  */
+			oldstMulti = old_cluster.controldata.chkpnt_oldstMulti;

If a cluster's upgrade history includes an upgrade from 9.3 to early 9.4, it
may have a wrong value here. Specifically, upgrades done before a61daa14
(2014-07 commit) have that hazard. We still have backend code to detect such
cases and reduce damage:

ereport(LOG,
(errmsg("cannot truncate up to MultiXact %u because it does not exist on disk, skipping truncation",
newOldestMulti)));

However, the pg_upgrade side from the v19 commit lacks such protection. If
heap tuples still reference older multixacts than the faulty control data
suggests, pg_upgrade will copy too small a range, making affected tuples
unreadable.

I also had Opus 4.8 look for defects in this change and write test cases. It
didn't find the above problem, but it had other findings. I'm attaching the
full report. I recommend fixing at least these before release:

+| 5 | `resetwal-nextmxoff-zero` | `src/bin/pg_resetwal/pg_resetwal.c:703`, and `-O` with no zero check at 297-307 | `pg_resetwal -f` (or `-O 0`) leaves `nextMultiOffset = 0`, the reserved "invalid offset"; first multixact created afterwards is permanently unreadable: `ERROR: MultiXact 1 has invalid offset` | Yes — `src/test/modules/test_slru/t/003_multixact_offset.pl` |

+| 7 | `members-truncation-apparent-wraparound` | `src/backend/access/transam/multixact.c:2642-2647` | When `nextOffset` lands on a members page boundary, truncation logs `could not truncate directory "pg_multixact/members": apparent wraparound` (impossible for a 64-bit counter) and reclaims nothing | Yes — `003_multixact_offset.pl` |

Others are more optional or already reported. In particular, two other
findings were already reported and apparently fixed after the Opus run, in
thread "pg_upgrade silently truncates nextMultiOffset to 32 bits":
/messages/by-id/CAD21AoCvzerscfU8o4ARQ793yAGHpQ72r2x5apeC_W2-k=SLCQ@mail.gmail.com

Attachments:

bd8d9c9-MultiXactOffset-tests-v0.patchtext/plain; charset=utf-8Download+853-1
#2Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Noah Misch (#1)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

On 28/08/2026 02:17, Noah Misch wrote:

commit bd8d9c9 wrote:

Commit: Heikki Linnakangas <heikki.linnakangas@iki.fi>
CommitDate: Tue Dec 9 13:53:03 2025 +0200

Widen MultiXactOffset to 64 bits

--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
+		nxtmulti = old_cluster.controldata.chkpnt_nxtmulti;
+		if (old_cluster.controldata.cat_ver >= MULTIXACT_FORMATCHANGE_CAT_VER)
+		{
+			/* Versions 9.3 - 18: convert all multixids  */
+			oldstMulti = old_cluster.controldata.chkpnt_oldstMulti;

If a cluster's upgrade history includes an upgrade from 9.3 to early 9.4, it
may have a wrong value here. Specifically, upgrades done before a61daa14
(2014-07 commit) have that hazard. We still have backend code to detect such
cases and reduce damage:

ereport(LOG,
(errmsg("cannot truncate up to MultiXact %u because it does not exist on disk, skipping truncation",
newOldestMulti)));

However, the pg_upgrade side from the v19 commit lacks such protection. If
heap tuples still reference older multixacts than the faulty control data
suggests, pg_upgrade will copy too small a range, making affected tuples
unreadable.

Thanks, I'll look into this. My first reaction is that I think if
oldstMulti is incorrectly too old, the upgrade will fail because the
conversion routine will fail to find it. If it's too new, i.e. "in the
future", it will also fail to find it.

The third possibility is that the bogus oldstMulti value is within the
range of the "real" range. That can happen if multixid wraparound had
already happened before the (broken) 9.3 -> 9.4 upgrade. In that case,
even if the multixids are still readable on disk, you're one vacuum away
from truncating them. In other words, the damage has already been done,
or could be done at any minute. That's the scenario for which the commit
message 78db307bb2 says "this mechanism cannot save us".

But I'll do some testing of that. Let's ensure that the error message
makes sense at the very least.

I also had Opus 4.8 look for defects in this change and write test cases. It
didn't find the above problem, but it had other findings. I'm attaching the
full report. I recommend fixing at least these before release:

+| 5 | `resetwal-nextmxoff-zero` | `src/bin/pg_resetwal/pg_resetwal.c:703`, and `-O` with no zero check at 297-307 | `pg_resetwal -f` (or `-O 0`) leaves `nextMultiOffset = 0`, the reserved "invalid offset"; first multixact created afterwards is permanently unreadable: `ERROR: MultiXact 1 has invalid offset` | Yes — `src/test/modules/test_slru/t/003_multixact_offset.pl` |

This was already fixed in commit 2cad308cb8.

+| 7 | `members-truncation-apparent-wraparound` | `src/backend/access/transam/multixact.c:2642-2647` | When `nextOffset` lands on a members page boundary, truncation logs `could not truncate directory "pg_multixact/members": apparent wraparound` (impossible for a 64-bit counter) and reclaims nothing | Yes — `003_multixact_offset.pl` |

Ok, I'll look into this one, and the other ones that might not be fixed yet.

- Heikki

#3Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Noah Misch (#1)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

(adding Michael and Naga)

I'd actually attribute the below issue to commit 97b101776c "Add
pg_get_multixact_stats()". In a nutshell, MultiXactState->oldestOffset
is not updated during recovery, and that was true before widening
multixacts to 64 bits already. That's intentional and wasn't visible to
users until the pg_get_multixact_stats() function was added. If you run
pg_get_multixact_stats() in a standby, the 'num_members' and
'members_size' are calculated incorrectly, because oldestOffset is always 0.

To fix, we could update MultiXactState->oldestOffset during recovery
too, from the control file at start of recovery, and whenever we see a
XLOG_MULTIXACT_TRUNCATE_ID record. We should probably still calculate a
fresh value at end of recovery, like we currently do.

Full LLM report extracted from Noah's markdown file below:

On 28/08/2026 02:17, Noah Misch wrote:

### 3.3 `standby-oldestoffset-zero-stats` — `pg_get_multixact_stats()` on a hot standby reports lifetime members, not retained members

**Location:** `src/backend/access/transam/multixact.c:2545-2559` and its SQL consumer
`src/backend/utils/adt/multixactfuncs.c:127-130`.

```
2545 void
2546 GetMultiXactInfo(uint32 *multixacts, MultiXactOffset *nextOffset,
2547 MultiXactId *oldestMultiXactId, MultiXactOffset *oldestOffset)
2548 {
2549 MultiXactId nextMultiXactId;
2550
2551 LWLockAcquire(MultiXactGenLock, LW_SHARED);
2552 *nextOffset = MultiXactState->nextOffset;
2553 *oldestMultiXactId = MultiXactState->oldestMultiXactId;
2554 nextMultiXactId = MultiXactState->nextMXact;
2555 *oldestOffset = MultiXactState->oldestOffset;
2556 LWLockRelease(MultiXactGenLock);
```

```
127 GetMultiXactInfo(&multixacts, &nextOffset, &oldestMultiXactId, &oldestOffset);
128 members = nextOffset - oldestOffset;
129 membersBytes = MultiXactOffsetStorageSize(nextOffset, oldestOffset);
```

**Mechanism.** `MultiXactState->oldestOffset` is assigned in exactly two places:
`SetOldestOffset()` at `multixact.c:2488`, and `TruncateMultiXact()` at `multixact.c:2788`. Neither
runs in recovery:

* `TruncateMultiXact()` opens with `Assert(!RecoveryInProgress());` (`multixact.c:2685`).
* `SetOldestOffset()` is called only from `SetMultiXactIdLimit()` at `multixact.c:2173`, which is
*after* the early return at `multixact.c:2161-2162`:
`if (!MultiXactState->finishedStartup) return;`. `finishedStartup` is set only by
`TrimMultiXact()` (`multixact.c:2006`), whose sole caller is `xlog.c:6527`, at end of recovery.
* The redo path (`multixact.c:2985-2990`) calls `SetMultiXactIdLimit(xlrec.oldestMulti, …)` — which
correctly stores `oldestMultiXactId` at `multixact.c:2140` before the early return — and then
`PerformMembersTruncation(xlrec.oldestOffset)` / `PerformOffsetsTruncation()`, using the WAL
record's value directly. It never stores `xlrec.oldestOffset` into shared memory.

So on a never-promoted standby, `MultiXactState->oldestOffset` keeps its zeroed shmem value for the
life of the server while `nextOffset` is advanced continuously by redo. `num_members` therefore comes
out as `nextOffset - 0` and `members_size` as `5 * nextOffset` bytes
(`MultiXactOffsetStorageSize()`, `multixact_internal.h:125-134`: `MULTIXACT_MEMBERGROUP_SIZE /
MULTIXACT_MEMBERS_PER_MEMBERGROUP` = 20/4 = 5). `num_mxids` and `oldest_multixact` are correct.

The overreport is exactly the primary's current `oldestOffset`, and since bd8d9c9 made offsets
64-bit and monotonic, that quantity climbs for the life of the installation. Meanwhile redo *does*
physically delete the members segments, so the standby reports gigabytes of members it does not have.

**Trigger (reproduced).** Primary with `wal_level = replica`; create a table, take `FOR SHARE` from
concurrent sessions to build multixacts; `pg_basebackup -R` a standby and start it; on the primary
`ALTER DATABASE template0 ALLOW_CONNECTIONS true`, `VACUUM FREEZE` in every database, `CHECKPOINT`,
so `TruncateMultiXact()` advances `oldestOffset`. Observed: primary `num_members` 0, `members_size` 0;
standby `num_members` 3, `members_size` 15, `pg_is_in_recovery()` true. A larger run (1000 multixacts
of 2 members, `nextOffset` 2001) showed the standby reporting 2001/10005 against a primary reporting
0/0. Promoting the standby corrects it immediately, confirming the `finishedStartup` mechanism.

**User-visible consequence.** `SELECT * FROM pg_get_multixact_stats()` on a hot standby contradicts
the same query on its primary for byte-identical data, and contradicts
`doc/src/sgml/func/func-info.sgml:3010-3013`, which defines `num_members` as "the total number of
multixact member entries **currently present in the system**" and `members_size` as "the storage
occupied by `num_members` in the `pg_multixact/members` directory". Monitoring built on this function
sees a standby that appears to be holding hundreds of GB of members it does not hold.

**Attribution — stated honestly.** This is the weakest attribution in the set, and it survived
2/3 refutation votes rather than 3/3. The dissenting reading is correct on the facts and should be
recorded: the `finishedStartup` gate and the redo path are **identical in `bd8d9c9^`**, so the
standby's stuck `oldestOffset` predates the commit; pre-commit it was harmless because the only
consumer, `MultiXactMemberFreezeThreshold()`, never runs in recovery. What bd8d9c9 contributed is
deleting `MultiXactStateData.oldestOffsetKnown` and changing `GetMultiXactInfo()` from returning
`bool` to `void` — it used to short-circuit with `*members = 0; … return false;` so callers could
distinguish "unknown" from "zero". `97b10177` (2025-12-30) then wired the now-unqualified value into
SQL with no `RecoveryInProgress()` guard and no NULL path. A committer may reasonably file this
against `97b10177`; the missing "unknown" signal is bd8d9c9's.

**Why it is not covered by a test.** Testing it needs a full streaming primary/standby pair plus the
`template0`-connectable `VACUUM FREEZE` dance to move the primary's `oldestOffset` off zero — that is
a `src/test/recovery`-style test, not something that fits in `test_slru`'s single-node harness or in
`007_multixact_conversion.pl`. It also cannot be asserted until the intended semantics are decided:
should the standby return NULL for these two columns, or should redo store `xlrec.oldestOffset`?
Writing a test first would bake in whichever answer the test author guessed.

**Test someone should write** (in `src/test/recovery/t/`, once the semantics are settled):

```
init primary (wal_level=replica), create t, produce N multixacts via concurrent FOR SHARE
pg_basebackup -R -> standby, start, wait for catchup
on primary: ALTER DATABASE template0 ALLOW_CONNECTIONS true; VACUUM FREEZE in all dbs; CHECKPOINT
wait_for_catchup
is(standby: SELECT num_members FROM pg_get_multixact_stats(),
primary: same) # or: expect NULL on the standby
promote standby; re-run; expect it to match
```

The middle assertion is the whole test; it fails today with standby = `nextOffset`,
primary = `nextOffset - oldestOffset`.

---

- Heikki

#4Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Noah Misch (#1)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

On 28/08/2026 02:17, Noah Misch wrote:

Others are more optional or already reported. In particular, two other
findings were already reported and apparently fixed after the Opus run, in
thread "pg_upgrade silently truncates nextMultiOffset to 32 bits":
/messages/by-id/CAD21AoCvzerscfU8o4ARQ793yAGHpQ72r2x5apeC_W2-k=SLCQ@mail.gmail.com

For the documentation issues, I propose the attached.

- Heikki

Attachments:

t253584_4
0001-Fix-documentation-related-to-widening-multixid-offse.patchtext/x-patch; charset=UTF-8; name=0001-Fix-documentation-related-to-widening-multixid-offse.patchDownload+8-12
#5Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Heikki Linnakangas (#2)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

On 16/09/2026 14:33, Heikki Linnakangas wrote:

On 28/08/2026 02:17, Noah Misch wrote:

commit bd8d9c9 wrote:

Commit:     Heikki Linnakangas <heikki.linnakangas@iki.fi>
CommitDate: Tue Dec 9 13:53:03 2025 +0200

     Widen MultiXactOffset to 64 bits

--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
+        nxtmulti = old_cluster.controldata.chkpnt_nxtmulti;
+        if (old_cluster.controldata.cat_ver >= 
MULTIXACT_FORMATCHANGE_CAT_VER)
+        {
+            /* Versions 9.3 - 18: convert all multixids  */
+            oldstMulti = old_cluster.controldata.chkpnt_oldstMulti;

If a cluster's upgrade history includes an upgrade from 9.3 to early
9.4, it
may have a wrong value here.  Specifically, upgrades done before a61daa14
(2014-07 commit) have that hazard.  We still have backend code to
detect such
cases and reduce damage:

        ereport(LOG,
                (errmsg("cannot truncate up to MultiXact %u because it
does not exist on disk, skipping truncation",
                        newOldestMulti)));

However, the pg_upgrade side from the v19 commit lacks such
protection.  If
heap tuples still reference older multixacts than the faulty control data
suggests, pg_upgrade will copy too small a range, making affected tuples
unreadable.

Thanks, I'll look into this. My first reaction is that I think if
oldstMulti is incorrectly too old, the upgrade will fail because the
conversion routine will fail to find it. If it's too new, i.e. "in the
future", it will also fail to find it.

The third possibility is that the bogus oldstMulti value is within the
range of the "real" range. That can happen if multixid wraparound had
already happened before the (broken) 9.3 -> 9.4 upgrade. In that case,
even if the multixids are still readable on disk, you're one vacuum away
from truncating them. In other words, the damage has already been done,
or could be done at any minute. That's the scenario for which the commit
message 78db307bb2 says "this mechanism cannot save us".

But I'll do some testing of that. Let's ensure that the error message
makes sense at the very least.

Ok, I reconstructed that scenario artificially with the attached script.
It creates a new cluster, runs a test program to create a bunch of
multixids, runs vacuum freeze to truncate the multixids SLRU and advance
oldestMulti. Finally, it uses pg_resetwal to reset oldestMulti back to
1, to simulate what you'd get with the buggy 9.3 pg_upgrade. I tested
this with a v18 source cluster, but I don't think the source version
matters. (It must've been pg_upgraded to 9.3 in the past, but it
could've gone through more pg_upgrades since)

When you run pg_upgrade on that, it fails like this:

Converting pg_multixact files
could not open file "/tmp/pgcluster/pg_multixact/offsets/0000": No such file or directory

Not an ideal error message, but at least you get an error rather than
silently losing data.

With the attached patch, you get an error message crafted specifically
for that case.

Converting pg_multixact files fatal

Segment file containing multixid 1 does not exist.
This can happen if an old version of pg_upgrade was used in the past
to upgrade the cluster from version 9.3 to 9.4. If that is the cause,
it can be fixed by running VACUUM FREEZE before upgrade.
Failure, exiting

I'm not sure this is worth committing though. How many clusters are
there out there that have been pg_upgraded to version 9.3.0-9.3.4, and
haven't since been running for long enough to fix the oldestMultiXid
value? Having that error message seems pretty harmless too, but if you
actually hit that case that the file is missing, it might actually be
more likely that you've hit some other bug than that old pg_upgrade issue.

Another approach would be to ignore the oldestMulti in the control file
and scan pg_multixact/offsets for the oldest segment that actually
exists. But I don't like that. Unless you actually hit that old
pg_upgrade bug, a missing file is a sign of trouble, and it's better
that you get alerted about it than upgrading the broken cluster to a new
version.

- Heikki

Attachments:

t253584_5
init_multixids.shapplication/x-shellscript; name=init_multixids.shDownload
generate_multixids.pytext/x-python; charset=UTF-8; name=generate_multixids.pyDownload
0001-Give-a-special-error-if-multixid-1-is-missing-on-pg_.patchtext/x-patch; charset=UTF-8; name=0001-Give-a-special-error-if-multixid-1-is-missing-on-pg_.patchDownload+82-23
#6Noah Misch
noah@leadboat.com
In reply to: Heikki Linnakangas (#5)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

On Thu, Sep 17, 2026 at 06:25:58PM +0300, Heikki Linnakangas wrote:

On 16/09/2026 14:33, Heikki Linnakangas wrote:

On 28/08/2026 02:17, Noah Misch wrote:

commit bd8d9c9 wrote:

Commit:     Heikki Linnakangas <heikki.linnakangas@iki.fi>
CommitDate: Tue Dec 9 13:53:03 2025 +0200

     Widen MultiXactOffset to 64 bits

--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
+        nxtmulti = old_cluster.controldata.chkpnt_nxtmulti;
+        if (old_cluster.controldata.cat_ver >=
MULTIXACT_FORMATCHANGE_CAT_VER)
+        {
+            /* Versions 9.3 - 18: convert all multixids  */
+            oldstMulti = old_cluster.controldata.chkpnt_oldstMulti;

If a cluster's upgrade history includes an upgrade from 9.3 to early
9.4, it
may have a wrong value here.  Specifically, upgrades done before a61daa14
(2014-07 commit) have that hazard.  We still have backend code to
detect such
cases and reduce damage:

        ereport(LOG,
                (errmsg("cannot truncate up to MultiXact %u because
it does not exist on disk, skipping truncation",
                        newOldestMulti)));

However, the pg_upgrade side from the v19 commit lacks such
protection.  If
heap tuples still reference older multixacts than the faulty control data
suggests, pg_upgrade will copy too small a range, making affected tuples
unreadable.

Thanks, I'll look into this. My first reaction is that I think if
oldstMulti is incorrectly too old, the upgrade will fail because the
conversion routine will fail to find it. If it's too new, i.e. "in the
future", it will also fail to find it.

Agreed, those cases fail cleanly. No concerns on those.

The third possibility is that the bogus oldstMulti value is within the
range of the "real" range.

That's the only scenario I felt was at risk. For example, the range of MXIDs
in tuple headers is [(unsigned)-100M,200M], but oldestMulti is 1.

That can happen if multixid wraparound had
already happened before the (broken) 9.3 -> 9.4 upgrade. In that case,
even if the multixids are still readable on disk, you're one vacuum away
from truncating them. In other words, the damage has already been done,
or could be done at any minute.

Okay. I briefly tried to verify how rapidly one can expect v18 to truncate
the SLRUs in this scenario, but I didn't get very far. I will rely on your
analysis. Thanks for studying it.

With the attached patch, you get an error message crafted specifically for
that case.

Converting pg_multixact files fatal

Segment file containing multixid 1 does not exist.
This can happen if an old version of pg_upgrade was used in the past
to upgrade the cluster from version 9.3 to 9.4. If that is the cause,
it can be fixed by running VACUUM FREEZE before upgrade.
Failure, exiting

I'm not sure this is worth committing though. How many clusters are there
out there that have been pg_upgraded to version 9.3.0-9.3.4, and haven't
since been running for long enough to fix the oldestMultiXid value? Having
that error message seems pretty harmless too, but if you actually hit that
case that the file is missing, it might actually be more likely that you've
hit some other bug than that old pg_upgrade issue.

I wouldn't commit the more-specific error message. If it did something for
the "third possibility", that might be different. I think we can consider
closed the topic of how 9.3 upgrades interact with 64-bit MultiXactOffset.

#7Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Noah Misch (#6)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

On 17/09/2026 19:21, Noah Misch wrote:

On Thu, Sep 17, 2026 at 06:25:58PM +0300, Heikki Linnakangas wrote:

On 16/09/2026 14:33, Heikki Linnakangas wrote:

On 28/08/2026 02:17, Noah Misch wrote:

commit bd8d9c9 wrote:

Commit:     Heikki Linnakangas <heikki.linnakangas@iki.fi>
CommitDate: Tue Dec 9 13:53:03 2025 +0200

     Widen MultiXactOffset to 64 bits

--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
+        nxtmulti = old_cluster.controldata.chkpnt_nxtmulti;
+        if (old_cluster.controldata.cat_ver >=
MULTIXACT_FORMATCHANGE_CAT_VER)
+        {
+            /* Versions 9.3 - 18: convert all multixids  */
+            oldstMulti = old_cluster.controldata.chkpnt_oldstMulti;

If a cluster's upgrade history includes an upgrade from 9.3 to early
9.4, it
may have a wrong value here.  Specifically, upgrades done before a61daa14
(2014-07 commit) have that hazard.  We still have backend code to
detect such
cases and reduce damage:

        ereport(LOG,
                (errmsg("cannot truncate up to MultiXact %u because
it does not exist on disk, skipping truncation",
                        newOldestMulti)));

However, the pg_upgrade side from the v19 commit lacks such
protection.  If
heap tuples still reference older multixacts than the faulty control data
suggests, pg_upgrade will copy too small a range, making affected tuples
unreadable.

Thanks, I'll look into this. My first reaction is that I think if
oldstMulti is incorrectly too old, the upgrade will fail because the
conversion routine will fail to find it. If it's too new, i.e. "in the
future", it will also fail to find it.

Agreed, those cases fail cleanly. No concerns on those.

The third possibility is that the bogus oldstMulti value is within the
range of the "real" range.

That's the only scenario I felt was at risk. For example, the range of MXIDs
in tuple headers is [(unsigned)-100M,200M], but oldestMulti is 1.

That can happen if multixid wraparound had
already happened before the (broken) 9.3 -> 9.4 upgrade. In that case,
even if the multixids are still readable on disk, you're one vacuum away
from truncating them. In other words, the damage has already been done,
or could be done at any minute.

Okay. I briefly tried to verify how rapidly one can expect v18 to truncate
the SLRUs in this scenario, but I didn't get very far. I will rely on your
analysis. Thanks for studying it.

Thinking about this some more, you were right originally, there is
danger here:

If you're in that situation, where oldestMulti is higher than
min(datminmxid), VACUUM will _not_ truncate the slrus. That was the
whole idea of commit 78db307bb2, to refrain from truncating. So you're
not in much danger, unless you consume so many multixids that you start
to overwrite the tail.

But v19 pg_upgrade would indeed effectively truncate them away, by not
converting them to the new format.

A better fix here is to add a pre-check that oldestMulti <
min(datminmxid). That seems like a good sanity check anyway, and it will
catch that ancient 9.3 scenario.

Barring objections, I'll commit the attached to add that check. I didn't
include a hint about that 9.3 upgrade bug, since I don't think you're
very likely to hit that anymore. But it's good to have the check to
prevent damage if you do manage to hit it, or some other bug.

- Heikki

Attachments:

t253584_7
0001-Check-that-oldestXID-and-oldestMulti-are-consistent-.patchtext/x-patch; charset=UTF-8; name=0001-Check-that-oldestXID-and-oldestMulti-are-consistent-.patchDownload+97-32
#8Naga Appani
nagnrik@gmail.com
In reply to: Heikki Linnakangas (#3)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

On Wed, Sep 16, 2026 at 8:37 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:

(adding Michael and Naga)

I'd actually attribute the below issue to commit 97b101776c "Add
pg_get_multixact_stats()". In a nutshell, MultiXactState->oldestOffset
is not updated during recovery, and that was true before widening
multixacts to 64 bits already. That's intentional and wasn't visible to
users until the pg_get_multixact_stats() function was added. If you run
pg_get_multixact_stats() in a standby, the 'num_members' and
'members_size' are calculated incorrectly, because oldestOffset is always 0.

To fix, we could update MultiXactState->oldestOffset during recovery
too, from the control file at start of recovery, and whenever we see a
XLOG_MULTIXACT_TRUNCATE_ID record. We should probably still calculate a
fresh value at end of recovery, like we currently do.

Hi Heikki, Noah,

Thanks for the detailed analysis and for looping me in, and sorry for the
delayed response. This is about the pg_get_multixact_stats() standby issue
above, not the 9.3 pg_upgrade topic the thread has since moved to.

I reproduced it on a primary/standby pair built from REL_19_BETA3. On a
primary with wal_level=replica I generated a lot of multixacts with pgbench
running SELECT ... FOR SHARE (100 clients), took a pg_basebackup -R standby
and started it (still in recovery), then on the primary ran VACUUM FREEZE
in all databases (including template0) and a CHECKPOINT to move oldestOffset
forward.

Results (unpatched REL_19_BETA3)
--------------------------------
=== PRIMARY ===
num_mxids | num_members | members_size | oldest_multixact
-----------+-------------+--------------+------------------
0 | 0 | 0 | 409396

=== STANDBY (in recovery) ===
pg_is_in_recovery | num_mxids | num_members | members_size |
oldest_multixact
-------------------+-----------+-------------+--------------+------------------
t | 0 | 17563829 | 87819145 |
409396

=== STANDBY after promote ===
pg_is_in_recovery | num_mxids | num_members | members_size |
oldest_multixact
-------------------+-----------+-------------+--------------+------------------
f | 0 | 0 | 0 |
409396

The standby shows about 17.5M members / ~84 MB while the primary correctly
shows zero for the same data. oldestOffset is stuck at 0 on the standby, so
num_members comes out as nextOffset instead of nextOffset minus
oldestOffset. Promoting the standby fixes it right away, since the correct
oldestOffset is only computed at end of recovery.

Fix
---
Following your suggestion, the attached patch updates
MultiXactState->oldestOffset while replaying XLOG_MULTIXACT_TRUNCATE_ID, so
the value stays current during recovery. TrimMultiXact() still computes a
fresh value at end of recovery.

With the patch, the same workload gives matching results on the standby:

=== PRIMARY ===
num_mxids | num_members | members_size | oldest_multixact
-----------+-------------+--------------+------------------
0 | 0 | 0 | 397660

=== STANDBY (in recovery) ===
pg_is_in_recovery | num_mxids | num_members | members_size |
oldest_multixact
-------------------+-----------+-------------+--------------+------------------
t | 0 | 0 | 0 |
397660

I did not add the start-of-recovery initialization from the control file
you mentioned. With only the redo-path update there is still a short window
before the first truncation record is replayed where oldestOffset reads 0.
Handling that at startup needs the value without the find_multixact_start()
lookup, which asserts finishedStartup, so I left it out for now.

I would love to help see this through. I may not be able to stay very
hands-on over the next while though, so if it needs follow-up and I am slow
to respond, please do not wait on me. I am happy for someone to build on it
or take a different approach, and glad to review and help where I can.

Thank you!

Best regards,
Naga Appani

Attachments:

t253584_8
v1-0001-Maintain-MultiXact-oldestOffset-during-recovery.patchapplication/octet-stream; name=v1-0001-Maintain-MultiXact-oldestOffset-during-recovery.patchDownload+9-1
#9Noah Misch
noah@leadboat.com
In reply to: Heikki Linnakangas (#7)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

On Fri, Sep 18, 2026 at 09:15:38PM +0300, Heikki Linnakangas wrote:

From 866999c1251fbbde2f1d9db2799734c7c6e8fe54 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date: Fri, 18 Sep 2026 20:57:26 +0300
Subject: [PATCH 1/1] Check that oldestXID and oldestMulti are consistent at
pg_upgrade

I read this patch. It looks reasonable, and I expect it provides the intended
defense. Thank you.

#10Tom Lane
tgl@sss.pgh.pa.us
In reply to: Noah Misch (#9)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

Noah Misch <noah@leadboat.com> writes:

On Fri, Sep 18, 2026 at 09:15:38PM +0300, Heikki Linnakangas wrote:

From 866999c1251fbbde2f1d9db2799734c7c6e8fe54 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date: Fri, 18 Sep 2026 20:57:26 +0300
Subject: [PATCH 1/1] Check that oldestXID and oldestMulti are consistent at
pg_upgrade

I read this patch. It looks reasonable, and I expect it provides the intended
defense. Thank you.

The BF animals that run cross-version upgrade tests say that this
fails when upgrading from 9.2 [1]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&amp;dt=2026-09-19%2014%3A17%3A23:

Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database connection settings ok
Checking for unsupported encodings ok
Checking names of databases, roles, and tablespaces ok
Checking database user is the install user ok
Checking for prepared transactions ok
Checking oldestXID and oldestMultiXid consistency SQL command failed
SELECT datname, datfrozenxid, datminmxid FROM pg_catalog.pg_database
ERROR: column "datminmxid" does not exist
LINE 1: SELECT datname, datfrozenxid, datminmxid FROM pg_catalog.pg_...
^
Failure, exiting

Now, we've dropped support for upgrading from 9.2 in HEAD, but we
have not done so in v19, so I think this needs a fix in v19.
If nothing else works, we could just skip the whole check when source
server is <= 9.2.

(I don't think this is worth breaking beta4 release freeze for, but
it should get fixed afterwards.)

regards, tom lane

[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&amp;dt=2026-09-19%2014%3A17%3A23

#11Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Tom Lane (#10)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

On 20/09/2026 05:12, Tom Lane wrote:

Noah Misch <noah@leadboat.com> writes:

On Fri, Sep 18, 2026 at 09:15:38PM +0300, Heikki Linnakangas wrote:

From 866999c1251fbbde2f1d9db2799734c7c6e8fe54 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date: Fri, 18 Sep 2026 20:57:26 +0300
Subject: [PATCH 1/1] Check that oldestXID and oldestMulti are consistent at
pg_upgrade

I read this patch. It looks reasonable, and I expect it provides the intended
defense. Thank you.

The BF animals that run cross-version upgrade tests say that this
fails when upgrading from 9.2 [1]:

Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database connection settings ok
Checking for unsupported encodings ok
Checking names of databases, roles, and tablespaces ok
Checking database user is the install user ok
Checking for prepared transactions ok
Checking oldestXID and oldestMultiXid consistency SQL command failed
SELECT datname, datfrozenxid, datminmxid FROM pg_catalog.pg_database
ERROR: column "datminmxid" does not exist
LINE 1: SELECT datname, datfrozenxid, datminmxid FROM pg_catalog.pg_...
^
Failure, exiting

Now, we've dropped support for upgrading from 9.2 in HEAD, but we
have not done so in v19, so I think this needs a fix in v19.
If nothing else works, we could just skip the whole check when source
server is <= 9.2.

Ugh, that's what I get for not actually testing with a real 9.2
installation..

Here's what I came up with for this. Thanks Andrew for also reporting
this off-list; this is essentially what you proposed too.

(I don't think this is worth breaking beta4 release freeze for, but
it should get fixed afterwards.)

I'm actually inclined to push this now, I don't see much risk of causing
more last-minute breakage. Isn't finding this kind of last-minute issues
exactly the reason we have the freeze? But that's just a weak opinion,
it also seems totally fine to me to leave this broken for beta4 and fix
later.

- Heikki

Attachments:

t253584_11
0001-Fix-pg_upgrade-from-version-9.2.patchtext/x-patch; charset=UTF-8; name=0001-Fix-pg_upgrade-from-version-9.2.patchDownload+9-1
#12Andrew Dunstan
andrew@dunslane.net
In reply to: Heikki Linnakangas (#11)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

On 2026-09-20 Su 8:59 AM, Heikki Linnakangas wrote:

Here's what I came up with for this. Thanks Andrew for also reporting
this off-list; this is essentially what you proposed too.

(I don't think this is worth breaking beta4 release freeze for, but
it should get fixed afterwards.)

I'm actually inclined to push this now, I don't see much risk of
causing more last-minute breakage. Isn't finding this kind of
last-minute issues exactly the reason we have the freeze? But that's
just a weak opinion, it also seems totally fine to me to leave this
broken for beta4 and fix later.

Hi Heikki,

Code freeze is over :-) Please push so the buildfarm can go green again.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

#13Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Andrew Dunstan (#12)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

On 23/09/2026 23:12, Andrew Dunstan wrote:

On 2026-09-20 Su 8:59 AM, Heikki Linnakangas wrote:

Here's what I came up with for this. Thanks Andrew for also reporting
this off-list; this is essentially what you proposed too.

(I don't think this is worth breaking beta4 release freeze for, but
it should get fixed afterwards.)

I'm actually inclined to push this now, I don't see much risk of
causing more last-minute breakage. Isn't finding this kind of last-
minute issues exactly the reason we have the freeze? But that's just a
weak opinion, it also seems totally fine to me to leave this broken
for beta4 and fix later.

Hi Heikki,

Code freeze is over :-) Please push so the buildfarm can go green again.

Pushed, thanks for the reminder!

- Heikki

#14Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Heikki Linnakangas (#4)
Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

On 16/09/2026 19:49, Heikki Linnakangas wrote:

On 28/08/2026 02:17, Noah Misch wrote:

Others are more optional or already reported.  In particular, two other
findings were already reported and apparently fixed after the Opus
run, in
thread "pg_upgrade silently truncates nextMultiOffset to 32 bits":
https://www.postgresql.org/message-id/
CAD21AoCvzerscfU8o4ARQ793yAGHpQ72r2x5apeC_W2-k%3DSLCQ%40mail.gmail.com

For the documentation issues, I propose the attached.

I committed this with minor rewording, and some related comment fixes
(commit b69356cd78). With that, all the issues from this thread have
been addressed; I'll mark the Open Item as resolved.

- Heikki