[Patch] Fix pg_get_multixact_stats() over-reporting members on a hot standby

Started by Naga Appaniabout 1 month ago8 messageshackers
Beta feature

Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.

won't retrysuccessCI history

This thread has been committed, so CI has stopped here. Anything below is the last result it produced.

You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:

docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t253876
psql -h localhost -U postgres

Built from patchset v1 (message #1), September 23, 2026 at 04:55 AM.

Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:

git clone --branch t253876_1 https://github.com/hackorum-dev/postgres.git

In a checkout you already have, add the fork once:

git remote add hackorum https://github.com/hackorum-dev/postgres.git

then, for this patchset and every later one:

git fetch hackorum t253876_1 && git checkout t253876_1

Patchset v1 (message #1) is on t253876_1

Jump to latest
#1Naga Appani
nagnrik@gmail.com

Hi,

Splitting this out into its own thread for CommitFest tracking. It was
first discussed in [0]/messages/by-id/05d064e8-219d-40bd-9fdf-652b648ba317@iki.fi, where Heikki pointed out that
pg_get_multixact_stats() reports wrong member counts on a hot standby.

MultiXactState->oldestOffset is not maintained during recovery, so on a hot
standby it stays at 0 for the life of the server. pg_get_multixact_stats()
then computes num_members as nextOffset instead of nextOffset -
oldestOffset, and over-reports members and members_size. num_mxids and
oldest_multixact are correct.

I reproduced it on REL_19_BETA3 by building up multixacts on a primary,
setting up a standby, then advancing the primary's oldestOffset (VACUUM
FREEZE across all databases and a CHECKPOINT) and comparing
pg_get_multixact_stats() on both:

=== PRIMARY ===
num_mxids | num_members | members_size | oldest_multixact
-----------+-------------+--------------+------------------
0 | 0 | 0 | 409396

=== STANDBY (in recovery) ===
pg_is_in_recovery | num_mxids | num_members | members_size |
oldest_multixact
-------------------+-----------+-------------+--------------+------------------
t | 0 | 17563829 | 87819145 |
409396

The standby shows about 17.5M members / ~84 MB while the primary correctly
shows zero for the same data. Promoting the standby fixes it right away,
since the correct oldestOffset is only computed at end of recovery.

The attached patch updates MultiXactState->oldestOffset while replaying
XLOG_MULTIXACT_TRUNCATE_ID, as Heikki suggested, so the value stays current
during recovery. TrimMultiXact() still computes a fresh value at end of
recovery. With the patch, the same workload gives matching results on the
standby:

=== PRIMARY ===
num_mxids | num_members | members_size | oldest_multixact
-----------+-------------+--------------+------------------
0 | 0 | 0 | 397660

=== STANDBY (in recovery) ===
pg_is_in_recovery | num_mxids | num_members | members_size |
oldest_multixact
-------------------+-----------+-------------+--------------+------------------
t | 0 | 0 | 0 |
397660

I did not add the start-of-recovery initialization from the control file
Heikki mentioned [0]/messages/by-id/05d064e8-219d-40bd-9fdf-652b648ba317@iki.fi. With only the redo-path update there is still a short
window before the first truncation record is replayed where oldestOffset
reads 0. Handling that at startup needs the value without the
find_multixact_start() lookup, which asserts finishedStartup, so I left it
out for now.

I would love to help see this through. I may not be able to stay very
hands-on over the next while though, so if it needs follow-up and I am slow
to respond, please do not wait on me. I am happy for someone to build on it
or take a different approach, and glad to review and help where I can.

Patch attached.

[0]: /messages/by-id/05d064e8-219d-40bd-9fdf-652b648ba317@iki.fi

Best regards,
Naga Appani

Attachments:

t253876_1
v1-0001-Maintain-MultiXact-oldestOffset-during-recovery.patchapplication/octet-stream; name=v1-0001-Maintain-MultiXact-oldestOffset-during-recovery.patchDownload+9-1
#2Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Naga Appani (#1)
Re: [Patch] Fix pg_get_multixact_stats() over-reporting members on a hot standby

On 21/09/2026 07:43, Naga Appani wrote:

I did not add the start-of-recovery initialization from the control file
Heikki mentioned [0]. With only the redo-path update there is still a short
window before the first truncation record is replayed where oldestOffset
reads 0. Handling that at startup needs the value without the
find_multixact_start() lookup, which asserts finishedStartup, so I left it
out for now.

Hmm, the window could be very long, there's no guarantee that you'll see
a MULTIXACT_TRUNCATE_ID record any time soon. And after restart, it'll
revert back to 0 again. If we can't get a reasonable value at startup, I
wonder if we should just always show NULL in the standby.

- Heikki

#3Michael Paquier
michael@paquier.xyz
In reply to: Heikki Linnakangas (#2)
Re: [Patch] Fix pg_get_multixact_stats() over-reporting members on a hot standby

On Mon, Sep 21, 2026 at 10:19:47AM +0300, Heikki Linnakangas wrote:

Hmm, the window could be very long, there's no guarantee that you'll see a
MULTIXACT_TRUNCATE_ID record any time soon. And after restart, it'll revert
back to 0 again. If we can't get a reasonable value at startup, I wonder if
we should just always show NULL in the standby.

I'd like to think that we should just throw an ERROR in this case with
a ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE, similarly to a lot of the
WAL functions. NULL is useful when for example using such a function
with a JOIN on catalogs, when called for a lot of tuples, but that's
not the case here.
--
Michael

#4Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Michael Paquier (#3)
Re: [Patch] Fix pg_get_multixact_stats() over-reporting members on a hot standby

On 21/09/2026 10:30, Michael Paquier wrote:

On Mon, Sep 21, 2026 at 10:19:47AM +0300, Heikki Linnakangas wrote:

Hmm, the window could be very long, there's no guarantee that you'll see a
MULTIXACT_TRUNCATE_ID record any time soon. And after restart, it'll revert
back to 0 again. If we can't get a reasonable value at startup, I wonder if
we should just always show NULL in the standby.

I'd like to think that we should just throw an ERROR in this case with
a ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE, similarly to a lot of the
WAL functions. NULL is useful when for example using such a function
with a JOIN on catalogs, when called for a lot of tuples, but that's
not the case here.

That would disable pg_get_multixact_stats() in a standby altogether, but
only the members-related fields (num_members, members_size) have this
problem. So that would throw some babies out with the bathwater. It's a
new function so we could still do it though, if that's what we want.

Another idea is to add the oldestOffset field to the control file. I
actually find it a little weird that we don't have it there already. But
I'm uncomfortable making that change so late in the release cycle, just
for this function, in particular because it would require a catversion bump.

- Heikki

#5Naga Appani
nagnrik@gmail.com
In reply to: Heikki Linnakangas (#4)
Re: [Patch] Fix pg_get_multixact_stats() over-reporting members on a hot standby

On Mon, Sep 21, 2026 at 10:21 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:

On 21/09/2026 10:30, Michael Paquier wrote:

On Mon, Sep 21, 2026 at 10:19:47AM +0300, Heikki Linnakangas wrote:

Hmm, the window could be very long, there's no guarantee that you'll see a
MULTIXACT_TRUNCATE_ID record any time soon. And after restart, it'll revert
back to 0 again. If we can't get a reasonable value at startup, I wonder if
we should just always show NULL in the standby.

I'd like to think that we should just throw an ERROR in this case with
a ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE, similarly to a lot of the
WAL functions. NULL is useful when for example using such a function
with a JOIN on catalogs, when called for a lot of tuples, but that's
not the case here.

That would disable pg_get_multixact_stats() in a standby altogether, but
only the members-related fields (num_members, members_size) have this
problem. So that would throw some babies out with the bathwater. It's a
new function so we could still do it though, if that's what we want.

Another idea is to add the oldestOffset field to the control file. I
actually find it a little weird that we don't have it there already. But
I'm uncomfortable making that change so late in the release cycle, just
for this function, in particular because it would require a catversion bump.

- Heikki

Thanks both for looking into this!

I lean toward ERROR over NULL, but not strongly.

AFAICS none of this is really actionable on a standby either way, since
freezing and vacuum are only actionable on the primary. An error seems
more honest than returning numbers that look real but don't drive any
decision. Could be missing a use case though, please let me know what you
think.

Thanks,
Naga

#6Michael Paquier
michael@paquier.xyz
In reply to: Naga Appani (#5)
Re: [Patch] Fix pg_get_multixact_stats() over-reporting members on a hot standby

On Mon, Sep 21, 2026 at 10:55:02AM -0500, Naga Appani wrote:

AFAICS none of this is really actionable on a standby either way, since
freezing and vacuum are only actionable on the primary. An error seems
more honest than returning numbers that look real but don't drive any
decision. Could be missing a use case though, please let me know what you
think.

Yeah, I'd just choose an error at this stage rather than trying to
provide partially-true numbers. If somebody is interested in
providing these numbers on a standby, that could always be done later
on.
--
Michael

#7Michael Paquier
michael@paquier.xyz
In reply to: Naga Appani (#5)
Re: [Patch] Fix pg_get_multixact_stats() over-reporting members on a hot standby

On Mon, Sep 21, 2026 at 10:55:02AM -0500, Naga Appani wrote:

I lean toward ERROR over NULL, but not strongly.

I'll go switch that to an error if called while RecoveryInProgress(),
then. Bonus points regarding the fact that we can reuse some of the
existing error messages from xlogfuncs.c, meaning no extra translation
required.
--
Michael

#8Naga Appani
nagnrik@gmail.com
In reply to: Michael Paquier (#7)
Re: [Patch] Fix pg_get_multixact_stats() over-reporting members on a hot standby

On Wed, Sep 23, 2026 at 7:06 PM Michael Paquier <michael@paquier.xyz> wrote:

On Mon, Sep 21, 2026 at 10:55:02AM -0500, Naga Appani wrote:

I lean toward ERROR over NULL, but not strongly.

I'll go switch that to an error if called while RecoveryInProgress(),
then. Bonus points regarding the fact that we can reuse some of the
existing error messages from xlogfuncs.c, meaning no extra translation
required.
--
Michael

Thanks Michael, that matches what I had in v2 as well (same
errmsg/errhint pattern from xlogfuncs.c).

Thanks,
Naga