BUG #19764: ManualBug3 Assert failure in pgstat_setup_shared_refs crashes initdb post-bootstrap subprocess

On 10/15/26 17:05, PG Bug reporting form wrote:

The following bug has been logged on the website:

Bug reference: 19764
Logged by: Yuelin Wang
Email address: 1217816127@qq.com
PostgreSQL version: 19rc1
Operating system: Linux (Ubuntu 24.04, x86_64)
Description:

### What happens?

During `initdb`, the "performing post-bootstrap initialization" subprocess
calls `pgstat_initialize()` (which sets `pgStatLocal.shmem->gc_request_count
= 1` via `StatsShmemInit`), then calls `StartupXLOG()`.
Inside `StartupXLOG()`, `pgstat_restore_stats()` is invoked.
After `pgstat_restore_stats()` completes, `gc_request_count` has been zeroed
out to 0 (verified via debug instrumentation in the working-tree build).
Later, during catalog initialization (`InitPostgres` calls
`table_beginscan_catalog`, which calls `heap_beginscan`, which calls
`pgstat_assoc_relation`), `pgstat_get_entry_ref` is invoked for the first
time in this subprocess.
This triggers `pgstat_setup_shared_refs()`
(src/backend/utils/activity/pgstat_shmem.c), which reads `pgStatSharedRefAge
= pg_atomic_read_u64(&pgStatLocal.shmem->gc_request_count)` and gets 0.
Immediately after, the function asserts `Assert(pgStatSharedRefAge != 0)` at
pgstat_shmem.c:939, which fires and aborts the subprocess, causing initdb to
fail and clean up the data directory.
The violated invariant is: `gc_request_count` must remain at its initialized
value (1) and never fall to 0, since `pgstat_setup_shared_refs` relies on a
non-zero value to guarantee `pgStatSharedRefAge != 0`.
The `own_hash` feature introduced in commit 44539786b8 added `entry_counts`,
`kind_hash_handles`, and `raw_kind_dsa_area` arrays to
`PgStat_ShmemControl`, altering the struct layout and the initialization
sequence in `pgstat_restore_stats`/`pgstat_read_statsfile`.
The debug trace confirms that `gc_request_count` transitions from 1 to 0
precisely during `pgstat_restore_stats()`.
This matches the Entry91 root-cause pattern: a field that must be
initialized to a sentinel non-zero value before use is inadvertently reset
to the uninitialized zero value by a code path that was modified by the same
recent commit.

### To Reproduce

Start a fresh PostgreSQL ASAN instance with initdb:

```sh
TMPDIR=$(mktemp -d /tmp/pg_scan_XXXX)
DATADIR=$TMPDIR/pgdata
PGUSER=$(whoami)

ASAN_OPTIONS=detect_leaks=0:abort_on_error=0:log_path=$TMPDIR/asan \
/data/ylwang/Projects/DirectedDB/latesttarget/postgre/build_asan/install/bin/initdb
\
-D $DATADIR -U $PGUSER --no-locale --encoding=UTF8 2>&1
```

Trigger output:

```
The files belonging to this database system will be owned by user
"yuelinwang".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default text search configuration will be set to "english".

Data page checksums are enabled.

creating directory .../pgdata ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Asia/Singapore
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... TRAP: failed
Assert("pgStatSharedRefAge != 0"), File: "pgstat_shmem.c", Line: 939, PID:
3146498
/lib/x86_64-linux-gnu/libasan.so.8(+0x831e0)[...]
.../postgres(ExceptionalCondition+0x165)[...]
.../postgres(+0x14c282a)[...]
.../postgres(pgstat_get_entry_ref+0x33b)[...]
.../postgres(pgstat_prep_pending_entry+0x37)[...]
.../postgres(+0x14b9ee9)[...]
.../postgres(pgstat_assoc_relation+0x1f7)[...]
.../postgres(+0x5a3a24)[...]
.../postgres(heap_beginscan+0x8c0)[...]
.../postgres(+0x6e832b)[...]
.../postgres(table_beginscan_catalog+0x9a)[...]
.../postgres(+0x1963b82)[...]
.../postgres(InitPostgres+0x63a)[...]
.../postgres(PostgresMain+0x553)[...]
.../postgres(main+0x5d5)[...]
Aborted (core dumped)
child process exited with exit code 134
initdb: removing data directory ".../pgdata"
```

The assertion failure prevents initdb from completing and the data directory
is cleaned up automatically.

I can't reproduce this. Please provide step-by-step instructions how to
trigger the issue.

I tried this:

CC=clang ./configure --prefix=/home/user/builds/asan \
--enable-debug --enable-cassert \
CFLAGS="-O0 -g -fno-omit-frame-pointer -fsanitize=address,undefined \
-fno-sanitize=function -fno-sanitize-recover=all" \
LDFLAGS="-fsanitize=address,undefined"

and then after build

export ASAN_OPTIONS=detect_leaks=0:abort_on_error=0
initdb tmp/data2 --no-locale --encoding=UTF8 -U test

No crash / assert failure. Furthermore, I fail to understand why would
this require ASAN build, if it crashes on an assert.

regards

--
Tomas Vondra

On 10/15/26 18:34, Tomas Vondra wrote:

On 10/15/26 17:05, PG Bug reporting form wrote:

The following bug has been logged on the website:

Bug reference: 19764
Logged by: Yuelin Wang
Email address: 1217816127@qq.com
PostgreSQL version: 19rc1
Operating system: Linux (Ubuntu 24.04, x86_64)
Description:

### What happens?

During `initdb`, the "performing post-bootstrap initialization" subprocess
calls `pgstat_initialize()` (which sets `pgStatLocal.shmem->gc_request_count
= 1` via `StatsShmemInit`), then calls `StartupXLOG()`.
Inside `StartupXLOG()`, `pgstat_restore_stats()` is invoked.
After `pgstat_restore_stats()` completes, `gc_request_count` has been zeroed
out to 0 (verified via debug instrumentation in the working-tree build).
Later, during catalog initialization (`InitPostgres` calls
`table_beginscan_catalog`, which calls `heap_beginscan`, which calls
`pgstat_assoc_relation`), `pgstat_get_entry_ref` is invoked for the first
time in this subprocess.
This triggers `pgstat_setup_shared_refs()`
(src/backend/utils/activity/pgstat_shmem.c), which reads `pgStatSharedRefAge
= pg_atomic_read_u64(&pgStatLocal.shmem->gc_request_count)` and gets 0.
Immediately after, the function asserts `Assert(pgStatSharedRefAge != 0)` at
pgstat_shmem.c:939, which fires and aborts the subprocess, causing initdb to
fail and clean up the data directory.
The violated invariant is: `gc_request_count` must remain at its initialized
value (1) and never fall to 0, since `pgstat_setup_shared_refs` relies on a
non-zero value to guarantee `pgStatSharedRefAge != 0`.
The `own_hash` feature introduced in commit 44539786b8 added `entry_counts`,
`kind_hash_handles`, and `raw_kind_dsa_area` arrays to
`PgStat_ShmemControl`, altering the struct layout and the initialization
sequence in `pgstat_restore_stats`/`pgstat_read_statsfile`.
The debug trace confirms that `gc_request_count` transitions from 1 to 0
precisely during `pgstat_restore_stats()`.
This matches the Entry91 root-cause pattern: a field that must be
initialized to a sentinel non-zero value before use is inadvertently reset
to the uninitialized zero value by a code path that was modified by the same
recent commit.

### To Reproduce

Start a fresh PostgreSQL ASAN instance with initdb:

```sh
TMPDIR=$(mktemp -d /tmp/pg_scan_XXXX)
DATADIR=$TMPDIR/pgdata
PGUSER=$(whoami)

ASAN_OPTIONS=detect_leaks=0:abort_on_error=0:log_path=$TMPDIR/asan \
/data/ylwang/Projects/DirectedDB/latesttarget/postgre/build_asan/install/bin/initdb
\
-D $DATADIR -U $PGUSER --no-locale --encoding=UTF8 2>&1
```

Trigger output:

```
The files belonging to this database system will be owned by user
"yuelinwang".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default text search configuration will be set to "english".

Data page checksums are enabled.

creating directory .../pgdata ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Asia/Singapore
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... TRAP: failed
Assert("pgStatSharedRefAge != 0"), File: "pgstat_shmem.c", Line: 939, PID:
3146498
/lib/x86_64-linux-gnu/libasan.so.8(+0x831e0)[...]
.../postgres(ExceptionalCondition+0x165)[...]
.../postgres(+0x14c282a)[...]
.../postgres(pgstat_get_entry_ref+0x33b)[...]
.../postgres(pgstat_prep_pending_entry+0x37)[...]
.../postgres(+0x14b9ee9)[...]
.../postgres(pgstat_assoc_relation+0x1f7)[...]
.../postgres(+0x5a3a24)[...]
.../postgres(heap_beginscan+0x8c0)[...]
.../postgres(+0x6e832b)[...]
.../postgres(table_beginscan_catalog+0x9a)[...]
.../postgres(+0x1963b82)[...]
.../postgres(InitPostgres+0x63a)[...]
.../postgres(PostgresMain+0x553)[...]
.../postgres(main+0x5d5)[...]
Aborted (core dumped)
child process exited with exit code 134
initdb: removing data directory ".../pgdata"
```

The assertion failure prevents initdb from completing and the data directory
is cleaned up automatically.

I can't reproduce this. Please provide step-by-step instructions how to
trigger the issue.

I tried this:

CC=clang ./configure --prefix=/home/user/builds/asan \
--enable-debug --enable-cassert \
CFLAGS="-O0 -g -fno-omit-frame-pointer -fsanitize=address,undefined \
-fno-sanitize=function -fno-sanitize-recover=all" \
LDFLAGS="-fsanitize=address,undefined"

and then after build

export ASAN_OPTIONS=detect_leaks=0:abort_on_error=0
initdb tmp/data2 --no-locale --encoding=UTF8 -U test

No crash / assert failure. Furthermore, I fail to understand why would
this require ASAN build, if it crashes on an assert.

I've done some more experiments, and I strongly suspect the AI tool
you're relying on simply works on a broken build. If I checkout and
build at 44539786b8^, and then update and do an incremental build at
44539786b8, I get very similar crashes:

----
TRAP: failed Assert("pgStatSharedRefAge != 0"), File:
"/tmp/pgstat_stale_tKJQ/postgres/build/../src/backend/utils/activity/pgstat_shmem.c",
Line: 505, PID: 498693
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(ExceptionalCondition+0x9e)
[0xbff043]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres() [0xa4660c]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(pgstat_get_entry_ref+0xc0)
[0xa469a0]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(pgstat_create_transactional+0x2e)
[0xa495ca]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(pgstat_create_relation+0x57)
[0xa42139]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(heap_create+0x2f1)
[0x5476d5]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(boot_yyparse+0x91a)
[0x531572]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(BootstrapModeMain+0x5e5)
[0x535cbb]
/tmp/pgstat_stale_tKJQ/postgres/build/install/bin/postgres(main+0x2eb)
[0x7a353c]
/lib64/libc.so.6(+0x35b5) [0x79ff8c2d45b5]
----

Your report has a slightly different backtrace, but it also says Line
939, and that line does not have that assert at all (there's a different
assert on line 938).

Thus my theory is - this is a broken incremental build, due to the AI
thing not using --enable-depend (or doing make clean). Chances are the
backtraces differ because of that too. And if that assessment is
correct, it means the whole elaborate "What happens?" section is likely
just a pile of hallucinated garbage.

I may be wrong, of course. An easy way to prove that is by providing a
working reproducer. (But I find that unlikely - the chance of us not
noticing a crash in initdb, which *every* test run does multiple times,
is rather low IMO.)

regards

--
Tomas Vondra

On Thu, Oct 15, 2026 at 09:36:24PM +0200, Tomas Vondra wrote:

I may be wrong, of course. An easy way to prove that is by providing a
working reproducer. (But I find that unlikely - the chance of us not
noticing a crash in initdb, which *every* test run does multiple times,
is rather low IMO.)

These code paths run under initdb in the buildfarm and the CI on a
daily basis hundreds of times or even more than that. Of course there
could be a bug, but I just suspect a false positive.
--
Michael

On Fri, Oct 16, 2026 at 1:00 PM Michael Paquier wrote:

These code paths run under initdb in the buildfarm and the CI on a
daily basis hundreds of times or even more than that. Of course
there could be a bug, but I just suspect a false positive.

I ran controlled builds to settle the clean vs incremental build question.

Clean builds (git clean -fdx; ./configure --enable-cassert --enable-debug
--enable-depend; make; make install), initdb run 50 times serially and 50 times
8-way parallel on each:

REL_19_RC1 (0b8bff042a), REL_19_STABLE (58d16be8ac), master (23e7ca78a0)
and 44539786b8: 400/400 OK, no assertion failure.

Incremental build as Tomas described (build at 44539786b8^, check out
44539786b8, make again, configured without --enable-depend): initdb fails
10/10 with

TRAP: failed Assert("pgStatSharedRefAge != 0"), File: "pgstat_shmem.c", Line: 505

The same incremental update with --enable-depend: 100/100 OK.

What goes wrong: 44539786b8 adds kind_hash_handles[] and raw_kind_dsa_area[]
to PgStat_ShmemControl ahead of is_shutdown, which moves gc_request_count from
offset 24 to 552. Without dependency tracking, only pgstat.o and pgstat_shmem.o
get rebuilt; the other pgstat_*.o keep the old layout. The stale
pgstat_archiver_reset_all_cb() still addresses archiver at shmem+288, so its
reset_offset covers 448..584, which now includes gc_request_count. When there
is no stats file, pgstat_restore_stats() -> pgstat_reset_after_failure() calls
that callback, it copies the (zero) stats over reset_offset, and
gc_request_count drops from 1 to 0. That is the sequence in the report; with gdb
I see gc_request_count = 1 on entry to pgstat_archiver_reset_all_cb and 0 in the
next pgstat_setup_shared_refs(). With a clean build the same callback runs and
the counter stays at 1.

Leaving only pgstat_archiver.o stale reproduces it 20/20. Rebuilding only
pgstat_archiver.o makes this assertion go away, but then the other stale objects
corrupt other parts of shared memory (Assert("list->head != procno") in
proclist.h), so a tree in that state isn't usable at all.

So I agree with Tomas and Michael: this is a build artifact, not a bug in 19rc1
or master. 44539786b8 is only on master, and in 19rc1 this assertion is at
pgstat_shmem.c:391, so the trace in the report seems to come from a different
tree than 19rc1.

Regards,
Manu

Sorry for the two-day delay in responding, and thank you for the detailed investigation.

I agree with your conclusion. The assertion failure was caused by my incremental build without dependency tracking, which left some `pgstat_*.o` files compiled against the old `PgStat_ShmemControl` layout. It was therefore a build artifact caused by stale object files, not a bug in PostgreSQL 19rc1 or master.

The line-number mismatch also confirms that the backtrace I reported did not come from the claimed REL_19_RC1 tree. I should have verified the exact source tree and reproduced the issue with a clean build before reporting it.

I apologize for the confusion and for the time spent investigating this. Please consider my original report withdrawn.

violin
1217816127@qq.com

Original

From: Manu <manuelreyesbravo@gmail.com&gt;
Date: 2026-10-17 15:05
To: pgsql-bugs <pgsql-bugs@lists.postgresql.org&gt;, Michael Paquier <michael@paquier.xyz&gt;, Tomas Vondra <tomas@vondra.me&gt;
Cc: 1217816127 <1217816127@qq.com&gt;, PG Bug reporting form <noreply@postgresql.org&gt;
Subject: Re: BUG #19764: ManualBug3 Assert failure in pgstat_setup_shared_refs crashes initdb

On&nbsp;Fri,&nbsp;Oct&nbsp;16,&nbsp;2026&nbsp;at&nbsp;1:00&nbsp;PM&nbsp;Michael&nbsp;Paquier&nbsp;wrote:
&gt;&nbsp;These&nbsp;code&nbsp;paths&nbsp;run&nbsp;under&nbsp;initdb&nbsp;in&nbsp;the&nbsp;buildfarm&nbsp;and&nbsp;the&nbsp;CI&nbsp;on&nbsp;a
&gt;&nbsp;daily&nbsp;basis&nbsp;hundreds&nbsp;of&nbsp;times&nbsp;or&nbsp;even&nbsp;more&nbsp;than&nbsp;that.&nbsp;&nbsp;Of&nbsp;course
&gt;&nbsp;there&nbsp;could&nbsp;be&nbsp;a&nbsp;bug,&nbsp;but&nbsp;I&nbsp;just&nbsp;suspect&nbsp;a&nbsp;false&nbsp;positive.

I&nbsp;ran&nbsp;controlled&nbsp;builds&nbsp;to&nbsp;settle&nbsp;the&nbsp;clean&nbsp;vs&nbsp;incremental&nbsp;build&nbsp;question.

Clean&nbsp;builds&nbsp;(git&nbsp;clean&nbsp;-fdx;&nbsp;./configure&nbsp;--enable-cassert&nbsp;--enable-debug
--enable-depend;&nbsp;make;&nbsp;make&nbsp;install),&nbsp;initdb&nbsp;run&nbsp;50&nbsp;times&nbsp;serially&nbsp;and&nbsp;50&nbsp;times
8-way&nbsp;parallel&nbsp;on&nbsp;each:

&nbsp;&nbsp;REL_19_RC1&nbsp;(0b8bff042a),&nbsp;REL_19_STABLE&nbsp;(58d16be8ac),&nbsp;master&nbsp;(23e7ca78a0)
&nbsp;&nbsp;and&nbsp;44539786b8:&nbsp;400/400&nbsp;OK,&nbsp;no&nbsp;assertion&nbsp;failure.

Incremental&nbsp;build&nbsp;as&nbsp;Tomas&nbsp;described&nbsp;(build&nbsp;at&nbsp;44539786b8^,&nbsp;check&nbsp;out
44539786b8,&nbsp;make&nbsp;again,&nbsp;configured&nbsp;without&nbsp;--enable-depend):&nbsp;initdb&nbsp;fails
10/10&nbsp;with

&nbsp;&nbsp;TRAP:&nbsp;failed&nbsp;Assert("pgStatSharedRefAge&nbsp;!=&nbsp;0"),&nbsp;File:&nbsp;"pgstat_shmem.c",&nbsp;Line:&nbsp;505

The&nbsp;same&nbsp;incremental&nbsp;update&nbsp;with&nbsp;--enable-depend:&nbsp;100/100&nbsp;OK.

What&nbsp;goes&nbsp;wrong:&nbsp;44539786b8&nbsp;adds&nbsp;kind_hash_handles[]&nbsp;and&nbsp;raw_kind_dsa_area[]
to&nbsp;PgStat_ShmemControl&nbsp;ahead&nbsp;of&nbsp;is_shutdown,&nbsp;which&nbsp;moves&nbsp;gc_request_count&nbsp;from
offset&nbsp;24&nbsp;to&nbsp;552.&nbsp;Without&nbsp;dependency&nbsp;tracking,&nbsp;only&nbsp;pgstat.o&nbsp;and&nbsp;pgstat_shmem.o
get&nbsp;rebuilt;&nbsp;the&nbsp;other&nbsp;pgstat_*.o&nbsp;keep&nbsp;the&nbsp;old&nbsp;layout.&nbsp;The&nbsp;stale
pgstat_archiver_reset_all_cb()&nbsp;still&nbsp;addresses&nbsp;archiver&nbsp;at&nbsp;shmem+288,&nbsp;so&nbsp;its
reset_offset&nbsp;covers&nbsp;448..584,&nbsp;which&nbsp;now&nbsp;includes&nbsp;gc_request_count.&nbsp;When&nbsp;there
is&nbsp;no&nbsp;stats&nbsp;file,&nbsp;pgstat_restore_stats()&nbsp;-&gt;&nbsp;pgstat_reset_after_failure()&nbsp;calls
that&nbsp;callback,&nbsp;it&nbsp;copies&nbsp;the&nbsp;(zero)&nbsp;stats&nbsp;over&nbsp;reset_offset,&nbsp;and
gc_request_count&nbsp;drops&nbsp;from&nbsp;1&nbsp;to&nbsp;0.&nbsp;That&nbsp;is&nbsp;the&nbsp;sequence&nbsp;in&nbsp;the&nbsp;report;&nbsp;with&nbsp;gdb
I&nbsp;see&nbsp;gc_request_count&nbsp;=&nbsp;1&nbsp;on&nbsp;entry&nbsp;to&nbsp;pgstat_archiver_reset_all_cb&nbsp;and&nbsp;0&nbsp;in&nbsp;the
next&nbsp;pgstat_setup_shared_refs().&nbsp;With&nbsp;a&nbsp;clean&nbsp;build&nbsp;the&nbsp;same&nbsp;callback&nbsp;runs&nbsp;and
the&nbsp;counter&nbsp;stays&nbsp;at&nbsp;1.

Leaving&nbsp;only&nbsp;pgstat_archiver.o&nbsp;stale&nbsp;reproduces&nbsp;it&nbsp;20/20.&nbsp;Rebuilding&nbsp;only
pgstat_archiver.o&nbsp;makes&nbsp;this&nbsp;assertion&nbsp;go&nbsp;away,&nbsp;but&nbsp;then&nbsp;the&nbsp;other&nbsp;stale&nbsp;objects
corrupt&nbsp;other&nbsp;parts&nbsp;of&nbsp;shared&nbsp;memory&nbsp;(Assert("list-&gt;head&nbsp;!=&nbsp;procno")&nbsp;in
proclist.h),&nbsp;so&nbsp;a&nbsp;tree&nbsp;in&nbsp;that&nbsp;state&nbsp;isn't&nbsp;usable&nbsp;at&nbsp;all.

So&nbsp;I&nbsp;agree&nbsp;with&nbsp;Tomas&nbsp;and&nbsp;Michael:&nbsp;this&nbsp;is&nbsp;a&nbsp;build&nbsp;artifact,&nbsp;not&nbsp;a&nbsp;bug&nbsp;in&nbsp;19rc1
or&nbsp;master.&nbsp;44539786b8&nbsp;is&nbsp;only&nbsp;on&nbsp;master,&nbsp;and&nbsp;in&nbsp;19rc1&nbsp;this&nbsp;assertion&nbsp;is&nbsp;at
pgstat_shmem.c:391,&nbsp;so&nbsp;the&nbsp;trace&nbsp;in&nbsp;the&nbsp;report&nbsp;seems&nbsp;to&nbsp;come&nbsp;from&nbsp;a&nbsp;different
tree&nbsp;than&nbsp;19rc1.

Regards,
Manu

On Fri, Oct 17, 2026 violin wrote:

I should have verified the exact source tree and reproduced the issue
with a clean build before reporting it. [...] Please consider my
original report withdrawn.

No need to apologize -- an incremental build without --enable-depend
leaving stale object files is an easy trap, and your report was detailed
enough (the exact call path and the gc_request_count invariant) that it
was quick to track down. --enable-depend, or a clean build, avoids it.
Thanks for following up and closing the loop.

Regards,
Manu