pgstat: allow a stats kind to use its own dedicated dsa/dshash

Started by Sami Imseih4 months ago24 messageshackers
Jump to latest
#1Sami Imseih
samimseih@gmail.com

Hi,

As I have been working on [1]/messages/by-id/CAA5RZ0vZwR_dSK6fo0P2-EnskUVN0NjLHnGnJMFDPC8-kEW3sQ@mail.gmail.com in which pg_stat_statements can use
a custom cumulative stats kind, I realized it will be beneficial for a
kind to have a dedicated dsa area/dshash in which to store their stats.

A kind like pg_stat_statements will have thousands of entries and
likely will have more entries than all other kinds combined. If combined
in the same dsa/dshash with everything else, sequential scans looking
for pg_stat_statements entries will have to skip over all other kinds,
and vice versa. A dedicated hash also reduces lock contention on
unrelated kinds, particularly for garbage-collection.

There is also a need for custom kinds to independently size their own
dsa areas, so they can manage them carefully. If we want to set limits
on how much space the entries can use for a kind like pg_stat_statements,
we need to be able to set the size directly on its dsa.

The attached proposal implements a new stats kind option, own_hash, which
when set to true allocates an independent dsa/dshash for that stats kind.

Also, 2 new external APIs are introduced:
- pgstat_get_hash_for_kind(PgStat_Kind kind)
- pgstat_get_dsa_for_kind(PgStat_Kind kind)
These allow the extension to retrieve the dshash and dsa pointers for
both iterating through the entries and for setting the size limit on
the dsa. Otherwise, the management of these separate structures is
transparent to the extension and managed by the pgstat infrastructure.

These APIs are necessary because we don't have higher-level abstractions
in pgstats to do things like "iterate through this kind" or "set size
limit for this stats kind"; such abstractions may be good to have to
avoid using dshash_seq_* or dsa_set_size_limit directly, but I am not
sure they are worth it at this point.

Only variable-length stats can opt into own_hash. Fixed-length allocates their
data into PgStat_ShmemControl.

[1]: /messages/by-id/CAA5RZ0vZwR_dSK6fo0P2-EnskUVN0NjLHnGnJMFDPC8-kEW3sQ@mail.gmail.com

--
Sami Imseih
Amazon Web Services (AWS)

Attachments:

t253087_1
v1-0001-pgstat-allow-a-stats-kind-to-use-its-own-dedicate.patchapplication/octet-stream; name=v1-0001-pgstat-allow-a-stats-kind-to-use-its-own-dedicate.patchDownload+545-252
#2Bertrand Drouvot
bertranddrouvot.pg@gmail.com
In reply to: Sami Imseih (#1)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

Hi,

On Wed, Jul 15, 2026 at 12:16:26PM -0500, Sami Imseih wrote:

The attached proposal implements a new stats kind option, own_hash, which
when set to true allocates an independent dsa/dshash for that stats kind.

Thanks for the patch!

The dedicated hash idea makes sense to me.

A few comments:

=== 1

- shhashent = dshash_find_or_insert(pgStatLocal.shared_hash, &key, &shfound);
+ shhashent = dshash_find_or_insert(hash, &key, &shfound);

With the patch, an extension can set a size limit on its dedicated DSA, making
an allocation failure much easier to reach. Such a failure in dshash_find_or_insert()
would raise ERROR, leaving the backend local cache entry with a NULL shared_entry.

A later pgstat_gc_entry_refs() call could then dereference the NULL
entry_ref->shared_entry pointer.

I wonder if this should use dshash_find_or_insert_extended(..., DSHASH_INSERT_NO_OOM)
and release the local entry if it returns NULL?

=== 2

@@ -2080,12 +2090,12 @@ pgstat_read_statsfile(void)
                     * putting all stats into checkpointer's
                     * pgStatEntryRefHash would be wasted effort and memory.
                     */
-                   p = dshash_find_or_insert(pgStatLocal.shared_hash, &key, &found);
+                   p = dshash_find_or_insert(pgstat_get_hash_for_kind(key.kind), &key, &found);

If the dedicated DSA cannot accommodate the persisted entries, dshash_find_or_insert()
raises ERROR and startup can fail.

I wonder if this case should discard the statistics that no longer fit rather
than prevent the server from starting?

=== 3

+   /* Add per-kind DSA space for own_hash kinds */
+   for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
+   {
+       const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);

I wonder if it wouldn't make more sense for own_hash to create a dedicated dshash
in the existing pgstat DSA, with a separate option for a dedicated DSA?

That would retain independent iteration and hash partition locks while avoiding
an additional shared memory allocation and per backend DSA state when independent
memory accounting is not needed.

=== 4

+ * Returns the DSA area for a given kind.  Kinds with own_hash set have
+ * a dedicated DSA; others use the shared DSA.
+ */
+static inline dsa_area *
+pgstat_get_dsa_for_kind(PgStat_Kind kind)
+{
+   if (pgStatLocal.kind_dsa[kind] != NULL)
+       return pgStatLocal.kind_dsa[kind];
+

Current master provides dshash_get_dsa_area() since 762e329e83f, so this one
looks now redundant.

=== 5

pgstat_reset_matching_entries(bool (*do_reset) (PgStatShared_HashEntry *, Datum),
@@ -1173,26 +1282,29 @@ pgstat_reset_matching_entries(bool (*do_reset) (PgStatShared_HashEntry *, Datum)
PgStatShared_HashEntry *p;

    /* dshash entry is not modified, take shared lock */
-   dshash_seq_init(&hstat, pgStatLocal.shared_hash, false);
-   while ((p = dshash_seq_next(&hstat)) != NULL)
+   for (int h = 0; h < pgStatLocal.num_hashes; h++)
    {
-       PgStatShared_Common *header;
+       dshash_seq_init(&hstat, pgStatLocal.all_hashes[h], false);
+       while ((p = dshash_seq_next(&hstat)) != NULL)
+       {

pgstat_reset_entries_of_kind() uses this routine, so resetting one kind still
scans every hash, including unrelated dedicated hashes.

Could pgstat_reset_entries_of_kind() scan only the hash returned by
pgstat_get_hash_for_kind(kind)? The kind filter would still be needed when
that returns the shared hash.

=== 6

-   /* Register custom statistics kind */
-   pgstat_register_kind(PGSTAT_KIND_TEST_CUSTOM_VAR_STATS, &custom_stats);
+   /* Must be loaded via shared_preload_libraries */
+   if (!process_shared_preload_libraries_in_progress)
+       return;
+

This restores the behavior removed by 5045d9ff3b5. The SQL functions remain
callable with the kind unregistered.

I think the branch should be removed so pgstat_register_kind() reports the
intended prerequisite error.

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

#3Michael Paquier
michael@paquier.xyz
In reply to: Sami Imseih (#1)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

On Wed, Jul 15, 2026 at 12:16:26PM -0500, Sami Imseih wrote:

Also, 2 new external APIs are introduced:
- pgstat_get_hash_for_kind(PgStat_Kind kind)
- pgstat_get_dsa_for_kind(PgStat_Kind kind)
These allow the extension to retrieve the dshash and dsa pointers for
both iterating through the entries and for setting the size limit on
the dsa. Otherwise, the management of these separate structures is
transparent to the extension and managed by the pgstat infrastructure.

These APIs are necessary because we don't have higher-level abstractions
in pgstats to do things like "iterate through this kind" or "set size
limit for this stats kind"; such abstractions may be good to have to
avoid using dshash_seq_* or dsa_set_size_limit directly, but I am not
sure they are worth it at this point.

Only variable-length stats can opt into own_hash. Fixed-length allocates their
data into PgStat_ShmemControl.

Please note that this patch does not apply anymore. Could you rebase,
please?
--
Michael

#4Sami Imseih
samimseih.pg@gmail.com
In reply to: Michael Paquier (#3)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

Please note that this patch does not apply anymore. Could you rebase,
please?

Done.

I also addressed some of Bertrand's points.

=== 1

I wonder if this should use
dshash_find_or_insert_extended(..., DSHASH_INSERT_NO_OOM)
and release the local entry if it returns NULL?

Done. The insertion paths now use
dshash_find_or_insert_extended(..., DSHASH_INSERT_NO_OOM). In
pgstat_get_entry_ref(), the local entry is released if insertion returns NULL.

=== 4

Current master provides dshash_get_dsa_area() since 762e329e83f, so this one
looks now redundant.

Done. pgstat_get_dsa_for_kind() now uses

dshash_get_dsa_area(pgstat_get_hash_for_kind(kind))

=== 5

Could pgstat_reset_entries_of_kind() scan only the hash returned by
pgstat_get_hash_for_kind(kind)?

Done. I added pgstat_reset_matching_entries_in_hash().
pgstat_reset_matching_entries() still scans all hashes by calling that helper
for each hash. pgstat_reset_entries_of_kind() now scans only
pgstat_get_hash_for_kind(kind), instead of scanning all hashes and filtering by
kind.

=== 6

I think the branch should be removed so pgstat_register_kind() reports the
intended prerequisite error.

Done.

For the other points.

=== 2

I wonder if this case should discard the statistics that no longer fit rather
than prevent the server from starting?

I left this unchanged for now. Discarding persisted stats instead of failing
startup changes the failure semantics, so I think this needs more discussion.
I also think the existing behavior is sane: either the stats file is loaded, or
it is discarded as a whole.

=== 3

That would retain independent iteration and hash partition locks while avoiding
an additional shared memory allocation and per backend DSA state when independent
memory accounting is not needed.

I kept own_hash tied to a dedicated DSA for now. I am not sure we would want to
support that mix and match behavior. A stats kind that asks for its own hash is
already asking to be isolated from the shared stats table, and using a dedicated
DSA keeps the hash storage and entry storage isolated in the same way.

I have not thought of a good use case for splitting those controls, but perhaps
there is one.

Attached is v2

--
Sami Imseih
Amazon Web Services AWS

Attachments:

t253087_4
v2-0001-pgstat-allow-a-stats-kind-to-use-its-own-dedicate.patchapplication/octet-stream; name=v2-0001-pgstat-allow-a-stats-kind-to-use-its-own-dedicate.patchDownload+570-263
#5Sami Imseih
samimseih.pg@gmail.com
In reply to: Sami Imseih (#4)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

Sorry for the noise. Removing my old email address from this thread.

--
Sami

#6Michael Paquier
michael@paquier.xyz
In reply to: Sami Imseih (#4)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

On Mon, Oct 05, 2026 at 04:13:37PM -0500, Sami Imseih wrote:

I kept own_hash tied to a dedicated DSA for now. I am not sure we would want to
support that mix and match behavior. A stats kind that asks for its own hash is
already asking to be isolated from the shared stats table, and using a dedicated
DSA keeps the hash storage and entry storage isolated in the same way.

I have not thought of a good use case for splitting those controls, but perhaps
there is one.

Attached is v2

+	dsa_area   *kind_dsa[PGSTAT_KIND_MAX + 1];
 	dshash_table *shared_hash;
+	dshash_table *kind_hash[PGSTAT_KIND_MAX + 1];
[...]
+pgstat_get_hash_for_kind(PgStat_Kind kind)
+{
+	if (pgStatLocal.kind_hash[kind] != NULL)
+		return pgStatLocal.kind_hash[kind];
+
+	Assert(!pgstat_get_kind_info(kind)->own_hash);
+	return pgStatLocal.shared_hash;
+}
[...]
+	/* Attach dedicated DSAs and hash tables */
+	for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
[...]
+		else
+		{
+			pgStatLocal.kind_dsa[kind] = NULL;
+			pgStatLocal.kind_hash[kind] = NULL;
+		}

Having both shared_hash and kind_dsa/kind_hash feels duplicated. Hmm.
How about enforcing a policy so as kind_dsa/kind_hash are always set
for variable-sized stats kinds? If kind_hash_valid is true, point
these to the main shared hashtable instead of using NULL. That would
make pgstat_get_hash_for_kind() unnecessary. And then perhaps remove
the shared hashtable pointer all together from the main structure?
It looks like the decision-making could just be driven by
kind_hash_valid, for the most parts.

pgstat_get_dsa_for_kind() is a nice wrapper to keep, as that means
less dshash_get_dsa_area() footprint.

+	/* All hash tables to iterate (shared + per-kind), built at attach time */
+	dshash_table *all_hashes[PGSTAT_KIND_MAX + 1];
+	int			num_hashes;

It looks like you are right to have this optimization with an
additional array. That speeds the flush of the stats by not having to
go through all PGSTAT_KIND_MAX entries, which I suspect could become
costly with very short queries a-la "SELECT 1".

/* dshash entry is not modified, take shared lock */
- dshash_seq_init(&hstat, pgStatLocal.shared_hash, false);
- while ((p = dshash_seq_next(&hstat)) != NULL)

This code has been moved to pgstat_reset_matching_entries_in_hash(),
and the corresponding comment is gone. That's important to keep
documented.

 	/* entries are removed, take an exclusive lock */
-	dshash_seq_init(&hstat, pgStatLocal.shared_hash, true);
-	while ((ps = dshash_seq_next(&hstat)) != NULL)
+	for (int h = 0; h < pgStatLocal.num_hashes; h++)
 	{
-		if (ps->dropped)
-			continue;
+		dshash_seq_init(&hstat, pgStatLocal.all_hashes[h], true);

Similar remark here. The exclusive lock part applies to
dshash_seq_init().
--
Michael

#7Sami Imseih
samimseih.pg@gmail.com
In reply to: Michael Paquier (#6)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

Having both shared_hash and kind_dsa/kind_hash feels duplicated. Hmm.
How about enforcing a policy so as kind_dsa/kind_hash are always set
for variable-sized stats kinds? If kind_hash_valid is true, point
these to the main shared hashtable instead of using NULL. That would
make pgstat_get_hash_for_kind() unnecessary. And then perhaps remove
the shared hashtable pointer all together from the main structure?

Agreed. I have changed this in the attached v3 so that kind_hash[] is
populated for all variable-numbered stats kinds at attach time. Kinds
with own_hash use their dedicated hash, and the others point to the
shared hash. I also removed the shared_hash and dsa fields from
PgStat_LocalState.

I kept pgstat_get_hash_for_kind() as a small inline wrapper around
kind_hash[] for now, mainly to keep the assertion in one place.

pgstat_get_dsa_for_kind() is a nice wrapper to keep, as that means
less dshash_get_dsa_area() footprint.

I went back and forth on this one. In the v3 changes, we already have
the selected hash when we need the DSA, so dshash_get_dsa_area(hash)
avoids looking up the hash for the kind again. I can restore the
wrapper if you prefer it for readability, but I prefer this approach
with one less helper.

It looks like you are right to have this optimization with an
additional array. That speeds the flush of the stats by not having to
go through all PGSTAT_KIND_MAX entries, which I suspect could become
costly with very short queries a-la "SELECT 1".

Thanks, I kept all_hashes[] for that reason. It is built once at attach
time and used by paths that need to scan all variable-numbered stats.

This code has been moved to pgstat_reset_matching_entries_in_hash(),
and the corresponding comment is gone. That's important to keep
documented.

Similar remark here. The exclusive lock part applies to
dshash_seq_init().

Agreed. I restored both comments in v3.

Other changes in v3:

- Exposed own_hash in pg_stat_kind_info, with the test updated for the
new column.
- The custom stats test now repeats the variable-stats restart/crash
checks for both shared and dedicated hash configurations. It also
adds reset coverage for the variable-stats kind.

--
Sami Imseih
Amazon Web Services (AWS)

Attachments:

t253087_7
v3-0001-pgstat-allow-a-stats-kind-to-use-its-own-dedicate.patchapplication/octet-stream; name=v3-0001-pgstat-allow-a-stats-kind-to-use-its-own-dedicate.patchDownload+704-362
#8Michael Paquier
michael@paquier.xyz
In reply to: Sami Imseih (#7)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

On Tue, Oct 06, 2026 at 11:20:16AM -0500, Sami Imseih wrote:

Agreed. I restored both comments in v3.

Other changes in v3:

+                    hash = pgstat_get_hash_for_kind(key.kind);
+                    dsa = dshash_get_dsa_area(hash);
[...]
-                    chunk = pgstat_alloc_entry_body(key.kind);
+                    chunk = pgstat_alloc_entry_body(key.kind, dsa);

There is a lot of back-and-forth with dshash_get_dsa_area() as a
matter to replace pgStatLocal.dsa. Hmm, dshash_get_dsa_area() only
retrieves a dshash_table->area, but wouldn't it be simpler to replace
the single pgStatLocal.dsa with an area based on the kind ID? Passing
around a dsa_area in pgstat_alloc_entry_body() and pgstat_init_entry()
feels unproductive if we should know the area based on the kind ID.
Knowing the dsa_area matters for the dsa_free() in
pgstat_get_entry_ref(), sure, but I'd probably just use an array that
holds all the references to limit the footprint and use of
dshash_get_dsa_area(). So that would mean to replace all the
pgStatLocal.dsa by an array-based lookup with the kind ID. Or
perhaps introduce an inline pgstat_get_dsaarea_for_kind() with an
Assert(), something that v2 did not do either.

-                    pgstat_get_kind_info(shent->key.kind)->name,
+                    kind_info ? kind_info->name : "unknown",

Ugh. Is that something worth a backpatch? Why does this new failure
mode matter?

+   if (kind < PGSTAT_KIND_MIN || kind > PGSTAT_KIND_MAX ||
+       pgStatLocal.kind_hash[kind] == NULL)

In pgstat_drop_entry(), it seems to me that this is a no-op, as in a
should-never-happen scenario.

+static inline dshash_table *
+pgstat_get_hash_for_kind(PgStat_Kind kind)
+{
+    Assert(pgStatLocal.kind_hash[kind] != NULL);
+
+    return pgStatLocal.kind_hash[kind];
+}

Hmm, okay by me if you want to keep this one, but honestly I am biased
about how useful it is. That's here really only for the assert() to
catch programming mistakes, and we would crash shortly anyway after
calling pgstat_get_hash_for_kind().

+foreach my $use_own_hash (qw(true false)) 
[...]
+   pgstat_register_kind(PGSTAT_KIND_TEST_CUSTOM_VAR_STATS,
+                        test_custom_stats_use_own_hash ?
+                        &custom_stats_own_hash : &custom_stats_shared_hash);

Rather than a boolean given to the input function and a GUC, just
specify a string with a name. I'd imagine a point where a third
variable-sized custom kind gets introduced. Perhaps also assign a
different name to the new custom_stats_own_hash? Relying on one kind
ID for this module is a good idea. That's less bits eaten in the
available range.

+   $node->safe_psql('postgres',
+       q(select test_custom_stats_var_update('entry1')));
+   $node->safe_psql('postgres',
+       q(select test_custom_stats_var_update('entry1')));
+   $node->safe_psql('postgres',

Perhaps use a generate_series() to reduce the number of queries and
the number of duplicated lines?

No need for a catversion bump in patches posted for reviews. These
would conflict in the CI or when applied on HEAD. It conflicted
already on HEAD as of e9c03f0f1bf6.

The change for pg_stat_get_kind_info could be split into its own
patch. Minor issue and easy to split, as different code paths are
touched.
--
Michael

#9Sami Imseih
samimseih.pg@gmail.com
In reply to: Michael Paquier (#8)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

There is a lot of back-and-forth with dshash_get_dsa_area() as a
matter to replace pgStatLocal.dsa. Hmm, dshash_get_dsa_area() only
retrieves a dshash_table->area, but wouldn't it be simpler to replace
the single pgStatLocal.dsa with an area based on the kind ID?

Done in v4. pgStatLocal.dsa is replaced by pgStatLocal.kind_dsa[],
so pgStatLocal.kind_hash[kind] and pgStatLocal.kind_dsa[kind] can be
used directly instead of passing a dsa_area around.

Or perhaps introduce an inline pgstat_get_dsaarea_for_kind() with an
Assert(), something that v2 did not do either.

I used the array lookup directly instead. This also addresses your later
point about the value of an assert-only helper, and keeps the DSA lookup
consistent with the hash lookup after removing pgstat_get_hash_for_kind().

-                    pgstat_get_kind_info(shent->key.kind)->name,
+                    kind_info ? kind_info->name : "unknown",

Ugh. Is that something worth a backpatch? Why does this new failure
mode matter?

Removed in v4. This is not a failure mode we should expect on this
path, so the fallback was just defensive noise.

+   if (kind < PGSTAT_KIND_MIN || kind > PGSTAT_KIND_MAX ||
+       pgStatLocal.kind_hash[kind] == NULL)

In pgstat_drop_entry(), it seems to me that this is a no-op, as in a
should-never-happen scenario.

Removed in v4.

Rather than a boolean given to the input function and a GUC, just
specify a string with a name. I'd imagine a point where a third
variable-sized custom kind gets introduced. Perhaps also assign a
different name to the new custom_stats_own_hash? Relying on one kind
ID for this module is a good idea. That's less bits eaten in the
available range.

Done in v4. I replaced the boolean GUC with
test_custom_var_stats.kind, so the test selects a custom stats kind by
name. The TAP test now uses a Perl array with the two names and expected
own_hash values. More cases can be added to the array later while still
sharing the same custom kind ID.

Perhaps use a generate_series() to reduce the number of queries and
the number of duplicated lines?

Done.

No need for a catversion bump in patches posted for reviews.

Removed the catversion bump from v4.

The change for pg_stat_get_kind_info could be split into its own
patch.

Done. v4 is split into two patches.

--
Sami

Attachments:

t253087_9
v4-0002-pgstat-expose-own_hash-in-pg_stat_kind_info.patchapplication/octet-stream; name=v4-0002-pgstat-expose-own_hash-in-pg_stat_kind_info.patchDownload+31-8
v4-0001-pgstat-allow-a-stats-kind-to-use-its-own-dedicate.patchapplication/octet-stream; name=v4-0001-pgstat-allow-a-stats-kind-to-use-its-own-dedicate.patchDownload+664-347
#10Michael Paquier
michael@paquier.xyz
In reply to: Sami Imseih (#9)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

On Wed, Oct 07, 2026 at 04:55:21PM -0500, Sami Imseih wrote:

Done. v4 is split into two patches.

I have spent some time on that, still in the middle of it, and for now
I am attaching my edits as of a v5-0003 that can apply on top of your
set of two patches (re-attached, these are untouched).

Another doubt that I have is about test_custom_stats_var_is_own_hash()
and its limited value. It is only a check that mirrors if a custom
stats kind uses its own_hash or not, but that does not seem worthwhile
to me due to the backend side restrictions already in place, so I'd be
tempted to drop it, keeping the test module focused on what it is
doing (insert, drops, create, etc.).
--
Michael

Attachments:

t253087_10
v5-0001-pgstat-allow-a-stats-kind-to-use-its-own-dedicate.patchtext/plain; charset=us-asciiDownload+669-352
v5-0002-pgstat-expose-own_hash-in-pg_stat_kind_info.patchtext/plain; charset=us-asciiDownload+31-8
v5-0003-Michael-s-edits.patchtext/plain; charset=us-asciiDownload+41-52
#11Sami Imseih
samimseih.pg@gmail.com
In reply to: Michael Paquier (#10)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

Done. v4 is split into two patches.

I have spent some time on that, still in the middle of it, and for now
I am attaching my edits as of a v5-0003 that can apply on top of your
set of two patches (re-attached, these are untouched).

Thanks!

v6 brings in your edits. I agree with them. I also made some minor
comment changes.
The other substantial changes are:

v6-0001:

1/

@@ -1062,9 +1178,15 @@ pgstat_drop_entry(PgStat_Kind kind, Oid dboid,
uint64 objid,
bool missing_ok)
{
PgStat_HashKey key = {0};
+ dshash_table *hash;
PgStatShared_HashEntry *shent;
bool freed = true;

+       Assert(kind >= PGSTAT_KIND_MIN && kind <= PGSTAT_KIND_MAX);
+       Assert(pgStatLocal.kind_hash[kind] != NULL);
+
+       hash = pgStatLocal.kind_hash[kind];
+

These asserts should be an ERROR instead. pgstat_drop_entry() can be called
before the kind is registered, as is the case in
pgstat_execute_transactional_drops.
For example if a custom stats extension is removed from config. In that case
kind_hash[kind] is NULL, and this would dereference it before
missing_ok is reached.
I changed this to return true when missing_ok is set, and ERROR
otherwise. I could
not find any other paths that need this handling.

Of course, there is also the alternate case where we associate a kind
id with a completely
different kind on startup, and in that case, it's ok to drop the
entries. All WAL/2PC records
store is a kind_id, and I don't think we need anything more complex.

2/

Renamed all_hashes/num_hashes to var_hashes/num_var_hashes.

3/

Removed test_custom_stats_var_is_own_hash(). I agree with you. I was thinking
about this one yesterday, but we have enough safeguards in place.

v6-0002:

4/ The existing TAP test was missing the own_hash column.

--
Sami

Attachments:

t253087_11
v6-0001-pgstat-allow-a-stats-kind-to-use-its-own-dedicate.patchapplication/octet-stream; name=v6-0001-pgstat-allow-a-stats-kind-to-use-its-own-dedicate.patchDownload+618-352
v6-0002-pgstat-expose-own_hash-in-pg_stat_kind_info.patchapplication/octet-stream; name=v6-0002-pgstat-expose-own_hash-in-pg_stat_kind_info.patchDownload+26-11
#12Michael Paquier
michael@paquier.xyz
In reply to: Sami Imseih (#11)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

On Thu, Oct 08, 2026 at 12:00:34PM -0500, Sami Imseih wrote:

For example if a custom stats extension is removed from config. In that case
kind_hash[kind] is NULL, and this would dereference it before
missing_ok is reached.
I changed this to return true when missing_ok is set, and ERROR
otherwise. I could not find any other paths that need this handling.

Of course, there is also the alternate case where we associate a kind
id with a completely
different kind on startup, and in that case, it's ok to drop the
entries. All WAL/2PC records
store is a kind_id, and I don't think we need anything more complex.

I'm halfly fine regarding this line of argument. Checking that the
kind_hash is assigned for the kind ID in the record and failing (or
silently skip if missing_ok is set) sounds fine. But the kind ID has
no need to be bound to a hard check: WAL records are protected by a
CRC, meaning that an incorrect kind ID would come from the origin, and
we trust the origin, being the WALs themselves.

v6-0002:

4/ The existing TAP test was missing the own_hash column.

The tests used duplicated descriptions across the loop, so I have
appended a $kind_name. A second thing that felt missing was a query
on pg_stat_kind_info to check that own_hash is set, once for each
loop. It's true that v6 included something at the top, but only
covered that own_hash=false, and not the second mode.

I have done another pass, adjusting a few things on the way, and
applied all that.
--
Michael

#13Sami Imseih
samimseih.pg@gmail.com
In reply to: Michael Paquier (#12)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

I have done another pass, adjusting a few things on the way, and
applied all that.
--
Michael

Thanks!

--
Sami

#14Zsolt Parragi
zsolt.parragi@percona.com
In reply to: Sami Imseih (#13)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

I have done another pass, adjusting a few things on the way, and
applied all that.
--
Michael

Thanks!

Hello!

An automatic claude review pointed out that a stats file hash entry
for a fixed-numbered kind now crashes during startup.

The attached patch fixes this and includes a simple test case in it to
show the issue.

Attachments:

nocfbot-v1-0001-Fix-crash-at-startup-with-stats-file-entry-of-fix.patchapplication/octet-stream; name=nocfbot-v1-0001-Fix-crash-at-startup-with-stats-file-entry-of-fix.patchDownload+69-11
#15Sami Imseih
samimseih.pg@gmail.com
In reply to: Zsolt Parragi (#14)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

Hi,

An automatic claude review pointed out that a stats file hash entry
for a fixed-numbered kind now crashes during startup.

Good find.

The attached patch fixes this and includes a simple test case in it to
show the issue.

A test is good here. I don't think we have enough test coverage for
pgstat startup.

However, I think we could keep the proposed test simpler, and flip the
kind from
variable to fixed once, which is enough.

Also, despite the comments in pgstat_internal.h:

```
/*
* For variable-numbered stats. Optional.
*/
void (*to_serialized_name) (const PgStat_HashKey *key,
const PgStatShared_Common *header, NameData *name);
bool (*from_serialized_name) (const NameData *name, PgStat_HashKey *key);

```

we do not prevent a fixed-numbered kind from defining these callbacks.
I think we should harden that. With it in place, the new check is
only needed for PGSTAT_FILE_ENTRY_HASH, as a name entry for a
fixed-numbered kind will then always fail the !from_serialized_name
check when PGSTAT_FILE_ENTRY_NAME.

0001 disallows the serialized name callbacks for fixed-numbered kinds.
0002 is your fix with the check in the hash entry branch and the
simplified test.

--
Sami

Attachments:

t253087_15
v1-0001-Disallow-serialized-name-callbacks-for-fixed-numb.patchapplication/octet-stream; name=v1-0001-Disallow-serialized-name-callbacks-for-fixed-numb.patchDownload+4-1
v1-0002-Fix-crash-at-startup-with-stats-file-entry-of-fix.patchapplication/octet-stream; name=v1-0002-Fix-crash-at-startup-with-stats-file-entry-of-fix.patchDownload+64-11
#16Michael Paquier
michael@paquier.xyz
In reply to: Sami Imseih (#15)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

On Wed, Oct 14, 2026 at 02:32:44PM -0500, Sami Imseih wrote:

A test is good here. I don't think we have enough test coverage for
pgstat startup.

FWIW, I am not convinced that the gain in coverage is worth the cycles
on this one. This is only one WARNING with many more patterns to
check how pgstat_read_statsfile() is robust with changes in
shared_preload_libraries and reuses of the same IDs.

However, I think we could keep the proposed test simpler, and flip the
kind from
variable to fixed once, which is enough.

Another idea that I may have mentioned on the lists (or, actually,
perhaps may not have mentioned at all on the lists yet), would be to
have a test suite that writes arbitrary file patterns that we check on
restart. The fact that by design pgstat_read_statsfile() discards
everything once one entry is inconsistent makes the check of more than
one pattern harder to scale as we would need to force multiple
startups.. But if we just added a set of SQL functions able to read
or edit a stats file and began publishing pgstat_read_statsfile() or
an equivalent of it, it would be a different story. Whether it is
worth having or not is a different story.

we do not prevent a fixed-numbered kind from defining these callbacks.
I think we should harden that. With it in place, the new check is
only needed for PGSTAT_FILE_ENTRY_HASH, as a name entry for a
fixed-numbered kind will then always fail the !from_serialized_name
check when PGSTAT_FILE_ENTRY_NAME.

Hmm. Yep, I don't see why we should not do that as well at startup.
It's one of these startup hardenings.

Saying all that, adding an extra check based on fixed_amount if we are
dealing with an ENTRY_HASH is of course fine here. If one has the
idea to reuse a kind ID, a crash would be surprising, like any of the
other checks of this kind.
--
Michael

#17Zsolt Parragi
zsolt.parragi@percona.com
In reply to: Michael Paquier (#16)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

FWIW, I am not convinced that the gain in coverage is worth the cycles
on this one. This is only one WARNING with many more patterns to
check how pgstat_read_statsfile() is robust with changes in
shared_preload_libraries and reuses of the same IDs.

I agree, I included the test mainly because it's an easy reproducer for the issue, otherwise it's very specific. I only kept it as a single patch because the entire thing is small and it is easy to separate the pgstat.c changes from it.

#18Sami Imseih
samimseih.pg@gmail.com
In reply to: Zsolt Parragi (#17)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

FWIW, I am not convinced that the gain in coverage is worth the cycles
on this one. This is only one WARNING with many more patterns to
check how pgstat_read_statsfile() is robust with changes in
shared_preload_libraries and reuses of the same IDs.

I agree, I included the test mainly because it's an easy reproducer
for the issue, otherwise it's very specific. I only kept it as a
single patch because the entire thing is small and it is easy to
separate the pgstat.c changes from it.

Not including the test is fine by me. Coverage of pgstat startup is
thin overall,
and a test for this one case doesn't get us much closer to covering the many
ways the stats file can go bad.

But if we just added a set of SQL functions able to read
or edit a stats file and began publishing pgstat_read_statsfile() or
an equivalent of it, it would be a different story. Whether it is
worth having or not is a different story.

Right, I think that would be the better way to get coverage here.
Low priority, but nice to have.

Attached v2 removes the test from 0002; the pgstat.c fix and
0001 are unchanged.

--
Sami

Attachments:

t253087_18
v2-0001-Disallow-serialized-name-callbacks-for-fixed-numb.patchapplication/octet-stream; name=v2-0001-Disallow-serialized-name-callbacks-for-fixed-numb.patchDownload+4-1
v2-0002-Fix-crash-at-startup-with-stats-file-entry-of-fix.patchapplication/octet-stream; name=v2-0002-Fix-crash-at-startup-with-stats-file-entry-of-fix.patchDownload+8-1
#19Tom Lane
tgl@sss.pgh.pa.us
In reply to: Sami Imseih (#18)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

Say folks,

The currently committed version of test_custom_stats is unstable
on my BF animal mamba [1]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mamba&dt=2026-10-17%2006%3A12%3A12.

Running the test by hand 30 times on that machine, I got 20 passes,
7 runs that failed at line 94 only, and 3 that duplicated the
buildfarm report:

# Failed test 'variable-sized stats of entry1 for test_custom_var_stats'
# at t/001_custom_stats.pl line 94.
# got: 'entry1|0|Test entry 1'
# expected: 'entry1|2|Test entry 1'

# Failed test 'variable-sized stats of entry2 for test_custom_var_stats'
# at t/001_custom_stats.pl line 100.
# got: 'entry2|0|Test entry 2'
# expected: 'entry2|3|Test entry 2'

What it looks like to me is that the test script makes no attempt to
force the locally-applied stats out to shared memory, and the relative
failure rates here correspond to the probability that we can
execute 1, 2, or more test_custom_stats_var_report() queries before
a background stats flush happens. If that's a correct diagnosis,
we'll probably see this on other slow animals before too much longer.
Otherwise, maybe there's something happening here that's specific
to PPC and/or NetBSD?

regards, tom lane

[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mamba&dt=2026-10-17%2006%3A12%3A12

#20Sami Imseih
samimseih.pg@gmail.com
In reply to: Tom Lane (#19)
Re: pgstat: allow a stats kind to use its own dedicated dsa/dshash

If that's a correct diagnosis,
we'll probably see this on other slow animals before too much longer.

This is the correct diagnosis.

+               q(select test_custom_stats_var_update(name)
+                 from (values ('entry1', 2), ('entry2', 3),
+                              ('entry3', 2), ('entry4', 3)) as stats(name, n),
+                      generate_series(1, n)));

These entry updates are batched now, which was not done before. This
leaves them as pending stats in the updating backend, and the next
safe_psql() runs in a different backend and checks the values before
the updating backend has finished shutdown and flushed its pending stats.

To confirm, I tested by slowing down pgstat_shutdown_hook(), and I was
able to reproduce similar failures.

+++ b/src/backend/utils/activity/pgstat.c
@@ -658,6 +658,9 @@ pgstat_shutdown_hook(int code, Datum arg)
        if (OidIsValid(MyDatabaseId))
                pgstat_report_disconnect(MyDatabaseId);
+       if (!dlist_is_empty(&pgStatPending))
+               pg_usleep(1000000L);
+
        pgstat_report_stat(true);

We need to force a flush when test_custom_stats_var_update(), which calls
pgstat_prep_pending_entry(), is followed by test_custom_stats_var_report()
from another backend. This is only the case for the batched updates.

--
Sami

Attachments:

t253087_20
v1-0001-pgstat-Stabilize-test_custom_stats-counter-checks.patchapplication/octet-stream; name=v1-0001-pgstat-Stabilize-test_custom_stats-counter-checks.patchDownload+2-2
#21Michael Paquier
michael@paquier.xyz
In reply to: Sami Imseih (#20)
#22Alexander Lakhin
exclusion@gmail.com
In reply to: Sami Imseih (#20)
#23Michael Paquier
michael@paquier.xyz
In reply to: Alexander Lakhin (#22)
#24Alexander Lakhin
exclusion@gmail.com
In reply to: Michael Paquier (#23)