Pluggable cumulative statistics
Hi all,
While looking at ways to make pg_stat_statements more scalable and
dynamically manageable (no more PGC_POSTMASTER for the max number of
entries), which came out as using a dshash, Andres has mentioned me
off-list (on twitter/X) that we'd better plug in it to the shmem
pgstats facility, moving the text file that holds the query strings
into memory (with size restrictions for the query strings, for
example). This has challenges on its own (query ID is 8 bytes
incompatible with the dboid/objid hash key used by pgstats, discard of
entries when maximum). Anyway, this won't happen if we don't do one
of these two things:
1) Move pg_stat_statements into core, adapting pgstats for its
requirements.
2) Make the shmem pgstats pluggable so as it is possible for extensions
to register their own stats kinds.
1) may have its advantages, still I am not sure if we want to do that.
And 2) is actually something that can be used for more things than
just pg_stat_statements, because people love extensions and
statistics (spoiler: I do). The idea is simple: any extension
defining a custom stats kind would be able to rely on all the in-core
facilities we use for the existing in-core kinds:
a) Snapshotting and caching of the stats, via stats_fetch_consistency.
b) Native handling and persistency of the custom stats data.
c) Reuse stats after a crash, pointing at this comment in xlog.c:
* TODO: With a bit of extra work we could just start with a pgstat file
* associated with the checkpoint redo location we're starting from.
This means that we always remove the stats after a crash. That's
something I have a patch for, not for this thread, but the idea is
that custom stats would also benefit from this property.
The implementation is based on the following ideas:
* A structure in shared memory that tracks the IDs of the custom stats
kinds with their names. These are incremented starting from
PGSTAT_KIND_LAST.
* Processes use a local array cache that keeps tracks of all the
custom PgStat_KindInfos, indexed by (kind_id - PGSTAT_KIND_LAST).
* The kind IDs may change across restarts, meaning that any stats data
associated to a custom kind is stored with the *name* of the custom
stats kind. Depending on the discussion happening here, I'd be open
to use the same concept as custom RMGRs, where custom kind IDs are
"reserved", fixed in time, and tracked in the Postgres wiki. It is
cheaper to store the stats this way, as well, while managing conflicts
across extensions available in the community ecosystem.
* Custom stats can be added without shared_preload_libraries,
loading them from a shmem startup hook with shared_preload_libraries
is also possible.
* The shmem pgstats defines two types of statistics: the ones in a
dshash and what's called a "fixed" type like for archiver, WAL, etc.
pointing to areas of shared memory. All the fixed types are linked to
structures for snapshotting and shmem tracking. As a matter of
simplification and because I could not really see a case where I'd
want to plug in a fixed stats kind, the patch forbids this case. This
case could be allowed, but I'd rather refactor the structures of
pgstat_internal.h so as we don't have traces of the "fixed" stats
structures in so many areas.
* Making custom stats data persistent is an interesting problem, and
there are a couple of approaches I've considered:
** Allow custom kinds to define callbacks to read and write data from
a source they'd want, like their own file through a fd. This has the
disadvantage to remove the benefit of c) above.
** Store everything in the existing stats file, adding one type of
entry like 'S' and 'N' with a "custom" type, where the *name* of the
custom stats kind is stored instead of its ID computed from shared
memory.
A mix of both? The patch attached has used the second approach. If
the process reading/writing the stats does not know about the custom
stats data, the data is discarded.
* pgstat.c has a big array called pgstat_kind_infos to define all the
existing stats kinds. Perhaps the code should be refactored to use
this new API? That would make the code more consistent with what we
do for resource managers, for one, while moving the KindInfos into
their own file. With that in mind, storing the kind ID in KindInfos
feels intuitive.
While thinking about a use case to show what these APIs can do, I have
decided to add statistics to the existing module injection_points
rather than implement a new test module, gathering data about them and
have tests that could use this data (like tracking the number of times
a point is taken). This is simple enough that it can be used as a
template, as well. There is a TAP test checking the data persistence
across restarts, so I did not mess up this part much, hopefully.
Please find attached a patch set implementing these ideas:
- 0001 switches PgStat_Kind from an enum to a uint32, for the internal
counters.
- 0002 is some cleanup for the hardcoded S, N and E in pgstat.c.
- 0003 introduces the backend-side APIs, with the shmem table counter
and the routine to give code paths a way to register their own stats
kind (see pgstat_add_kind).
- 0004 implements an example of how to use these APIs, see
injection_stats.c in src/test/modules/injection_points/.
- 0005 adds some docs.
- 0006 is an idea of how to make this custom stats data persistent.
This will hopefully spark a discussion, and I was looking for answers
regarding these questions:
- Should the pgstat_kind_infos array in pgstat.c be refactored to use
something similar to pgstat_add_kind?
- How should the persistence of the custom stats be achieved?
Callbacks to give custom stats kinds a way to write/read their data,
push everything into a single file, or support both?
- Should this do like custom RMGRs and assign to each stats kinds ID
that are set in stone rather than dynamic ones?
Thanks for reading.
--
Michael
Attachments:
0001-Switch-PgStat_Kind-from-enum-to-uint32.patchtext/x-diff; charset=us-asciiDownload
From 0847075d42b9f9ad5ba882d51a3c3de984f8e563 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 13 Jun 2024 13:25:05 +0900
Subject: [PATCH 1/6] Switch PgStat_Kind from enum to uint32
A follow-up patch is planned to make this counter extensible, and
keeping a trace of the kind behind a type is useful in the internal
routines used by pgstats. While on it, switch pgstat_is_kind_valid() to
use PgStat_Kind, to be more consistent with its callers.
---
src/include/pgstat.h | 35 ++++++++++++++---------------
src/backend/utils/activity/pgstat.c | 6 ++---
2 files changed, 20 insertions(+), 21 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 2136239710..2d30fadaf1 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -32,26 +32,25 @@
#define PG_STAT_TMP_DIR "pg_stat_tmp"
/* The types of statistics entries */
-typedef enum PgStat_Kind
-{
- /* use 0 for INVALID, to catch zero-initialized data */
- PGSTAT_KIND_INVALID = 0,
+#define PgStat_Kind uint32
- /* stats for variable-numbered objects */
- PGSTAT_KIND_DATABASE, /* database-wide statistics */
- PGSTAT_KIND_RELATION, /* per-table statistics */
- PGSTAT_KIND_FUNCTION, /* per-function statistics */
- PGSTAT_KIND_REPLSLOT, /* per-slot statistics */
- PGSTAT_KIND_SUBSCRIPTION, /* per-subscription statistics */
+/* use 0 for INVALID, to catch zero-initialized data */
+#define PGSTAT_KIND_INVALID 0
- /* stats for fixed-numbered objects */
- PGSTAT_KIND_ARCHIVER,
- PGSTAT_KIND_BGWRITER,
- PGSTAT_KIND_CHECKPOINTER,
- PGSTAT_KIND_IO,
- PGSTAT_KIND_SLRU,
- PGSTAT_KIND_WAL,
-} PgStat_Kind;
+/* stats for variable-numbered objects */
+#define PGSTAT_KIND_DATABASE 1 /* database-wide statistics */
+#define PGSTAT_KIND_RELATION 2 /* per-table statistics */
+#define PGSTAT_KIND_FUNCTION 3 /* per-function statistics */
+#define PGSTAT_KIND_REPLSLOT 4 /* per-slot statistics */
+#define PGSTAT_KIND_SUBSCRIPTION 5 /* per-subscription statistics */
+
+/* stats for fixed-numbered objects */
+#define PGSTAT_KIND_ARCHIVER 6
+#define PGSTAT_KIND_BGWRITER 7
+#define PGSTAT_KIND_CHECKPOINTER 8
+#define PGSTAT_KIND_IO 9
+#define PGSTAT_KIND_SLRU 10
+#define PGSTAT_KIND_WAL 11
#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index dcc2ad8d95..d558cc1414 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -173,7 +173,7 @@ static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
static void pgstat_build_snapshot_fixed(PgStat_Kind kind);
-static inline bool pgstat_is_kind_valid(int ikind);
+static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
/* ----------
@@ -1254,9 +1254,9 @@ pgstat_get_kind_from_str(char *kind_str)
}
static inline bool
-pgstat_is_kind_valid(int ikind)
+pgstat_is_kind_valid(PgStat_Kind kind)
{
- return ikind >= PGSTAT_KIND_FIRST_VALID && ikind <= PGSTAT_KIND_LAST;
+ return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
}
const PgStat_KindInfo *
--
2.43.0
0002-Replace-hardcoded-identifiers-in-pgstats-file-by-var.patchtext/x-diff; charset=us-asciiDownload
From dd7a98a990aa28a06a82607052180f69ee44e05f Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 13 Jun 2024 13:25:48 +0900
Subject: [PATCH 2/6] Replace hardcoded identifiers in pgstats file by
variables
This changes three variable types:
- N for named entries.
- S for entries identified by a hash.
- E for end-of-file
---
src/backend/utils/activity/pgstat.c | 23 +++++++++++++++--------
1 file changed, 15 insertions(+), 8 deletions(-)
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index d558cc1414..f03fee7cd5 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -127,6 +127,13 @@
#define PGSTAT_SNAPSHOT_HASH_SIZE 512
+/* ---------
+ * Identifiers in stats file.
+ * ---------
+ */
+#define PGSTAT_FILE_END 'E' /* end of file */
+#define PGSTAT_FILE_NAME 'N' /* stats entry identified by name */
+#define PGSTAT_FILE_SYSTEM 'S' /* stats entry identified by PgStat_HashKey */
/* hash table for statistics snapshots entry */
typedef struct PgStat_SnapshotEntry
@@ -1408,7 +1415,7 @@ pgstat_write_statsfile(void)
if (!kind_info->to_serialized_name)
{
/* normal stats entry, identified by PgStat_HashKey */
- fputc('S', fpout);
+ fputc(PGSTAT_FILE_SYSTEM, fpout);
write_chunk_s(fpout, &ps->key);
}
else
@@ -1418,7 +1425,7 @@ pgstat_write_statsfile(void)
kind_info->to_serialized_name(&ps->key, shstats, &name);
- fputc('N', fpout);
+ fputc(PGSTAT_FILE_NAME, fpout);
write_chunk_s(fpout, &ps->key.kind);
write_chunk_s(fpout, &name);
}
@@ -1435,7 +1442,7 @@ pgstat_write_statsfile(void)
* pgstat.stat with it. The ferror() check replaces testing for error
* after each individual fputc or fwrite (in write_chunk()) above.
*/
- fputc('E', fpout);
+ fputc(PGSTAT_FILE_END, fpout);
if (ferror(fpout))
{
@@ -1572,8 +1579,8 @@ pgstat_read_statsfile(void)
switch (t)
{
- case 'S':
- case 'N':
+ case PGSTAT_FILE_SYSTEM:
+ case PGSTAT_FILE_NAME:
{
PgStat_HashKey key;
PgStatShared_HashEntry *p;
@@ -1581,7 +1588,7 @@ pgstat_read_statsfile(void)
CHECK_FOR_INTERRUPTS();
- if (t == 'S')
+ if (t == PGSTAT_FILE_SYSTEM)
{
/* normal stats entry, identified by PgStat_HashKey */
if (!read_chunk_s(fpin, &key))
@@ -1647,8 +1654,8 @@ pgstat_read_statsfile(void)
break;
}
- case 'E':
- /* check that 'E' actually signals end of file */
+ case PGSTAT_FILE_END:
+ /* check that PGSTAT_FILE_END actually signals end of file */
if (fgetc(fpin) != EOF)
goto error;
--
2.43.0
0003-Introduce-pluggable-APIs-for-Cumulative-Statistics.patchtext/x-diff; charset=us-asciiDownload
From 8fbf5e027dfbcc9fd3ebc52cb82000a855c23064 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 13 Jun 2024 13:32:22 +0900
Subject: [PATCH 3/6] Introduce pluggable APIs for Cumulative Statistics
This commit adds support in the backend for $subject, allowing
out-of-core extensions to add their own custom statistics kinds.
---
src/include/pgstat.h | 2 +
src/include/storage/lwlocklist.h | 1 +
src/include/utils/pgstat_internal.h | 1 +
src/backend/storage/ipc/ipci.c | 2 +
src/backend/utils/activity/pgstat.c | 301 +++++++++++++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/tools/pgindent/typedefs.list | 3 +
7 files changed, 308 insertions(+), 3 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 2d30fadaf1..b3cdc0da6d 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -463,6 +463,8 @@ typedef struct PgStat_PendingWalStats
/* functions called from postmaster */
extern Size StatsShmemSize(void);
extern void StatsShmemInit(void);
+extern Size StatsKindShmemSize(void);
+extern void StatsKindShmemInit(void);
/* Functions called during server startup / shutdown */
extern void pgstat_restore_stats(void);
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 85f6568b9e..ed78f93683 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -83,3 +83,4 @@ PG_LWLOCK(49, WALSummarizer)
PG_LWLOCK(50, DSMRegistry)
PG_LWLOCK(51, InjectionPoint)
PG_LWLOCK(52, SerialControl)
+PG_LWLOCK(53, PgStatKind)
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index dbbca31602..21dfff740d 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -503,6 +503,7 @@ static inline void *pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common
*/
extern const PgStat_KindInfo *pgstat_get_kind_info(PgStat_Kind kind);
+extern PgStat_Kind pgstat_add_kind(const PgStat_KindInfo *kind_info);
#ifdef USE_ASSERT_CHECKING
extern void pgstat_assert_is_up(void);
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 521ed5418c..8b5023d9de 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -149,6 +149,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
size = add_size(size, StatsShmemSize());
+ size = add_size(size, StatsKindShmemSize());
size = add_size(size, WaitEventExtensionShmemSize());
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
@@ -355,6 +356,7 @@ CreateOrAttachShmemStructs(void)
SyncScanShmemInit();
AsyncShmemInit();
StatsShmemInit();
+ StatsKindShmemInit();
WaitEventExtensionShmemInit();
InjectionPointShmemInit();
}
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index f03fee7cd5..b96743ce84 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -58,6 +58,12 @@
* PGSTAT_FETCH_CONSISTENCY_SNAPSHOT statistics are stored in
* pgStatLocal.snapshot.
*
+ * It is possible for out-of-core modules to define custom statistics kinds,
+ * that can use the same properties as any in-core stats kinds. Each custom
+ * kind is assigned a unique PgStat_Kind stored in shared memory with the
+ * name of the statistics kind. Each PgStat_KindInfo is maintained in a
+ * local array cache known to the current process.
+ *
* To keep things manageable, stats handling is split across several
* files. Infrastructure pieces are in:
* - pgstat.c - this file, to tie it all together
@@ -94,13 +100,18 @@
#include <unistd.h>
#include "access/xact.h"
+#include "common/int.h"
#include "lib/dshash.h"
#include "pgstat.h"
#include "port/atomics.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/shmem.h"
+#include "storage/spin.h"
+#include "storage/s_lock.h"
#include "utils/guc_hooks.h"
+#include "utils/hsearch.h"
#include "utils/memutils.h"
#include "utils/pgstat_internal.h"
#include "utils/timestamp.h"
@@ -400,6 +411,121 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
};
+/* --------
+ * Hash tables for storing custom pgstats kinds
+ *
+ * PgStatKindHashById is used to find the name from a PgStat_Kind.
+ * Any backend can search it to find custom stats kinds
+ *
+ * PgStatKindHashByName is used to find the PgStat_Kind from a name.
+ * It is used to ensure that no duplicated entries are registered.
+ *
+ * The size of the hash table is based on the assumption that
+ * PGSTAT_KIND_HASH_INIT_SIZE is enough for most cases, and it seems
+ * unlikely that the number of entries will reach
+ * PGSTAT_KIND_HASH_MAX_SIZE.
+ * --------
+ */
+
+static HTAB *PgStatKindHashById; /* find PgStat_KindInfo from IDs */
+static HTAB *PgStatKindHashByName; /* find PgStat_Kind from names */
+
+#define PGSTAT_KIND_HASH_INIT_SIZE 16
+#define PGSTAT_KIND_HASH_MAX_SIZE 128
+
+/* hash table entries */
+typedef struct PgStatKindEntryById
+{
+ PgStat_Kind kind; /* hash key */
+ char kind_name[NAMEDATALEN]; /* stats kind name */
+} PgStatKindEntryById;
+
+typedef struct PgStatKindEntryByName
+{
+ char kind_name[NAMEDATALEN]; /* hash key */
+ PgStat_Kind kind; /* kind ID */
+} PgStatKindEntryByName;
+
+/* dynamic allocation counter for custom pgstats kinds */
+typedef struct PgStatKindCounterData
+{
+ int nextId; /* next ID to assign */
+ slock_t mutex; /* protects the counter */
+} PgStatKindCounterData;
+
+/* pointer to the shared memory */
+static PgStatKindCounterData *PgStatKindCounter;
+
+/* first ID of custom pgstats kinds, as stored in pgstats tables */
+#define PGSTAT_KIND_INITIAL_ID (PGSTAT_KIND_LAST + 1)
+
+/*
+ * Local array cache pointing to the custom PgStat_KindInfos known to the
+ * current process, indexed by kind ID minus PGSTAT_KIND_LAST. Any unused
+ * entries in the array will contain NULL.
+ */
+static const PgStat_KindInfo **PgStatKindCache = NULL;
+static uint32 PgStatKindCacheNum = 0;
+
+/* -----------------------------------------------------------
+ * Functions managing the shared memory for custom stats kinds
+ * -----------------------------------------------------------
+ */
+
+/*
+ * Compute shared memory space needed for custom stats kinds
+ */
+Size
+StatsKindShmemSize(void)
+{
+ Size sz;
+
+ sz = MAXALIGN(sizeof(PgStatKindCounterData));
+ sz = add_size(sz, hash_estimate_size(PGSTAT_KIND_HASH_MAX_SIZE,
+ sizeof(PgStatKindEntryById)));
+ sz = add_size(sz, hash_estimate_size(PGSTAT_KIND_HASH_MAX_SIZE,
+ sizeof(PgStatKindEntryByName)));
+ return sz;
+}
+
+/*
+ * Initialize shared memory area for custom stats kinds during startup
+ */
+void
+StatsKindShmemInit(void)
+{
+ bool found;
+ HASHCTL info;
+
+ PgStatKindCounter = (PgStatKindCounterData *)
+ ShmemInitStruct("PgStatKindCounterData",
+ sizeof(PgStatKindCounterData), &found);
+ if (!found)
+ {
+ /* initialize the allocation counter and its spinlock. */
+ PgStatKindCounter->nextId = PGSTAT_KIND_INITIAL_ID;
+ SpinLockInit(&PgStatKindCounter->mutex);
+ }
+
+ /* initialize or attach the hash tables to store custom stats kinds */
+ info.keysize = sizeof(PgStat_Kind);
+ info.entrysize = sizeof(PgStatKindEntryById);
+ PgStatKindHashById = ShmemInitHash("PgStatKind hash by id",
+ PGSTAT_KIND_HASH_INIT_SIZE,
+ PGSTAT_KIND_HASH_MAX_SIZE,
+ &info,
+ HASH_ELEM | HASH_BLOBS);
+
+ /* key is a NULL-terminated string */
+ info.keysize = sizeof(char[NAMEDATALEN]);
+ info.entrysize = sizeof(PgStatKindEntryByName);
+ PgStatKindHashByName = ShmemInitHash("PgStatKind hash by name",
+ PGSTAT_KIND_HASH_INIT_SIZE,
+ PGSTAT_KIND_HASH_MAX_SIZE,
+ &info,
+ HASH_ELEM | HASH_STRINGS);
+}
+
/* ------------------------------------------------------------
* Functions managing the state of the stats system for all backends.
* ------------------------------------------------------------
@@ -1254,6 +1380,18 @@ pgstat_get_kind_from_str(char *kind_str)
return kind;
}
+ /* Check the local cache if any */
+ if (PgStatKindCacheNum > 0)
+ {
+ for (int kind = 0; kind <= PgStatKindCacheNum; kind++)
+ {
+ if (PgStatKindCache[kind] == NULL)
+ continue;
+ if (pg_strcasecmp(kind_str, PgStatKindCache[kind]->name) == 0)
+ return kind + PGSTAT_KIND_INITIAL_ID;
+ }
+ }
+
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid statistics kind: \"%s\"", kind_str)));
@@ -1263,7 +1401,8 @@ pgstat_get_kind_from_str(char *kind_str)
static inline bool
pgstat_is_kind_valid(PgStat_Kind kind)
{
- return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
+ return kind >= PGSTAT_KIND_FIRST_VALID &&
+ kind <= (PGSTAT_KIND_LAST + PgStatKindCacheNum);
}
const PgStat_KindInfo *
@@ -1271,7 +1410,152 @@ pgstat_get_kind_info(PgStat_Kind kind)
{
Assert(pgstat_is_kind_valid(kind));
- return &pgstat_kind_infos[kind];
+ if (kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST)
+ return &pgstat_kind_infos[kind];
+
+ /* Look in local cache */
+ if (kind >= PGSTAT_KIND_INITIAL_ID &&
+ kind <= PGSTAT_KIND_INITIAL_ID + PgStatKindCacheNum)
+ return PgStatKindCache[kind - PGSTAT_KIND_INITIAL_ID];
+
+ Assert(false);
+ return NULL; /* keep compiler quiet */
+}
+
+/*
+ * Save a PgStat_KindInfo into the local cache, if not already done.
+ */
+static void
+pgstat_save_kind_info(PgStat_Kind kind, const PgStat_KindInfo *kind_info)
+{
+ /* This should only be called for user-defined stats kinds */
+ if (kind <= PGSTAT_KIND_LAST)
+ return;
+
+ /* Convert to array index */
+ kind -= PGSTAT_KIND_INITIAL_ID;
+
+ /* If necessary, create or enlarge local cache array. */
+ if (kind >= PgStatKindCacheNum)
+ {
+ uint32 newalloc;
+
+ /*
+ * Do a simple increment, to keep an exact count of the custom stats
+ * kinds stored rather than an upper-bound. This is more costly each
+ * time a new PgStat_KindInfo is added, but saves in correctness. This
+ * overflow should not happen as this is capped by
+ * PGSTAT_KIND_HASH_MAX_SIZE, but let's be safe.
+ */
+ if (pg_add_u32_overflow(kind, 1, &newalloc))
+ elog(ERROR, "could not allocate memory for custom pgstats");
+
+ if (PgStatKindCache == NULL)
+ PgStatKindCache = (const PgStat_KindInfo **)
+ MemoryContextAllocZero(TopMemoryContext,
+ newalloc * sizeof(PgStat_KindInfo *));
+ else
+ PgStatKindCache = repalloc0_array(PgStatKindCache,
+ const PgStat_KindInfo *,
+ PgStatKindCacheNum,
+ newalloc);
+ PgStatKindCacheNum = newalloc;
+ }
+
+ PgStatKindCache[kind] = kind_info;
+}
+
+/*
+ * Allocate a new stats kind and return its PgStat_Kind.
+ */
+PgStat_Kind
+pgstat_add_kind(const PgStat_KindInfo *kind_info)
+{
+ PgStat_Kind kind;
+ bool found;
+ PgStatKindEntryByName *entry_by_name;
+ PgStatKindEntryById *entry_by_id;
+
+ if (strlen(kind_info->name) >= NAMEDATALEN)
+ elog(ERROR,
+ "cannot use custom stats kind longer than %u characters",
+ NAMEDATALEN - 1);
+
+ /*
+ * These are not supported for now, as these point out to fixed areas of
+ * shared memory.
+ */
+ if (kind_info->fixed_amount)
+ elog(ERROR,
+ "cannot define custom stats kind with fixed amount of data");
+
+ /*
+ * Check if kind ID associated to the name is already defined, and return
+ * it if so.
+ */
+ LWLockAcquire(PgStatKindLock, LW_SHARED);
+ entry_by_name = (PgStatKindEntryByName *)
+ hash_search(PgStatKindHashByName, kind_info->name,
+ HASH_FIND, &found);
+ LWLockRelease(PgStatKindLock);
+
+ if (found)
+ {
+ pgstat_save_kind_info(entry_by_name->kind, kind_info);
+ return entry_by_name->kind;
+ }
+
+ /*
+ * Allocate and register a new stats kind. Recheck if this kind name
+ * exists, as it could be possible that a concurrent process has inserted
+ * one with the same name since the LWLock acquired again here was
+ * previously released.
+ */
+ LWLockAcquire(PgStatKindLock, LW_EXCLUSIVE);
+ entry_by_name = (PgStatKindEntryByName *)
+ hash_search(PgStatKindHashByName, kind_info->name,
+ HASH_FIND, &found);
+ if (found)
+ {
+ LWLockRelease(PgStatKindLock);
+ pgstat_save_kind_info(entry_by_name->kind, kind_info);
+ return entry_by_name->kind;
+ }
+
+ /* Allocate a new kind ID */
+ SpinLockAcquire(&PgStatKindCounter->mutex);
+
+ if (PgStatKindCounter->nextId >= PGSTAT_KIND_HASH_MAX_SIZE - PGSTAT_KIND_INITIAL_ID)
+ {
+ SpinLockRelease(&PgStatKindCounter->mutex);
+ ereport(ERROR,
+ errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("too many stats kinds"));
+ }
+
+ kind = PgStatKindCounter->nextId;
+ PgStatKindCounter->nextId++;
+
+ SpinLockRelease(&PgStatKindCounter->mutex);
+
+ /* Register the new stats kind */
+ entry_by_id = (PgStatKindEntryById *)
+ hash_search(PgStatKindHashById, &kind,
+ HASH_ENTER, &found);
+ Assert(!found);
+ strlcpy(entry_by_id->kind_name, kind_info->name,
+ sizeof(entry_by_id->kind_name));
+
+ entry_by_name = (PgStatKindEntryByName *)
+ hash_search(PgStatKindHashByName, kind_info->name,
+ HASH_ENTER, &found);
+ Assert(!found);
+ entry_by_name->kind = kind;
+
+ LWLockRelease(PgStatKindLock);
+
+ pgstat_save_kind_info(kind, kind_info);
+ return kind;
}
/*
@@ -1405,6 +1689,17 @@ pgstat_write_statsfile(void)
if (ps->dropped)
continue;
+ /*
+ * This discards data related to custom stats kinds that are unknown
+ * to this process.
+ */
+ if (!pgstat_is_kind_valid(ps->key.kind))
+ {
+ elog(WARNING, "found unknown stats entry %u/%u/%u",
+ ps->key.kind, ps->key.dboid, ps->key.objoid);
+ continue;
+ }
+
shstats = (PgStatShared_Common *) dsa_get_address(pgStatLocal.dsa, ps->body);
kind_info = pgstat_get_kind_info(ps->key.kind);
@@ -1639,7 +1934,7 @@ pgstat_read_statsfile(void)
if (found)
{
dshash_release_lock(pgStatLocal.shared_hash, p);
- elog(WARNING, "found duplicate stats entry %d/%u/%u",
+ elog(WARNING, "found duplicate stats entry %u/%u/%u",
key.kind, key.dboid, key.objoid);
goto error;
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 87cbca2811..32d6d8fd74 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -345,6 +345,7 @@ WALSummarizer "Waiting to read or update WAL summarization state."
DSMRegistry "Waiting to read or update the dynamic shared memory registry."
InjectionPoint "Waiting to read or update information related to injection points."
SerialControl "Waiting to read or update shared <filename>pg_serial</filename> state."
+PgStatKind "Waiting to read or update custom pgstats kind information."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 4f57078d13..8718ca54e0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2109,6 +2109,9 @@ PgFdwRelationInfo
PgFdwSamplingMethod
PgFdwScanState
PgIfAddrCallback
+PgStatKindCounterData
+PgStatKindEntryById
+PgStatKindEntryByName
PgStatShared_Archiver
PgStatShared_BgWriter
PgStatShared_Checkpointer
--
2.43.0
0004-injection_points-Add-statistics-for-custom-points.patchtext/x-diff; charset=us-asciiDownload
From 6728eb684760be8b883c69dedc7bb089fa090d7c Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 13 Jun 2024 13:37:24 +0900
Subject: [PATCH 4/6] injection_points: Add statistics for custom points
This acts as a template of what can be achieved with the pluggable
cumulative stats APIs, while being useful on its own for injection
points.
Currently, the only data gathered is the number of times an injection
point is called. This can be extended as required. All the routines
related to the stats are located in their own file, for clarity.
---
src/test/modules/injection_points/Makefile | 7 +-
.../expected/injection_points.out | 67 +++++++
.../injection_points--1.0.sql | 10 +
.../injection_points/injection_points.c | 17 ++
.../injection_points/injection_stats.c | 176 ++++++++++++++++++
.../injection_points/injection_stats.h | 22 +++
src/test/modules/injection_points/meson.build | 1 +
.../injection_points/sql/injection_points.sql | 15 ++
src/tools/pgindent/typedefs.list | 2 +
9 files changed, 315 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/injection_points/injection_stats.c
create mode 100644 src/test/modules/injection_points/injection_stats.h
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 31bd787994..676823a87f 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -1,7 +1,10 @@
# src/test/modules/injection_points/Makefile
-MODULES = injection_points
-
+MODULE_big = injection_points
+OBJS = \
+ $(WIN32RES) \
+ injection_points.o \
+ injection_stats.o
EXTENSION = injection_points
DATA = injection_points--1.0.sql
PGFILEDESC = "injection_points - facility for injection points"
diff --git a/src/test/modules/injection_points/expected/injection_points.out b/src/test/modules/injection_points/expected/injection_points.out
index dd9db06e10..cb50a664f9 100644
--- a/src/test/modules/injection_points/expected/injection_points.out
+++ b/src/test/modules/injection_points/expected/injection_points.out
@@ -208,5 +208,72 @@ SELECT injection_points_detach('TestConditionLocal1');
(1 row)
+-- Statistics
+SELECT injection_points_stats_numcalls('TestConditionStats'); -- nothing
+ injection_points_stats_numcalls
+---------------------------------
+
+(1 row)
+
+SELECT injection_points_attach('TestConditionStats', 'notice');
+ injection_points_attach
+-------------------------
+
+(1 row)
+
+SELECT injection_points_stats_numcalls('TestConditionStats'); -- returns 0
+ injection_points_stats_numcalls
+---------------------------------
+ 0
+(1 row)
+
+SELECT injection_points_run('TestConditionStats');
+NOTICE: notice triggered for injection point TestConditionStats
+ injection_points_run
+----------------------
+
+(1 row)
+
+SELECT injection_points_stats_numcalls('TestConditionStats'); -- returns 1
+ injection_points_stats_numcalls
+---------------------------------
+ 1
+(1 row)
+
+-- Check transactions with stats snapshots
+BEGIN;
+SET stats_fetch_consistency TO snapshot;
+SELECT injection_points_stats_numcalls('TestConditionStats'); -- returns 1
+ injection_points_stats_numcalls
+---------------------------------
+ 1
+(1 row)
+
+SELECT injection_points_run('TestConditionStats');
+NOTICE: notice triggered for injection point TestConditionStats
+ injection_points_run
+----------------------
+
+(1 row)
+
+SELECT injection_points_stats_numcalls('TestConditionStats'); -- still 1
+ injection_points_stats_numcalls
+---------------------------------
+ 1
+(1 row)
+
+COMMIT;
+SELECT injection_points_stats_numcalls('TestConditionStats'); -- now 2
+ injection_points_stats_numcalls
+---------------------------------
+ 2
+(1 row)
+
+SELECT injection_points_detach('TestConditionStats');
+ injection_points_detach
+-------------------------
+
+(1 row)
+
DROP EXTENSION injection_points;
DROP FUNCTION wait_pid;
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index c16a33b08d..deaf47d8ae 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -54,3 +54,13 @@ CREATE FUNCTION injection_points_detach(IN point_name TEXT)
RETURNS void
AS 'MODULE_PATHNAME', 'injection_points_detach'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+--
+-- injection_points_stats_numcalls()
+--
+-- Reports statistics, if any, related to the given injection point.
+--
+CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
+RETURNS bigint
+AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index 5c44625d1d..6346af6cf6 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -18,6 +18,7 @@
#include "postgres.h"
#include "fmgr.h"
+#include "injection_stats.h"
#include "miscadmin.h"
#include "nodes/pg_list.h"
#include "nodes/value.h"
@@ -170,6 +171,9 @@ injection_points_cleanup(int code, Datum arg)
char *name = strVal(lfirst(lc));
(void) InjectionPointDetach(name);
+
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
}
}
@@ -182,6 +186,8 @@ injection_error(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(ERROR, "error triggered for injection point %s", name);
}
@@ -193,6 +199,8 @@ injection_notice(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(NOTICE, "notice triggered for injection point %s", name);
}
@@ -211,6 +219,8 @@ injection_wait(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
/*
* Use the injection point name for this custom wait event. Note that
* this custom wait event name is not released, but we don't care much for
@@ -299,6 +309,10 @@ injection_points_attach(PG_FUNCTION_ARGS)
inj_list_local = lappend(inj_list_local, makeString(pstrdup(name)));
MemoryContextSwitchTo(oldctx);
}
+
+ /* Add entry for stats */
+ pgstat_create_inj(name);
+
PG_RETURN_VOID();
}
@@ -400,5 +414,8 @@ injection_points_detach(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldctx);
}
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
+
PG_RETURN_VOID();
}
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
new file mode 100644
index 0000000000..6314486d79
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -0,0 +1,176 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.c
+ * Code for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+
+#include "common/hashfn.h"
+#include "injection_stats.h"
+#include "pgstat.h"
+#include "utils/builtins.h"
+#include "utils/pgstat_internal.h"
+
+/* Structures for statistics of injection points */
+typedef struct PgStat_StatInjEntry
+{
+ PgStat_Counter numcalls; /* number of times point has been run */
+} PgStat_StatInjEntry;
+
+typedef struct PgStatShared_InjectionPoint
+{
+ PgStatShared_Common header;
+ PgStat_StatInjEntry stats;
+} PgStatShared_InjectionPoint;
+
+static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+
+static const PgStat_KindInfo injection_stats = {
+ .name = "injection_points",
+ .fixed_amount = false, /* Bounded by the number of points */
+
+ /* Injection points are system-wide */
+ .accessed_across_databases = true,
+
+ .shared_size = sizeof(PgStatShared_InjectionPoint),
+ .shared_data_off = offsetof(PgStatShared_InjectionPoint, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPoint *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatInjEntry),
+ .flush_pending_cb = injection_stats_flush_cb,
+};
+
+/*
+ * Compute stats entry idx from point name with a 4-byte hash.
+ */
+#define PGSTAT_INJ_IDX(name) hash_bytes((const unsigned char *) name, strlen(name))
+
+static PgStat_Kind inj_stats_kind = PGSTAT_KIND_INVALID;
+
+/*
+ * Callback for stats handling
+ */
+static bool
+injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ PgStat_StatInjEntry *localent;
+ PgStatShared_InjectionPoint *shfuncent;
+
+ localent = (PgStat_StatInjEntry *) entry_ref->pending;
+ shfuncent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ shfuncent->stats.numcalls += localent->numcalls;
+ return true;
+}
+
+/*
+ * Support function for the SQL-callable pgstat* functions. Returns
+ * a pointer to the injection point statistics struct.
+ */
+static PgStat_StatInjEntry *
+pgstat_fetch_stat_injentry(const char *name)
+{
+ PgStat_StatInjEntry *entry = NULL;
+
+ if (inj_stats_kind == PGSTAT_KIND_INVALID)
+ inj_stats_kind = pgstat_add_kind(&injection_stats);
+
+ /* Compile the lookup key as a hash of the point name */
+ entry = (PgStat_StatInjEntry *) pgstat_fetch_entry(inj_stats_kind,
+ InvalidOid,
+ PGSTAT_INJ_IDX(name));
+ return entry;
+}
+
+/*
+ * Report injection point creation.
+ */
+void
+pgstat_create_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+
+ if (inj_stats_kind == PGSTAT_KIND_INVALID)
+ inj_stats_kind = pgstat_add_kind(&injection_stats);
+
+ entry_ref = pgstat_get_entry_ref_locked(inj_stats_kind, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ /* initialize shared memory data */
+ memset(&shstatent->stats, 0, sizeof(shstatent->stats));
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * Report injection point drop.
+ */
+void
+pgstat_drop_inj(const char *name)
+{
+ if (inj_stats_kind == PGSTAT_KIND_INVALID)
+ inj_stats_kind = pgstat_add_kind(&injection_stats);
+
+ if (!pgstat_drop_entry(inj_stats_kind, InvalidOid,
+ PGSTAT_INJ_IDX(name)))
+ pgstat_request_entry_refs_gc();
+}
+
+/*
+ * Report statistics for injection point.
+ *
+ * This is simple because the set of stats to report currently is simple:
+ * track the number of times a point has been run.
+ */
+void
+pgstat_report_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+ PgStat_StatInjEntry *statent;
+
+ if (inj_stats_kind == PGSTAT_KIND_INVALID)
+ inj_stats_kind = pgstat_add_kind(&injection_stats);
+
+ entry_ref = pgstat_get_entry_ref_locked(inj_stats_kind, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+ statent = &shstatent->stats;
+
+ /* Update the injection point statistics */
+ statent->numcalls++;
+
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * SQL function returning the number of times an injection point
+ * has been called.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_numcalls);
+Datum
+injection_points_stats_numcalls(PG_FUNCTION_ARGS)
+{
+ char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_StatInjEntry *entry = pgstat_fetch_stat_injentry(name);
+
+ if (entry == NULL)
+ PG_RETURN_NULL();
+
+ PG_RETURN_INT64(entry->numcalls);
+}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
new file mode 100644
index 0000000000..15621f49fd
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -0,0 +1,22 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.h
+ * Definitions for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.h
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#ifndef INJECTION_STATS
+#define INJECTION_STATS
+
+extern void pgstat_create_inj(const char *name);
+extern void pgstat_drop_inj(const char *name);
+extern void pgstat_report_inj(const char *name);
+
+#endif
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index 8e1b5b4539..526fbc1457 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -6,6 +6,7 @@ endif
injection_points_sources = files(
'injection_points.c',
+ 'injection_stats.c',
)
if host_system == 'windows'
diff --git a/src/test/modules/injection_points/sql/injection_points.sql b/src/test/modules/injection_points/sql/injection_points.sql
index 71e2972a7e..04cf7f48db 100644
--- a/src/test/modules/injection_points/sql/injection_points.sql
+++ b/src/test/modules/injection_points/sql/injection_points.sql
@@ -66,5 +66,20 @@ SELECT injection_points_detach('TestConditionError');
SELECT injection_points_attach('TestConditionLocal1', 'error');
SELECT injection_points_detach('TestConditionLocal1');
+-- Statistics
+SELECT injection_points_stats_numcalls('TestConditionStats'); -- nothing
+SELECT injection_points_attach('TestConditionStats', 'notice');
+SELECT injection_points_stats_numcalls('TestConditionStats'); -- returns 0
+SELECT injection_points_run('TestConditionStats');
+SELECT injection_points_stats_numcalls('TestConditionStats'); -- returns 1
+-- Check transactions with stats snapshots
+BEGIN;
+SET stats_fetch_consistency TO snapshot;
+SELECT injection_points_stats_numcalls('TestConditionStats'); -- returns 1
+SELECT injection_points_run('TestConditionStats');
+SELECT injection_points_stats_numcalls('TestConditionStats'); -- still 1
+COMMIT;
+SELECT injection_points_stats_numcalls('TestConditionStats'); -- now 2
+SELECT injection_points_detach('TestConditionStats');
DROP EXTENSION injection_points;
DROP FUNCTION wait_pid;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8718ca54e0..29197fe212 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2119,6 +2119,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_InjectionPoint
PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
@@ -2150,6 +2151,7 @@ PgStat_Snapshot
PgStat_SnapshotEntry
PgStat_StatDBEntry
PgStat_StatFuncEntry
+PgStat_StatInjEntry
PgStat_StatReplSlotEntry
PgStat_StatSubEntry
PgStat_StatTabEntry
--
2.43.0
0005-doc-Add-section-for-Custom-Cumulative-Statistics-API.patchtext/x-diff; charset=us-asciiDownload
From 6b2c701f703febe129a444f43ca290b5ea6fcc7a Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 13 Jun 2024 13:37:33 +0900
Subject: [PATCH 5/6] doc: Add section for Custom Cumulative Statistics APIs
This provides a short description of what can be done, with a pointer to
the template in the tree.
---
doc/src/sgml/xfunc.sgml | 50 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 50 insertions(+)
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index a7c170476a..e611ab233e 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3676,6 +3676,56 @@ extern bool InjectionPointDetach(const char *name);
</para>
</sect2>
+ <sect2 id="xfunc-addin-custom-cumulative-statistics">
+ <title>Custom Cumulative Statistics</title>
+
+ <para>
+ Is is possible for add-ins written in C-language to use custom types
+ of statistics registered in the
+ <link linkend="monitoring-stats-setup">Cumulative Statistics System</link>.
+ </para>
+
+ <para>
+ First, define a <literal>PgStat_KindInfo</literal> that includes all
+ the information related to the custom type registered. For example:
+<programlisting>
+static const PgStat_KindInfo custom_stats = {
+ .name = "custom_stats",
+ .fixed_amount = false,
+ .shared_size = sizeof(PgStatShared_Custom),
+ .shared_data_off = offsetof(PgStatShared_Custom, stats),
+ .shared_data_len = sizeof(((PgStatShared_Custom *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatCustomEntry),
+}
+</programlisting>
+
+ Then, each backend that needs to use this custom type needs to register
+ it with <literal>pgstat_add_kind</literal>, that returns a unique ID
+ used to store the entries related to this type of statistics:
+<programlisting>
+extern PgStat_Kind pgstat_add_kind(const PgStat_KindInfo *kind_info);
+</programlisting>
+ </para>
+
+ <para>
+ The details of the API for <literal>PgStat_KindInfo</literal> can
+ be found in <filename>src/include/utils/pgstat_internal.h</filename>.
+ </para>
+
+ <para>
+ The type of statistics registered is associated with a name and a unique
+ ID shared across the server in shared memory. Each backend using a
+ custom type of statistics maintains a local cache storing the information
+ of each PgStat_KindInfo.
+ </para>
+
+ <para>
+ An example describing how to register and use custom statistics can be
+ found in
+ <filename>src/test/modules/injection_points/injection_stats.c</filename>.
+ </para>
+ </sect2>
+
<sect2 id="extend-cpp">
<title>Using C++ for Extensibility</title>
--
2.43.0
0006-Extend-custom-cumulative-stats-to-be-persistent.patchtext/x-diff; charset=us-asciiDownload
From 1a096ea5bff6550815d81e80134a8e6b9aa44d6e Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 13 Jun 2024 15:54:32 +0900
Subject: [PATCH 6/6] Extend custom cumulative stats to be persistent
This patch uses the approach of a unique file, so as all the stats are
stored in the existing PGSTAT_STAT_PERMANENT_FILENAME, extending its
format with a new entry type to be able to read on load and write at
shutdown the custom kinds of statistics registered:
- The name of the custom stats kinds is stored into the stats file
rather than its ID.
- The rest of the hash key, made of the database OID and the object OID,
is stored if the custom kind is not serialized. If serialized, where
callbacks are used to do a mapping of the on-disk name <-> hash key, the
serialized name is pushed instead.
The patch includes an example of what can be achieved with the test
module injection_points, with a TAP test checking if statistics can be
kept after a clean restart.
---
src/include/pgstat.h | 2 +-
src/backend/utils/activity/pgstat.c | 96 ++++++++++++++++++-
src/backend/utils/adt/pgstatfuncs.c | 2 +-
src/test/modules/injection_points/Makefile | 4 +
.../injection_points/injection_points.c | 29 +++++-
.../injection_points/injection_stats.c | 54 ++++++++++-
.../injection_points/injection_stats.h | 4 +
src/test/modules/injection_points/meson.build | 8 ++
.../modules/injection_points/t/001_stats.pl | 50 ++++++++++
9 files changed, 240 insertions(+), 9 deletions(-)
create mode 100644 src/test/modules/injection_points/t/001_stats.pl
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index b3cdc0da6d..6c9084b977 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -487,7 +487,7 @@ extern void pgstat_clear_snapshot(void);
extern TimestampTz pgstat_get_stat_snapshot_timestamp(bool *have_snapshot);
/* helpers */
-extern PgStat_Kind pgstat_get_kind_from_str(char *kind_str);
+extern PgStat_Kind pgstat_get_kind_from_str(char *kind_str, int elevel);
extern bool pgstat_have_entry(PgStat_Kind kind, Oid dboid, Oid objoid);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index b96743ce84..5ea7dbc64b 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -110,6 +110,7 @@
#include "storage/shmem.h"
#include "storage/spin.h"
#include "storage/s_lock.h"
+#include "utils/builtins.h"
#include "utils/guc_hooks.h"
#include "utils/hsearch.h"
#include "utils/memutils.h"
@@ -142,6 +143,7 @@
* Identifiers in stats file.
* ---------
*/
+#define PGSTAT_FILE_CUSTOM 'C' /* custom stats kind */
#define PGSTAT_FILE_END 'E' /* end of file */
#define PGSTAT_FILE_NAME 'N' /* stats entry identified by name */
#define PGSTAT_FILE_SYSTEM 'S' /* stats entry identified by PgStat_HashKey */
@@ -1372,7 +1374,7 @@ pgstat_flush_pending_entries(bool nowait)
*/
PgStat_Kind
-pgstat_get_kind_from_str(char *kind_str)
+pgstat_get_kind_from_str(char *kind_str, int elevel)
{
for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
{
@@ -1392,10 +1394,10 @@ pgstat_get_kind_from_str(char *kind_str)
}
}
- ereport(ERROR,
+ ereport(elevel,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid statistics kind: \"%s\"", kind_str)));
- return PGSTAT_KIND_DATABASE; /* avoid compiler warnings */
+ return PGSTAT_KIND_INVALID; /* avoid compiler warnings */
}
static inline bool
@@ -1478,7 +1480,7 @@ pgstat_add_kind(const PgStat_KindInfo *kind_info)
if (strlen(kind_info->name) >= NAMEDATALEN)
elog(ERROR,
- "cannot use custom stats kind longer than %u characters",
+ "cannot define custom stats kind name longer than %u characters",
NAMEDATALEN - 1);
/*
@@ -1489,6 +1491,15 @@ pgstat_add_kind(const PgStat_KindInfo *kind_info)
elog(ERROR,
"cannot define custom stats kind with fixed amount of data");
+ /*
+ * Custom stats entries are represented on disk with their kind name
+ * and their entry name, so these are mandatory.
+ */
+ if (kind_info->to_serialized_name == NULL ||
+ kind_info->from_serialized_name == NULL)
+ elog(ERROR,
+ "cannot define custom stats kind without serialization callbacks");
+
/*
* Check if kind ID associated to the name is already defined, and return
* it if so.
@@ -1707,7 +1718,37 @@ pgstat_write_statsfile(void)
/* if not dropped the valid-entry refcount should exist */
Assert(pg_atomic_read_u32(&ps->refcount) > 0);
- if (!kind_info->to_serialized_name)
+ if (ps->key.kind > PGSTAT_KIND_LAST)
+ {
+ /*
+ * Custom stats entry, identified by kind name and entry name on
+ * disk.
+ */
+ NameData name;
+ NameData kind_name;
+
+ fputc(PGSTAT_FILE_CUSTOM, fpout);
+
+ /* Kind name */
+ namestrcpy(&kind_name, kind_info->name);
+ write_chunk_s(fpout, &kind_name);
+
+ /*
+ * If serialized use the on-disk format, or use the hash
+ * key components for the database OID and object OID.
+ */
+ if (kind_info->to_serialized_name)
+ {
+ kind_info->to_serialized_name(&ps->key, shstats, &name);
+ write_chunk_s(fpout, &name);
+ }
+ else
+ {
+ write_chunk_s(fpout, &ps->key.dboid);
+ write_chunk_s(fpout, &ps->key.objoid);
+ }
+ }
+ else if (!kind_info->to_serialized_name)
{
/* normal stats entry, identified by PgStat_HashKey */
fputc(PGSTAT_FILE_SYSTEM, fpout);
@@ -1874,6 +1915,7 @@ pgstat_read_statsfile(void)
switch (t)
{
+ case PGSTAT_FILE_CUSTOM:
case PGSTAT_FILE_SYSTEM:
case PGSTAT_FILE_NAME:
{
@@ -1892,6 +1934,50 @@ pgstat_read_statsfile(void)
if (!pgstat_is_kind_valid(key.kind))
goto error;
}
+ else if (t == PGSTAT_FILE_CUSTOM)
+ {
+ NameData kind_name;
+ NameData name;
+ PgStat_Kind kind;
+ const PgStat_KindInfo *kind_info = NULL;
+
+ /* First comes the name of the stats */
+ if (!read_chunk_s(fpin, &kind_name))
+ goto error;
+
+ /* Check if it is a valid stats kind */
+ kind = pgstat_get_kind_from_str(NameStr(kind_name),
+ WARNING);
+ if (!pgstat_is_kind_valid(kind))
+ goto error;
+
+ kind_info = pgstat_get_kind_info(kind);
+
+ /* Then comes the entry name, if serialized */
+ if (kind_info->from_serialized_name)
+ {
+ if (!read_chunk_s(fpin, &name))
+ goto error;
+
+ /* Compile its key */
+ if (!kind_info->from_serialized_name(&name, &key))
+ {
+ /* skip over data for entry we don't care about */
+ if (fseek(fpin, pgstat_get_entry_len(kind), SEEK_CUR) != 0)
+ goto error;
+ continue;
+ }
+ }
+ else
+ {
+ /* Extract the rest of the hash key */
+ key.kind = kind;
+ if (!read_chunk_s(fpin, &key.dboid))
+ goto error;
+ if (!read_chunk_s(fpin, &key.objoid))
+ goto error;
+ }
+ }
else
{
/* stats entry identified by name on disk (e.g. slots) */
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 3876339ee1..59ef463687 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -2028,7 +2028,7 @@ pg_stat_have_stats(PG_FUNCTION_ARGS)
char *stats_type = text_to_cstring(PG_GETARG_TEXT_P(0));
Oid dboid = PG_GETARG_OID(1);
Oid objoid = PG_GETARG_OID(2);
- PgStat_Kind kind = pgstat_get_kind_from_str(stats_type);
+ PgStat_Kind kind = pgstat_get_kind_from_str(stats_type, ERROR);
PG_RETURN_BOOL(pgstat_have_entry(kind, dboid, objoid));
}
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 676823a87f..9d84d9aaf3 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -12,9 +12,13 @@ PGFILEDESC = "injection_points - facility for injection points"
REGRESS = injection_points
REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
+TAP_TESTS = 1
+
# The injection points are cluster-wide, so disable installcheck
NO_INSTALLCHECK = 1
+export enable_injection_points enable_injection_points
+
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index 6346af6cf6..f0c8c67826 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -34,9 +34,11 @@
PG_MODULE_MAGIC;
+/* Hooks */
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
/* Maximum number of waits usable in injection points at once */
#define INJ_MAX_WAIT 8
-#define INJ_NAME_MAXLEN 64
/*
* Conditions related to injection points. This tracks in shared memory the
@@ -419,3 +421,28 @@ injection_points_detach(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+
+static void
+injection_points_shmem_startup(void)
+{
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /*
+ * Note that this does not call injection_init_shmem(), as it relies
+ * on the DSM registry, which cannot be used in the postmaster context.
+ */
+
+ /* Register custom statistics */
+ pgstat_register_inj();
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = injection_points_shmem_startup;
+}
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
index 6314486d79..3acbccd32a 100644
--- a/src/test/modules/injection_points/injection_stats.c
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -22,9 +22,13 @@
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
+/* Maximum name length */
+#define INJ_NAME_MAXLEN 64
+
/* Structures for statistics of injection points */
typedef struct PgStat_StatInjEntry
{
+ char name[INJ_NAME_MAXLEN]; /* used for serialization */
PgStat_Counter numcalls; /* number of times point has been run */
} PgStat_StatInjEntry;
@@ -35,6 +39,11 @@ typedef struct PgStatShared_InjectionPoint
} PgStatShared_InjectionPoint;
static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+static void injection_stats_to_serialized_name_cb(const PgStat_HashKey *key,
+ const PgStatShared_Common *header,
+ NameData *name);
+static bool injection_stats_from_serialized_name_cb(const NameData *name,
+ PgStat_HashKey *key);
static const PgStat_KindInfo injection_stats = {
.name = "injection_points",
@@ -48,6 +57,8 @@ static const PgStat_KindInfo injection_stats = {
.shared_data_len = sizeof(((PgStatShared_InjectionPoint *) 0)->stats),
.pending_size = sizeof(PgStat_StatInjEntry),
.flush_pending_cb = injection_stats_flush_cb,
+ .to_serialized_name = injection_stats_to_serialized_name_cb,
+ .from_serialized_name = injection_stats_from_serialized_name_cb,
};
/*
@@ -58,7 +69,7 @@ static const PgStat_KindInfo injection_stats = {
static PgStat_Kind inj_stats_kind = PGSTAT_KIND_INVALID;
/*
- * Callback for stats handling
+ * Callbacks for stats handling
*/
static bool
injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
@@ -76,6 +87,35 @@ injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
return true;
}
+/*
+ * Transforms a hash key into a name that is then stored on disk.
+ */
+static void
+injection_stats_to_serialized_name_cb(const PgStat_HashKey *key,
+ const PgStatShared_Common *header,
+ NameData *name)
+{
+ PgStatShared_InjectionPoint *shstatent =
+ (PgStatShared_InjectionPoint *) header;
+
+ namestrcpy(name, shstatent->stats.name);
+}
+
+/*
+ * Computes the hash key used for this injection point name.
+ */
+static bool
+injection_stats_from_serialized_name_cb(const NameData *name,
+ PgStat_HashKey *key)
+{
+ uint32 idx = PGSTAT_INJ_IDX(NameStr(*name));
+
+ key->kind = inj_stats_kind;
+ key->dboid = InvalidOid;
+ key->objoid = idx;
+ return true;
+}
+
/*
* Support function for the SQL-callable pgstat* functions. Returns
* a pointer to the injection point statistics struct.
@@ -95,6 +135,16 @@ pgstat_fetch_stat_injentry(const char *name)
return entry;
}
+/*
+ * Workhorse to do the registration work, called in _PG_init().
+ */
+void
+pgstat_register_inj(void)
+{
+ if (inj_stats_kind == PGSTAT_KIND_INVALID)
+ inj_stats_kind = pgstat_add_kind(&injection_stats);
+}
+
/*
* Report injection point creation.
*/
@@ -113,6 +163,8 @@ pgstat_create_inj(const char *name)
/* initialize shared memory data */
memset(&shstatent->stats, 0, sizeof(shstatent->stats));
+ strlcpy(shstatent->stats.name, name, INJ_NAME_MAXLEN);
+
pgstat_unlock_entry(entry_ref);
}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
index 15621f49fd..b59ac4cf1c 100644
--- a/src/test/modules/injection_points/injection_stats.h
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -15,6 +15,10 @@
#ifndef INJECTION_STATS
#define INJECTION_STATS
+/* Maximum name length */
+#define INJ_NAME_MAXLEN 64
+
+extern void pgstat_register_inj(void);
extern void pgstat_create_inj(const char *name);
extern void pgstat_drop_inj(const char *name);
extern void pgstat_report_inj(const char *name);
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index 526fbc1457..612216c144 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -38,4 +38,12 @@ tests += {
# The injection points are cluster-wide, so disable installcheck
'runningcheck': false,
},
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_stats.pl',
+ ],
+ },
}
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
new file mode 100644
index 0000000000..f8d90a3869
--- /dev/null
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -0,0 +1,50 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+use locale;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Test persistency of statistics generated for injection points.
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# Node initialization
+my $node = PostgreSQL::Test::Cluster->new('master');
+$node->init;
+$node->append_conf('postgresql.conf',
+ "shared_preload_libraries = 'injection_points'");
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# This should count for two calls.
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('stats-notice', 'notice');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_run('stats-notice');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_run('stats-notice');");
+my $numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats calls');
+
+# Restart the node cleanly, stats should still be around.
+$node->restart;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats after clean restart');
+
+# On crash the stats are gone.
+$node->stop('immediate');
+$node->start;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '', 'number of stats after clean restart');
+
+done_testing();
--
2.43.0
On Thu, Jun 13, 2024 at 04:59:50PM +0900, Michael Paquier wrote:
- How should the persistence of the custom stats be achieved?
Callbacks to give custom stats kinds a way to write/read their data,
push everything into a single file, or support both?
- Should this do like custom RMGRs and assign to each stats kinds ID
that are set in stone rather than dynamic ones?
These two questions have been itching me in terms of how it would work
for extension developers, after noticing that custom RMGRs are used
more than I thought:
https://wiki.postgresql.org/wiki/CustomWALResourceManagers
The result is proving to be nicer, shorter by 300 lines in total and
much simpler when it comes to think about the way stats are flushed
because it is possible to achieve the same result as the first patch
set without manipulating any of the code paths doing the read and
write of the pgstats file.
In terms of implementation, pgstat.c's KindInfo data is divided into
two parts, for efficiency:
- The exiting in-core stats with designated initializers, renamed as
built-in stats kinds.
- The custom stats kinds are saved in TopMemoryContext, and can only
be registered with shared_preload_libraries. The patch reserves a set
of 128 harcoded slots for all the custom kinds making the lookups for
the KindInfos quite cheap. Upon registration, a custom stats kind
needs to assign a unique ID, with uniqueness on the names and IDs
checked at registration.
The backend code does ID -> information lookups in the hotter paths,
meaning that the code only checks if an ID is built-in or custom, then
redirects to the correct array where the information is stored.
There is one code path that does a name -> information lookup for the
undocumented SQL function pg_stat_have_stats() used in the tests,
which is a bit less efficient now, but that does not strike me as an
issue.
modules/injection_points/ works as previously as a template to show
how to use these APIs, with tests for the whole.
With that in mind, the patch set is more pleasant to the eye, and the
attached v2 consists of:
- 0001 and 0002 are some cleanups, same as previously to prepare for
the backend-side APIs.
- 0003 adds the backend support to plug-in custom stats.
- 0004 includes documentation.
- 0005 is an example of how to use them, with a TAP test providing
coverage.
Note that the patch I've proposed to make stats persistent at
checkpoint so as we don't discard everything after a crash is able to
work with the custom stats proposed on this thread:
https://commitfest.postgresql.org/48/5047/
--
Michael
Attachments:
v2-0001-Switch-PgStat_Kind-from-enum-to-uint32.patchtext/x-diff; charset=us-asciiDownload
From 4c065f73cb744d1735c01e6f276d658853810f2e Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 13 Jun 2024 13:25:05 +0900
Subject: [PATCH v2 1/5] Switch PgStat_Kind from enum to uint32
A follow-up patch is planned to make this counter extensible, and
keeping a trace of the kind behind a type is useful in the internal
routines used by pgstats. While on it, switch pgstat_is_kind_valid() to
use PgStat_Kind, to be more consistent with its callers.
---
src/include/pgstat.h | 35 ++++++++++++++---------------
src/backend/utils/activity/pgstat.c | 6 ++---
2 files changed, 20 insertions(+), 21 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 2136239710..2d30fadaf1 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -32,26 +32,25 @@
#define PG_STAT_TMP_DIR "pg_stat_tmp"
/* The types of statistics entries */
-typedef enum PgStat_Kind
-{
- /* use 0 for INVALID, to catch zero-initialized data */
- PGSTAT_KIND_INVALID = 0,
+#define PgStat_Kind uint32
- /* stats for variable-numbered objects */
- PGSTAT_KIND_DATABASE, /* database-wide statistics */
- PGSTAT_KIND_RELATION, /* per-table statistics */
- PGSTAT_KIND_FUNCTION, /* per-function statistics */
- PGSTAT_KIND_REPLSLOT, /* per-slot statistics */
- PGSTAT_KIND_SUBSCRIPTION, /* per-subscription statistics */
+/* use 0 for INVALID, to catch zero-initialized data */
+#define PGSTAT_KIND_INVALID 0
- /* stats for fixed-numbered objects */
- PGSTAT_KIND_ARCHIVER,
- PGSTAT_KIND_BGWRITER,
- PGSTAT_KIND_CHECKPOINTER,
- PGSTAT_KIND_IO,
- PGSTAT_KIND_SLRU,
- PGSTAT_KIND_WAL,
-} PgStat_Kind;
+/* stats for variable-numbered objects */
+#define PGSTAT_KIND_DATABASE 1 /* database-wide statistics */
+#define PGSTAT_KIND_RELATION 2 /* per-table statistics */
+#define PGSTAT_KIND_FUNCTION 3 /* per-function statistics */
+#define PGSTAT_KIND_REPLSLOT 4 /* per-slot statistics */
+#define PGSTAT_KIND_SUBSCRIPTION 5 /* per-subscription statistics */
+
+/* stats for fixed-numbered objects */
+#define PGSTAT_KIND_ARCHIVER 6
+#define PGSTAT_KIND_BGWRITER 7
+#define PGSTAT_KIND_CHECKPOINTER 8
+#define PGSTAT_KIND_IO 9
+#define PGSTAT_KIND_SLRU 10
+#define PGSTAT_KIND_WAL 11
#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index dcc2ad8d95..d558cc1414 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -173,7 +173,7 @@ static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
static void pgstat_build_snapshot_fixed(PgStat_Kind kind);
-static inline bool pgstat_is_kind_valid(int ikind);
+static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
/* ----------
@@ -1254,9 +1254,9 @@ pgstat_get_kind_from_str(char *kind_str)
}
static inline bool
-pgstat_is_kind_valid(int ikind)
+pgstat_is_kind_valid(PgStat_Kind kind)
{
- return ikind >= PGSTAT_KIND_FIRST_VALID && ikind <= PGSTAT_KIND_LAST;
+ return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
}
const PgStat_KindInfo *
--
2.45.1
v2-0002-Replace-hardcoded-identifiers-in-pgstats-file-by-.patchtext/x-diff; charset=us-asciiDownload
From 87d3c9a0f8197bcc962b8b74f85d5710a3a0dba2 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 13 Jun 2024 13:25:48 +0900
Subject: [PATCH v2 2/5] Replace hardcoded identifiers in pgstats file by
variables
This changes three variable types:
- N for named entries.
- S for entries identified by a hash.
- E for end-of-file
---
src/backend/utils/activity/pgstat.c | 23 +++++++++++++++--------
1 file changed, 15 insertions(+), 8 deletions(-)
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index d558cc1414..f03fee7cd5 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -127,6 +127,13 @@
#define PGSTAT_SNAPSHOT_HASH_SIZE 512
+/* ---------
+ * Identifiers in stats file.
+ * ---------
+ */
+#define PGSTAT_FILE_END 'E' /* end of file */
+#define PGSTAT_FILE_NAME 'N' /* stats entry identified by name */
+#define PGSTAT_FILE_SYSTEM 'S' /* stats entry identified by PgStat_HashKey */
/* hash table for statistics snapshots entry */
typedef struct PgStat_SnapshotEntry
@@ -1408,7 +1415,7 @@ pgstat_write_statsfile(void)
if (!kind_info->to_serialized_name)
{
/* normal stats entry, identified by PgStat_HashKey */
- fputc('S', fpout);
+ fputc(PGSTAT_FILE_SYSTEM, fpout);
write_chunk_s(fpout, &ps->key);
}
else
@@ -1418,7 +1425,7 @@ pgstat_write_statsfile(void)
kind_info->to_serialized_name(&ps->key, shstats, &name);
- fputc('N', fpout);
+ fputc(PGSTAT_FILE_NAME, fpout);
write_chunk_s(fpout, &ps->key.kind);
write_chunk_s(fpout, &name);
}
@@ -1435,7 +1442,7 @@ pgstat_write_statsfile(void)
* pgstat.stat with it. The ferror() check replaces testing for error
* after each individual fputc or fwrite (in write_chunk()) above.
*/
- fputc('E', fpout);
+ fputc(PGSTAT_FILE_END, fpout);
if (ferror(fpout))
{
@@ -1572,8 +1579,8 @@ pgstat_read_statsfile(void)
switch (t)
{
- case 'S':
- case 'N':
+ case PGSTAT_FILE_SYSTEM:
+ case PGSTAT_FILE_NAME:
{
PgStat_HashKey key;
PgStatShared_HashEntry *p;
@@ -1581,7 +1588,7 @@ pgstat_read_statsfile(void)
CHECK_FOR_INTERRUPTS();
- if (t == 'S')
+ if (t == PGSTAT_FILE_SYSTEM)
{
/* normal stats entry, identified by PgStat_HashKey */
if (!read_chunk_s(fpin, &key))
@@ -1647,8 +1654,8 @@ pgstat_read_statsfile(void)
break;
}
- case 'E':
- /* check that 'E' actually signals end of file */
+ case PGSTAT_FILE_END:
+ /* check that PGSTAT_FILE_END actually signals end of file */
if (fgetc(fpin) != EOF)
goto error;
--
2.45.1
v2-0003-Introduce-pluggable-APIs-for-Cumulative-Statistic.patchtext/x-diff; charset=us-asciiDownload
From 10ecf14b8bc51d870e2aaccb57386c3fda91615e Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 20 Jun 2024 09:19:54 +0900
Subject: [PATCH v2 3/5] Introduce pluggable APIs for Cumulative Statistics
This commit adds support in the backend for $subject, allowing
out-of-core extensions to add their own custom statistics kinds. The
stats kinds are divided into two parts for efficiency:
- The built-in stats kinds, with designated initializers.
- The custom kinds, able to use a range of IDs (128 slots available as
of this patch), with information saved in TopMemoryContext.
Custom cumulative statistics can only be loaded with
shared_preload_libraries at startup, and must allocate a unique ID
shared across all the PostgreSQL extension ecosystem with the following
wiki page:
https://wiki.postgresql.org/wiki/CustomCumulativeStats
---
src/include/pgstat.h | 32 +++++-
src/include/utils/pgstat_internal.h | 4 +-
src/backend/utils/activity/pgstat.c | 171 +++++++++++++++++++++++++---
src/backend/utils/adt/pgstatfuncs.c | 2 +-
4 files changed, 189 insertions(+), 20 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 2d30fadaf1..59abd422c3 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -52,9 +52,35 @@
#define PGSTAT_KIND_SLRU 10
#define PGSTAT_KIND_WAL 11
-#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
-#define PGSTAT_NUM_KINDS (PGSTAT_KIND_LAST + 1)
+#define PGSTAT_KIND_MIN_BUILTIN PGSTAT_KIND_DATABASE
+#define PGSTAT_KIND_MAX_BUILTIN PGSTAT_KIND_WAL
+
+/* Custom stats kinds */
+#define PGSTAT_KIND_MAX 256 /* Maximum ID allowed */
+
+/* Range of IDs allowed for custom stats kinds */
+#define PGSTAT_KIND_CUSTOM_MIN 128
+#define PGSTAT_KIND_CUSTOM_MAX PGSTAT_KIND_MAX
+#define PGSTAT_KIND_CUSTOM_SIZE (PGSTAT_KIND_CUSTOM_MAX - PGSTAT_KIND_CUSTOM_MIN + 1)
+
+/*
+ * PgStat_Kind to use for extensions that require an ID, but are still in
+ * development and have not reserved their own unique kind ID yet. See:
+ * https://wiki.postgresql.org/wiki/CustomCumulativeStats
+ */
+#define PGSTAT_KIND_EXPERIMENTAL 128
+
+static inline bool
+pgstat_is_kind_builtin(PgStat_Kind kind)
+{
+ return kind > PGSTAT_KIND_INVALID && kind <= PGSTAT_KIND_MAX_BUILTIN;
+}
+
+static inline bool
+pgstat_is_kind_custom(PgStat_Kind kind)
+{
+ return kind >= PGSTAT_KIND_CUSTOM_MIN && kind <= PGSTAT_KIND_CUSTOM_MAX;
+}
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index dbbca31602..0fcee1ff3d 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -446,7 +446,7 @@ typedef struct PgStat_Snapshot
/* time at which snapshot was taken */
TimestampTz snapshot_timestamp;
- bool fixed_valid[PGSTAT_NUM_KINDS];
+ bool fixed_valid[PGSTAT_KIND_MAX_BUILTIN + 1];
PgStat_ArchiverStats archiver;
@@ -503,6 +503,8 @@ static inline void *pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common
*/
extern const PgStat_KindInfo *pgstat_get_kind_info(PgStat_Kind kind);
+extern void pgstat_register_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
#ifdef USE_ASSERT_CHECKING
extern void pgstat_assert_is_up(void);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index f03fee7cd5..5b18d78782 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -49,8 +49,16 @@
* pgStatPending list. Pending statistics updates are flushed out by
* pgstat_report_stat().
*
+ * It is possible for out-of-core modules to define custom statistics kinds,
+ * that can use the same properties as any in-core stats kinds. Each custom
+ * stats kind needs to assign a unique ID to ensure that it does not overlap
+ * with other extensions. In order to reserve a unique stats kind ID, refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats.
+ *
* The behavior of different kinds of statistics is determined by the kind's
- * entry in pgstat_kind_infos, see PgStat_KindInfo for details.
+ * entry in pgstat_kind_builtin_infos for all statistics kinds defined in
+ * core, and pgstat_kind_custom_infos for custom kinds registered at startup
+ * by pgstat_register_kind(). See PgStat_KindInfo for details.
*
* The consistency of read accesses to statistics can be configured using the
* stats_fetch_consistency GUC (see config.sgml and monitoring.sgml for the
@@ -249,7 +257,7 @@ static bool pgstat_is_shutdown = false;
/*
- * The different kinds of statistics.
+ * The different kinds of built-in statistics.
*
* If reasonably possible, handling specific to one kind of stats should go
* through this abstraction, rather than making more of pgstat.c aware.
@@ -261,7 +269,7 @@ static bool pgstat_is_shutdown = false;
* seem to be a great way of doing that, given the split across multiple
* files.
*/
-static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
+static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_MAX_BUILTIN + 1] = {
/* stats kinds for variable-numbered objects */
@@ -399,6 +407,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
},
};
+/*
+ * Information about custom statistics kinds.
+ *
+ * These are saved in a different array than the built-in kinds to save
+ * in clarity with the initializations.
+ *
+ * Indexed by PGSTAT_KIND_CUSTOM_MIN, of size PGSTAT_KIND_CUSTOM_SIZE.
+ */
+static const PgStat_KindInfo **pgstat_kind_custom_infos = NULL;
/* ------------------------------------------------------------
* Functions managing the state of the stats system for all backends.
@@ -1051,7 +1068,7 @@ pgstat_build_snapshot(void)
/*
* Build snapshot of all fixed-numbered stats.
*/
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN_BUILTIN; kind <= PGSTAT_KIND_MAX_BUILTIN; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
@@ -1248,22 +1265,33 @@ pgstat_flush_pending_entries(bool nowait)
PgStat_Kind
pgstat_get_kind_from_str(char *kind_str)
{
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN_BUILTIN; kind <= PGSTAT_KIND_MAX_BUILTIN; kind++)
{
- if (pg_strcasecmp(kind_str, pgstat_kind_infos[kind].name) == 0)
+ if (pg_strcasecmp(kind_str, pgstat_kind_builtin_infos[kind].name) == 0)
return kind;
}
+ /* Check the custom set of cumulative stats */
+ if (pgstat_kind_custom_infos)
+ {
+ for (int kind = 0; kind < PGSTAT_KIND_CUSTOM_SIZE; kind++)
+ {
+ if (pgstat_kind_custom_infos[kind] &&
+ pg_strcasecmp(kind_str, pgstat_kind_custom_infos[kind]->name) == 0)
+ return kind + PGSTAT_KIND_CUSTOM_MIN;
+ }
+ }
+
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid statistics kind: \"%s\"", kind_str)));
- return PGSTAT_KIND_DATABASE; /* avoid compiler warnings */
+ return PGSTAT_KIND_INVALID; /* avoid compiler warnings */
}
static inline bool
pgstat_is_kind_valid(PgStat_Kind kind)
{
- return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
+ return pgstat_is_kind_builtin(kind) || pgstat_is_kind_custom(kind);
}
const PgStat_KindInfo *
@@ -1271,7 +1299,94 @@ pgstat_get_kind_info(PgStat_Kind kind)
{
Assert(pgstat_is_kind_valid(kind));
- return &pgstat_kind_infos[kind];
+ if (pgstat_is_kind_builtin(kind))
+ return &pgstat_kind_builtin_infos[kind];
+
+ if (pgstat_is_kind_custom(kind))
+ {
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (pgstat_kind_custom_infos == NULL &&
+ pgstat_kind_custom_infos[idx] == NULL)
+ return NULL;
+ return pgstat_kind_custom_infos[idx];
+ }
+
+ Assert(false);
+ return NULL; /* keep compiler quiet */
+}
+
+/*
+ * Register a new stats kind.
+ *
+ * PgStat_Kinds must be globally unique across all extensions. Refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats to reserve a
+ * unique ID for your extension, to avoid conflicts with other extension
+ * developers. During development, use PGSTAT_KIND_EXPERIMENTAL to avoid
+ * needlessly reserving a new ID.
+ */
+void
+pgstat_register_kind(PgStat_Kind kind, const PgStat_KindInfo *kind_info)
+{
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (kind_info->name == NULL || strlen(kind_info->name) == 0)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics name is invalid"),
+ errhint("Provide a non-empty name for the custom cumulative statistics.")));
+
+ if (!pgstat_is_kind_custom(kind))
+ ereport(ERROR, (errmsg("custom cumulative statistics ID %u is out of range", kind),
+ errhint("Provide a custom cumulative statistics ID between %u and %u.",
+ PGSTAT_KIND_CUSTOM_MIN, PGSTAT_KIND_CUSTOM_MAX)));
+
+ if (!process_shared_preload_libraries_in_progress)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics must be registered while initializing modules in \"shared_preload_libraries\".")));
+
+ /*
+ * These are not supported for now, as these point out to fixed areas of
+ * shared memory.
+ */
+ if (kind_info->fixed_amount)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics property is invalid"),
+ errhint("Custom cumulative statistics cannot use a fixed amount of data.")));
+
+ /*
+ * If pgstat_kind_custom_infos is not available yet, allocate it.
+ */
+ if (pgstat_kind_custom_infos == NULL)
+ {
+ pgstat_kind_custom_infos = (const PgStat_KindInfo **)
+ MemoryContextAllocZero(TopMemoryContext,
+ sizeof(PgStat_KindInfo *) * PGSTAT_KIND_CUSTOM_SIZE);
+ }
+
+ if (pgstat_kind_custom_infos[idx] != NULL &&
+ pgstat_kind_custom_infos[idx]->name != NULL)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom resource manager \"%s\" already registered with the same ID.",
+ pgstat_kind_custom_infos[idx]->name)));
+
+ /* check for existing custom stats with the same name */
+ for (int existing_kind = 0; existing_kind < PGSTAT_KIND_CUSTOM_SIZE; existing_kind++)
+ {
+ if (pgstat_kind_custom_infos[existing_kind] == NULL)
+ continue;
+ if (!pg_strcasecmp(pgstat_kind_custom_infos[existing_kind]->name, kind_info->name))
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Existing cumulative statistics with ID %u has the same name.", existing_kind + PGSTAT_KIND_CUSTOM_MIN)));
+ }
+
+ /* Register it */
+ pgstat_kind_custom_infos[idx] = kind_info;
+ ereport(LOG,
+ (errmsg("registered custom resource manager \"%s\" with ID %u",
+ kind_info->name, kind)));
}
/*
@@ -1349,7 +1464,7 @@ pgstat_write_statsfile(void)
/*
* XXX: The following could now be generalized to just iterate over
- * pgstat_kind_infos instead of knowing about the different kinds of
+ * pgstat_kind_builtin_infos instead of knowing about the different kinds of
* stats.
*/
@@ -1405,6 +1520,17 @@ pgstat_write_statsfile(void)
if (ps->dropped)
continue;
+ /*
+ * This discards data related to custom stats kinds that are unknown
+ * to this process.
+ */
+ if (!pgstat_is_kind_valid(ps->key.kind))
+ {
+ elog(WARNING, "found unknown stats entry %u/%u/%u",
+ ps->key.kind, ps->key.dboid, ps->key.objoid);
+ continue;
+ }
+
shstats = (PgStatShared_Common *) dsa_get_address(pgStatLocal.dsa, ps->body);
kind_info = pgstat_get_kind_info(ps->key.kind);
@@ -1529,8 +1655,8 @@ pgstat_read_statsfile(void)
/*
* XXX: The following could now be generalized to just iterate over
- * pgstat_kind_infos instead of knowing about the different kinds of
- * stats.
+ * pgstat_kind_builtin_infos instead of knowing about the different kinds
+ * of stats.
*/
/*
@@ -1639,7 +1765,7 @@ pgstat_read_statsfile(void)
if (found)
{
dshash_release_lock(pgStatLocal.shared_hash, p);
- elog(WARNING, "found duplicate stats entry %d/%u/%u",
+ elog(WARNING, "found duplicate stats entry %u/%u/%u",
key.kind, key.dboid, key.objoid);
goto error;
}
@@ -1692,8 +1818,8 @@ pgstat_reset_after_failure(void)
{
TimestampTz ts = GetCurrentTimestamp();
- /* reset fixed-numbered stats */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ /* reset fixed-numbered stats for built-in */
+ for (int kind = PGSTAT_KIND_MIN_BUILTIN; kind <= PGSTAT_KIND_MAX_BUILTIN; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
@@ -1703,6 +1829,21 @@ pgstat_reset_after_failure(void)
kind_info->reset_all_cb(ts);
}
+ /* do the same for custom stats */
+ if (pgstat_kind_custom_infos)
+ {
+ for (int kind = PGSTAT_KIND_CUSTOM_MIN; kind <= PGSTAT_KIND_CUSTOM_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info)
+ continue;
+ if (!kind_info->fixed_amount)
+ continue;
+ kind_info->reset_all_cb(ts);
+ }
+ }
+
/* and drop variable-numbered ones */
pgstat_drop_all_entries();
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 3876339ee1..3221137123 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1696,7 +1696,7 @@ pg_stat_reset(PG_FUNCTION_ARGS)
* Reset some shared cluster-wide counters
*
* When adding a new reset target, ideally the name should match that in
- * pgstat_kind_infos, if relevant.
+ * pgstat_kind_builtin_infos, if relevant.
*/
Datum
pg_stat_reset_shared(PG_FUNCTION_ARGS)
--
2.45.1
v2-0004-doc-Add-section-for-Custom-Cumulative-Statistics-.patchtext/x-diff; charset=us-asciiDownload
From e74cbbdedbb1cf4945b401057e5e111116bdba22 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 13 Jun 2024 13:37:33 +0900
Subject: [PATCH v2 4/5] doc: Add section for Custom Cumulative Statistics APIs
This provides a short description of what can be done, with a pointer to
the template in the tree.
---
doc/src/sgml/xfunc.sgml | 63 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 63 insertions(+)
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index 1d0b65193e..a1c2319857 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3682,6 +3682,69 @@ extern bool InjectionPointDetach(const char *name);
</para>
</sect2>
+ <sect2 id="xfunc-addin-custom-cumulative-statistics">
+ <title>Custom Cumulative Statistics</title>
+
+ <para>
+ Is is possible for add-ins written in C-language to use custom types
+ of cumulative statistics registered in the
+ <link linkend="monitoring-stats-setup">Cumulative Statistics System</link>.
+ </para>
+
+ <para>
+ First, define a <literal>PgStat_KindInfo</literal> that includes all
+ the information related to the custom type registered. For example:
+<programlisting>
+static const PgStat_KindInfo custom_stats = {
+ .name = "custom_stats",
+ .fixed_amount = false,
+ .shared_size = sizeof(PgStatShared_Custom),
+ .shared_data_off = offsetof(PgStatShared_Custom, stats),
+ .shared_data_len = sizeof(((PgStatShared_Custom *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatCustomEntry),
+}
+</programlisting>
+
+ Then, each backend that needs to use this custom type needs to register
+ it with <literal>pgstat_register_kind</literal> and a unique ID used to
+ store the entries related to this type of statistics:
+<programlisting>
+extern PgStat_Kind pgstat_add_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
+</programlisting>
+ While developing a new extension, use
+ <literal>PGSTAT_KIND_EXPERIMENTAL</literal> for
+ <parameter>kind</parameter>. When you are ready to release the extension
+ to users, reserve a kind ID at the
+ <ulink url="https://wiki.postgresql.org/wiki/CustomCumulativeStats">
+ Custom Cumulative Statistics</ulink> page.
+ </para>
+
+ <para>
+ The details of the API for <literal>PgStat_KindInfo</literal> can
+ be found in <filename>src/include/utils/pgstat_internal.h</filename>.
+ </para>
+
+ <para>
+ The type of statistics registered is associated with a name and a unique
+ ID shared across the server in shared memory. Each backend using a
+ custom type of statistics maintains a local cache storing the information
+ of each custom <literal>PgStat_KindInfo</literal>.
+ </para>
+
+ <para>
+ Place the extension module implementing the custom cumulative statistics
+ type in <xref linkend="guc-shared-preload-libraries"/> so that it will
+ be loaded early during <productname>PostgreSQL</productname> startup.
+ </para>
+
+ <para>
+ An example describing how to register and use custom statistics can be
+ found in
+ <filename>src/test/modules/injection_points/injection_stats.c</filename>.
+ </para>
+ </sect2>
+
<sect2 id="extend-cpp">
<title>Using C++ for Extensibility</title>
--
2.45.1
v2-0005-injection_points-Add-statistics-for-custom-points.patchtext/x-diff; charset=us-asciiDownload
From c798aba13372aa2afdf6b9452fa28b8915e6043e Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 20 Jun 2024 09:18:35 +0900
Subject: [PATCH v2 5/5] injection_points: Add statistics for custom points
This acts as a template of what can be achieved with the pluggable
cumulative stats APIs, while being useful on its own for injection
points.
Currently, the only data gathered is the number of times an injection
point is called. This can be extended as required. All the routines
related to the stats are located in their own file, for clarity.
A TAP test is included to provide coverage for these new APIs, showing
the persistency of the data across restarts.
---
src/test/modules/injection_points/Makefile | 11 +-
.../injection_points--1.0.sql | 10 +
.../injection_points/injection_points.c | 27 +++
.../injection_points/injection_stats.c | 194 ++++++++++++++++++
.../injection_points/injection_stats.h | 23 +++
src/test/modules/injection_points/meson.build | 9 +
.../modules/injection_points/t/001_stats.pl | 50 +++++
src/tools/pgindent/typedefs.list | 2 +
8 files changed, 324 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/injection_points/injection_stats.c
create mode 100644 src/test/modules/injection_points/injection_stats.h
create mode 100644 src/test/modules/injection_points/t/001_stats.pl
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 31bd787994..9d84d9aaf3 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -1,7 +1,10 @@
# src/test/modules/injection_points/Makefile
-MODULES = injection_points
-
+MODULE_big = injection_points
+OBJS = \
+ $(WIN32RES) \
+ injection_points.o \
+ injection_stats.o
EXTENSION = injection_points
DATA = injection_points--1.0.sql
PGFILEDESC = "injection_points - facility for injection points"
@@ -9,9 +12,13 @@ PGFILEDESC = "injection_points - facility for injection points"
REGRESS = injection_points
REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
+TAP_TESTS = 1
+
# The injection points are cluster-wide, so disable installcheck
NO_INSTALLCHECK = 1
+export enable_injection_points enable_injection_points
+
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index c16a33b08d..deaf47d8ae 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -54,3 +54,13 @@ CREATE FUNCTION injection_points_detach(IN point_name TEXT)
RETURNS void
AS 'MODULE_PATHNAME', 'injection_points_detach'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+--
+-- injection_points_stats_numcalls()
+--
+-- Reports statistics, if any, related to the given injection point.
+--
+CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
+RETURNS bigint
+AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index 5c44625d1d..46bec5059c 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -18,6 +18,7 @@
#include "postgres.h"
#include "fmgr.h"
+#include "injection_stats.h"
#include "miscadmin.h"
#include "nodes/pg_list.h"
#include "nodes/value.h"
@@ -170,6 +171,9 @@ injection_points_cleanup(int code, Datum arg)
char *name = strVal(lfirst(lc));
(void) InjectionPointDetach(name);
+
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
}
}
@@ -182,6 +186,8 @@ injection_error(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(ERROR, "error triggered for injection point %s", name);
}
@@ -193,6 +199,8 @@ injection_notice(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(NOTICE, "notice triggered for injection point %s", name);
}
@@ -211,6 +219,8 @@ injection_wait(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
/*
* Use the injection point name for this custom wait event. Note that
* this custom wait event name is not released, but we don't care much for
@@ -299,6 +309,10 @@ injection_points_attach(PG_FUNCTION_ARGS)
inj_list_local = lappend(inj_list_local, makeString(pstrdup(name)));
MemoryContextSwitchTo(oldctx);
}
+
+ /* Add entry for stats */
+ pgstat_create_inj(name);
+
PG_RETURN_VOID();
}
@@ -400,5 +414,18 @@ injection_points_detach(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldctx);
}
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
+
PG_RETURN_VOID();
}
+
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ pgstat_register_inj();
+}
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
new file mode 100644
index 0000000000..c37b0b33d3
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -0,0 +1,194 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.c
+ * Code for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+
+#include "common/hashfn.h"
+#include "injection_stats.h"
+#include "pgstat.h"
+#include "utils/builtins.h"
+#include "utils/pgstat_internal.h"
+
+/* Structures for statistics of injection points */
+typedef struct PgStat_StatInjEntry
+{
+ PgStat_Counter numcalls; /* number of times point has been run */
+} PgStat_StatInjEntry;
+
+typedef struct PgStatShared_InjectionPoint
+{
+ PgStatShared_Common header;
+ PgStat_StatInjEntry stats;
+} PgStatShared_InjectionPoint;
+
+static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+
+static const PgStat_KindInfo injection_stats = {
+ .name = "injection_points",
+ .fixed_amount = false, /* Bounded by the number of points */
+
+ /* Injection points are system-wide */
+ .accessed_across_databases = true,
+
+ .shared_size = sizeof(PgStatShared_InjectionPoint),
+ .shared_data_off = offsetof(PgStatShared_InjectionPoint, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPoint *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatInjEntry),
+ .flush_pending_cb = injection_stats_flush_cb,
+};
+
+/*
+ * Compute stats entry idx from point name with a 4-byte hash.
+ */
+#define PGSTAT_INJ_IDX(name) hash_bytes((const unsigned char *) name, strlen(name))
+
+#define PGSTAT_KIND_INJECTION PGSTAT_KIND_EXPERIMENTAL
+
+/* Track if stats are loaded */
+static bool inj_stats_loaded = false;
+
+/*
+ * Callback for stats handling
+ */
+static bool
+injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ PgStat_StatInjEntry *localent;
+ PgStatShared_InjectionPoint *shfuncent;
+
+ localent = (PgStat_StatInjEntry *) entry_ref->pending;
+ shfuncent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ shfuncent->stats.numcalls += localent->numcalls;
+ return true;
+}
+
+/*
+ * Support function for the SQL-callable pgstat* functions. Returns
+ * a pointer to the injection point statistics struct.
+ */
+static PgStat_StatInjEntry *
+pgstat_fetch_stat_injentry(const char *name)
+{
+ PgStat_StatInjEntry *entry = NULL;
+
+ if (!inj_stats_loaded)
+ return NULL;
+
+ /* Compile the lookup key as a hash of the point name */
+ entry = (PgStat_StatInjEntry *) pgstat_fetch_entry(PGSTAT_KIND_INJECTION,
+ InvalidOid,
+ PGSTAT_INJ_IDX(name));
+ return entry;
+}
+
+/*
+ * Workhorse to do the registration work, called in _PG_init().
+ */
+void
+pgstat_register_inj(void)
+{
+ pgstat_register_kind(PGSTAT_KIND_INJECTION, &injection_stats);
+
+ /* mark stats as loaded */
+ inj_stats_loaded = true;
+}
+
+/*
+ * Report injection point creation.
+ */
+void
+pgstat_create_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ /* initialize shared memory data */
+ memset(&shstatent->stats, 0, sizeof(shstatent->stats));
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * Report injection point drop.
+ */
+void
+pgstat_drop_inj(const char *name)
+{
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ if (!pgstat_drop_entry(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name)))
+ pgstat_request_entry_refs_gc();
+}
+
+/*
+ * Report statistics for injection point.
+ *
+ * This is simple because the set of stats to report currently is simple:
+ * track the number of times a point has been run.
+ */
+void
+pgstat_report_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+ PgStat_StatInjEntry *statent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+ statent = &shstatent->stats;
+
+ /* Update the injection point statistics */
+ statent->numcalls++;
+
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * SQL function returning the number of times an injection point
+ * has been called.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_numcalls);
+Datum
+injection_points_stats_numcalls(PG_FUNCTION_ARGS)
+{
+ char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_StatInjEntry *entry = pgstat_fetch_stat_injentry(name);
+
+ if (entry == NULL)
+ PG_RETURN_NULL();
+
+ PG_RETURN_INT64(entry->numcalls);
+}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
new file mode 100644
index 0000000000..3e99705483
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -0,0 +1,23 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.h
+ * Definitions for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.h
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#ifndef INJECTION_STATS
+#define INJECTION_STATS
+
+extern void pgstat_register_inj(void);
+extern void pgstat_create_inj(const char *name);
+extern void pgstat_drop_inj(const char *name);
+extern void pgstat_report_inj(const char *name);
+
+#endif
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index 8e1b5b4539..612216c144 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -6,6 +6,7 @@ endif
injection_points_sources = files(
'injection_points.c',
+ 'injection_stats.c',
)
if host_system == 'windows'
@@ -37,4 +38,12 @@ tests += {
# The injection points are cluster-wide, so disable installcheck
'runningcheck': false,
},
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_stats.pl',
+ ],
+ },
}
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
new file mode 100644
index 0000000000..f8d90a3869
--- /dev/null
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -0,0 +1,50 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+use locale;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Test persistency of statistics generated for injection points.
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# Node initialization
+my $node = PostgreSQL::Test::Cluster->new('master');
+$node->init;
+$node->append_conf('postgresql.conf',
+ "shared_preload_libraries = 'injection_points'");
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# This should count for two calls.
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('stats-notice', 'notice');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_run('stats-notice');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_run('stats-notice');");
+my $numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats calls');
+
+# Restart the node cleanly, stats should still be around.
+$node->restart;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats after clean restart');
+
+# On crash the stats are gone.
+$node->stop('immediate');
+$node->start;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '', 'number of stats after clean restart');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 61ad417cde..481baee16d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2116,6 +2116,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_InjectionPoint
PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
@@ -2147,6 +2148,7 @@ PgStat_Snapshot
PgStat_SnapshotEntry
PgStat_StatDBEntry
PgStat_StatFuncEntry
+PgStat_StatInjEntry
PgStat_StatReplSlotEntry
PgStat_StatSubEntry
PgStat_StatTabEntry
--
2.45.1
Hi,
On Thu, Jun 13, 2024 at 04:59:50PM +0900, Michael Paquier wrote:
Hi all,
2) Make the shmem pgstats pluggable so as it is possible for extensions
to register their own stats kinds.
Thanks for the patch! I like the idea of having custom stats (it has also been
somehow mentioned in [1]/messages/by-id/20220818195124.c7ipzf6c5v7vxymc@awork3.anarazel.de).
2) is actually something that can be used for more things than
just pg_stat_statements, because people love extensions and
statistics (spoiler: I do).
+1
* Making custom stats data persistent is an interesting problem, and
there are a couple of approaches I've considered:
** Allow custom kinds to define callbacks to read and write data from
a source they'd want, like their own file through a fd. This has the
disadvantage to remove the benefit of c) above.
** Store everything in the existing stats file, adding one type of
entry like 'S' and 'N' with a "custom" type, where the *name* of the
custom stats kind is stored instead of its ID computed from shared
memory.
What about having 2 files?
- One is the existing stats file
- One "predefined" for all the custom stats (so what you've done minus the
in-core stats). This one would not be configurable and the extensions will
not need to know about it.
Would that remove the benefit from c) that you mentioned up-thread?
[1]: /messages/by-id/20220818195124.c7ipzf6c5v7vxymc@awork3.anarazel.de
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Thu, Jun 20, 2024 at 09:46:30AM +0900, Michael Paquier wrote:
On Thu, Jun 13, 2024 at 04:59:50PM +0900, Michael Paquier wrote:
- How should the persistence of the custom stats be achieved?
Callbacks to give custom stats kinds a way to write/read their data,
push everything into a single file, or support both?
- Should this do like custom RMGRs and assign to each stats kinds ID
that are set in stone rather than dynamic ones?
These two questions have been itching me in terms of how it would work
for extension developers, after noticing that custom RMGRs are used
more than I thought:
https://wiki.postgresql.org/wiki/CustomWALResourceManagersThe result is proving to be nicer, shorter by 300 lines in total and
much simpler when it comes to think about the way stats are flushed
because it is possible to achieve the same result as the first patch
set without manipulating any of the code paths doing the read and
write of the pgstats file.
I think it makes sense to follow the same "behavior" as the custom
wal resource managers. That, indeed, looks much more simpler than v1.
In terms of implementation, pgstat.c's KindInfo data is divided into
two parts, for efficiency:
- The exiting in-core stats with designated initializers, renamed as
built-in stats kinds.
- The custom stats kinds are saved in TopMemoryContext,
Agree that a backend lifetime memory area is fine for that purpose.
and can only
be registered with shared_preload_libraries. The patch reserves a set
of 128 harcoded slots for all the custom kinds making the lookups for
the KindInfos quite cheap.
+ MemoryContextAllocZero(TopMemoryContext,
+ sizeof(PgStat_KindInfo *) * PGSTAT_KIND_CUSTOM_SIZE);
and that's only 8 * PGSTAT_KIND_CUSTOM_SIZE bytes in total.
I had a quick look at the patches (have in mind to do more):
With that in mind, the patch set is more pleasant to the eye, and the
attached v2 consists of:
- 0001 and 0002 are some cleanups, same as previously to prepare for
the backend-side APIs.
0001 and 0002 look pretty straightforward at a quick look.
- 0003 adds the backend support to plug-in custom stats.
1 ===
It looks to me that there is a mix of "in core" and "built-in" to name the
non custom stats. Maybe it's worth to just use one?
As I can see (and as you said above) this is mainly inspired by the custom
resource manager and 2 === and 3 === are probably copy/paste consequences.
2 ===
+ if (pgstat_kind_custom_infos[idx] != NULL &&
+ pgstat_kind_custom_infos[idx]->name != NULL)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom resource manager \"%s\" already registered with the same ID.",
+ pgstat_kind_custom_infos[idx]->name)));
s/Custom resource manager/Custom cumulative statistics/
3 ===
+ ereport(LOG,
+ (errmsg("registered custom resource manager \"%s\" with ID %u",
+ kind_info->name, kind)));
s/custom resource manager/custom cumulative statistics/
- 0004 includes documentation.
Did not look yet.
- 0005 is an example of how to use them, with a TAP test providing
coverage.
Did not look yet.
As I said, I've in mind to do a more in depth review. I've noted the above while
doing a quick read of the patches so thought it makes sense to share them
now while at it.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Thu, Jun 20, 2024 at 01:05:42PM +0000, Bertrand Drouvot wrote:
On Thu, Jun 13, 2024 at 04:59:50PM +0900, Michael Paquier wrote:
* Making custom stats data persistent is an interesting problem, and
there are a couple of approaches I've considered:
** Allow custom kinds to define callbacks to read and write data from
a source they'd want, like their own file through a fd. This has the
disadvantage to remove the benefit of c) above.
** Store everything in the existing stats file, adding one type of
entry like 'S' and 'N' with a "custom" type, where the *name* of the
custom stats kind is stored instead of its ID computed from shared
memory.What about having 2 files?
- One is the existing stats file
- One "predefined" for all the custom stats (so what you've done minus the
in-core stats). This one would not be configurable and the extensions will
not need to know about it.
Another thing that can be done here is to add a few callbacks to
control how an entry should be written out when the dshash is scanned
or read when the dshash is populated depending on the KindInfo.
That's not really complicated to do as the populate part could have a
cleanup phase if an error is found. I just did not do it yet because
this patch set is already covering a lot, just to get the basics in.
Would that remove the benefit from c) that you mentioned up-thread?
Yes, that can be slightly annoying. Splitting the stats across
multiple files would mean that each stats file would have to store the
redo LSN. That's not really complicated to implement, but really easy
to miss. Perhaps folks implementing their own stats kinds would be
aware anyway because we are going to need a callback to initialize the
file to write if we do that, and the redo LSN should be provided in
input of it. Giving more control to extension developers here would
be OK for me, especially since they could use their own format for
their output file(s).
--
Michael
On Thu, Jun 20, 2024 at 02:27:14PM +0000, Bertrand Drouvot wrote:
On Thu, Jun 20, 2024 at 09:46:30AM +0900, Michael Paquier wrote:
I think it makes sense to follow the same "behavior" as the custom
wal resource managers. That, indeed, looks much more simpler than v1.
Thanks for the feedback.
and can only
be registered with shared_preload_libraries. The patch reserves a set
of 128 harcoded slots for all the custom kinds making the lookups for
the KindInfos quite cheap.+ MemoryContextAllocZero(TopMemoryContext, + sizeof(PgStat_KindInfo *) * PGSTAT_KIND_CUSTOM_SIZE);and that's only 8 * PGSTAT_KIND_CUSTOM_SIZE bytes in total.
Enlarging that does not worry me much. Just not too much.
With that in mind, the patch set is more pleasant to the eye, and the
attached v2 consists of:
- 0001 and 0002 are some cleanups, same as previously to prepare for
the backend-side APIs.0001 and 0002 look pretty straightforward at a quick look.
0002 is quite independentn. Still, 0001 depends a bit on the rest.
Anyway, the Kind is already 4 bytes and it cleans up some APIs that
used int for the Kind, so enforcing signedness is just cleaner IMO.
- 0003 adds the backend support to plug-in custom stats.
1 ===
It looks to me that there is a mix of "in core" and "built-in" to name the
non custom stats. Maybe it's worth to just use one?
Right. Perhaps better to remove "in core" and stick to "builtin", as
I've used the latter for the variables and such.
As I can see (and as you said above) this is mainly inspired by the custom
resource manager and 2 === and 3 === are probably copy/paste consequences.2 ===
+ if (pgstat_kind_custom_infos[idx] != NULL && + pgstat_kind_custom_infos[idx]->name != NULL) + ereport(ERROR, + (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind), + errdetail("Custom resource manager \"%s\" already registered with the same ID.", + pgstat_kind_custom_infos[idx]->name)));s/Custom resource manager/Custom cumulative statistics/
3 ===
+ ereport(LOG, + (errmsg("registered custom resource manager \"%s\" with ID %u", + kind_info->name, kind)));s/custom resource manager/custom cumulative statistics/
Oops. Will fix.
--
Michael
At Thu, 13 Jun 2024 16:59:50 +0900, Michael Paquier <michael@paquier.xyz> wrote in
* The kind IDs may change across restarts, meaning that any stats data
associated to a custom kind is stored with the *name* of the custom
stats kind. Depending on the discussion happening here, I'd be open
to use the same concept as custom RMGRs, where custom kind IDs are
"reserved", fixed in time, and tracked in the Postgres wiki. It is
cheaper to store the stats this way, as well, while managing conflicts
across extensions available in the community ecosystem.
I prefer to avoid having a central database if possible.
If we don't intend to move stats data alone out of a cluster for use
in another one, can't we store the relationship between stats names
and numeric IDs (or index numbers) in a separate file, which is loaded
just before and synced just after extension preloading finishes?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Fri, Jun 21, 2024 at 01:09:10PM +0900, Kyotaro Horiguchi wrote:
At Thu, 13 Jun 2024 16:59:50 +0900, Michael Paquier <michael@paquier.xyz> wrote in
* The kind IDs may change across restarts, meaning that any stats data
associated to a custom kind is stored with the *name* of the custom
stats kind. Depending on the discussion happening here, I'd be open
to use the same concept as custom RMGRs, where custom kind IDs are
"reserved", fixed in time, and tracked in the Postgres wiki. It is
cheaper to store the stats this way, as well, while managing conflicts
across extensions available in the community ecosystem.I prefer to avoid having a central database if possible.
I was thinking the same originally, but the experience with custom
RMGRs has made me change my mind. There are more of these than I
thought originally:
https://wiki.postgresql.org/wiki/CustomWALResourceManagers
If we don't intend to move stats data alone out of a cluster for use
in another one, can't we store the relationship between stats names
and numeric IDs (or index numbers) in a separate file, which is loaded
just before and synced just after extension preloading finishes?
Yeah, I've implemented a prototype that does exactly something like
that with a restriction on the stats name to NAMEDATALEN, except that
I've added the kind ID <-> kind name mapping at the beginning of the
main stats file. At the end, it still felt weird and over-engineered
to me, like the v1 prototype of upthread, because we finish with a
strange mix when reloading the dshash where the builtin ID are handled
with fixed values, with more code paths required when doing the
serialize callback dance for stats kinds like replication slots,
because the custom kinds need to update their hash keys to the new
values based on the ID/name mapping stored at the beginning of the
file itself.
The equation complicates itself a bit more once you'd try to add more
ways to write some stats kinds to other places, depending on what a
custom kind wants to achieve. I can see the benefits of both
approaches, still fixing the IDs in time leads to a lot of simplicity
in this infra, which is very appealing on its own before tackling the
next issues where I would rely on the proposed APIs.
--
Michael
On Fri, Jun 21, 2024 at 08:13:15AM +0900, Michael Paquier wrote:
On Thu, Jun 20, 2024 at 02:27:14PM +0000, Bertrand Drouvot wrote:
On Thu, Jun 20, 2024 at 09:46:30AM +0900, Michael Paquier wrote:
I think it makes sense to follow the same "behavior" as the custom
wal resource managers. That, indeed, looks much more simpler than v1.Thanks for the feedback.
While looking at a different patch from Tristan in this area at [1]/messages/by-id/ZoNytpoHOzHGBLYi@paquier.xyz -- Michael, I
still got annoyed that this patch set was not able to support the case
of custom fixed-numbered stats, so as it is possible to plug in
pgstats things similar to the archiver, the checkpointer, WAL, etc.
These are plugged in shared memory, and are handled with copies in the
stats snapshots. After a good night of sleep, I have come up with a
good solution for that, among the following lines:
- PgStat_ShmemControl holds an array of void* indexed by
PGSTAT_NUM_KINDS, pointing to shared memory areas allocated for each
fixed-numbered stats. Each entry is allocated a size corresponding to
PgStat_KindInfo->shared_size.
- PgStat_Snapshot holds an array of void* also indexed by
PGSTAT_NUM_KINDS, pointing to the fixed stats stored in the
snapshots. These have a size of PgStat_KindInfo->shared_data_len, set
up when stats are initialized at process startup, so this reflects
everywhere.
- Fixed numbered stats now set shared_size, and we use this number to
determine the size to allocate for each fixed-numbered stats in shmem.
- A callback is added to initialize the shared memory assigned to each
fixed-numbered stats, consisting of LWLock initializations for the
current types of stats. So this initialization step is moved out of
pgstat.c into each stats kind file.
All that has been done in the rebased patch set as of 0001, which is
kind of a nice cleanup overall because it removes all the dependencies
to the fixed-numbered stats structures from the "main" pgstats code in
pgstat.c and pgstat_shmem.c.
The remaining patches consist of:
- 0002, Switch PgStat_Kind to a uint32. Cleanup.
- 0003 introduces the pluggable stats facility. Feeding on the
refactoring for the fixed-numbered stats in 0001, it is actually
possible to get support for these in the pluggable APIs by just
removing the restriction in the registration path. This extends the
void* arrays to store references that cover the range of custom kind
IDs.
- 0004 has some docs.
- 0005 includes an example of implementation for variable-numbered
stats with the injection_points module.
- 0006 is new for this thread, implementing an example for
fixed-numbered stats, using again the injection_points module. This
stuff gathers stats about the number of times points are run, attached
and detached. Perhaps that's useful in itself, I don't know, but it
provides the coverage I want for this facility.
While on it, I have applied one of the cleanup patches as
9fd02525793f.
[1]: /messages/by-id/ZoNytpoHOzHGBLYi@paquier.xyz -- Michael
--
Michael
Attachments:
v3-0004-doc-Add-section-for-Custom-Cumulative-Statistics-.patchtext/x-diff; charset=us-asciiDownload
From 2286f57cddd6e5b1cf7b6f771c9f08f9acb2a937 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 3 Jul 2024 13:49:53 +0900
Subject: [PATCH v3 4/6] doc: Add section for Custom Cumulative Statistics APIs
This provides a short description of what can be done, with a pointer to
the template in the tree.
---
doc/src/sgml/xfunc.sgml | 63 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 63 insertions(+)
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index f3a3e4e2f8..7e34c5020b 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3686,6 +3686,69 @@ extern bool InjectionPointDetach(const char *name);
</para>
</sect2>
+ <sect2 id="xfunc-addin-custom-cumulative-statistics">
+ <title>Custom Cumulative Statistics</title>
+
+ <para>
+ Is is possible for add-ins written in C-language to use custom types
+ of cumulative statistics registered in the
+ <link linkend="monitoring-stats-setup">Cumulative Statistics System</link>.
+ </para>
+
+ <para>
+ First, define a <literal>PgStat_KindInfo</literal> that includes all
+ the information related to the custom type registered. For example:
+<programlisting>
+static const PgStat_KindInfo custom_stats = {
+ .name = "custom_stats",
+ .fixed_amount = false,
+ .shared_size = sizeof(PgStatShared_Custom),
+ .shared_data_off = offsetof(PgStatShared_Custom, stats),
+ .shared_data_len = sizeof(((PgStatShared_Custom *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatCustomEntry),
+}
+</programlisting>
+
+ Then, each backend that needs to use this custom type needs to register
+ it with <literal>pgstat_register_kind</literal> and a unique ID used to
+ store the entries related to this type of statistics:
+<programlisting>
+extern PgStat_Kind pgstat_add_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
+</programlisting>
+ While developing a new extension, use
+ <literal>PGSTAT_KIND_EXPERIMENTAL</literal> for
+ <parameter>kind</parameter>. When you are ready to release the extension
+ to users, reserve a kind ID at the
+ <ulink url="https://wiki.postgresql.org/wiki/CustomCumulativeStats">
+ Custom Cumulative Statistics</ulink> page.
+ </para>
+
+ <para>
+ The details of the API for <literal>PgStat_KindInfo</literal> can
+ be found in <filename>src/include/utils/pgstat_internal.h</filename>.
+ </para>
+
+ <para>
+ The type of statistics registered is associated with a name and a unique
+ ID shared across the server in shared memory. Each backend using a
+ custom type of statistics maintains a local cache storing the information
+ of each custom <literal>PgStat_KindInfo</literal>.
+ </para>
+
+ <para>
+ Place the extension module implementing the custom cumulative statistics
+ type in <xref linkend="guc-shared-preload-libraries"/> so that it will
+ be loaded early during <productname>PostgreSQL</productname> startup.
+ </para>
+
+ <para>
+ An example describing how to register and use custom statistics can be
+ found in
+ <filename>src/test/modules/injection_points/injection_stats.c</filename>.
+ </para>
+ </sect2>
+
<sect2 id="extend-cpp">
<title>Using C++ for Extensibility</title>
--
2.45.2
v3-0005-injection_points-Add-statistics-for-custom-points.patchtext/x-diff; charset=us-asciiDownload
From 143d38f0292e08cde279a3073a5ea45844601cc7 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 3 Jul 2024 13:58:58 +0900
Subject: [PATCH v3 5/6] injection_points: Add statistics for custom points
This acts as a template of what can be achieved with the pluggable
cumulative stats APIs, while being useful on its own for injection
points.
Currently, the only data gathered is the number of times an injection
point is called. This can be extended as required. All the routines
related to the stats are located in their own file, for clarity.
A TAP test is included to provide coverage for these new APIs, showing
the persistency of the data across restarts.
---
src/test/modules/injection_points/Makefile | 11 +-
.../injection_points--1.0.sql | 10 +
.../injection_points/injection_points.c | 27 +++
.../injection_points/injection_stats.c | 194 ++++++++++++++++++
.../injection_points/injection_stats.h | 23 +++
src/test/modules/injection_points/meson.build | 9 +
.../modules/injection_points/t/001_stats.pl | 48 +++++
src/tools/pgindent/typedefs.list | 2 +
8 files changed, 322 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/injection_points/injection_stats.c
create mode 100644 src/test/modules/injection_points/injection_stats.h
create mode 100644 src/test/modules/injection_points/t/001_stats.pl
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 2ffd2f77ed..7b9cd12a2a 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -1,7 +1,10 @@
# src/test/modules/injection_points/Makefile
-MODULES = injection_points
-
+MODULE_big = injection_points
+OBJS = \
+ $(WIN32RES) \
+ injection_points.o \
+ injection_stats.o
EXTENSION = injection_points
DATA = injection_points--1.0.sql
PGFILEDESC = "injection_points - facility for injection points"
@@ -11,9 +14,13 @@ REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
ISOLATION = inplace
+TAP_TESTS = 1
+
# The injection points are cluster-wide, so disable installcheck
NO_INSTALLCHECK = 1
+export enable_injection_points enable_injection_points
+
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index c16a33b08d..deaf47d8ae 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -54,3 +54,13 @@ CREATE FUNCTION injection_points_detach(IN point_name TEXT)
RETURNS void
AS 'MODULE_PATHNAME', 'injection_points_detach'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+--
+-- injection_points_stats_numcalls()
+--
+-- Reports statistics, if any, related to the given injection point.
+--
+CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
+RETURNS bigint
+AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index 1b695a1820..f65b34ce82 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -18,6 +18,7 @@
#include "postgres.h"
#include "fmgr.h"
+#include "injection_stats.h"
#include "miscadmin.h"
#include "nodes/pg_list.h"
#include "nodes/value.h"
@@ -170,6 +171,9 @@ injection_points_cleanup(int code, Datum arg)
char *name = strVal(lfirst(lc));
(void) InjectionPointDetach(name);
+
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
}
}
@@ -182,6 +186,8 @@ injection_error(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(ERROR, "error triggered for injection point %s", name);
}
@@ -193,6 +199,8 @@ injection_notice(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(NOTICE, "notice triggered for injection point %s", name);
}
@@ -211,6 +219,8 @@ injection_wait(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
/*
* Use the injection point name for this custom wait event. Note that
* this custom wait event name is not released, but we don't care much for
@@ -299,6 +309,10 @@ injection_points_attach(PG_FUNCTION_ARGS)
inj_list_local = lappend(inj_list_local, makeString(pstrdup(name)));
MemoryContextSwitchTo(oldctx);
}
+
+ /* Add entry for stats */
+ pgstat_create_inj(name);
+
PG_RETURN_VOID();
}
@@ -400,5 +414,18 @@ injection_points_detach(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldctx);
}
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
+
PG_RETURN_VOID();
}
+
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ pgstat_register_inj();
+}
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
new file mode 100644
index 0000000000..c37b0b33d3
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -0,0 +1,194 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.c
+ * Code for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+
+#include "common/hashfn.h"
+#include "injection_stats.h"
+#include "pgstat.h"
+#include "utils/builtins.h"
+#include "utils/pgstat_internal.h"
+
+/* Structures for statistics of injection points */
+typedef struct PgStat_StatInjEntry
+{
+ PgStat_Counter numcalls; /* number of times point has been run */
+} PgStat_StatInjEntry;
+
+typedef struct PgStatShared_InjectionPoint
+{
+ PgStatShared_Common header;
+ PgStat_StatInjEntry stats;
+} PgStatShared_InjectionPoint;
+
+static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+
+static const PgStat_KindInfo injection_stats = {
+ .name = "injection_points",
+ .fixed_amount = false, /* Bounded by the number of points */
+
+ /* Injection points are system-wide */
+ .accessed_across_databases = true,
+
+ .shared_size = sizeof(PgStatShared_InjectionPoint),
+ .shared_data_off = offsetof(PgStatShared_InjectionPoint, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPoint *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatInjEntry),
+ .flush_pending_cb = injection_stats_flush_cb,
+};
+
+/*
+ * Compute stats entry idx from point name with a 4-byte hash.
+ */
+#define PGSTAT_INJ_IDX(name) hash_bytes((const unsigned char *) name, strlen(name))
+
+#define PGSTAT_KIND_INJECTION PGSTAT_KIND_EXPERIMENTAL
+
+/* Track if stats are loaded */
+static bool inj_stats_loaded = false;
+
+/*
+ * Callback for stats handling
+ */
+static bool
+injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ PgStat_StatInjEntry *localent;
+ PgStatShared_InjectionPoint *shfuncent;
+
+ localent = (PgStat_StatInjEntry *) entry_ref->pending;
+ shfuncent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ shfuncent->stats.numcalls += localent->numcalls;
+ return true;
+}
+
+/*
+ * Support function for the SQL-callable pgstat* functions. Returns
+ * a pointer to the injection point statistics struct.
+ */
+static PgStat_StatInjEntry *
+pgstat_fetch_stat_injentry(const char *name)
+{
+ PgStat_StatInjEntry *entry = NULL;
+
+ if (!inj_stats_loaded)
+ return NULL;
+
+ /* Compile the lookup key as a hash of the point name */
+ entry = (PgStat_StatInjEntry *) pgstat_fetch_entry(PGSTAT_KIND_INJECTION,
+ InvalidOid,
+ PGSTAT_INJ_IDX(name));
+ return entry;
+}
+
+/*
+ * Workhorse to do the registration work, called in _PG_init().
+ */
+void
+pgstat_register_inj(void)
+{
+ pgstat_register_kind(PGSTAT_KIND_INJECTION, &injection_stats);
+
+ /* mark stats as loaded */
+ inj_stats_loaded = true;
+}
+
+/*
+ * Report injection point creation.
+ */
+void
+pgstat_create_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ /* initialize shared memory data */
+ memset(&shstatent->stats, 0, sizeof(shstatent->stats));
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * Report injection point drop.
+ */
+void
+pgstat_drop_inj(const char *name)
+{
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ if (!pgstat_drop_entry(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name)))
+ pgstat_request_entry_refs_gc();
+}
+
+/*
+ * Report statistics for injection point.
+ *
+ * This is simple because the set of stats to report currently is simple:
+ * track the number of times a point has been run.
+ */
+void
+pgstat_report_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+ PgStat_StatInjEntry *statent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+ statent = &shstatent->stats;
+
+ /* Update the injection point statistics */
+ statent->numcalls++;
+
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * SQL function returning the number of times an injection point
+ * has been called.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_numcalls);
+Datum
+injection_points_stats_numcalls(PG_FUNCTION_ARGS)
+{
+ char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_StatInjEntry *entry = pgstat_fetch_stat_injentry(name);
+
+ if (entry == NULL)
+ PG_RETURN_NULL();
+
+ PG_RETURN_INT64(entry->numcalls);
+}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
new file mode 100644
index 0000000000..3e99705483
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -0,0 +1,23 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.h
+ * Definitions for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.h
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#ifndef INJECTION_STATS
+#define INJECTION_STATS
+
+extern void pgstat_register_inj(void);
+extern void pgstat_create_inj(const char *name);
+extern void pgstat_drop_inj(const char *name);
+extern void pgstat_report_inj(const char *name);
+
+#endif
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index 3c23c14d81..a52fe5121e 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -6,6 +6,7 @@ endif
injection_points_sources = files(
'injection_points.c',
+ 'injection_stats.c',
)
if host_system == 'windows'
@@ -42,4 +43,12 @@ tests += {
'inplace',
],
},
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_stats.pl',
+ ],
+ },
}
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
new file mode 100644
index 0000000000..7d5a96e522
--- /dev/null
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -0,0 +1,48 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+use locale;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Test persistency of statistics generated for injection points.
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# Node initialization
+my $node = PostgreSQL::Test::Cluster->new('master');
+$node->init;
+$node->append_conf('postgresql.conf',
+ "shared_preload_libraries = 'injection_points'");
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# This should count for two calls.
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('stats-notice', 'notice');");
+$node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
+$node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
+my $numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats calls');
+
+# Restart the node cleanly, stats should still be around.
+$node->restart;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats after clean restart');
+
+# On crash the stats are gone.
+$node->stop('immediate');
+$node->start;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '', 'number of stats after clean restart');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e6c1caf649..b81c137479 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2116,6 +2116,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_InjectionPoint
PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
@@ -2147,6 +2148,7 @@ PgStat_Snapshot
PgStat_SnapshotEntry
PgStat_StatDBEntry
PgStat_StatFuncEntry
+PgStat_StatInjEntry
PgStat_StatReplSlotEntry
PgStat_StatSubEntry
PgStat_StatTabEntry
--
2.45.2
v3-0006-Add-support-for-fixed-numbered-statistics-in-plug.patchtext/x-diff; charset=us-asciiDownload
From 3cda5fe5588a20c3f037334a4879289ca57ec222 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 3 Jul 2024 18:26:25 +0900
Subject: [PATCH v3 6/6] Add support for fixed-numbered statistics in pluggable
pgstats facility
These kinds of stats (like WAL, bgwriter, checkpointer, etc.) are stored
in shared memory with a fixed amount of entries. With the previous
commits in place, this commit only lifts a past restriction in place.
The meat of the change is in the test module injection_points, that
provides an example of how to implement fixed-numbered stats kinds.
---
src/backend/utils/activity/pgstat.c | 9 --
.../injection_points--1.0.sql | 11 ++
.../injection_points/injection_points.c | 3 +
.../injection_points/injection_stats.c | 153 +++++++++++++++++-
.../injection_points/injection_stats.h | 3 +
.../modules/injection_points/t/001_stats.pl | 11 +-
6 files changed, 178 insertions(+), 12 deletions(-)
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 994aa2ba51..d87999d091 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -1395,15 +1395,6 @@ pgstat_register_kind(PgStat_Kind kind, const PgStat_KindInfo *kind_info)
(errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
errdetail("Custom cumulative statistics must be registered while initializing modules in \"shared_preload_libraries\".")));
- /*
- * These are not supported for now, as these point out to fixed areas of
- * shared memory.
- */
- if (kind_info->fixed_amount)
- ereport(ERROR,
- (errmsg("custom cumulative statistics property is invalid"),
- errhint("Custom cumulative statistics cannot use a fixed amount of data.")));
-
/*
* If pgstat_kind_custom_infos is not available yet, allocate it.
*/
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index deaf47d8ae..a52b7a6b7c 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -64,3 +64,14 @@ CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
RETURNS bigint
AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
LANGUAGE C STRICT;
+
+--
+-- injection_points_stats_fixed()
+--
+-- Reports fixed-numbered statistics for injection points.
+CREATE FUNCTION injection_points_stats_fixed(OUT numattach int8,
+ OUT numdetach int8,
+ OUT numrun int8)
+RETURNS record
+AS 'MODULE_PATHNAME', 'injection_points_stats_fixed'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index f65b34ce82..c71ad0d367 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -297,6 +297,7 @@ injection_points_attach(PG_FUNCTION_ARGS)
condition.pid = MyProcPid;
}
+ pgstat_report_inj_fixed(1, 0, 0);
InjectionPointAttach(name, "injection_points", function, &condition,
sizeof(InjectionPointCondition));
@@ -325,6 +326,7 @@ injection_points_run(PG_FUNCTION_ARGS)
{
char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ pgstat_report_inj_fixed(0, 0, 1);
INJECTION_POINT(name);
PG_RETURN_VOID();
@@ -401,6 +403,7 @@ injection_points_detach(PG_FUNCTION_ARGS)
{
char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ pgstat_report_inj_fixed(0, 1, 0);
if (!InjectionPointDetach(name))
elog(ERROR, "could not detach injection point \"%s\"", name);
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
index c37b0b33d3..53440f5184 100644
--- a/src/test/modules/injection_points/injection_stats.c
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -17,12 +17,13 @@
#include "fmgr.h"
#include "common/hashfn.h"
+#include "funcapi.h"
#include "injection_stats.h"
#include "pgstat.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
-/* Structures for statistics of injection points */
+/* Structures for statistics of injection points, variable-size */
typedef struct PgStat_StatInjEntry
{
PgStat_Counter numcalls; /* number of times point has been run */
@@ -35,6 +36,9 @@ typedef struct PgStatShared_InjectionPoint
} PgStatShared_InjectionPoint;
static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+static void injection_stats_fixed_init_shmem_cb(void *stats);
+static void injection_stats_fixed_reset_all_cb(TimestampTz ts);
+static void injection_stats_fixed_snapshot_cb(void);
static const PgStat_KindInfo injection_stats = {
.name = "injection_points",
@@ -50,18 +54,49 @@ static const PgStat_KindInfo injection_stats = {
.flush_pending_cb = injection_stats_flush_cb,
};
+/* Structures for statistics of injection points, fixed-size */
+typedef struct PgStat_StatInjFixedEntry
+{
+ PgStat_Counter numattach; /* number of points attached */
+ PgStat_Counter numdetach; /* number of points detached */
+ PgStat_Counter numrun; /* number of points run */
+ TimestampTz stat_reset_timestamp;
+} PgStat_StatInjFixedEntry;
+
+typedef struct PgStatShared_InjectionPointFixed
+{
+ LWLock lock; /* protects all the counters */
+ uint32 changecount;
+ PgStat_StatInjFixedEntry stats;
+ PgStat_StatInjFixedEntry reset_offset;
+} PgStatShared_InjectionPointFixed;
+
+static const PgStat_KindInfo injection_stats_fixed = {
+ .name = "injection_points_fixed",
+ .fixed_amount = true,
+
+ .shared_size = sizeof(PgStat_StatInjFixedEntry),
+ .shared_data_off = offsetof(PgStatShared_InjectionPointFixed, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPointFixed *) 0)->stats),
+
+ .init_shmem_cb = injection_stats_fixed_init_shmem_cb,
+ .reset_all_cb = injection_stats_fixed_reset_all_cb,
+ .snapshot_cb = injection_stats_fixed_snapshot_cb,
+};
+
/*
* Compute stats entry idx from point name with a 4-byte hash.
*/
#define PGSTAT_INJ_IDX(name) hash_bytes((const unsigned char *) name, strlen(name))
#define PGSTAT_KIND_INJECTION PGSTAT_KIND_EXPERIMENTAL
+#define PGSTAT_KIND_INJECTION_FIXED (PGSTAT_KIND_EXPERIMENTAL + 1)
/* Track if stats are loaded */
static bool inj_stats_loaded = false;
/*
- * Callback for stats handling
+ * Callbacks for stats handling
*/
static bool
injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
@@ -79,6 +114,59 @@ injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
return true;
}
+static void
+injection_stats_fixed_init_shmem_cb(void *stats)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+static void
+injection_stats_fixed_reset_all_cb(TimestampTz ts)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_INJECTION_FIXED];
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ pgstat_copy_changecounted_stats(&stats_shmem->reset_offset,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+ stats_shmem->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&stats_shmem->lock);
+}
+
+static void
+injection_stats_fixed_snapshot_cb(void)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_INJECTION_FIXED];
+ PgStat_StatInjFixedEntry *stat_snap = (PgStat_StatInjFixedEntry *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_INJECTION_FIXED];
+ PgStat_StatInjFixedEntry *reset_offset = &stats_shmem->reset_offset;
+ PgStat_StatInjFixedEntry reset;
+
+ pgstat_copy_changecounted_stats(stat_snap,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+
+ LWLockAcquire(&stats_shmem->lock, LW_SHARED);
+ memcpy(&reset, reset_offset, sizeof(stats_shmem->stats));
+ LWLockRelease(&stats_shmem->lock);
+
+ /* compensate by reset offsets */
+#define FIXED_COMP(fld) stat_snap->fld -= reset.fld;
+ FIXED_COMP(numattach);
+ FIXED_COMP(numdetach);
+ FIXED_COMP(numrun);
+#undef FIXED_COMP
+}
+
/*
* Support function for the SQL-callable pgstat* functions. Returns
* a pointer to the injection point statistics struct.
@@ -105,6 +193,7 @@ void
pgstat_register_inj(void)
{
pgstat_register_kind(PGSTAT_KIND_INJECTION, &injection_stats);
+ pgstat_register_kind(PGSTAT_KIND_INJECTION_FIXED, &injection_stats_fixed);
/* mark stats as loaded */
inj_stats_loaded = true;
@@ -176,6 +265,30 @@ pgstat_report_inj(const char *name)
pgstat_unlock_entry(entry_ref);
}
+/*
+ * Report fixed number of statistics for an injection point.
+ */
+void
+pgstat_report_inj_fixed(uint32 numattach,
+ uint32 numdetach,
+ uint32 numrun)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ stats_shmem = (PgStatShared_InjectionPointFixed *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_INJECTION_FIXED];
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numattach += numattach;
+ stats_shmem->stats.numdetach += numdetach;
+ stats_shmem->stats.numrun += numrun;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+}
+
/*
* SQL function returning the number of times an injection point
* has been called.
@@ -192,3 +305,39 @@ injection_points_stats_numcalls(PG_FUNCTION_ARGS)
PG_RETURN_INT64(entry->numcalls);
}
+
+/*
+ * SQL function returning fixed-numbered statistics for injection points.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_fixed);
+Datum
+injection_points_stats_fixed(PG_FUNCTION_ARGS)
+{
+ TupleDesc tupdesc;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ PgStat_StatInjFixedEntry *stats;
+
+ pgstat_snapshot_fixed(PGSTAT_KIND_INJECTION_FIXED);
+ stats = pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_INJECTION_FIXED];
+
+ /* Initialise attributes information in the tuple descriptor */
+ tupdesc = CreateTemplateTupleDesc(3);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "numattach",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "numdetach",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "numrun",
+ INT8OID, -1, 0);
+ BlessTupleDesc(tupdesc);
+
+ values[0] = Int64GetDatum(stats->numattach);
+ values[1] = Int64GetDatum(stats->numdetach);
+ values[2] = Int64GetDatum(stats->numrun);
+ nulls[0] = false;
+ nulls[1] = false;
+ nulls[2] = false;
+
+ /* Returns the record as Datum */
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
index 3e99705483..cf68b25f7b 100644
--- a/src/test/modules/injection_points/injection_stats.h
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -19,5 +19,8 @@ extern void pgstat_register_inj(void);
extern void pgstat_create_inj(const char *name);
extern void pgstat_drop_inj(const char *name);
extern void pgstat_report_inj(const char *name);
+extern void pgstat_report_inj_fixed(uint32 numattach,
+ uint32 numdetach,
+ uint32 numrun);
#endif
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
index 7d5a96e522..e3c69b94ca 100644
--- a/src/test/modules/injection_points/t/001_stats.pl
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -31,18 +31,27 @@ $node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
my $numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
is($numcalls, '2', 'number of stats calls');
+my $fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '1|0|2', 'number of fixed stats');
# Restart the node cleanly, stats should still be around.
$node->restart;
$numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
is($numcalls, '2', 'number of stats after clean restart');
+$fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '1|0|2', 'number of fixed stats after clean restart');
# On crash the stats are gone.
$node->stop('immediate');
$node->start;
$numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
-is($numcalls, '', 'number of stats after clean restart');
+is($numcalls, '', 'number of stats after crash');
+$fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '0|0|0', 'number of fixed stats after crash');
done_testing();
--
2.45.2
v3-0001-Rework-handling-of-fixed-numbered-statistics-in-p.patchtext/x-diff; charset=us-asciiDownload
From 596424c30af63e96f3ab4274541ab6cb6cfaab0e Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 3 Jul 2024 13:26:22 +0900
Subject: [PATCH v3 1/6] Rework handling of fixed-numbered statistics in
pgstats
This commit removes fixed-numbered statistics structures from
PgStat_ShmemControl, the central shared memory structure of pgstats and
PgStat_Snapshot, in charge of holding stats snapshots for backends,
based on copies of what is in shared memory.
This has the advantage to simplify the read and write of the pgstats
file by not tracking the types of fixed-numbered stats in pgstat.c
itself, replacing it with what PgStat_KindInfo already knows for each
one of them.
This refactoring makes possible the introduction of more pluggable APIs
into pgstats for fixed-numbered stats. The following changes are
applied to the internal structures:
- PgStat_ShmemControl holds an array of void* indexed by
PGSTAT_NUM_KINDS, pointing to shared memory areas allocated for each
fixed-numbered stats. Each entry is allocated a size corresponding to
PgStat_KindInfo->shared_size.
- PgStat_Snapshot holds an array of void* also indexed by
PGSTAT_NUM_KINDS, pointing to the fixed stats stored in the snapshots.
These have a size of PgStat_KindInfo->shared_data_len.
- Fixed numbered stats now set shared_size, so as
- A callback is added to initialize the shared memory assigned to each
fixed-numbered stats, consisting of LWLock initializations for the
current types of stats. So this initialization is moved out of pgstat.c
into each stats kind file.
---
src/include/utils/pgstat_internal.h | 52 +++----
src/backend/utils/activity/pgstat.c | 130 ++++++++++--------
src/backend/utils/activity/pgstat_archiver.c | 23 +++-
src/backend/utils/activity/pgstat_bgwriter.c | 26 +++-
.../utils/activity/pgstat_checkpointer.c | 26 +++-
src/backend/utils/activity/pgstat_io.c | 44 ++++--
src/backend/utils/activity/pgstat_shmem.c | 33 +++--
src/backend/utils/activity/pgstat_slru.c | 25 +++-
src/backend/utils/activity/pgstat_wal.c | 26 +++-
9 files changed, 250 insertions(+), 135 deletions(-)
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index e21ef4e2c9..b8b2152d71 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -195,16 +195,11 @@ typedef struct PgStat_KindInfo
/*
* The size of an entry in the shared stats hash table (pointed to by
- * PgStatShared_HashEntry->body).
+ * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
+ * the size of an entry in PgStat_ShmemControl->fixed_data.
*/
uint32 shared_size;
- /*
- * The offset of the statistics struct in the containing shared memory
- * control structure PgStat_ShmemControl, for fixed-numbered statistics.
- */
- uint32 shared_ctl_off;
-
/*
* The offset/size of statistics inside the shared stats entry. Used when
* [de-]serializing statistics to / from disk respectively. Separate from
@@ -245,6 +240,13 @@ typedef struct PgStat_KindInfo
const PgStatShared_Common *header, NameData *name);
bool (*from_serialized_name) (const NameData *name, PgStat_HashKey *key);
+ /*
+ * For fixed-numbered statistics: Initialize shared memory state.
+ *
+ * "stats" is the pointer to the allocated shared memory area.
+ */
+ void (*init_shmem_cb) (void *stats);
+
/*
* For fixed-numbered statistics: Reset All.
*/
@@ -425,14 +427,12 @@ typedef struct PgStat_ShmemControl
pg_atomic_uint64 gc_request_count;
/*
- * Stats data for fixed-numbered objects.
+ * Stats data for fixed-numbered objects, indexed by PgStat_Kind.
+ *
+ * Each entry has a size of PgStat_KindInfo->shared_size.
*/
- PgStatShared_Archiver archiver;
- PgStatShared_BgWriter bgwriter;
- PgStatShared_Checkpointer checkpointer;
- PgStatShared_IO io;
- PgStatShared_SLRU slru;
- PgStatShared_Wal wal;
+ void *fixed_data[PGSTAT_NUM_KINDS];
+
} PgStat_ShmemControl;
@@ -446,19 +446,13 @@ typedef struct PgStat_Snapshot
/* time at which snapshot was taken */
TimestampTz snapshot_timestamp;
+ /*
+ * Data in snapshot for fixed-numbered statistics, indexed by PgStat_Kind.
+ * Each entry is allocated in TopMemoryContext, for a size of
+ * shared_data_len.
+ */
bool fixed_valid[PGSTAT_NUM_KINDS];
-
- PgStat_ArchiverStats archiver;
-
- PgStat_BgWriterStats bgwriter;
-
- PgStat_CheckpointerStats checkpointer;
-
- PgStat_IO io;
-
- PgStat_SLRUStats slru[SLRU_NUM_ELEMENTS];
-
- PgStat_WalStats wal;
+ void *fixed_data[PGSTAT_NUM_KINDS];
/* to free snapshot in bulk */
MemoryContext context;
@@ -522,6 +516,7 @@ extern void pgstat_snapshot_fixed(PgStat_Kind kind);
* Functions in pgstat_archiver.c
*/
+extern void pgstat_archiver_init_shmem_cb(void *stats);
extern void pgstat_archiver_reset_all_cb(TimestampTz ts);
extern void pgstat_archiver_snapshot_cb(void);
@@ -530,6 +525,7 @@ extern void pgstat_archiver_snapshot_cb(void);
* Functions in pgstat_bgwriter.c
*/
+extern void pgstat_bgwriter_init_shmem_cb(void *stats);
extern void pgstat_bgwriter_reset_all_cb(TimestampTz ts);
extern void pgstat_bgwriter_snapshot_cb(void);
@@ -538,6 +534,7 @@ extern void pgstat_bgwriter_snapshot_cb(void);
* Functions in pgstat_checkpointer.c
*/
+extern void pgstat_checkpointer_init_shmem_cb(void *stats);
extern void pgstat_checkpointer_reset_all_cb(TimestampTz ts);
extern void pgstat_checkpointer_snapshot_cb(void);
@@ -568,6 +565,7 @@ extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
*/
extern bool pgstat_flush_io(bool nowait);
+extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
@@ -626,6 +624,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind,
*/
extern bool pgstat_slru_flush(bool nowait);
+extern void pgstat_slru_init_shmem_cb(void *stats);
extern void pgstat_slru_reset_all_cb(TimestampTz ts);
extern void pgstat_slru_snapshot_cb(void);
@@ -638,6 +637,7 @@ extern bool pgstat_flush_wal(bool nowait);
extern void pgstat_init_wal(void);
extern bool pgstat_have_pending_wal(void);
+extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index c37c11b2ec..13ddbcdcfb 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -132,6 +132,7 @@
* ---------
*/
#define PGSTAT_FILE_ENTRY_END 'E' /* end of file */
+#define PGSTAT_FILE_ENTRY_FIXED 'F' /* fixed-numbered stats entry */
#define PGSTAT_FILE_ENTRY_NAME 'N' /* stats entry identified by name */
#define PGSTAT_FILE_ENTRY_HASH 'S' /* stats entry identified by
* PgStat_HashKey */
@@ -173,6 +174,8 @@ typedef struct PgStat_SnapshotEntry
static void pgstat_write_statsfile(void);
static void pgstat_read_statsfile(void);
+static void pgstat_init_snapshot(void);
+
static void pgstat_reset_after_failure(void);
static bool pgstat_flush_pending_entries(bool nowait);
@@ -349,10 +352,11 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = true,
- .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver),
+ .shared_size = sizeof(PgStatShared_Archiver),
.shared_data_off = offsetof(PgStatShared_Archiver, stats),
.shared_data_len = sizeof(((PgStatShared_Archiver *) 0)->stats),
+ .init_shmem_cb = pgstat_archiver_init_shmem_cb,
.reset_all_cb = pgstat_archiver_reset_all_cb,
.snapshot_cb = pgstat_archiver_snapshot_cb,
},
@@ -362,10 +366,11 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = true,
- .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter),
+ .shared_size = sizeof(PgStatShared_BgWriter),
.shared_data_off = offsetof(PgStatShared_BgWriter, stats),
.shared_data_len = sizeof(((PgStatShared_BgWriter *) 0)->stats),
+ .init_shmem_cb = pgstat_bgwriter_init_shmem_cb,
.reset_all_cb = pgstat_bgwriter_reset_all_cb,
.snapshot_cb = pgstat_bgwriter_snapshot_cb,
},
@@ -375,10 +380,11 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = true,
- .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer),
+ .shared_size = sizeof(PgStatShared_Checkpointer),
.shared_data_off = offsetof(PgStatShared_Checkpointer, stats),
.shared_data_len = sizeof(((PgStatShared_Checkpointer *) 0)->stats),
+ .init_shmem_cb = pgstat_checkpointer_init_shmem_cb,
.reset_all_cb = pgstat_checkpointer_reset_all_cb,
.snapshot_cb = pgstat_checkpointer_snapshot_cb,
},
@@ -388,10 +394,11 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = true,
- .shared_ctl_off = offsetof(PgStat_ShmemControl, io),
+ .shared_size = sizeof(PgStatShared_IO),
.shared_data_off = offsetof(PgStatShared_IO, stats),
.shared_data_len = sizeof(((PgStatShared_IO *) 0)->stats),
+ .init_shmem_cb = pgstat_io_init_shmem_cb,
.reset_all_cb = pgstat_io_reset_all_cb,
.snapshot_cb = pgstat_io_snapshot_cb,
},
@@ -401,10 +408,11 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = true,
- .shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
+ .shared_size = sizeof(PgStatShared_SLRU),
.shared_data_off = offsetof(PgStatShared_SLRU, stats),
.shared_data_len = sizeof(((PgStatShared_SLRU *) 0)->stats),
+ .init_shmem_cb = pgstat_slru_init_shmem_cb,
.reset_all_cb = pgstat_slru_reset_all_cb,
.snapshot_cb = pgstat_slru_snapshot_cb,
},
@@ -414,10 +422,11 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = true,
- .shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
+ .shared_size = sizeof(PgStatShared_Wal),
.shared_data_off = offsetof(PgStatShared_Wal, stats),
.shared_data_len = sizeof(((PgStatShared_Wal *) 0)->stats),
+ .init_shmem_cb = pgstat_wal_init_shmem_cb,
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
},
@@ -571,6 +580,8 @@ pgstat_initialize(void)
pgstat_attach_shmem();
+ pgstat_init_snapshot();
+
pgstat_init_wal();
/* Set up a process-exit hook to clean up */
@@ -982,6 +993,22 @@ pgstat_snapshot_fixed(PgStat_Kind kind)
Assert(pgStatLocal.snapshot.fixed_valid[kind]);
}
+static void
+pgstat_init_snapshot(void)
+{
+ /* Initialize fixed-numbered statistics data in snapshots */
+ for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info->fixed_amount)
+ continue;
+
+ pgStatLocal.snapshot.fixed_data[kind] =
+ MemoryContextAlloc(TopMemoryContext, kind_info->shared_data_len);
+ }
+}
+
static void
pgstat_prep_snapshot(void)
{
@@ -1371,47 +1398,25 @@ pgstat_write_statsfile(void)
format_id = PGSTAT_FILE_FORMAT_ID;
write_chunk_s(fpout, &format_id);
- /*
- * XXX: The following could now be generalized to just iterate over
- * pgstat_kind_infos instead of knowing about the different kinds of
- * stats.
- */
+ /* Write various stats structs with fixed number of objects */
+ for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ {
+ char *ptr;
+ const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
- /*
- * Write archiver stats struct
- */
- pgstat_build_snapshot_fixed(PGSTAT_KIND_ARCHIVER);
- write_chunk_s(fpout, &pgStatLocal.snapshot.archiver);
+ if (!info->fixed_amount)
+ continue;
- /*
- * Write bgwriter stats struct
- */
- pgstat_build_snapshot_fixed(PGSTAT_KIND_BGWRITER);
- write_chunk_s(fpout, &pgStatLocal.snapshot.bgwriter);
+ Assert(info->shared_size != 0 && info->shared_data_len != 0);
- /*
- * Write checkpointer stats struct
- */
- pgstat_build_snapshot_fixed(PGSTAT_KIND_CHECKPOINTER);
- write_chunk_s(fpout, &pgStatLocal.snapshot.checkpointer);
+ /* prepare snapshot data and write it */
+ pgstat_build_snapshot_fixed(kind);
+ ptr = pgStatLocal.snapshot.fixed_data[kind];
- /*
- * Write IO stats struct
- */
- pgstat_build_snapshot_fixed(PGSTAT_KIND_IO);
- write_chunk_s(fpout, &pgStatLocal.snapshot.io);
-
- /*
- * Write SLRU stats struct
- */
- pgstat_build_snapshot_fixed(PGSTAT_KIND_SLRU);
- write_chunk_s(fpout, &pgStatLocal.snapshot.slru);
-
- /*
- * Write WAL stats struct
- */
- pgstat_build_snapshot_fixed(PGSTAT_KIND_WAL);
- write_chunk_s(fpout, &pgStatLocal.snapshot.wal);
+ fputc(PGSTAT_FILE_ENTRY_FIXED, fpout);
+ write_chunk_s(fpout, &kind);
+ write_chunk(fpout, ptr, info->shared_data_len);
+ }
/*
* Walk through the stats entries
@@ -1551,22 +1556,6 @@ pgstat_read_statsfile(void)
format_id != PGSTAT_FILE_FORMAT_ID)
goto error;
- /* Read various stats structs with fixed number of objects */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
- {
- char *ptr;
- const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
-
- if (!info->fixed_amount)
- continue;
-
- Assert(info->shared_ctl_off != 0);
-
- ptr = ((char *) shmem) + info->shared_ctl_off + info->shared_data_off;
- if (!read_chunk(fpin, ptr, info->shared_data_len))
- goto error;
- }
-
/*
* We found an existing statistics file. Read it and put all the hash
* table entries into place.
@@ -1577,6 +1566,29 @@ pgstat_read_statsfile(void)
switch (t)
{
+ case PGSTAT_FILE_ENTRY_FIXED:
+ {
+ PgStat_Kind kind;
+ const PgStat_KindInfo *info;
+ char *ptr;
+
+ if (!read_chunk_s(fpin, &kind))
+ goto error;
+
+ if (!pgstat_is_kind_valid(kind))
+ goto error;
+
+ info = pgstat_get_kind_info(kind);
+ Assert(info->fixed_amount);
+
+ /* Load back stats into shared memory */
+ ptr = ((char *) shmem->fixed_data[kind]) + info->shared_data_off;
+ if (!read_chunk(fpin, ptr,
+ info->shared_data_len))
+ goto error;
+
+ break;
+ }
case PGSTAT_FILE_ENTRY_HASH:
case PGSTAT_FILE_ENTRY_NAME:
{
diff --git a/src/backend/utils/activity/pgstat_archiver.c b/src/backend/utils/activity/pgstat_archiver.c
index 66398b20e5..99cd461f81 100644
--- a/src/backend/utils/activity/pgstat_archiver.c
+++ b/src/backend/utils/activity/pgstat_archiver.c
@@ -27,7 +27,8 @@
void
pgstat_report_archiver(const char *xlog, bool failed)
{
- PgStatShared_Archiver *stats_shmem = &pgStatLocal.shmem->archiver;
+ PgStatShared_Archiver *stats_shmem = (PgStatShared_Archiver *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_ARCHIVER];
TimestampTz now = GetCurrentTimestamp();
pgstat_begin_changecount_write(&stats_shmem->changecount);
@@ -59,13 +60,23 @@ pgstat_fetch_stat_archiver(void)
{
pgstat_snapshot_fixed(PGSTAT_KIND_ARCHIVER);
- return &pgStatLocal.snapshot.archiver;
+ return (PgStat_ArchiverStats *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_ARCHIVER];
+}
+
+void
+pgstat_archiver_init_shmem_cb(void *stats)
+{
+ PgStatShared_Archiver *stats_shmem = (PgStatShared_Archiver *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
}
void
pgstat_archiver_reset_all_cb(TimestampTz ts)
{
- PgStatShared_Archiver *stats_shmem = &pgStatLocal.shmem->archiver;
+ PgStatShared_Archiver *stats_shmem = (PgStatShared_Archiver *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_ARCHIVER];
/* see explanation above PgStatShared_Archiver for the reset protocol */
LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
@@ -80,8 +91,10 @@ pgstat_archiver_reset_all_cb(TimestampTz ts)
void
pgstat_archiver_snapshot_cb(void)
{
- PgStatShared_Archiver *stats_shmem = &pgStatLocal.shmem->archiver;
- PgStat_ArchiverStats *stat_snap = &pgStatLocal.snapshot.archiver;
+ PgStatShared_Archiver *stats_shmem = (PgStatShared_Archiver *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_ARCHIVER];
+ PgStat_ArchiverStats *stat_snap = (PgStat_ArchiverStats *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_ARCHIVER];
PgStat_ArchiverStats *reset_offset = &stats_shmem->reset_offset;
PgStat_ArchiverStats reset;
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index 7d2432e4fa..778d3f0b0e 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -29,7 +29,8 @@ PgStat_BgWriterStats PendingBgWriterStats = {0};
void
pgstat_report_bgwriter(void)
{
- PgStatShared_BgWriter *stats_shmem = &pgStatLocal.shmem->bgwriter;
+ PgStatShared_BgWriter *stats_shmem = (PgStatShared_BgWriter *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_BGWRITER];
static const PgStat_BgWriterStats all_zeroes;
Assert(!pgStatLocal.shmem->is_shutdown);
@@ -72,13 +73,23 @@ pgstat_fetch_stat_bgwriter(void)
{
pgstat_snapshot_fixed(PGSTAT_KIND_BGWRITER);
- return &pgStatLocal.snapshot.bgwriter;
+ return (PgStat_BgWriterStats *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_BGWRITER];
+}
+
+void
+pgstat_bgwriter_init_shmem_cb(void *stats)
+{
+ PgStatShared_BgWriter *stats_shmem = (PgStatShared_BgWriter *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
}
void
pgstat_bgwriter_reset_all_cb(TimestampTz ts)
{
- PgStatShared_BgWriter *stats_shmem = &pgStatLocal.shmem->bgwriter;
+ PgStatShared_BgWriter *stats_shmem = (PgStatShared_BgWriter *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_BGWRITER];
/* see explanation above PgStatShared_BgWriter for the reset protocol */
LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
@@ -93,11 +104,14 @@ pgstat_bgwriter_reset_all_cb(TimestampTz ts)
void
pgstat_bgwriter_snapshot_cb(void)
{
- PgStatShared_BgWriter *stats_shmem = &pgStatLocal.shmem->bgwriter;
+ PgStatShared_BgWriter *stats_shmem = (PgStatShared_BgWriter *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_BGWRITER];
+ PgStat_BgWriterStats *stat_snap = (PgStat_BgWriterStats *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_BGWRITER];
PgStat_BgWriterStats *reset_offset = &stats_shmem->reset_offset;
PgStat_BgWriterStats reset;
- pgstat_copy_changecounted_stats(&pgStatLocal.snapshot.bgwriter,
+ pgstat_copy_changecounted_stats(stat_snap,
&stats_shmem->stats,
sizeof(stats_shmem->stats),
&stats_shmem->changecount);
@@ -107,7 +121,7 @@ pgstat_bgwriter_snapshot_cb(void)
LWLockRelease(&stats_shmem->lock);
/* compensate by reset offsets */
-#define BGWRITER_COMP(fld) pgStatLocal.snapshot.bgwriter.fld -= reset.fld;
+#define BGWRITER_COMP(fld) stat_snap->fld -= reset.fld;
BGWRITER_COMP(buf_written_clean);
BGWRITER_COMP(maxwritten_clean);
BGWRITER_COMP(buf_alloc);
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 30a8110e38..6e86255b8d 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -31,7 +31,8 @@ pgstat_report_checkpointer(void)
{
/* We assume this initializes to zeroes */
static const PgStat_CheckpointerStats all_zeroes;
- PgStatShared_Checkpointer *stats_shmem = &pgStatLocal.shmem->checkpointer;
+ PgStatShared_Checkpointer *stats_shmem = (PgStatShared_Checkpointer *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_CHECKPOINTER];
Assert(!pgStatLocal.shmem->is_shutdown);
pgstat_assert_is_up();
@@ -81,13 +82,23 @@ pgstat_fetch_stat_checkpointer(void)
{
pgstat_snapshot_fixed(PGSTAT_KIND_CHECKPOINTER);
- return &pgStatLocal.snapshot.checkpointer;
+ return (PgStat_CheckpointerStats *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_CHECKPOINTER];
+}
+
+void
+pgstat_checkpointer_init_shmem_cb(void *stats)
+{
+ PgStatShared_Checkpointer *stats_shmem = (PgStatShared_Checkpointer *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
}
void
pgstat_checkpointer_reset_all_cb(TimestampTz ts)
{
- PgStatShared_Checkpointer *stats_shmem = &pgStatLocal.shmem->checkpointer;
+ PgStatShared_Checkpointer *stats_shmem = (PgStatShared_Checkpointer *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_CHECKPOINTER];
/* see explanation above PgStatShared_Checkpointer for the reset protocol */
LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
@@ -102,11 +113,14 @@ pgstat_checkpointer_reset_all_cb(TimestampTz ts)
void
pgstat_checkpointer_snapshot_cb(void)
{
- PgStatShared_Checkpointer *stats_shmem = &pgStatLocal.shmem->checkpointer;
+ PgStatShared_Checkpointer *stats_shmem = (PgStatShared_Checkpointer *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_CHECKPOINTER];
+ PgStat_CheckpointerStats *stat_snap = (PgStat_CheckpointerStats *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_CHECKPOINTER];
PgStat_CheckpointerStats *reset_offset = &stats_shmem->reset_offset;
PgStat_CheckpointerStats reset;
- pgstat_copy_changecounted_stats(&pgStatLocal.snapshot.checkpointer,
+ pgstat_copy_changecounted_stats(stat_snap,
&stats_shmem->stats,
sizeof(stats_shmem->stats),
&stats_shmem->changecount);
@@ -116,7 +130,7 @@ pgstat_checkpointer_snapshot_cb(void)
LWLockRelease(&stats_shmem->lock);
/* compensate by reset offsets */
-#define CHECKPOINTER_COMP(fld) pgStatLocal.snapshot.checkpointer.fld -= reset.fld;
+#define CHECKPOINTER_COMP(fld) stat_snap->fld -= reset.fld;
CHECKPOINTER_COMP(num_timed);
CHECKPOINTER_COMP(num_requested);
CHECKPOINTER_COMP(restartpoints_timed);
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 9d6e067382..6cbbc2094a 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -158,7 +158,8 @@ pgstat_fetch_stat_io(void)
{
pgstat_snapshot_fixed(PGSTAT_KIND_IO);
- return &pgStatLocal.snapshot.io;
+ return (PgStat_IO *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_IO];
}
/*
@@ -174,13 +175,16 @@ pgstat_flush_io(bool nowait)
{
LWLock *bktype_lock;
PgStat_BktypeIO *bktype_shstats;
+ PgStatShared_IO *stat_shmem;
if (!have_iostats)
return false;
- bktype_lock = &pgStatLocal.shmem->io.locks[MyBackendType];
- bktype_shstats =
- &pgStatLocal.shmem->io.stats.stats[MyBackendType];
+ stat_shmem = (PgStatShared_IO *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_IO];
+
+ bktype_lock = &stat_shmem->locks[MyBackendType];
+ bktype_shstats = &stat_shmem->stats.stats[MyBackendType];
if (!nowait)
LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
@@ -251,13 +255,25 @@ pgstat_get_io_object_name(IOObject io_object)
pg_unreachable();
}
+void
+pgstat_io_init_shmem_cb(void *stats)
+{
+ PgStatShared_IO *stat_shmem = (PgStatShared_IO *) stats;
+
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ LWLockInitialize(&stat_shmem->locks[i], LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_io_reset_all_cb(TimestampTz ts)
{
+ PgStatShared_IO *stat_shmem = (PgStatShared_IO *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_IO];
+
for (int i = 0; i < BACKEND_NUM_TYPES; i++)
{
- LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
- PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
+ LWLock *bktype_lock = &stat_shmem->locks[i];
+ PgStat_BktypeIO *bktype_shstats = &stat_shmem->stats.stats[i];
LWLockAcquire(bktype_lock, LW_EXCLUSIVE);
@@ -266,7 +282,7 @@ pgstat_io_reset_all_cb(TimestampTz ts)
* the reset timestamp as well.
*/
if (i == 0)
- pgStatLocal.shmem->io.stats.stat_reset_timestamp = ts;
+ stat_shmem->stats.stat_reset_timestamp = ts;
memset(bktype_shstats, 0, sizeof(*bktype_shstats));
LWLockRelease(bktype_lock);
@@ -276,11 +292,16 @@ pgstat_io_reset_all_cb(TimestampTz ts)
void
pgstat_io_snapshot_cb(void)
{
+ PgStatShared_IO *stat_shmem = (PgStatShared_IO *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_IO];
+ PgStat_IO *stat_snap = (PgStat_IO *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_IO];
+
for (int i = 0; i < BACKEND_NUM_TYPES; i++)
{
- LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i];
- PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i];
- PgStat_BktypeIO *bktype_snap = &pgStatLocal.snapshot.io.stats[i];
+ LWLock *bktype_lock = &stat_shmem->locks[i];
+ PgStat_BktypeIO *bktype_shstats = &stat_shmem->stats.stats[i];
+ PgStat_BktypeIO *bktype_snap = &stat_snap->stats[i];
LWLockAcquire(bktype_lock, LW_SHARED);
@@ -289,8 +310,7 @@ pgstat_io_snapshot_cb(void)
* the reset timestamp as well.
*/
if (i == 0)
- pgStatLocal.snapshot.io.stat_reset_timestamp =
- pgStatLocal.shmem->io.stats.stat_reset_timestamp;
+ stat_snap->stat_reset_timestamp = stat_shmem->stats.stat_reset_timestamp;
/* using struct assignment due to better type safety */
*bktype_snap = *bktype_shstats;
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 634b967820..bd62c4b72a 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -131,6 +131,19 @@ StatsShmemSize(void)
sz = MAXALIGN(sizeof(PgStat_ShmemControl));
sz = add_size(sz, pgstat_dsa_init_size());
+ /* Add shared memory for all the fixed-numbered statistics */
+ for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info->fixed_amount)
+ continue;
+
+ Assert(kind_info->shared_size != 0);
+
+ sz += MAXALIGN(kind_info->shared_size);
+ }
+
return sz;
}
@@ -196,17 +209,19 @@ StatsShmemInit(void)
pg_atomic_init_u64(&ctl->gc_request_count, 1);
+ /* initialize fixed-numbered statistics */
+ for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
- /* initialize fixed-numbered stats */
- LWLockInitialize(&ctl->archiver.lock, LWTRANCHE_PGSTATS_DATA);
- LWLockInitialize(&ctl->bgwriter.lock, LWTRANCHE_PGSTATS_DATA);
- LWLockInitialize(&ctl->checkpointer.lock, LWTRANCHE_PGSTATS_DATA);
- LWLockInitialize(&ctl->slru.lock, LWTRANCHE_PGSTATS_DATA);
- LWLockInitialize(&ctl->wal.lock, LWTRANCHE_PGSTATS_DATA);
+ if (!kind_info->fixed_amount)
+ continue;
- for (int i = 0; i < BACKEND_NUM_TYPES; i++)
- LWLockInitialize(&ctl->io.locks[i],
- LWTRANCHE_PGSTATS_DATA);
+ Assert(kind_info->shared_size != 0);
+
+ ctl->fixed_data[kind] = ShmemAlloc(kind_info->shared_size);
+ kind_info->init_shmem_cb(ctl->fixed_data[kind]);
+ }
}
else
{
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 56ea1c3378..d50589c1de 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -106,7 +106,8 @@ pgstat_fetch_slru(void)
{
pgstat_snapshot_fixed(PGSTAT_KIND_SLRU);
- return pgStatLocal.snapshot.slru;
+ return (PgStat_SLRUStats *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_SLRU];
}
/*
@@ -155,7 +156,8 @@ pgstat_get_slru_index(const char *name)
bool
pgstat_slru_flush(bool nowait)
{
- PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru;
+ PgStatShared_SLRU *stats_shmem = (PgStatShared_SLRU *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_SLRU];
int i;
if (!have_slrustats)
@@ -192,6 +194,14 @@ pgstat_slru_flush(bool nowait)
return false;
}
+void
+pgstat_slru_init_shmem_cb(void *stats)
+{
+ PgStatShared_SLRU *stats_shmem = (PgStatShared_SLRU *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_slru_reset_all_cb(TimestampTz ts)
{
@@ -202,12 +212,14 @@ pgstat_slru_reset_all_cb(TimestampTz ts)
void
pgstat_slru_snapshot_cb(void)
{
- PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru;
+ PgStatShared_SLRU *stats_shmem = (PgStatShared_SLRU *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_SLRU];
+ PgStat_SLRUStats *stat_snap = (PgStat_SLRUStats *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_SLRU];
LWLockAcquire(&stats_shmem->lock, LW_SHARED);
- memcpy(pgStatLocal.snapshot.slru, &stats_shmem->stats,
- sizeof(stats_shmem->stats));
+ memcpy(stat_snap, &stats_shmem->stats, sizeof(stats_shmem->stats));
LWLockRelease(&stats_shmem->lock);
}
@@ -237,7 +249,8 @@ get_slru_entry(int slru_idx)
static void
pgstat_reset_slru_counter_internal(int index, TimestampTz ts)
{
- PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru;
+ PgStatShared_SLRU *stats_shmem = (PgStatShared_SLRU *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_SLRU];
LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 0e374f133a..030b339e47 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -68,7 +68,8 @@ pgstat_fetch_stat_wal(void)
{
pgstat_snapshot_fixed(PGSTAT_KIND_WAL);
- return &pgStatLocal.snapshot.wal;
+ return (PgStat_WalStats *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_WAL];
}
/*
@@ -81,7 +82,8 @@ pgstat_fetch_stat_wal(void)
bool
pgstat_flush_wal(bool nowait)
{
- PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
+ PgStatShared_Wal *stats_shmem = (PgStatShared_Wal *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_WAL];
WalUsage wal_usage_diff = {0};
Assert(IsUnderPostmaster || !IsPostmasterEnvironment);
@@ -163,10 +165,20 @@ pgstat_have_pending_wal(void)
PendingWalStats.wal_sync != 0;
}
+void
+pgstat_wal_init_shmem_cb(void *stats)
+{
+ PgStatShared_Wal *stats_shmem = (PgStatShared_Wal *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+
void
pgstat_wal_reset_all_cb(TimestampTz ts)
{
- PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
+ PgStatShared_Wal *stats_shmem = (PgStatShared_Wal *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_WAL];
LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
memset(&stats_shmem->stats, 0, sizeof(stats_shmem->stats));
@@ -177,10 +189,12 @@ pgstat_wal_reset_all_cb(TimestampTz ts)
void
pgstat_wal_snapshot_cb(void)
{
- PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal;
+ PgStatShared_Wal *stats_shmem = (PgStatShared_Wal *)
+ pgStatLocal.shmem->fixed_data[PGSTAT_KIND_WAL];
+ PgStat_WalStats *stat_snap = (PgStat_WalStats *)
+ pgStatLocal.snapshot.fixed_data[PGSTAT_KIND_WAL];
LWLockAcquire(&stats_shmem->lock, LW_SHARED);
- memcpy(&pgStatLocal.snapshot.wal, &stats_shmem->stats,
- sizeof(pgStatLocal.snapshot.wal));
+ memcpy(stat_snap, &stats_shmem->stats, sizeof(PgStat_WalStats));
LWLockRelease(&stats_shmem->lock);
}
--
2.45.2
v3-0002-Switch-PgStat_Kind-from-enum-to-uint32.patchtext/x-diff; charset=us-asciiDownload
From 790d900cca796a82aa3761da25597bf1496c2615 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 3 Jul 2024 13:39:01 +0900
Subject: [PATCH v3 2/6] Switch PgStat_Kind from enum to uint32
A follow-up patch is planned to make this counter extensible, and
keeping a trace of the kind behind a type is useful in the internal
routines used by pgstats. While on it, switch pgstat_is_kind_valid() to
use PgStat_Kind, to be more consistent with its callers.
---
src/include/pgstat.h | 35 ++++++++++++++---------------
src/backend/utils/activity/pgstat.c | 6 ++---
2 files changed, 20 insertions(+), 21 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 2136239710..2d30fadaf1 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -32,26 +32,25 @@
#define PG_STAT_TMP_DIR "pg_stat_tmp"
/* The types of statistics entries */
-typedef enum PgStat_Kind
-{
- /* use 0 for INVALID, to catch zero-initialized data */
- PGSTAT_KIND_INVALID = 0,
+#define PgStat_Kind uint32
- /* stats for variable-numbered objects */
- PGSTAT_KIND_DATABASE, /* database-wide statistics */
- PGSTAT_KIND_RELATION, /* per-table statistics */
- PGSTAT_KIND_FUNCTION, /* per-function statistics */
- PGSTAT_KIND_REPLSLOT, /* per-slot statistics */
- PGSTAT_KIND_SUBSCRIPTION, /* per-subscription statistics */
+/* use 0 for INVALID, to catch zero-initialized data */
+#define PGSTAT_KIND_INVALID 0
- /* stats for fixed-numbered objects */
- PGSTAT_KIND_ARCHIVER,
- PGSTAT_KIND_BGWRITER,
- PGSTAT_KIND_CHECKPOINTER,
- PGSTAT_KIND_IO,
- PGSTAT_KIND_SLRU,
- PGSTAT_KIND_WAL,
-} PgStat_Kind;
+/* stats for variable-numbered objects */
+#define PGSTAT_KIND_DATABASE 1 /* database-wide statistics */
+#define PGSTAT_KIND_RELATION 2 /* per-table statistics */
+#define PGSTAT_KIND_FUNCTION 3 /* per-function statistics */
+#define PGSTAT_KIND_REPLSLOT 4 /* per-slot statistics */
+#define PGSTAT_KIND_SUBSCRIPTION 5 /* per-subscription statistics */
+
+/* stats for fixed-numbered objects */
+#define PGSTAT_KIND_ARCHIVER 6
+#define PGSTAT_KIND_BGWRITER 7
+#define PGSTAT_KIND_CHECKPOINTER 8
+#define PGSTAT_KIND_IO 9
+#define PGSTAT_KIND_SLRU 10
+#define PGSTAT_KIND_WAL 11
#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 13ddbcdcfb..fb27272e86 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -184,7 +184,7 @@ static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
static void pgstat_build_snapshot_fixed(PgStat_Kind kind);
-static inline bool pgstat_is_kind_valid(int ikind);
+static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
/* ----------
@@ -1312,9 +1312,9 @@ pgstat_get_kind_from_str(char *kind_str)
}
static inline bool
-pgstat_is_kind_valid(int ikind)
+pgstat_is_kind_valid(PgStat_Kind kind)
{
- return ikind >= PGSTAT_KIND_FIRST_VALID && ikind <= PGSTAT_KIND_LAST;
+ return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
}
const PgStat_KindInfo *
--
2.45.2
v3-0003-Introduce-pluggable-APIs-for-Cumulative-Statistic.patchtext/x-diff; charset=us-asciiDownload
From 173976d81c0adeeef8768d708099b0b5f4584144 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 3 Jul 2024 15:04:45 +0900
Subject: [PATCH v3 3/6] Introduce pluggable APIs for Cumulative Statistics
This commit adds support in the backend for $subject, allowing
out-of-core extensions to add their own custom statistics kinds. The
stats kinds are divided into two parts for efficiency:
- The built-in stats kinds, with designated initializers.
- The custom kinds, able to use a range of IDs (128 slots available as
of this patch), with information saved in TopMemoryContext.
Custom cumulative statistics can only be loaded with
shared_preload_libraries at startup, and must allocate a unique ID
shared across all the PostgreSQL extension ecosystem with the following
wiki page:
https://wiki.postgresql.org/wiki/CustomCumulativeStats
As of this patch, fixed-numbered stats kind (like WAL, archiver,
bgwriter) are not supported, still some infrastructure is added to make
its support easier, as the fixed_data areas for the shmem and snapshot
structures are extended to cover custom and builtin stats kinds.
---
src/include/pgstat.h | 35 ++++-
src/include/utils/pgstat_internal.h | 8 +-
src/backend/utils/activity/pgstat.c | 161 +++++++++++++++++++---
src/backend/utils/activity/pgstat_shmem.c | 8 +-
src/backend/utils/adt/pgstatfuncs.c | 2 +-
5 files changed, 186 insertions(+), 28 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 2d30fadaf1..8d523607a4 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -34,6 +34,10 @@
/* The types of statistics entries */
#define PgStat_Kind uint32
+/* Range of IDs allowed, for built-in and custom kinds */
+#define PGSTAT_KIND_MIN 1 /* Minimum ID allowed */
+#define PGSTAT_KIND_MAX 256 /* Maximum ID allowed */
+
/* use 0 for INVALID, to catch zero-initialized data */
#define PGSTAT_KIND_INVALID 0
@@ -52,9 +56,34 @@
#define PGSTAT_KIND_SLRU 10
#define PGSTAT_KIND_WAL 11
-#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
-#define PGSTAT_NUM_KINDS (PGSTAT_KIND_LAST + 1)
+#define PGSTAT_KIND_MIN_BUILTIN PGSTAT_KIND_DATABASE
+#define PGSTAT_KIND_MAX_BUILTIN PGSTAT_KIND_WAL
+
+/* Custom stats kinds */
+
+/* Range of IDs allowed for custom stats kinds */
+#define PGSTAT_KIND_CUSTOM_MIN 128
+#define PGSTAT_KIND_CUSTOM_MAX PGSTAT_KIND_MAX
+#define PGSTAT_KIND_CUSTOM_SIZE (PGSTAT_KIND_CUSTOM_MAX - PGSTAT_KIND_CUSTOM_MIN + 1)
+
+/*
+ * PgStat_Kind to use for extensions that require an ID, but are still in
+ * development and have not reserved their own unique kind ID yet. See:
+ * https://wiki.postgresql.org/wiki/CustomCumulativeStats
+ */
+#define PGSTAT_KIND_EXPERIMENTAL 128
+
+static inline bool
+pgstat_is_kind_builtin(PgStat_Kind kind)
+{
+ return kind > PGSTAT_KIND_INVALID && kind <= PGSTAT_KIND_MAX_BUILTIN;
+}
+
+static inline bool
+pgstat_is_kind_custom(PgStat_Kind kind)
+{
+ return kind >= PGSTAT_KIND_CUSTOM_MIN && kind <= PGSTAT_KIND_CUSTOM_MAX;
+}
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index b8b2152d71..01b43a80e0 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -431,7 +431,7 @@ typedef struct PgStat_ShmemControl
*
* Each entry has a size of PgStat_KindInfo->shared_size.
*/
- void *fixed_data[PGSTAT_NUM_KINDS];
+ void *fixed_data[PGSTAT_KIND_MAX + 1];
} PgStat_ShmemControl;
@@ -451,8 +451,8 @@ typedef struct PgStat_Snapshot
* Each entry is allocated in TopMemoryContext, for a size of
* shared_data_len.
*/
- bool fixed_valid[PGSTAT_NUM_KINDS];
- void *fixed_data[PGSTAT_NUM_KINDS];
+ bool fixed_valid[PGSTAT_KIND_MAX + 1];
+ void *fixed_data[PGSTAT_KIND_MAX + 1];
/* to free snapshot in bulk */
MemoryContext context;
@@ -497,6 +497,8 @@ static inline void *pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common
*/
extern const PgStat_KindInfo *pgstat_get_kind_info(PgStat_Kind kind);
+extern void pgstat_register_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
#ifdef USE_ASSERT_CHECKING
extern void pgstat_assert_is_up(void);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index fb27272e86..994aa2ba51 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -49,8 +49,16 @@
* pgStatPending list. Pending statistics updates are flushed out by
* pgstat_report_stat().
*
+ * It is possible for external modules to define custom statistics kinds,
+ * that can use the same properties as any built-in stats kinds. Each custom
+ * stats kind needs to assign a unique ID to ensure that it does not overlap
+ * with other extensions. In order to reserve a unique stats kind ID, refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats.
+ *
* The behavior of different kinds of statistics is determined by the kind's
- * entry in pgstat_kind_infos, see PgStat_KindInfo for details.
+ * entry in pgstat_kind_builtin_infos for all the built-in statistics kinds
+ * defined, and pgstat_kind_custom_infos for custom kinds registered at
+ * startup by pgstat_register_kind(). See PgStat_KindInfo for details.
*
* The consistency of read accesses to statistics can be configured using the
* stats_fetch_consistency GUC (see config.sgml and monitoring.sgml for the
@@ -253,7 +261,7 @@ static bool pgstat_is_shutdown = false;
/*
- * The different kinds of statistics.
+ * The different kinds of built-in statistics.
*
* If reasonably possible, handling specific to one kind of stats should go
* through this abstraction, rather than making more of pgstat.c aware.
@@ -265,7 +273,7 @@ static bool pgstat_is_shutdown = false;
* seem to be a great way of doing that, given the split across multiple
* files.
*/
-static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
+static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_MAX_BUILTIN + 1] = {
/* stats kinds for variable-numbered objects */
@@ -432,6 +440,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
},
};
+/*
+ * Information about custom statistics kinds.
+ *
+ * These are saved in a different array than the built-in kinds to save
+ * in clarity with the initializations.
+ *
+ * Indexed by PGSTAT_KIND_CUSTOM_MIN, of size PGSTAT_KIND_CUSTOM_SIZE.
+ */
+static const PgStat_KindInfo **pgstat_kind_custom_infos = NULL;
/* ------------------------------------------------------------
* Functions managing the state of the stats system for all backends.
@@ -997,11 +1014,11 @@ static void
pgstat_init_snapshot(void)
{
/* Initialize fixed-numbered statistics data in snapshots */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
pgStatLocal.snapshot.fixed_data[kind] =
@@ -1102,10 +1119,12 @@ pgstat_build_snapshot(void)
/*
* Build snapshot of all fixed-numbered stats.
*/
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ if (!kind_info)
+ continue;
if (!kind_info->fixed_amount)
{
Assert(kind_info->snapshot_cb == NULL);
@@ -1299,30 +1318,125 @@ pgstat_flush_pending_entries(bool nowait)
PgStat_Kind
pgstat_get_kind_from_str(char *kind_str)
{
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN_BUILTIN; kind <= PGSTAT_KIND_MAX_BUILTIN; kind++)
{
- if (pg_strcasecmp(kind_str, pgstat_kind_infos[kind].name) == 0)
+ if (pg_strcasecmp(kind_str, pgstat_kind_builtin_infos[kind].name) == 0)
return kind;
}
+ /* Check the custom set of cumulative stats */
+ if (pgstat_kind_custom_infos)
+ {
+ for (int kind = 0; kind < PGSTAT_KIND_CUSTOM_SIZE; kind++)
+ {
+ if (pgstat_kind_custom_infos[kind] &&
+ pg_strcasecmp(kind_str, pgstat_kind_custom_infos[kind]->name) == 0)
+ return kind + PGSTAT_KIND_CUSTOM_MIN;
+ }
+ }
+
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid statistics kind: \"%s\"", kind_str)));
- return PGSTAT_KIND_DATABASE; /* avoid compiler warnings */
+ return PGSTAT_KIND_INVALID; /* avoid compiler warnings */
}
static inline bool
pgstat_is_kind_valid(PgStat_Kind kind)
{
- return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
+ return pgstat_is_kind_builtin(kind) || pgstat_is_kind_custom(kind);
}
const PgStat_KindInfo *
pgstat_get_kind_info(PgStat_Kind kind)
{
- Assert(pgstat_is_kind_valid(kind));
+ if (pgstat_is_kind_builtin(kind))
+ return &pgstat_kind_builtin_infos[kind];
- return &pgstat_kind_infos[kind];
+ if (pgstat_is_kind_custom(kind))
+ {
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (pgstat_kind_custom_infos == NULL ||
+ pgstat_kind_custom_infos[idx] == NULL)
+ return NULL;
+ return pgstat_kind_custom_infos[idx];
+ }
+
+ return NULL;
+}
+
+/*
+ * Register a new stats kind.
+ *
+ * PgStat_Kinds must be globally unique across all extensions. Refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats to reserve a
+ * unique ID for your extension, to avoid conflicts with other extension
+ * developers. During development, use PGSTAT_KIND_EXPERIMENTAL to avoid
+ * needlessly reserving a new ID.
+ */
+void
+pgstat_register_kind(PgStat_Kind kind, const PgStat_KindInfo *kind_info)
+{
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (kind_info->name == NULL || strlen(kind_info->name) == 0)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics name is invalid"),
+ errhint("Provide a non-empty name for the custom cumulative statistics.")));
+
+ if (!pgstat_is_kind_custom(kind))
+ ereport(ERROR, (errmsg("custom cumulative statistics ID %u is out of range", kind),
+ errhint("Provide a custom cumulative statistics ID between %u and %u.",
+ PGSTAT_KIND_CUSTOM_MIN, PGSTAT_KIND_CUSTOM_MAX)));
+
+ if (!process_shared_preload_libraries_in_progress)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics must be registered while initializing modules in \"shared_preload_libraries\".")));
+
+ /*
+ * These are not supported for now, as these point out to fixed areas of
+ * shared memory.
+ */
+ if (kind_info->fixed_amount)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics property is invalid"),
+ errhint("Custom cumulative statistics cannot use a fixed amount of data.")));
+
+ /*
+ * If pgstat_kind_custom_infos is not available yet, allocate it.
+ */
+ if (pgstat_kind_custom_infos == NULL)
+ {
+ pgstat_kind_custom_infos = (const PgStat_KindInfo **)
+ MemoryContextAllocZero(TopMemoryContext,
+ sizeof(PgStat_KindInfo *) * PGSTAT_KIND_CUSTOM_SIZE);
+ }
+
+ if (pgstat_kind_custom_infos[idx] != NULL &&
+ pgstat_kind_custom_infos[idx]->name != NULL)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics \"%s\" already registered with the same ID.",
+ pgstat_kind_custom_infos[idx]->name)));
+
+ /* check for existing custom stats with the same name */
+ for (int existing_kind = 0; existing_kind < PGSTAT_KIND_CUSTOM_SIZE; existing_kind++)
+ {
+ if (pgstat_kind_custom_infos[existing_kind] == NULL)
+ continue;
+ if (!pg_strcasecmp(pgstat_kind_custom_infos[existing_kind]->name, kind_info->name))
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Existing cumulative statistics with ID %u has the same name.", existing_kind + PGSTAT_KIND_CUSTOM_MIN)));
+ }
+
+ /* Register it */
+ pgstat_kind_custom_infos[idx] = kind_info;
+ ereport(LOG,
+ (errmsg("registered custom cumulative statistics \"%s\" with ID %u",
+ kind_info->name, kind)));
}
/*
@@ -1399,12 +1513,12 @@ pgstat_write_statsfile(void)
write_chunk_s(fpout, &format_id);
/* Write various stats structs with fixed number of objects */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
char *ptr;
const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
- if (!info->fixed_amount)
+ if (!info || !info->fixed_amount)
continue;
Assert(info->shared_size != 0 && info->shared_data_len != 0);
@@ -1434,6 +1548,17 @@ pgstat_write_statsfile(void)
if (ps->dropped)
continue;
+ /*
+ * This discards data related to custom stats kinds that are unknown
+ * to this process.
+ */
+ if (!pgstat_is_kind_valid(ps->key.kind))
+ {
+ elog(WARNING, "found unknown stats entry %u/%u/%u",
+ ps->key.kind, ps->key.dboid, ps->key.objoid);
+ continue;
+ }
+
shstats = (PgStatShared_Common *) dsa_get_address(pgStatLocal.dsa, ps->body);
kind_info = pgstat_get_kind_info(ps->key.kind);
@@ -1649,7 +1774,7 @@ pgstat_read_statsfile(void)
if (found)
{
dshash_release_lock(pgStatLocal.shared_hash, p);
- elog(WARNING, "found duplicate stats entry %d/%u/%u",
+ elog(WARNING, "found duplicate stats entry %u/%u/%u",
key.kind, key.dboid, key.objoid);
goto error;
}
@@ -1706,12 +1831,12 @@ pgstat_reset_after_failure(void)
{
TimestampTz ts = GetCurrentTimestamp();
- /* reset fixed-numbered stats */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ /* reset fixed-numbered stats for built-in */
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
kind_info->reset_all_cb(ts);
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index bd62c4b72a..0b50d58cce 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -132,10 +132,12 @@ StatsShmemSize(void)
sz = add_size(sz, pgstat_dsa_init_size());
/* Add shared memory for all the fixed-numbered statistics */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ if (!kind_info)
+ continue;
if (!kind_info->fixed_amount)
continue;
@@ -210,11 +212,11 @@ StatsShmemInit(void)
pg_atomic_init_u64(&ctl->gc_request_count, 1);
/* initialize fixed-numbered statistics */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
Assert(kind_info->shared_size != 0);
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 3876339ee1..3221137123 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1696,7 +1696,7 @@ pg_stat_reset(PG_FUNCTION_ARGS)
* Reset some shared cluster-wide counters
*
* When adding a new reset target, ideally the name should match that in
- * pgstat_kind_infos, if relevant.
+ * pgstat_kind_builtin_infos, if relevant.
*/
Datum
pg_stat_reset_shared(PG_FUNCTION_ARGS)
--
2.45.2
On 6/13/24 14:59, Michael Paquier wrote:
This will hopefully spark a discussion, and I was looking for answers
regarding these questions:
- Should the pgstat_kind_infos array in pgstat.c be refactored to use
something similar to pgstat_add_kind?
- How should the persistence of the custom stats be achieved?
Callbacks to give custom stats kinds a way to write/read their data,
push everything into a single file, or support both?
- Should this do like custom RMGRs and assign to each stats kinds ID
that are set in stone rather than dynamic ones?
It is a feature my extensions (which usually change planning behaviour)
definitely need. It is a problem to show the user if the extension does
something or not because TPS smooths the execution time of a single
query and performance cliffs.
BTW, we have 'labelled DSM segments', which allowed extensions to be
'lightweight' - not necessarily be loaded on startup, stay backend-local
and utilise shared resources. It was a tremendous win for me. Is it
possible to design this extension in the same way?
--
regards, Andrei Lepikhov
On Thu, Jul 04, 2024 at 10:11:02AM +0700, Andrei Lepikhov wrote:
It is a feature my extensions (which usually change planning behaviour)
definitely need. It is a problem to show the user if the extension does
something or not because TPS smooths the execution time of a single query
and performance cliffs.
Yeah, I can get that. pgstat.c is quite good regarding that as it
delays stats flushes until commit by holding pending entries (see the
pgStatPending business for variable-size stats). Custom stats kinds
registered would just rely on these facilities, including snapshot
APIs, etc.
BTW, we have 'labelled DSM segments', which allowed extensions to be
'lightweight' - not necessarily be loaded on startup, stay backend-local and
utilise shared resources. It was a tremendous win for me.Is it possible to design this extension in the same way?
I am not sure how this would be useful when it comes to cumulative
statistics, TBH. These stats are global by design, and especially
since these most likely need to be flushed at shutdown (as of HEAD)
and read at startup, the simplest way to achieve that to let the
checkpointer and the startup process know about them is to restrict
the registration of custom stats types via _PG_init when loading
shared libraries. That's what we do for custom WAL RMGRs, for
example.
I would not be against a new flag in KindInfo to state that a given
stats type should not be flushed, as much as a set of callbacks that
offers the possibility to redirect some stats kinds to somewhere else
than pgstat.stat, like pg_stat_statements. That would be a separate
patch than what's proposed here.
--
Michael
Hi,
On Wed, Jul 03, 2024 at 06:47:15PM +0900, Michael Paquier wrote:
While looking at a different patch from Tristan in this area at [1], I
still got annoyed that this patch set was not able to support the case
of custom fixed-numbered stats, so as it is possible to plug in
pgstats things similar to the archiver, the checkpointer, WAL, etc.
These are plugged in shared memory, and are handled with copies in the
stats snapshots. After a good night of sleep, I have come up with a
good solution for that,
Great!
among the following lines:
- PgStat_ShmemControl holds an array of void* indexed by
PGSTAT_NUM_KINDS, pointing to shared memory areas allocated for each
fixed-numbered stats. Each entry is allocated a size corresponding to
PgStat_KindInfo->shared_size.
That makes sense to me, and that's just a 96 bytes overhead (8 * PGSTAT_NUM_KINDS)
as compared to now.
- PgStat_Snapshot holds an array of void* also indexed by
PGSTAT_NUM_KINDS, pointing to the fixed stats stored in the
snapshots.
Same, that's just a 96 bytes overhead (8 * PGSTAT_NUM_KINDS) as compared to now.
These have a size of PgStat_KindInfo->shared_data_len, set
up when stats are initialized at process startup, so this reflects
everywhere.
Yeah.
- Fixed numbered stats now set shared_size, and we use this number to
determine the size to allocate for each fixed-numbered stats in shmem.
- A callback is added to initialize the shared memory assigned to each
fixed-numbered stats, consisting of LWLock initializations for the
current types of stats. So this initialization step is moved out of
pgstat.c into each stats kind file.
That looks a reasonable approach to me.
All that has been done in the rebased patch set as of 0001, which is
kind of a nice cleanup overall because it removes all the dependencies
to the fixed-numbered stats structures from the "main" pgstats code in
pgstat.c and pgstat_shmem.c.
Looking at 0001:
1 ===
In the commit message:
- Fixed numbered stats now set shared_size, so as
Is something missing in that sentence?
2 ===
@@ -425,14 +427,12 @@ typedef struct PgStat_ShmemControl
pg_atomic_uint64 gc_request_count;
/*
- * Stats data for fixed-numbered objects.
+ * Stats data for fixed-numbered objects, indexed by PgStat_Kind.
+ *
+ * Each entry has a size of PgStat_KindInfo->shared_size.
*/
- PgStatShared_Archiver archiver;
- PgStatShared_BgWriter bgwriter;
- PgStatShared_Checkpointer checkpointer;
- PgStatShared_IO io;
- PgStatShared_SLRU slru;
- PgStatShared_Wal wal;
+ void *fixed_data[PGSTAT_NUM_KINDS];
Can we move from PGSTAT_NUM_KINDS to the exact number of fixed stats? (add
a new define PGSTAT_NUM_FIXED_KINDS for example). That's not a big deal but we
are allocating some space for pointers that we won't use. Would need to change
the "indexing" logic though.
3 ===
Same as 2 === but for PgStat_Snapshot.
4 ===
+static void pgstat_init_snapshot(void);
what about pgstat_init_snapshot_fixed? (as it is for fixed-numbered statistics
only).
5 ===
+ /* Write various stats structs with fixed number of objects */
s/Write various stats/Write the stats/? (not coming from your patch but they
all were listed before though).
6 ===
+ for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ {
+ char *ptr;
+ const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
+
+ if (!info->fixed_amount)
+ continue;
Nit: Move the "ptr" declaration into an extra else? (useless to declare it
if it's not a fixed number stat)
7 ===
+ /* prepare snapshot data and write it */
+ pgstat_build_snapshot_fixed(kind);
What about changing pgstat_build_snapshot_fixed() to accept a PgStat_KindInfo
parameter (instead of the current PgStat_Kind one)? Reason is that
pgstat_get_kind_info() is already called/known in pgstat_snapshot_fixed(),
pgstat_build_snapshot() and pgstat_write_statsfile(). That would avoid
pgstat_build_snapshot_fixed() to retrieve (again) the kind_info.
8 ===
/*
* Reads in existing statistics file into the shared stats hash.
This comment above pgstat_read_statsfile() is not correct, fixed stats
are not going to the hash (was there before your patch though).
9 ===
+pgstat_archiver_init_shmem_cb(void *stats)
+{
+ PgStatShared_Archiver *stats_shmem = (PgStatShared_Archiver *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
Nit: Almost all the pgstat_XXX_init_shmem_cb() look very similar, I wonder if we
could use a macro to avoid code duplication.
10 ===
Remark not related to this patch: I think we could get rid of the shared_data_off
for the fixed stats (by moving the "stats" part at the header of their dedicated
struct). That would mean having things like:
"
typedef struct PgStatShared_Archiver
{
PgStat_ArchiverStats stats;
/* lock protects ->reset_offset as well as stats->stat_reset_timestamp */
LWLock lock;
uint32 changecount;
PgStat_ArchiverStats reset_offset;
} PgStatShared_Archiver;
"
Not sure that's worth it though.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On 2024-07-03 18:47:15 +0900, Michael Paquier wrote:
While looking at a different patch from Tristan in this area at [1], I
still got annoyed that this patch set was not able to support the case
of custom fixed-numbered stats, so as it is possible to plug in
pgstats things similar to the archiver, the checkpointer, WAL, etc.
These are plugged in shared memory, and are handled with copies in the
stats snapshots. After a good night of sleep, I have come up with a
good solution for that, among the following lines:
- PgStat_ShmemControl holds an array of void* indexed by
PGSTAT_NUM_KINDS, pointing to shared memory areas allocated for each
fixed-numbered stats. Each entry is allocated a size corresponding to
PgStat_KindInfo->shared_size.
I am dubious this is a good idea. The more indirection you add, the more
expensive it gets to count stuff, the more likely it is that we end up with
backend-local "caching" in front of the stats system.
IOW, I am against making builtin stats pay the price for pluggable
fixed-numbered stats.
It also substantially reduces type-safety, making it harder to refactor. Note
that you had to add static casts in a good number of additional places.
Greetings,
Andres Freund
Hi,
On 2024-06-13 16:59:50 +0900, Michael Paquier wrote:
* Making custom stats data persistent is an interesting problem, and
there are a couple of approaches I've considered:
** Allow custom kinds to define callbacks to read and write data from
a source they'd want, like their own file through a fd. This has the
disadvantage to remove the benefit of c) above.
I am *strongly* against this. That'll make it much harder to do stuff like not
resetting stats after crashes and just generally will make it harder to
improve the stats facility further.
I think that pluggable users of the stats facility should only have control
over how data is stored via quite generic means.
Greetings,
Andres Freund
Hi,
On 2024-07-04 14:00:47 -0700, Andres Freund wrote:
On 2024-06-13 16:59:50 +0900, Michael Paquier wrote:
* Making custom stats data persistent is an interesting problem, and
there are a couple of approaches I've considered:
** Allow custom kinds to define callbacks to read and write data from
a source they'd want, like their own file through a fd. This has the
disadvantage to remove the benefit of c) above.I am *strongly* against this. That'll make it much harder to do stuff like not
resetting stats after crashes and just generally will make it harder to
improve the stats facility further.I think that pluggable users of the stats facility should only have control
over how data is stored via quite generic means.
I forgot to say: In general I am highly supportive of this effort and thankful
to Michael for tackling it. The above was just about that one aspect.
Greetings,
Andres Freund
On Thu, Jul 04, 2024 at 02:00:47PM -0700, Andres Freund wrote:
On 2024-06-13 16:59:50 +0900, Michael Paquier wrote:
* Making custom stats data persistent is an interesting problem, and
there are a couple of approaches I've considered:
** Allow custom kinds to define callbacks to read and write data from
a source they'd want, like their own file through a fd. This has the
disadvantage to remove the benefit of c) above.I am *strongly* against this. That'll make it much harder to do stuff like not
resetting stats after crashes and just generally will make it harder to
improve the stats facility further.I think that pluggable users of the stats facility should only have control
over how data is stored via quite generic means.
I'm pretty much on the same line here, I think. If the redo logic is
changed, then any stats kinds pushing their stats into their own file
would need to copy/paste the same logic as the main file. And that's
more error prone.
I can get why some people would get that they don't want some stats
kinds to never be flushed at shutdown or even read at startup. Adding
more callbacks in this area is a separate discussion.
--
Michael
On Thu, Jul 04, 2024 at 02:08:25PM -0700, Andres Freund wrote:
I forgot to say: In general I am highly supportive of this effort and thankful
to Michael for tackling it. The above was just about that one aspect.
Thanks. Let's discuss how people want this stuff to be shaped, and
how much we want to cover. Better to do it one small step at a time.
--
Michael
On Thu, Jul 04, 2024 at 01:56:52PM -0700, Andres Freund wrote:
On 2024-07-03 18:47:15 +0900, Michael Paquier wrote:
- PgStat_ShmemControl holds an array of void* indexed by
PGSTAT_NUM_KINDS, pointing to shared memory areas allocated for each
fixed-numbered stats. Each entry is allocated a size corresponding to
PgStat_KindInfo->shared_size.I am dubious this is a good idea. The more indirection you add, the more
expensive it gets to count stuff, the more likely it is that we end up with
backend-local "caching" in front of the stats system.IOW, I am against making builtin stats pay the price for pluggable
fixed-numbered stats.
Okay, noted. So, if I get that right, you would prefer an approach
where we add an extra member in the snapshot and shmem control area
dedicated only to the custom kind IDs, indexed based on the range
of the custom kind IDs, leaving the built-in fixed structures in
PgStat_ShmemControl and PgStat_Snapshot?
I was feeling a bit uncomfortable with the extra redirection for the
built-in fixed kinds, still the temptation of making that more generic
was here, so..
Having the custom fixed types point to their own array in the snapshot
and ShmemControl adds a couple more null-ness checks depending on if
you're dealing with a builtin or custom ID range. That's mostly the
path in charge of retrieving the KindInfos.
It also substantially reduces type-safety, making it harder to refactor. Note
that you had to add static casts in a good number of additional places.
Not sure on this one, because that's the same issue as
variable-numbered stats, no? The central dshash only knows about the
size of the shared stats entries for each kind, with an offset to the
stats data that gets copied to the snapshots. So I don't quite get
the worry here.
Separately from that, I think that read/write of the fixed-numbered
stats would gain in clarity if we update them to be closer to the
variable-numbers by storing entries with a specific character ('F' in
0001). If we keep track of the fixed-numbered structures in
PgStat_Snapshot, that means adding an extra field in PgStat_KindInfo
to point to the offset in PgStat_Snapshot for the write part. Note
that the addition of the init_shmem callback simplifies shmem init,
and it is also required for the fixed-numbered pluggable part.
--
Michael
On Thu, Jul 04, 2024 at 11:30:17AM +0000, Bertrand Drouvot wrote:
On Wed, Jul 03, 2024 at 06:47:15PM +0900, Michael Paquier wrote:
among the following lines:
- PgStat_ShmemControl holds an array of void* indexed by
PGSTAT_NUM_KINDS, pointing to shared memory areas allocated for each
fixed-numbered stats. Each entry is allocated a size corresponding to
PgStat_KindInfo->shared_size.That makes sense to me, and that's just a 96 bytes overhead (8 * PGSTAT_NUM_KINDS)
as compared to now.
pgstat_io.c is by far the largest chunk.
- PgStat_Snapshot holds an array of void* also indexed by
PGSTAT_NUM_KINDS, pointing to the fixed stats stored in the
snapshots.Same, that's just a 96 bytes overhead (8 * PGSTAT_NUM_KINDS) as compared to now.
Still Andres does not seem to like that much, well ;)
Looking at 0001:
1 ===
In the commit message:
- Fixed numbered stats now set shared_size, so as
Is something missing in that sentence?
Right. This is missing a piece.
- PgStatShared_Archiver archiver; - PgStatShared_BgWriter bgwriter; - PgStatShared_Checkpointer checkpointer; - PgStatShared_IO io; - PgStatShared_SLRU slru; - PgStatShared_Wal wal; + void *fixed_data[PGSTAT_NUM_KINDS];Can we move from PGSTAT_NUM_KINDS to the exact number of fixed stats? (add
a new define PGSTAT_NUM_FIXED_KINDS for example). That's not a big deal but we
are allocating some space for pointers that we won't use. Would need to change
the "indexing" logic though.3 ===
Same as 2 === but for PgStat_Snapshot.
True for both. Based on the first inputs I got from Andres, the
built-in fixed stats structures would be kept as they are now, and we
could just add an extra member here for the custom fixed stats. That
still results in a few bytes wasted as not all custom stats want fixed
stats, but that's much cheaper.
4 ===
+static void pgstat_init_snapshot(void);
what about pgstat_init_snapshot_fixed? (as it is for fixed-numbered statistics
only).
Sure.
5 ===
+ /* Write various stats structs with fixed number of objects */
s/Write various stats/Write the stats/? (not coming from your patch but they
all were listed before though).
Yes, there are a few more things about that.
6 ===
+ for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++) + { + char *ptr; + const PgStat_KindInfo *info = pgstat_get_kind_info(kind); + + if (!info->fixed_amount) + continue;Nit: Move the "ptr" declaration into an extra else? (useless to declare it
if it's not a fixed number stat)
Comes down to one's taste. I think that this is OK as-is, but that's
my taste.
7 ===
+ /* prepare snapshot data and write it */ + pgstat_build_snapshot_fixed(kind);What about changing pgstat_build_snapshot_fixed() to accept a PgStat_KindInfo
parameter (instead of the current PgStat_Kind one)? Reason is that
pgstat_get_kind_info() is already called/known in pgstat_snapshot_fixed(),
pgstat_build_snapshot() and pgstat_write_statsfile(). That would avoid
pgstat_build_snapshot_fixed() to retrieve (again) the kind_info.
pgstat_snapshot_fixed() only calls pgstat_get_kind_info() with
assertions enabled. Perhaps we could do that, just that it does not
seem that critical to me.
8 ===
/*
* Reads in existing statistics file into the shared stats hash.This comment above pgstat_read_statsfile() is not correct, fixed stats
are not going to the hash (was there before your patch though).
Good catch. Let's adjust that separately.
9 ===
+pgstat_archiver_init_shmem_cb(void *stats) +{ + PgStatShared_Archiver *stats_shmem = (PgStatShared_Archiver *) stats; + + LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);Nit: Almost all the pgstat_XXX_init_shmem_cb() look very similar, I wonder if we
could use a macro to avoid code duplication.
They are very similar, still can do different things like pgstat_io.
I am not sure that the macro would bring more readability.
Remark not related to this patch: I think we could get rid of the shared_data_off
for the fixed stats (by moving the "stats" part at the header of their dedicated
struct). That would mean having things like:"
typedef struct PgStatShared_Archiver
{
PgStat_ArchiverStats stats;
/* lock protects ->reset_offset as well as stats->stat_reset_timestamp */
LWLock lock;
uint32 changecount;
PgStat_ArchiverStats reset_offset;
} PgStatShared_Archiver;
"
I'm not really convinced that it is a good idea to force the ordering
of the members in the shared structures for the fixed-numbered stats,
requiring these "stats" fields to always be first.
--
Michael
On Fri, Jun 21, 2024 at 01:28:11PM +0900, Michael Paquier wrote:
On Fri, Jun 21, 2024 at 01:09:10PM +0900, Kyotaro Horiguchi wrote:At Thu, 13 Jun 2024 16:59:50 +0900, Michael Paquier <michael@paquier.xyz> wrote in
* The kind IDs may change across restarts, meaning that any stats data
associated to a custom kind is stored with the *name* of the custom
stats kind. Depending on the discussion happening here, I'd be open
to use the same concept as custom RMGRs, where custom kind IDs are
"reserved", fixed in time, and tracked in the Postgres wiki. It is
cheaper to store the stats this way, as well, while managing conflicts
across extensions available in the community ecosystem.I prefer to avoid having a central database if possible.
I was thinking the same originally, but the experience with custom
RMGRs has made me change my mind. There are more of these than I
thought originally:
https://wiki.postgresql.org/wiki/CustomWALResourceManagers
From what I understand, coordinating custom RmgrIds via a wiki page was
made under the assumption that implementing a table AM with custom WAL
requires significant efforts, which limits the demand for ids. This
might not be same for custom stats -- I've got an impression it's easier
to create one, and there could be multiple kinds of stats per an
extension (one per component), right? This would mean more kind Ids to
manage and more efforts required to do that.
I agree though that it makes sense to start this way, it's just simpler.
But maybe it's worth thinking about some other solution in the long
term, taking the over-engineered prototype as a sign that more
refactoring is needed.
On Sun, Jul 07, 2024 at 12:21:26PM +0200, Dmitry Dolgov wrote:
From what I understand, coordinating custom RmgrIds via a wiki page was
made under the assumption that implementing a table AM with custom WAL
requires significant efforts, which limits the demand for ids. This
might not be same for custom stats -- I've got an impression it's easier
to create one, and there could be multiple kinds of stats per an
extension (one per component), right? This would mean more kind Ids to
manage and more efforts required to do that.
A given module will likely have one single RMGR because it is possible
to divide the RMGR into multiple records. Yes, this cannot really be
said for stats, and a set of stats kinds in one module may want
different kinds because these could have different properties.
My guess is that a combination of one fixed-numbered to track a global
state and one variable-numbered would be the combination most likely
to happen. Also, my impression about pg_stat_statements is that we'd
need this combination, actually, to track the number of entries in a
tighter way because scanning all the partitions of the central dshash
for entries with a specific KindInfo would have a high concurrency
cost.
I agree though that it makes sense to start this way, it's just simpler.
But maybe it's worth thinking about some other solution in the long
term, taking the over-engineered prototype as a sign that more
refactoring is needed.
The three possible methods I can think of here are, knowing that we
use a central, unique, file to store the stats (per se the arguments
on the redo thread for the stats):
- Store the name of the stats kinds with each entry. This is very
costly with many entries, and complicates the read-write paths because
currently we rely on the KindInfo.
- Store a mapping between the stats kind name and the KindInfo in the
file at write, then use the mapping at read and compare it reassemble
the entries stored. KindInfos are assigned at startup with a unique
counter in shmem. As mentioned upthread, I've implemented something
like that while making the custom stats being registered in the
shmem_startup_hook with requests in shmem_request_hook. That felt
over-engineered considering that the startup process needs to know the
stats kinds very early anyway, so we need _PG_init() and should
encourage its use.
- Fix the KindInfos in time and centralize the values assigned. This
eases the error control and can force the custom stats kinds to be
registered when shared_preload_libraries is loaded. The read is
faster as there is no need to re-check the mapping to reassemble
the stats entries.
At the end, fixing the KindInfos in time is the most reliable method
here (debugging could be slightly easier, less complicated than with
the mapping stored, still doable for all three methods).
--
Michael
On Fri, Jul 05, 2024 at 09:35:19AM +0900, Michael Paquier wrote:
On Thu, Jul 04, 2024 at 11:30:17AM +0000, Bertrand Drouvot wrote:
On Wed, Jul 03, 2024 at 06:47:15PM +0900, Michael Paquier wrote:
- PgStat_Snapshot holds an array of void* also indexed by
PGSTAT_NUM_KINDS, pointing to the fixed stats stored in the
snapshots.Same, that's just a 96 bytes overhead (8 * PGSTAT_NUM_KINDS) as compared to now.
Still Andres does not seem to like that much, well ;)
Please find attached a rebased patch set labelled v4. Built-in
fixed-numbered stats are still attached to the snapshot and shmem
control structures, and custom fixed stats kinds are tracked in the
same way as v3 with new members tracking data stored in
TopMemoryContext for the snapshots and shmem for the control data.
So, the custom and built-in stats kinds are separated into separate
parts of the structures, including the "valid" flags for the
snapshots. And this avoids any redirection when looking at the
built-in fixed-numbered stats.
I've tried at address all the previous comments (there could be stuff
I've missed while rebasing, of course).
The first three patches are refactoring pieces to make the rest more
edible, while 0004~ implement the main logic with templates in
modules/injection_points:
- 0001 refactors pgstat_write_statsfile() so as a loop om
PgStat_KindInfo is used to write the data. This is done with the
addition of snapshot_ctl_off in PgStat_KindInfo, to point to the area
in PgStat_Snapshot where the data is located for fixed stats.
9004abf6206e has done the same for the read part.
- 0002 adds an init_shmem callback, to let stats kinds initialize
states based on what's been allocated.
- 0003 refactors the read/write to use a new entry type in the stats
file for fixed-numbered stats.
- 0004 switches PgStat_Kind from an enum to uint32, adding a better
type for pluggability.
- 0005 is the main implementation.
- 0006 adds some docs.
- 0007 (variable-numbered stats) and 0008 (fixed-numbered stats) are
the examples demonstrating how to make pluggable stats for both types,
with tests of their own.
--
Michael
Attachments:
v4-0005-Introduce-pluggable-APIs-for-Cumulative-Statistic.patchtext/x-diff; charset=us-asciiDownload
From b21fd6cfd839595d0987674402a8a266350b0192 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 14:03:17 +0900
Subject: [PATCH v4 5/8] Introduce pluggable APIs for Cumulative Statistics
This commit adds support in the backend for $subject, allowing
out-of-core extensions to add their own custom statistics kinds. The
stats kinds are divided into two parts for efficiency:
- The built-in stats kinds, with designated initializers.
- The custom kinds, able to use a range of IDs (128 slots available as
of this patch), with information saved in TopMemoryContext.
Custom cumulative statistics can only be loaded with
shared_preload_libraries at startup, and must allocate a unique ID
shared across all the PostgreSQL extension ecosystem with the following
wiki page:
https://wiki.postgresql.org/wiki/CustomCumulativeStats
This is able to support fixed-numbered (like WAL, archiver, bgwriter)
and variable-numbered stats kinds.
---
src/include/pgstat.h | 35 +++-
src/include/utils/pgstat_internal.h | 22 ++-
src/backend/utils/activity/pgstat.c | 231 +++++++++++++++++++---
src/backend/utils/activity/pgstat_shmem.c | 31 ++-
src/backend/utils/adt/pgstatfuncs.c | 2 +-
5 files changed, 287 insertions(+), 34 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 2d30fadaf1..8d523607a4 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -34,6 +34,10 @@
/* The types of statistics entries */
#define PgStat_Kind uint32
+/* Range of IDs allowed, for built-in and custom kinds */
+#define PGSTAT_KIND_MIN 1 /* Minimum ID allowed */
+#define PGSTAT_KIND_MAX 256 /* Maximum ID allowed */
+
/* use 0 for INVALID, to catch zero-initialized data */
#define PGSTAT_KIND_INVALID 0
@@ -52,9 +56,34 @@
#define PGSTAT_KIND_SLRU 10
#define PGSTAT_KIND_WAL 11
-#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
-#define PGSTAT_NUM_KINDS (PGSTAT_KIND_LAST + 1)
+#define PGSTAT_KIND_MIN_BUILTIN PGSTAT_KIND_DATABASE
+#define PGSTAT_KIND_MAX_BUILTIN PGSTAT_KIND_WAL
+
+/* Custom stats kinds */
+
+/* Range of IDs allowed for custom stats kinds */
+#define PGSTAT_KIND_CUSTOM_MIN 128
+#define PGSTAT_KIND_CUSTOM_MAX PGSTAT_KIND_MAX
+#define PGSTAT_KIND_CUSTOM_SIZE (PGSTAT_KIND_CUSTOM_MAX - PGSTAT_KIND_CUSTOM_MIN + 1)
+
+/*
+ * PgStat_Kind to use for extensions that require an ID, but are still in
+ * development and have not reserved their own unique kind ID yet. See:
+ * https://wiki.postgresql.org/wiki/CustomCumulativeStats
+ */
+#define PGSTAT_KIND_EXPERIMENTAL 128
+
+static inline bool
+pgstat_is_kind_builtin(PgStat_Kind kind)
+{
+ return kind > PGSTAT_KIND_INVALID && kind <= PGSTAT_KIND_MAX_BUILTIN;
+}
+
+static inline bool
+pgstat_is_kind_custom(PgStat_Kind kind)
+{
+ return kind >= PGSTAT_KIND_CUSTOM_MIN && kind <= PGSTAT_KIND_CUSTOM_MAX;
+}
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 778f625ca1..39f63362a3 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -195,7 +195,8 @@ typedef struct PgStat_KindInfo
/*
* The size of an entry in the shared stats hash table (pointed to by
- * PgStatShared_HashEntry->body).
+ * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
+ * the size of an entry in PgStat_ShmemControl->custom_data.
*/
uint32 shared_size;
@@ -446,6 +447,13 @@ typedef struct PgStat_ShmemControl
PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
+
+ /*
+ * Custom stats data with fixed-numbered objects, indexed by (PgStat_Kind
+ * - PGSTAT_KIND_CUSTOM_MIN).
+ */
+ void *custom_data[PGSTAT_KIND_CUSTOM_SIZE];
+
} PgStat_ShmemControl;
@@ -459,7 +467,7 @@ typedef struct PgStat_Snapshot
/* time at which snapshot was taken */
TimestampTz snapshot_timestamp;
- bool fixed_valid[PGSTAT_NUM_KINDS];
+ bool fixed_valid[PGSTAT_KIND_MAX_BUILTIN + 1];
PgStat_ArchiverStats archiver;
@@ -473,6 +481,14 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ /*
+ * Data in snapshot for custom fixed-numbered statistics, indexed by
+ * (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
+ * TopMemoryContext, for a size of shared_data_len.
+ */
+ bool custom_valid[PGSTAT_KIND_CUSTOM_SIZE];
+ void *custom_data[PGSTAT_KIND_CUSTOM_SIZE];
+
/* to free snapshot in bulk */
MemoryContext context;
struct pgstat_snapshot_hash *stats;
@@ -516,6 +532,8 @@ static inline void *pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common
*/
extern const PgStat_KindInfo *pgstat_get_kind_info(PgStat_Kind kind);
+extern void pgstat_register_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
#ifdef USE_ASSERT_CHECKING
extern void pgstat_assert_is_up(void);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index f572a87a61..3b85fe0fc8 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -49,8 +49,16 @@
* pgStatPending list. Pending statistics updates are flushed out by
* pgstat_report_stat().
*
+ * It is possible for external modules to define custom statistics kinds,
+ * that can use the same properties as any built-in stats kinds. Each custom
+ * stats kind needs to assign a unique ID to ensure that it does not overlap
+ * with other extensions. In order to reserve a unique stats kind ID, refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats.
+ *
* The behavior of different kinds of statistics is determined by the kind's
- * entry in pgstat_kind_infos, see PgStat_KindInfo for details.
+ * entry in pgstat_kind_builtin_infos for all the built-in statistics kinds
+ * defined, and pgstat_kind_custom_infos for custom kinds registered at
+ * startup by pgstat_register_kind(). See PgStat_KindInfo for details.
*
* The consistency of read accesses to statistics can be configured using the
* stats_fetch_consistency GUC (see config.sgml and monitoring.sgml for the
@@ -174,6 +182,8 @@ typedef struct PgStat_SnapshotEntry
static void pgstat_write_statsfile(void);
static void pgstat_read_statsfile(void);
+static void pgstat_init_snapshot_fixed(void);
+
static void pgstat_reset_after_failure(void);
static bool pgstat_flush_pending_entries(bool nowait);
@@ -251,7 +261,7 @@ static bool pgstat_is_shutdown = false;
/*
- * The different kinds of statistics.
+ * The different kinds of built-in statistics.
*
* If reasonably possible, handling specific to one kind of stats should go
* through this abstraction, rather than making more of pgstat.c aware.
@@ -263,7 +273,7 @@ static bool pgstat_is_shutdown = false;
* seem to be a great way of doing that, given the split across multiple
* files.
*/
-static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
+static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_MAX_BUILTIN + 1] = {
/* stats kinds for variable-numbered objects */
@@ -436,6 +446,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
},
};
+/*
+ * Information about custom statistics kinds.
+ *
+ * These are saved in a different array than the built-in kinds to save
+ * in clarity with the initializations.
+ *
+ * Indexed by PGSTAT_KIND_CUSTOM_MIN, of size PGSTAT_KIND_CUSTOM_SIZE.
+ */
+static const PgStat_KindInfo **pgstat_kind_custom_infos = NULL;
/* ------------------------------------------------------------
* Functions managing the state of the stats system for all backends.
@@ -586,6 +605,8 @@ pgstat_initialize(void)
pgstat_init_wal();
+ pgstat_init_snapshot_fixed();
+
/* Set up a process-exit hook to clean up */
before_shmem_exit(pgstat_shutdown_hook, 0);
@@ -829,6 +850,8 @@ pgstat_clear_snapshot(void)
memset(&pgStatLocal.snapshot.fixed_valid, 0,
sizeof(pgStatLocal.snapshot.fixed_valid));
+ memset(&pgStatLocal.snapshot.custom_valid, 0,
+ sizeof(pgStatLocal.snapshot.custom_valid));
pgStatLocal.snapshot.stats = NULL;
pgStatLocal.snapshot.mode = PGSTAT_FETCH_CONSISTENCY_NONE;
@@ -992,7 +1015,29 @@ pgstat_snapshot_fixed(PgStat_Kind kind)
else
pgstat_build_snapshot_fixed(kind);
- Assert(pgStatLocal.snapshot.fixed_valid[kind]);
+ if (pgstat_is_kind_builtin(kind))
+ Assert(pgStatLocal.snapshot.fixed_valid[kind]);
+ else if (pgstat_is_kind_custom(kind))
+ Assert(pgStatLocal.snapshot.custom_valid[kind - PGSTAT_KIND_CUSTOM_MIN]);
+}
+
+static void
+pgstat_init_snapshot_fixed(void)
+{
+ /*
+ * Initialize fixed-numbered statistics data in snapshots, only for custom
+ * stats kinds.
+ */
+ for (int kind = PGSTAT_KIND_CUSTOM_MIN; kind <= PGSTAT_KIND_CUSTOM_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->fixed_amount)
+ continue;
+
+ pgStatLocal.snapshot.custom_data[kind - PGSTAT_KIND_CUSTOM_MIN] =
+ MemoryContextAlloc(TopMemoryContext, kind_info->shared_data_len);
+ }
}
static void
@@ -1088,10 +1133,12 @@ pgstat_build_snapshot(void)
/*
* Build snapshot of all fixed-numbered stats.
*/
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ if (!kind_info)
+ continue;
if (!kind_info->fixed_amount)
{
Assert(kind_info->snapshot_cb == NULL);
@@ -1108,6 +1155,20 @@ static void
pgstat_build_snapshot_fixed(PgStat_Kind kind)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ int idx;
+ bool *valid;
+
+ /* Position in fixed_valid or custom_valid */
+ if (pgstat_is_kind_builtin(kind))
+ {
+ idx = kind;
+ valid = pgStatLocal.snapshot.fixed_valid;
+ }
+ else
+ {
+ idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+ valid = pgStatLocal.snapshot.custom_valid;
+ }
Assert(kind_info->fixed_amount);
Assert(kind_info->snapshot_cb != NULL);
@@ -1115,21 +1176,21 @@ pgstat_build_snapshot_fixed(PgStat_Kind kind)
if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE)
{
/* rebuild every time */
- pgStatLocal.snapshot.fixed_valid[kind] = false;
+ valid[idx] = false;
}
- else if (pgStatLocal.snapshot.fixed_valid[kind])
+ else if (valid[idx])
{
/* in snapshot mode we shouldn't get called again */
Assert(pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_CACHE);
return;
}
- Assert(!pgStatLocal.snapshot.fixed_valid[kind]);
+ Assert(!valid[idx]);
kind_info->snapshot_cb();
- Assert(!pgStatLocal.snapshot.fixed_valid[kind]);
- pgStatLocal.snapshot.fixed_valid[kind] = true;
+ Assert(!valid[idx]);
+ valid[idx] = true;
}
@@ -1285,30 +1346,127 @@ pgstat_flush_pending_entries(bool nowait)
PgStat_Kind
pgstat_get_kind_from_str(char *kind_str)
{
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN_BUILTIN; kind <= PGSTAT_KIND_MAX_BUILTIN; kind++)
{
- if (pg_strcasecmp(kind_str, pgstat_kind_infos[kind].name) == 0)
+ if (pg_strcasecmp(kind_str, pgstat_kind_builtin_infos[kind].name) == 0)
return kind;
}
+ /* Check the custom set of cumulative stats */
+ if (pgstat_kind_custom_infos)
+ {
+ for (int kind = 0; kind < PGSTAT_KIND_CUSTOM_SIZE; kind++)
+ {
+ if (pgstat_kind_custom_infos[kind] &&
+ pg_strcasecmp(kind_str, pgstat_kind_custom_infos[kind]->name) == 0)
+ return kind + PGSTAT_KIND_CUSTOM_MIN;
+ }
+ }
+
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid statistics kind: \"%s\"", kind_str)));
- return PGSTAT_KIND_DATABASE; /* avoid compiler warnings */
+ return PGSTAT_KIND_INVALID; /* avoid compiler warnings */
}
static inline bool
pgstat_is_kind_valid(PgStat_Kind kind)
{
- return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
+ return pgstat_is_kind_builtin(kind) || pgstat_is_kind_custom(kind);
}
const PgStat_KindInfo *
pgstat_get_kind_info(PgStat_Kind kind)
{
- Assert(pgstat_is_kind_valid(kind));
+ if (pgstat_is_kind_builtin(kind))
+ return &pgstat_kind_builtin_infos[kind];
- return &pgstat_kind_infos[kind];
+ if (pgstat_is_kind_custom(kind))
+ {
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (pgstat_kind_custom_infos == NULL ||
+ pgstat_kind_custom_infos[idx] == NULL)
+ return NULL;
+ return pgstat_kind_custom_infos[idx];
+ }
+
+ return NULL;
+}
+
+/*
+ * Register a new stats kind.
+ *
+ * PgStat_Kinds must be globally unique across all extensions. Refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats to reserve a
+ * unique ID for your extension, to avoid conflicts with other extension
+ * developers. During development, use PGSTAT_KIND_EXPERIMENTAL to avoid
+ * needlessly reserving a new ID.
+ */
+void
+pgstat_register_kind(PgStat_Kind kind, const PgStat_KindInfo *kind_info)
+{
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (kind_info->name == NULL || strlen(kind_info->name) == 0)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics name is invalid"),
+ errhint("Provide a non-empty name for the custom cumulative statistics.")));
+
+ if (!pgstat_is_kind_custom(kind))
+ ereport(ERROR, (errmsg("custom cumulative statistics ID %u is out of range", kind),
+ errhint("Provide a custom cumulative statistics ID between %u and %u.",
+ PGSTAT_KIND_CUSTOM_MIN, PGSTAT_KIND_CUSTOM_MAX)));
+
+ if (!process_shared_preload_libraries_in_progress)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics must be registered while initializing modules in \"shared_preload_libraries\".")));
+
+ /*
+ * Check some data for fixed-numbered stats.
+ */
+ if (kind_info->fixed_amount)
+ {
+ if (kind_info->shared_size == 0)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics property is invalid"),
+ errhint("Custom cumulative statistics require a shared memory size for fixed-numbered objects.")));
+ }
+
+ /*
+ * If pgstat_kind_custom_infos is not available yet, allocate it.
+ */
+ if (pgstat_kind_custom_infos == NULL)
+ {
+ pgstat_kind_custom_infos = (const PgStat_KindInfo **)
+ MemoryContextAllocZero(TopMemoryContext,
+ sizeof(PgStat_KindInfo *) * PGSTAT_KIND_CUSTOM_SIZE);
+ }
+
+ if (pgstat_kind_custom_infos[idx] != NULL &&
+ pgstat_kind_custom_infos[idx]->name != NULL)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics \"%s\" already registered with the same ID.",
+ pgstat_kind_custom_infos[idx]->name)));
+
+ /* check for existing custom stats with the same name */
+ for (int existing_kind = 0; existing_kind < PGSTAT_KIND_CUSTOM_SIZE; existing_kind++)
+ {
+ if (pgstat_kind_custom_infos[existing_kind] == NULL)
+ continue;
+ if (!pg_strcasecmp(pgstat_kind_custom_infos[existing_kind]->name, kind_info->name))
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Existing cumulative statistics with ID %u has the same name.", existing_kind + PGSTAT_KIND_CUSTOM_MIN)));
+ }
+
+ /* Register it */
+ pgstat_kind_custom_infos[idx] = kind_info;
+ ereport(LOG,
+ (errmsg("registered custom cumulative statistics \"%s\" with ID %u",
+ kind_info->name, kind)));
}
/*
@@ -1385,18 +1543,22 @@ pgstat_write_statsfile(void)
write_chunk_s(fpout, &format_id);
/* Write various stats structs for fixed number of objects */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
char *ptr;
const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
- if (!info->fixed_amount)
+ if (!info || !info->fixed_amount)
continue;
- Assert(info->snapshot_ctl_off != 0);
+ if (pgstat_is_kind_builtin(kind))
+ Assert(info->snapshot_ctl_off != 0);
pgstat_build_snapshot_fixed(kind);
- ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+ else
+ ptr = pgStatLocal.snapshot.custom_data[kind - PGSTAT_KIND_CUSTOM_MIN];
fputc(PGSTAT_FILE_ENTRY_FIXED, fpout);
write_chunk_s(fpout, &kind);
@@ -1419,6 +1581,17 @@ pgstat_write_statsfile(void)
if (ps->dropped)
continue;
+ /*
+ * This discards data related to custom stats kinds that are unknown
+ * to this process.
+ */
+ if (!pgstat_is_kind_valid(ps->key.kind))
+ {
+ elog(WARNING, "found unknown stats entry %u/%u/%u",
+ ps->key.kind, ps->key.dboid, ps->key.objoid);
+ continue;
+ }
+
shstats = (PgStatShared_Common *) dsa_get_address(pgStatLocal.dsa, ps->body);
kind_info = pgstat_get_kind_info(ps->key.kind);
@@ -1566,8 +1739,16 @@ pgstat_read_statsfile(void)
Assert(info->fixed_amount);
/* Load back stats into shared memory */
- ptr = ((char *) shmem) + info->shared_ctl_off +
- info->shared_data_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) shmem) + info->shared_ctl_off +
+ info->shared_data_off;
+ else
+ {
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ ptr = ((char *) shmem->custom_data[idx]) +
+ info->shared_data_off;
+ }
if (!read_chunk(fpin, ptr,
info->shared_data_len))
@@ -1635,7 +1816,7 @@ pgstat_read_statsfile(void)
if (found)
{
dshash_release_lock(pgStatLocal.shared_hash, p);
- elog(WARNING, "found duplicate stats entry %d/%u/%u",
+ elog(WARNING, "found duplicate stats entry %u/%u/%u",
key.kind, key.dboid, key.objoid);
goto error;
}
@@ -1693,11 +1874,11 @@ pgstat_reset_after_failure(void)
TimestampTz ts = GetCurrentTimestamp();
/* reset fixed-numbered stats */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
kind_info->reset_all_cb(ts);
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 256f177f5a..c8c5a5e98e 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -131,6 +131,21 @@ StatsShmemSize(void)
sz = MAXALIGN(sizeof(PgStat_ShmemControl));
sz = add_size(sz, pgstat_dsa_init_size());
+ /* Add shared memory for all the custom fixed-numbered statistics */
+ for (int kind = PGSTAT_KIND_CUSTOM_MIN; kind <= PGSTAT_KIND_CUSTOM_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info)
+ continue;
+ if (!kind_info->fixed_amount)
+ continue;
+
+ Assert(kind_info->shared_size != 0);
+
+ sz += MAXALIGN(kind_info->shared_size);
+ }
+
return sz;
}
@@ -197,15 +212,25 @@ StatsShmemInit(void)
pg_atomic_init_u64(&ctl->gc_request_count, 1);
/* initialize fixed-numbered stats */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
char *ptr;
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
- ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ else
+ {
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ Assert(kind_info->shared_size != 0);
+ ctl->custom_data[idx] = ShmemAlloc(kind_info->shared_size);
+ ptr = ctl->custom_data[idx];
+ }
+
kind_info->init_shmem_cb((void *) ptr);
}
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 3876339ee1..3221137123 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1696,7 +1696,7 @@ pg_stat_reset(PG_FUNCTION_ARGS)
* Reset some shared cluster-wide counters
*
* When adding a new reset target, ideally the name should match that in
- * pgstat_kind_infos, if relevant.
+ * pgstat_kind_builtin_infos, if relevant.
*/
Datum
pg_stat_reset_shared(PG_FUNCTION_ARGS)
--
2.45.2
v4-0006-doc-Add-section-for-Custom-Cumulative-Statistics-.patchtext/x-diff; charset=us-asciiDownload
From f79cd779b66489b9ffc1c45e4e775c376f722ab2 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 3 Jul 2024 13:49:53 +0900
Subject: [PATCH v4 6/8] doc: Add section for Custom Cumulative Statistics APIs
This provides a short description of what can be done, with a pointer to
the template in the tree.
---
doc/src/sgml/xfunc.sgml | 63 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 63 insertions(+)
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index 756a9d07fb..878722af51 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3700,6 +3700,69 @@ extern bool InjectionPointDetach(const char *name);
</para>
</sect2>
+ <sect2 id="xfunc-addin-custom-cumulative-statistics">
+ <title>Custom Cumulative Statistics</title>
+
+ <para>
+ Is is possible for add-ins written in C-language to use custom types
+ of cumulative statistics registered in the
+ <link linkend="monitoring-stats-setup">Cumulative Statistics System</link>.
+ </para>
+
+ <para>
+ First, define a <literal>PgStat_KindInfo</literal> that includes all
+ the information related to the custom type registered. For example:
+<programlisting>
+static const PgStat_KindInfo custom_stats = {
+ .name = "custom_stats",
+ .fixed_amount = false,
+ .shared_size = sizeof(PgStatShared_Custom),
+ .shared_data_off = offsetof(PgStatShared_Custom, stats),
+ .shared_data_len = sizeof(((PgStatShared_Custom *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatCustomEntry),
+}
+</programlisting>
+
+ Then, each backend that needs to use this custom type needs to register
+ it with <literal>pgstat_register_kind</literal> and a unique ID used to
+ store the entries related to this type of statistics:
+<programlisting>
+extern PgStat_Kind pgstat_add_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
+</programlisting>
+ While developing a new extension, use
+ <literal>PGSTAT_KIND_EXPERIMENTAL</literal> for
+ <parameter>kind</parameter>. When you are ready to release the extension
+ to users, reserve a kind ID at the
+ <ulink url="https://wiki.postgresql.org/wiki/CustomCumulativeStats">
+ Custom Cumulative Statistics</ulink> page.
+ </para>
+
+ <para>
+ The details of the API for <literal>PgStat_KindInfo</literal> can
+ be found in <filename>src/include/utils/pgstat_internal.h</filename>.
+ </para>
+
+ <para>
+ The type of statistics registered is associated with a name and a unique
+ ID shared across the server in shared memory. Each backend using a
+ custom type of statistics maintains a local cache storing the information
+ of each custom <literal>PgStat_KindInfo</literal>.
+ </para>
+
+ <para>
+ Place the extension module implementing the custom cumulative statistics
+ type in <xref linkend="guc-shared-preload-libraries"/> so that it will
+ be loaded early during <productname>PostgreSQL</productname> startup.
+ </para>
+
+ <para>
+ An example describing how to register and use custom statistics can be
+ found in
+ <filename>src/test/modules/injection_points/injection_stats.c</filename>.
+ </para>
+ </sect2>
+
<sect2 id="extend-cpp">
<title>Using C++ for Extensibility</title>
--
2.45.2
v4-0007-injection_points-Add-statistics-for-custom-points.patchtext/x-diff; charset=us-asciiDownload
From 9f6ec2e76ffbe5c45d94b957f3d30ee79a77368a Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 13:36:06 +0900
Subject: [PATCH v4 7/8] injection_points: Add statistics for custom points
This acts as a template of what can be achieved with the pluggable
cumulative stats APIs, while being useful on its own for injection
points.
Currently, the only data gathered is the number of times an injection
point is called. This can be extended as required. All the routines
related to the stats are located in their own file, for clarity.
A TAP test is included to provide coverage for these new APIs, showing
the persistency of the data across restarts.
---
src/test/modules/injection_points/Makefile | 11 +-
.../injection_points--1.0.sql | 10 +
.../injection_points/injection_points.c | 27 +++
.../injection_points/injection_stats.c | 194 ++++++++++++++++++
.../injection_points/injection_stats.h | 23 +++
src/test/modules/injection_points/meson.build | 9 +
.../modules/injection_points/t/001_stats.pl | 48 +++++
src/tools/pgindent/typedefs.list | 2 +
8 files changed, 322 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/injection_points/injection_stats.c
create mode 100644 src/test/modules/injection_points/injection_stats.h
create mode 100644 src/test/modules/injection_points/t/001_stats.pl
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 2ffd2f77ed..7b9cd12a2a 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -1,7 +1,10 @@
# src/test/modules/injection_points/Makefile
-MODULES = injection_points
-
+MODULE_big = injection_points
+OBJS = \
+ $(WIN32RES) \
+ injection_points.o \
+ injection_stats.o
EXTENSION = injection_points
DATA = injection_points--1.0.sql
PGFILEDESC = "injection_points - facility for injection points"
@@ -11,9 +14,13 @@ REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
ISOLATION = inplace
+TAP_TESTS = 1
+
# The injection points are cluster-wide, so disable installcheck
NO_INSTALLCHECK = 1
+export enable_injection_points enable_injection_points
+
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index e275c2cf5b..747a64e812 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -64,3 +64,13 @@ CREATE FUNCTION injection_points_detach(IN point_name TEXT)
RETURNS void
AS 'MODULE_PATHNAME', 'injection_points_detach'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+--
+-- injection_points_stats_numcalls()
+--
+-- Reports statistics, if any, related to the given injection point.
+--
+CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
+RETURNS bigint
+AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index b6c8e89324..eb411b9d44 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -18,6 +18,7 @@
#include "postgres.h"
#include "fmgr.h"
+#include "injection_stats.h"
#include "miscadmin.h"
#include "nodes/pg_list.h"
#include "nodes/value.h"
@@ -170,6 +171,9 @@ injection_points_cleanup(int code, Datum arg)
char *name = strVal(lfirst(lc));
(void) InjectionPointDetach(name);
+
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
}
}
@@ -182,6 +186,8 @@ injection_error(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(ERROR, "error triggered for injection point %s", name);
}
@@ -193,6 +199,8 @@ injection_notice(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(NOTICE, "notice triggered for injection point %s", name);
}
@@ -211,6 +219,8 @@ injection_wait(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
/*
* Use the injection point name for this custom wait event. Note that
* this custom wait event name is not released, but we don't care much for
@@ -299,6 +309,10 @@ injection_points_attach(PG_FUNCTION_ARGS)
inj_list_local = lappend(inj_list_local, makeString(pstrdup(name)));
MemoryContextSwitchTo(oldctx);
}
+
+ /* Add entry for stats */
+ pgstat_create_inj(name);
+
PG_RETURN_VOID();
}
@@ -417,5 +431,18 @@ injection_points_detach(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldctx);
}
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
+
PG_RETURN_VOID();
}
+
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ pgstat_register_inj();
+}
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
new file mode 100644
index 0000000000..c37b0b33d3
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -0,0 +1,194 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.c
+ * Code for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+
+#include "common/hashfn.h"
+#include "injection_stats.h"
+#include "pgstat.h"
+#include "utils/builtins.h"
+#include "utils/pgstat_internal.h"
+
+/* Structures for statistics of injection points */
+typedef struct PgStat_StatInjEntry
+{
+ PgStat_Counter numcalls; /* number of times point has been run */
+} PgStat_StatInjEntry;
+
+typedef struct PgStatShared_InjectionPoint
+{
+ PgStatShared_Common header;
+ PgStat_StatInjEntry stats;
+} PgStatShared_InjectionPoint;
+
+static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+
+static const PgStat_KindInfo injection_stats = {
+ .name = "injection_points",
+ .fixed_amount = false, /* Bounded by the number of points */
+
+ /* Injection points are system-wide */
+ .accessed_across_databases = true,
+
+ .shared_size = sizeof(PgStatShared_InjectionPoint),
+ .shared_data_off = offsetof(PgStatShared_InjectionPoint, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPoint *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatInjEntry),
+ .flush_pending_cb = injection_stats_flush_cb,
+};
+
+/*
+ * Compute stats entry idx from point name with a 4-byte hash.
+ */
+#define PGSTAT_INJ_IDX(name) hash_bytes((const unsigned char *) name, strlen(name))
+
+#define PGSTAT_KIND_INJECTION PGSTAT_KIND_EXPERIMENTAL
+
+/* Track if stats are loaded */
+static bool inj_stats_loaded = false;
+
+/*
+ * Callback for stats handling
+ */
+static bool
+injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ PgStat_StatInjEntry *localent;
+ PgStatShared_InjectionPoint *shfuncent;
+
+ localent = (PgStat_StatInjEntry *) entry_ref->pending;
+ shfuncent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ shfuncent->stats.numcalls += localent->numcalls;
+ return true;
+}
+
+/*
+ * Support function for the SQL-callable pgstat* functions. Returns
+ * a pointer to the injection point statistics struct.
+ */
+static PgStat_StatInjEntry *
+pgstat_fetch_stat_injentry(const char *name)
+{
+ PgStat_StatInjEntry *entry = NULL;
+
+ if (!inj_stats_loaded)
+ return NULL;
+
+ /* Compile the lookup key as a hash of the point name */
+ entry = (PgStat_StatInjEntry *) pgstat_fetch_entry(PGSTAT_KIND_INJECTION,
+ InvalidOid,
+ PGSTAT_INJ_IDX(name));
+ return entry;
+}
+
+/*
+ * Workhorse to do the registration work, called in _PG_init().
+ */
+void
+pgstat_register_inj(void)
+{
+ pgstat_register_kind(PGSTAT_KIND_INJECTION, &injection_stats);
+
+ /* mark stats as loaded */
+ inj_stats_loaded = true;
+}
+
+/*
+ * Report injection point creation.
+ */
+void
+pgstat_create_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ /* initialize shared memory data */
+ memset(&shstatent->stats, 0, sizeof(shstatent->stats));
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * Report injection point drop.
+ */
+void
+pgstat_drop_inj(const char *name)
+{
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ if (!pgstat_drop_entry(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name)))
+ pgstat_request_entry_refs_gc();
+}
+
+/*
+ * Report statistics for injection point.
+ *
+ * This is simple because the set of stats to report currently is simple:
+ * track the number of times a point has been run.
+ */
+void
+pgstat_report_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+ PgStat_StatInjEntry *statent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+ statent = &shstatent->stats;
+
+ /* Update the injection point statistics */
+ statent->numcalls++;
+
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * SQL function returning the number of times an injection point
+ * has been called.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_numcalls);
+Datum
+injection_points_stats_numcalls(PG_FUNCTION_ARGS)
+{
+ char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_StatInjEntry *entry = pgstat_fetch_stat_injentry(name);
+
+ if (entry == NULL)
+ PG_RETURN_NULL();
+
+ PG_RETURN_INT64(entry->numcalls);
+}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
new file mode 100644
index 0000000000..3e99705483
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -0,0 +1,23 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.h
+ * Definitions for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.h
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#ifndef INJECTION_STATS
+#define INJECTION_STATS
+
+extern void pgstat_register_inj(void);
+extern void pgstat_create_inj(const char *name);
+extern void pgstat_drop_inj(const char *name);
+extern void pgstat_report_inj(const char *name);
+
+#endif
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index 3c23c14d81..a52fe5121e 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -6,6 +6,7 @@ endif
injection_points_sources = files(
'injection_points.c',
+ 'injection_stats.c',
)
if host_system == 'windows'
@@ -42,4 +43,12 @@ tests += {
'inplace',
],
},
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_stats.pl',
+ ],
+ },
}
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
new file mode 100644
index 0000000000..7d5a96e522
--- /dev/null
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -0,0 +1,48 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+use locale;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Test persistency of statistics generated for injection points.
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# Node initialization
+my $node = PostgreSQL::Test::Cluster->new('master');
+$node->init;
+$node->append_conf('postgresql.conf',
+ "shared_preload_libraries = 'injection_points'");
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# This should count for two calls.
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('stats-notice', 'notice');");
+$node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
+$node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
+my $numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats calls');
+
+# Restart the node cleanly, stats should still be around.
+$node->restart;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats after clean restart');
+
+# On crash the stats are gone.
+$node->stop('immediate');
+$node->start;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '', 'number of stats after clean restart');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9320e4d808..c2211d9804 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2117,6 +2117,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_InjectionPoint
PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
@@ -2148,6 +2149,7 @@ PgStat_Snapshot
PgStat_SnapshotEntry
PgStat_StatDBEntry
PgStat_StatFuncEntry
+PgStat_StatInjEntry
PgStat_StatReplSlotEntry
PgStat_StatSubEntry
PgStat_StatTabEntry
--
2.45.2
v4-0008-injection_points-Add-example-for-fixed-numbered-s.patchtext/x-diff; charset=us-asciiDownload
From db9dc4179f56e6448b28a0898a228761013bf765 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 13:44:14 +0900
Subject: [PATCH v4 8/8] injection_points: Add example for fixed-numbered
statistics
This acts as a template to show what can be achieved with fixed-numbered
stats (like WAL, bgwriter, etc.) for pluggable cumulative statistics.
---
.../injection_points--1.0.sql | 11 ++
.../injection_points/injection_points.c | 3 +
.../injection_points/injection_stats.c | 155 +++++++++++++++++-
.../injection_points/injection_stats.h | 3 +
.../modules/injection_points/t/001_stats.pl | 11 +-
5 files changed, 180 insertions(+), 3 deletions(-)
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index 747a64e812..fa0b1d06ae 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -74,3 +74,14 @@ CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
RETURNS bigint
AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
LANGUAGE C STRICT;
+
+--
+-- injection_points_stats_fixed()
+--
+-- Reports fixed-numbered statistics for injection points.
+CREATE FUNCTION injection_points_stats_fixed(OUT numattach int8,
+ OUT numdetach int8,
+ OUT numrun int8)
+RETURNS record
+AS 'MODULE_PATHNAME', 'injection_points_stats_fixed'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index eb411b9d44..f65bd83cc3 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -297,6 +297,7 @@ injection_points_attach(PG_FUNCTION_ARGS)
condition.pid = MyProcPid;
}
+ pgstat_report_inj_fixed(1, 0, 0);
InjectionPointAttach(name, "injection_points", function, &condition,
sizeof(InjectionPointCondition));
@@ -342,6 +343,7 @@ injection_points_run(PG_FUNCTION_ARGS)
{
char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ pgstat_report_inj_fixed(0, 0, 1);
INJECTION_POINT(name);
PG_RETURN_VOID();
@@ -418,6 +420,7 @@ injection_points_detach(PG_FUNCTION_ARGS)
{
char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ pgstat_report_inj_fixed(0, 1, 0);
if (!InjectionPointDetach(name))
elog(ERROR, "could not detach injection point \"%s\"", name);
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
index c37b0b33d3..69560b7fdc 100644
--- a/src/test/modules/injection_points/injection_stats.c
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -17,12 +17,13 @@
#include "fmgr.h"
#include "common/hashfn.h"
+#include "funcapi.h"
#include "injection_stats.h"
#include "pgstat.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
-/* Structures for statistics of injection points */
+/* Structures for statistics of injection points, variable-size */
typedef struct PgStat_StatInjEntry
{
PgStat_Counter numcalls; /* number of times point has been run */
@@ -35,6 +36,9 @@ typedef struct PgStatShared_InjectionPoint
} PgStatShared_InjectionPoint;
static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+static void injection_stats_fixed_init_shmem_cb(void *stats);
+static void injection_stats_fixed_reset_all_cb(TimestampTz ts);
+static void injection_stats_fixed_snapshot_cb(void);
static const PgStat_KindInfo injection_stats = {
.name = "injection_points",
@@ -50,18 +54,51 @@ static const PgStat_KindInfo injection_stats = {
.flush_pending_cb = injection_stats_flush_cb,
};
+/* Structures for statistics of injection points, fixed-size */
+typedef struct PgStat_StatInjFixedEntry
+{
+ PgStat_Counter numattach; /* number of points attached */
+ PgStat_Counter numdetach; /* number of points detached */
+ PgStat_Counter numrun; /* number of points run */
+ TimestampTz stat_reset_timestamp;
+} PgStat_StatInjFixedEntry;
+
+typedef struct PgStatShared_InjectionPointFixed
+{
+ LWLock lock; /* protects all the counters */
+ uint32 changecount;
+ PgStat_StatInjFixedEntry stats;
+ PgStat_StatInjFixedEntry reset_offset;
+} PgStatShared_InjectionPointFixed;
+
+static const PgStat_KindInfo injection_stats_fixed = {
+ .name = "injection_points_fixed",
+ .fixed_amount = true,
+
+ .shared_size = sizeof(PgStat_StatInjFixedEntry),
+ .shared_data_off = offsetof(PgStatShared_InjectionPointFixed, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPointFixed *) 0)->stats),
+
+ .init_shmem_cb = injection_stats_fixed_init_shmem_cb,
+ .reset_all_cb = injection_stats_fixed_reset_all_cb,
+ .snapshot_cb = injection_stats_fixed_snapshot_cb,
+};
+
/*
* Compute stats entry idx from point name with a 4-byte hash.
*/
#define PGSTAT_INJ_IDX(name) hash_bytes((const unsigned char *) name, strlen(name))
#define PGSTAT_KIND_INJECTION PGSTAT_KIND_EXPERIMENTAL
+#define PGSTAT_KIND_INJECTION_FIXED (PGSTAT_KIND_EXPERIMENTAL + 1)
+/* Position of fixed-numbered data in internal structures */
+#define PGSTAT_KIND_INJECTION_IDX (PGSTAT_KIND_INJECTION_FIXED - PGSTAT_KIND_CUSTOM_MIN)
/* Track if stats are loaded */
static bool inj_stats_loaded = false;
/*
- * Callback for stats handling
+ * Callbacks for stats handling
*/
static bool
injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
@@ -79,6 +116,59 @@ injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
return true;
}
+static void
+injection_stats_fixed_init_shmem_cb(void *stats)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+static void
+injection_stats_fixed_reset_all_cb(TimestampTz ts)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *)
+ pgStatLocal.shmem->custom_data[PGSTAT_KIND_INJECTION_IDX];
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ pgstat_copy_changecounted_stats(&stats_shmem->reset_offset,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+ stats_shmem->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&stats_shmem->lock);
+}
+
+static void
+injection_stats_fixed_snapshot_cb(void)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *)
+ pgStatLocal.shmem->custom_data[PGSTAT_KIND_INJECTION_IDX];
+ PgStat_StatInjFixedEntry *stat_snap = (PgStat_StatInjFixedEntry *)
+ pgStatLocal.snapshot.custom_data[PGSTAT_KIND_INJECTION_IDX];
+ PgStat_StatInjFixedEntry *reset_offset = &stats_shmem->reset_offset;
+ PgStat_StatInjFixedEntry reset;
+
+ pgstat_copy_changecounted_stats(stat_snap,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+
+ LWLockAcquire(&stats_shmem->lock, LW_SHARED);
+ memcpy(&reset, reset_offset, sizeof(stats_shmem->stats));
+ LWLockRelease(&stats_shmem->lock);
+
+ /* compensate by reset offsets */
+#define FIXED_COMP(fld) stat_snap->fld -= reset.fld;
+ FIXED_COMP(numattach);
+ FIXED_COMP(numdetach);
+ FIXED_COMP(numrun);
+#undef FIXED_COMP
+}
+
/*
* Support function for the SQL-callable pgstat* functions. Returns
* a pointer to the injection point statistics struct.
@@ -105,6 +195,7 @@ void
pgstat_register_inj(void)
{
pgstat_register_kind(PGSTAT_KIND_INJECTION, &injection_stats);
+ pgstat_register_kind(PGSTAT_KIND_INJECTION_FIXED, &injection_stats_fixed);
/* mark stats as loaded */
inj_stats_loaded = true;
@@ -176,6 +267,30 @@ pgstat_report_inj(const char *name)
pgstat_unlock_entry(entry_ref);
}
+/*
+ * Report fixed number of statistics for an injection point.
+ */
+void
+pgstat_report_inj_fixed(uint32 numattach,
+ uint32 numdetach,
+ uint32 numrun)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ stats_shmem = (PgStatShared_InjectionPointFixed *)
+ pgStatLocal.shmem->custom_data[PGSTAT_KIND_INJECTION_IDX];
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numattach += numattach;
+ stats_shmem->stats.numdetach += numdetach;
+ stats_shmem->stats.numrun += numrun;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+}
+
/*
* SQL function returning the number of times an injection point
* has been called.
@@ -192,3 +307,39 @@ injection_points_stats_numcalls(PG_FUNCTION_ARGS)
PG_RETURN_INT64(entry->numcalls);
}
+
+/*
+ * SQL function returning fixed-numbered statistics for injection points.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_fixed);
+Datum
+injection_points_stats_fixed(PG_FUNCTION_ARGS)
+{
+ TupleDesc tupdesc;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ PgStat_StatInjFixedEntry *stats;
+
+ pgstat_snapshot_fixed(PGSTAT_KIND_INJECTION_FIXED);
+ stats = pgStatLocal.snapshot.custom_data[PGSTAT_KIND_INJECTION_IDX];
+
+ /* Initialise attributes information in the tuple descriptor */
+ tupdesc = CreateTemplateTupleDesc(3);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "numattach",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "numdetach",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "numrun",
+ INT8OID, -1, 0);
+ BlessTupleDesc(tupdesc);
+
+ values[0] = Int64GetDatum(stats->numattach);
+ values[1] = Int64GetDatum(stats->numdetach);
+ values[2] = Int64GetDatum(stats->numrun);
+ nulls[0] = false;
+ nulls[1] = false;
+ nulls[2] = false;
+
+ /* Returns the record as Datum */
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
index 3e99705483..cf68b25f7b 100644
--- a/src/test/modules/injection_points/injection_stats.h
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -19,5 +19,8 @@ extern void pgstat_register_inj(void);
extern void pgstat_create_inj(const char *name);
extern void pgstat_drop_inj(const char *name);
extern void pgstat_report_inj(const char *name);
+extern void pgstat_report_inj_fixed(uint32 numattach,
+ uint32 numdetach,
+ uint32 numrun);
#endif
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
index 7d5a96e522..e3c69b94ca 100644
--- a/src/test/modules/injection_points/t/001_stats.pl
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -31,18 +31,27 @@ $node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
my $numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
is($numcalls, '2', 'number of stats calls');
+my $fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '1|0|2', 'number of fixed stats');
# Restart the node cleanly, stats should still be around.
$node->restart;
$numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
is($numcalls, '2', 'number of stats after clean restart');
+$fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '1|0|2', 'number of fixed stats after clean restart');
# On crash the stats are gone.
$node->stop('immediate');
$node->start;
$numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
-is($numcalls, '', 'number of stats after clean restart');
+is($numcalls, '', 'number of stats after crash');
+$fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '0|0|0', 'number of fixed stats after crash');
done_testing();
--
2.45.2
v4-0001-Use-pgstat_kind_infos-to-write-fixed-shared-stati.patchtext/x-diff; charset=us-asciiDownload
From d9fe88f22478012cac305612d06304c724c0265d Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 11:21:42 +0900
Subject: [PATCH v4 1/8] Use pgstat_kind_infos to write fixed shared statistics
This is similar to 9004abf6206e, but this time for the write part. This
requires the addition to an offset in PgStat_KindInfo to track the
location of the stats to write in PgStat_Snapshot.
---
src/include/utils/pgstat_internal.h | 6 ++++
src/backend/utils/activity/pgstat.c | 56 ++++++++++-------------------
2 files changed, 24 insertions(+), 38 deletions(-)
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index e21ef4e2c9..43d6deb03c 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -199,6 +199,12 @@ typedef struct PgStat_KindInfo
*/
uint32 shared_size;
+ /*
+ * The offset of the statistics struct in the cached statistics snapshot
+ * PgStat_Snapshot, for fixed-numbered statistics.
+ */
+ uint32 snapshot_ctl_off;
+
/*
* The offset of the statistics struct in the containing shared memory
* control structure PgStat_ShmemControl, for fixed-numbered statistics.
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index c37c11b2ec..65b937a85f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -349,6 +349,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = true,
+ .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver),
.shared_ctl_off = offsetof(PgStat_ShmemControl, archiver),
.shared_data_off = offsetof(PgStatShared_Archiver, stats),
.shared_data_len = sizeof(((PgStatShared_Archiver *) 0)->stats),
@@ -362,6 +363,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = true,
+ .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter),
.shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter),
.shared_data_off = offsetof(PgStatShared_BgWriter, stats),
.shared_data_len = sizeof(((PgStatShared_BgWriter *) 0)->stats),
@@ -375,6 +377,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = true,
+ .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer),
.shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer),
.shared_data_off = offsetof(PgStatShared_Checkpointer, stats),
.shared_data_len = sizeof(((PgStatShared_Checkpointer *) 0)->stats),
@@ -388,6 +391,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = true,
+ .snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
.shared_data_off = offsetof(PgStatShared_IO, stats),
.shared_data_len = sizeof(((PgStatShared_IO *) 0)->stats),
@@ -401,6 +405,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = true,
+ .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
.shared_data_off = offsetof(PgStatShared_SLRU, stats),
.shared_data_len = sizeof(((PgStatShared_SLRU *) 0)->stats),
@@ -414,6 +419,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = true,
+ .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
.shared_data_off = offsetof(PgStatShared_Wal, stats),
.shared_data_len = sizeof(((PgStatShared_Wal *) 0)->stats),
@@ -1371,47 +1377,21 @@ pgstat_write_statsfile(void)
format_id = PGSTAT_FILE_FORMAT_ID;
write_chunk_s(fpout, &format_id);
- /*
- * XXX: The following could now be generalized to just iterate over
- * pgstat_kind_infos instead of knowing about the different kinds of
- * stats.
- */
+ /* Write various stats structs for fixed number of objects */
+ for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ {
+ char *ptr;
+ const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
- /*
- * Write archiver stats struct
- */
- pgstat_build_snapshot_fixed(PGSTAT_KIND_ARCHIVER);
- write_chunk_s(fpout, &pgStatLocal.snapshot.archiver);
+ if (!info->fixed_amount)
+ continue;
- /*
- * Write bgwriter stats struct
- */
- pgstat_build_snapshot_fixed(PGSTAT_KIND_BGWRITER);
- write_chunk_s(fpout, &pgStatLocal.snapshot.bgwriter);
+ Assert(info->snapshot_ctl_off != 0);
- /*
- * Write checkpointer stats struct
- */
- pgstat_build_snapshot_fixed(PGSTAT_KIND_CHECKPOINTER);
- write_chunk_s(fpout, &pgStatLocal.snapshot.checkpointer);
-
- /*
- * Write IO stats struct
- */
- pgstat_build_snapshot_fixed(PGSTAT_KIND_IO);
- write_chunk_s(fpout, &pgStatLocal.snapshot.io);
-
- /*
- * Write SLRU stats struct
- */
- pgstat_build_snapshot_fixed(PGSTAT_KIND_SLRU);
- write_chunk_s(fpout, &pgStatLocal.snapshot.slru);
-
- /*
- * Write WAL stats struct
- */
- pgstat_build_snapshot_fixed(PGSTAT_KIND_WAL);
- write_chunk_s(fpout, &pgStatLocal.snapshot.wal);
+ pgstat_build_snapshot_fixed(kind);
+ ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+ write_chunk(fpout, ptr, info->shared_data_len);
+ }
/*
* Walk through the stats entries
--
2.45.2
v4-0002-Add-PgStat_KindInfo.init_shmem_cb.patchtext/x-diff; charset=us-asciiDownload
From 9491b4eabe583d9fde97cb702ec88990d8b5b54d Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 11:41:25 +0900
Subject: [PATCH v4 2/8] Add PgStat_KindInfo.init_shmem_cb
This new callback gives fixed-numbered stats the possibility to take
actions based on the area of shared memory they have been allocated.
This removes from pgstat_shmem.c any knowledge specific to the types
of fixed-numbered stats, and the initializations happen in their own
files.
This is handy to make this area of the code more pluggable (as in custom
stats kinds with a fixed number of objects).
---
src/include/utils/pgstat_internal.h | 13 +++++++++++++
src/backend/utils/activity/pgstat.c | 6 ++++++
src/backend/utils/activity/pgstat_archiver.c | 8 ++++++++
src/backend/utils/activity/pgstat_bgwriter.c | 8 ++++++++
.../utils/activity/pgstat_checkpointer.c | 8 ++++++++
src/backend/utils/activity/pgstat_io.c | 9 +++++++++
src/backend/utils/activity/pgstat_shmem.c | 19 ++++++++++---------
src/backend/utils/activity/pgstat_slru.c | 8 ++++++++
src/backend/utils/activity/pgstat_wal.c | 8 ++++++++
9 files changed, 78 insertions(+), 9 deletions(-)
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 43d6deb03c..778f625ca1 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -251,6 +251,13 @@ typedef struct PgStat_KindInfo
const PgStatShared_Common *header, NameData *name);
bool (*from_serialized_name) (const NameData *name, PgStat_HashKey *key);
+ /*
+ * For fixed-numbered statistics: Initialize shared memory state.
+ *
+ * "stats" is the pointer to the allocated shared memory area.
+ */
+ void (*init_shmem_cb) (void *stats);
+
/*
* For fixed-numbered statistics: Reset All.
*/
@@ -528,6 +535,7 @@ extern void pgstat_snapshot_fixed(PgStat_Kind kind);
* Functions in pgstat_archiver.c
*/
+extern void pgstat_archiver_init_shmem_cb(void *stats);
extern void pgstat_archiver_reset_all_cb(TimestampTz ts);
extern void pgstat_archiver_snapshot_cb(void);
@@ -536,6 +544,7 @@ extern void pgstat_archiver_snapshot_cb(void);
* Functions in pgstat_bgwriter.c
*/
+extern void pgstat_bgwriter_init_shmem_cb(void *stats);
extern void pgstat_bgwriter_reset_all_cb(TimestampTz ts);
extern void pgstat_bgwriter_snapshot_cb(void);
@@ -544,6 +553,7 @@ extern void pgstat_bgwriter_snapshot_cb(void);
* Functions in pgstat_checkpointer.c
*/
+extern void pgstat_checkpointer_init_shmem_cb(void *stats);
extern void pgstat_checkpointer_reset_all_cb(TimestampTz ts);
extern void pgstat_checkpointer_snapshot_cb(void);
@@ -574,6 +584,7 @@ extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
*/
extern bool pgstat_flush_io(bool nowait);
+extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
@@ -632,6 +643,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind,
*/
extern bool pgstat_slru_flush(bool nowait);
+extern void pgstat_slru_init_shmem_cb(void *stats);
extern void pgstat_slru_reset_all_cb(TimestampTz ts);
extern void pgstat_slru_snapshot_cb(void);
@@ -644,6 +656,7 @@ extern bool pgstat_flush_wal(bool nowait);
extern void pgstat_init_wal(void);
extern bool pgstat_have_pending_wal(void);
+extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 65b937a85f..ef435167f7 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -354,6 +354,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.shared_data_off = offsetof(PgStatShared_Archiver, stats),
.shared_data_len = sizeof(((PgStatShared_Archiver *) 0)->stats),
+ .init_shmem_cb = pgstat_archiver_init_shmem_cb,
.reset_all_cb = pgstat_archiver_reset_all_cb,
.snapshot_cb = pgstat_archiver_snapshot_cb,
},
@@ -368,6 +369,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.shared_data_off = offsetof(PgStatShared_BgWriter, stats),
.shared_data_len = sizeof(((PgStatShared_BgWriter *) 0)->stats),
+ .init_shmem_cb = pgstat_bgwriter_init_shmem_cb,
.reset_all_cb = pgstat_bgwriter_reset_all_cb,
.snapshot_cb = pgstat_bgwriter_snapshot_cb,
},
@@ -382,6 +384,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.shared_data_off = offsetof(PgStatShared_Checkpointer, stats),
.shared_data_len = sizeof(((PgStatShared_Checkpointer *) 0)->stats),
+ .init_shmem_cb = pgstat_checkpointer_init_shmem_cb,
.reset_all_cb = pgstat_checkpointer_reset_all_cb,
.snapshot_cb = pgstat_checkpointer_snapshot_cb,
},
@@ -396,6 +399,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.shared_data_off = offsetof(PgStatShared_IO, stats),
.shared_data_len = sizeof(((PgStatShared_IO *) 0)->stats),
+ .init_shmem_cb = pgstat_io_init_shmem_cb,
.reset_all_cb = pgstat_io_reset_all_cb,
.snapshot_cb = pgstat_io_snapshot_cb,
},
@@ -410,6 +414,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.shared_data_off = offsetof(PgStatShared_SLRU, stats),
.shared_data_len = sizeof(((PgStatShared_SLRU *) 0)->stats),
+ .init_shmem_cb = pgstat_slru_init_shmem_cb,
.reset_all_cb = pgstat_slru_reset_all_cb,
.snapshot_cb = pgstat_slru_snapshot_cb,
},
@@ -424,6 +429,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.shared_data_off = offsetof(PgStatShared_Wal, stats),
.shared_data_len = sizeof(((PgStatShared_Wal *) 0)->stats),
+ .init_shmem_cb = pgstat_wal_init_shmem_cb,
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
},
diff --git a/src/backend/utils/activity/pgstat_archiver.c b/src/backend/utils/activity/pgstat_archiver.c
index 66398b20e5..60e04711c4 100644
--- a/src/backend/utils/activity/pgstat_archiver.c
+++ b/src/backend/utils/activity/pgstat_archiver.c
@@ -62,6 +62,14 @@ pgstat_fetch_stat_archiver(void)
return &pgStatLocal.snapshot.archiver;
}
+void
+pgstat_archiver_init_shmem_cb(void *stats)
+{
+ PgStatShared_Archiver *stats_shmem = (PgStatShared_Archiver *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_archiver_reset_all_cb(TimestampTz ts)
{
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index 7d2432e4fa..364a7a2024 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -75,6 +75,14 @@ pgstat_fetch_stat_bgwriter(void)
return &pgStatLocal.snapshot.bgwriter;
}
+void
+pgstat_bgwriter_init_shmem_cb(void *stats)
+{
+ PgStatShared_BgWriter *stats_shmem = (PgStatShared_BgWriter *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_bgwriter_reset_all_cb(TimestampTz ts)
{
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 30a8110e38..bbfc9c7e18 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -84,6 +84,14 @@ pgstat_fetch_stat_checkpointer(void)
return &pgStatLocal.snapshot.checkpointer;
}
+void
+pgstat_checkpointer_init_shmem_cb(void *stats)
+{
+ PgStatShared_Checkpointer *stats_shmem = (PgStatShared_Checkpointer *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_checkpointer_reset_all_cb(TimestampTz ts)
{
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 9d6e067382..8af55989ee 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -251,6 +251,15 @@ pgstat_get_io_object_name(IOObject io_object)
pg_unreachable();
}
+void
+pgstat_io_init_shmem_cb(void *stats)
+{
+ PgStatShared_IO *stat_shmem = (PgStatShared_IO *) stats;
+
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ LWLockInitialize(&stat_shmem->locks[i], LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_io_reset_all_cb(TimestampTz ts)
{
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 634b967820..256f177f5a 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -196,17 +196,18 @@ StatsShmemInit(void)
pg_atomic_init_u64(&ctl->gc_request_count, 1);
-
/* initialize fixed-numbered stats */
- LWLockInitialize(&ctl->archiver.lock, LWTRANCHE_PGSTATS_DATA);
- LWLockInitialize(&ctl->bgwriter.lock, LWTRANCHE_PGSTATS_DATA);
- LWLockInitialize(&ctl->checkpointer.lock, LWTRANCHE_PGSTATS_DATA);
- LWLockInitialize(&ctl->slru.lock, LWTRANCHE_PGSTATS_DATA);
- LWLockInitialize(&ctl->wal.lock, LWTRANCHE_PGSTATS_DATA);
+ for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ char *ptr;
- for (int i = 0; i < BACKEND_NUM_TYPES; i++)
- LWLockInitialize(&ctl->io.locks[i],
- LWTRANCHE_PGSTATS_DATA);
+ if (!kind_info->fixed_amount)
+ continue;
+
+ ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ kind_info->init_shmem_cb((void *) ptr);
+ }
}
else
{
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 56ea1c3378..6f922a85bf 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -192,6 +192,14 @@ pgstat_slru_flush(bool nowait)
return false;
}
+void
+pgstat_slru_init_shmem_cb(void *stats)
+{
+ PgStatShared_SLRU *stats_shmem = (PgStatShared_SLRU *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_slru_reset_all_cb(TimestampTz ts)
{
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 0e374f133a..e2a3f6b865 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -163,6 +163,14 @@ pgstat_have_pending_wal(void)
PendingWalStats.wal_sync != 0;
}
+void
+pgstat_wal_init_shmem_cb(void *stats)
+{
+ PgStatShared_Wal *stats_shmem = (PgStatShared_Wal *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_wal_reset_all_cb(TimestampTz ts)
{
--
2.45.2
v4-0003-Add-a-new-F-type-of-entry-in-pgstats-file.patchtext/x-diff; charset=us-asciiDownload
From aec71d0c382689a6527de8769fa7670f615e18e9 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 12:24:44 +0900
Subject: [PATCH v4 3/8] Add a new 'F' type of entry in pgstats file
This new entry type is used for fixed-numbered statistics, making an
upcoming change to add support for custom pluggable stats kinds more
palatable.
---
src/backend/utils/activity/pgstat.c | 44 ++++++++++++++++++-----------
1 file changed, 28 insertions(+), 16 deletions(-)
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index ef435167f7..ed32ee4b6f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -132,6 +132,7 @@
* ---------
*/
#define PGSTAT_FILE_ENTRY_END 'E' /* end of file */
+#define PGSTAT_FILE_ENTRY_FIXED 'F' /* fixed-numbered stats entry */
#define PGSTAT_FILE_ENTRY_NAME 'N' /* stats entry identified by name */
#define PGSTAT_FILE_ENTRY_HASH 'S' /* stats entry identified by
* PgStat_HashKey */
@@ -1396,6 +1397,9 @@ pgstat_write_statsfile(void)
pgstat_build_snapshot_fixed(kind);
ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+
+ fputc(PGSTAT_FILE_ENTRY_FIXED, fpout);
+ write_chunk_s(fpout, &kind);
write_chunk(fpout, ptr, info->shared_data_len);
}
@@ -1537,22 +1541,6 @@ pgstat_read_statsfile(void)
format_id != PGSTAT_FILE_FORMAT_ID)
goto error;
- /* Read various stats structs with fixed number of objects */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
- {
- char *ptr;
- const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
-
- if (!info->fixed_amount)
- continue;
-
- Assert(info->shared_ctl_off != 0);
-
- ptr = ((char *) shmem) + info->shared_ctl_off + info->shared_data_off;
- if (!read_chunk(fpin, ptr, info->shared_data_len))
- goto error;
- }
-
/*
* We found an existing statistics file. Read it and put all the hash
* table entries into place.
@@ -1563,6 +1551,30 @@ pgstat_read_statsfile(void)
switch (t)
{
+ case PGSTAT_FILE_ENTRY_FIXED:
+ {
+ PgStat_Kind kind;
+ const PgStat_KindInfo *info;
+ char *ptr;
+
+ if (!read_chunk_s(fpin, &kind))
+ goto error;
+
+ if (!pgstat_is_kind_valid(kind))
+ goto error;
+ info = pgstat_get_kind_info(kind);
+ Assert(info->fixed_amount);
+
+ /* Load back stats into shared memory */
+ ptr = ((char *) shmem) + info->shared_ctl_off +
+ info->shared_data_off;
+
+ if (!read_chunk(fpin, ptr,
+ info->shared_data_len))
+ goto error;
+
+ break;
+ }
case PGSTAT_FILE_ENTRY_HASH:
case PGSTAT_FILE_ENTRY_NAME:
{
--
2.45.2
v4-0004-Switch-PgStat_Kind-from-enum-to-uint32.patchtext/x-diff; charset=us-asciiDownload
From cc18cfc3180a70695820f7824534a2dd27821dd1 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 12:28:26 +0900
Subject: [PATCH v4 4/8] Switch PgStat_Kind from enum to uint32
A follow-up patch is planned to make this counter extensible, and
keeping a trace of the kind behind a type is useful in the internal
routines used by pgstats. While on it, switch pgstat_is_kind_valid() to
use PgStat_Kind, to be more consistent with its callers.
---
src/include/pgstat.h | 35 ++++++++++++++---------------
src/backend/utils/activity/pgstat.c | 6 ++---
2 files changed, 20 insertions(+), 21 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 2136239710..2d30fadaf1 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -32,26 +32,25 @@
#define PG_STAT_TMP_DIR "pg_stat_tmp"
/* The types of statistics entries */
-typedef enum PgStat_Kind
-{
- /* use 0 for INVALID, to catch zero-initialized data */
- PGSTAT_KIND_INVALID = 0,
+#define PgStat_Kind uint32
- /* stats for variable-numbered objects */
- PGSTAT_KIND_DATABASE, /* database-wide statistics */
- PGSTAT_KIND_RELATION, /* per-table statistics */
- PGSTAT_KIND_FUNCTION, /* per-function statistics */
- PGSTAT_KIND_REPLSLOT, /* per-slot statistics */
- PGSTAT_KIND_SUBSCRIPTION, /* per-subscription statistics */
+/* use 0 for INVALID, to catch zero-initialized data */
+#define PGSTAT_KIND_INVALID 0
- /* stats for fixed-numbered objects */
- PGSTAT_KIND_ARCHIVER,
- PGSTAT_KIND_BGWRITER,
- PGSTAT_KIND_CHECKPOINTER,
- PGSTAT_KIND_IO,
- PGSTAT_KIND_SLRU,
- PGSTAT_KIND_WAL,
-} PgStat_Kind;
+/* stats for variable-numbered objects */
+#define PGSTAT_KIND_DATABASE 1 /* database-wide statistics */
+#define PGSTAT_KIND_RELATION 2 /* per-table statistics */
+#define PGSTAT_KIND_FUNCTION 3 /* per-function statistics */
+#define PGSTAT_KIND_REPLSLOT 4 /* per-slot statistics */
+#define PGSTAT_KIND_SUBSCRIPTION 5 /* per-subscription statistics */
+
+/* stats for fixed-numbered objects */
+#define PGSTAT_KIND_ARCHIVER 6
+#define PGSTAT_KIND_BGWRITER 7
+#define PGSTAT_KIND_CHECKPOINTER 8
+#define PGSTAT_KIND_IO 9
+#define PGSTAT_KIND_SLRU 10
+#define PGSTAT_KIND_WAL 11
#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index ed32ee4b6f..f572a87a61 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -182,7 +182,7 @@ static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
static void pgstat_build_snapshot_fixed(PgStat_Kind kind);
-static inline bool pgstat_is_kind_valid(int ikind);
+static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
/* ----------
@@ -1298,9 +1298,9 @@ pgstat_get_kind_from_str(char *kind_str)
}
static inline bool
-pgstat_is_kind_valid(int ikind)
+pgstat_is_kind_valid(PgStat_Kind kind)
{
- return ikind >= PGSTAT_KIND_FIRST_VALID && ikind <= PGSTAT_KIND_LAST;
+ return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
}
const PgStat_KindInfo *
--
2.45.2
Hi,
On Mon, Jul 08, 2024 at 02:30:23PM +0900, Michael Paquier wrote:
On Fri, Jul 05, 2024 at 09:35:19AM +0900, Michael Paquier wrote:
On Thu, Jul 04, 2024 at 11:30:17AM +0000, Bertrand Drouvot wrote:
On Wed, Jul 03, 2024 at 06:47:15PM +0900, Michael Paquier wrote:
- PgStat_Snapshot holds an array of void* also indexed by
PGSTAT_NUM_KINDS, pointing to the fixed stats stored in the
snapshots.Same, that's just a 96 bytes overhead (8 * PGSTAT_NUM_KINDS) as compared to now.
Still Andres does not seem to like that much, well ;)
Please find attached a rebased patch set labelled v4.
Thanks!
Built-in
fixed-numbered stats are still attached to the snapshot and shmem
control structures, and custom fixed stats kinds are tracked in the
same way as v3 with new members tracking data stored in
TopMemoryContext for the snapshots and shmem for the control data.
So, the custom and built-in stats kinds are separated into separate
parts of the structures, including the "valid" flags for the
snapshots. And this avoids any redirection when looking at the
built-in fixed-numbered stats.
Yeap.
I've tried at address all the previous comments (there could be stuff
I've missed while rebasing, of course).
Thanks!
The first three patches are refactoring pieces to make the rest more
edible, while 0004~ implement the main logic with templates in
modules/injection_points:
- 0001 refactors pgstat_write_statsfile() so as a loop om
PgStat_KindInfo is used to write the data. This is done with the
addition of snapshot_ctl_off in PgStat_KindInfo, to point to the area
in PgStat_Snapshot where the data is located for fixed stats.
9004abf6206e has done the same for the read part.
Looking at 0001:
1 ==
+ for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ {
+ char *ptr;
+ const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
I wonder if we could avoid going through stats that are not fixed ones. What about
doing something like?
"
for (int kind = <first fixed>; kind <= <last fixed>; kind++);
"
Would probably need to change the indexing logic though.
and then we could replace:
+ if (!info->fixed_amount)
+ continue;
with an assert instead.
Same would apply for the read part added in 9004abf6206e.
2 ===
+ pgstat_build_snapshot_fixed(kind);
+ ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+ write_chunk(fpout, ptr, info->shared_data_len);
I think that using "shared_data_len" is confusing here (was not the case in the
context of 9004abf6206e). I mean it is perfectly correct but the wording "shared"
looks weird to me when being used here. What it really is, is the size of the
stats. What about renaming shared_data_len with stats_data_len?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Mon, Jul 08, 2024 at 06:39:56AM +0000, Bertrand Drouvot wrote:
+ for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++) + { + char *ptr; + const PgStat_KindInfo *info = pgstat_get_kind_info(kind);I wonder if we could avoid going through stats that are not fixed ones. What about
doing something like?
Same would apply for the read part added in 9004abf6206e.
This becomes more relevant when the custom stats are added, as this
performs a scan across the full range of IDs supported. So this
choice is here for consistency, and to ease the pluggability.
2 ===
+ pgstat_build_snapshot_fixed(kind); + ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off; + write_chunk(fpout, ptr, info->shared_data_len);I think that using "shared_data_len" is confusing here (was not the case in the
context of 9004abf6206e). I mean it is perfectly correct but the wording "shared"
looks weird to me when being used here. What it really is, is the size of the
stats. What about renaming shared_data_len with stats_data_len?
It is the stats data associated to a shared entry. I think that's OK,
but perhaps I'm just used to it as I've been staring at this area for
days.
--
Michael
Hi,
On Mon, Jul 08, 2024 at 03:49:34PM +0900, Michael Paquier wrote:
On Mon, Jul 08, 2024 at 06:39:56AM +0000, Bertrand Drouvot wrote:
+ for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++) + { + char *ptr; + const PgStat_KindInfo *info = pgstat_get_kind_info(kind);I wonder if we could avoid going through stats that are not fixed ones. What about
doing something like?
Same would apply for the read part added in 9004abf6206e.This becomes more relevant when the custom stats are added, as this
performs a scan across the full range of IDs supported. So this
choice is here for consistency, and to ease the pluggability.
Gotcha.
2 ===
+ pgstat_build_snapshot_fixed(kind); + ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off; + write_chunk(fpout, ptr, info->shared_data_len);I think that using "shared_data_len" is confusing here (was not the case in the
context of 9004abf6206e). I mean it is perfectly correct but the wording "shared"
looks weird to me when being used here. What it really is, is the size of the
stats. What about renaming shared_data_len with stats_data_len?It is the stats data associated to a shared entry. I think that's OK,
but perhaps I'm just used to it as I've been staring at this area for
days.
Yeah, what I meant to say is that one could think for example that's the
PgStatShared_Archiver size while in fact it's the PgStat_ArchiverStats size.
I think it's more confusing when writing the stats. Here we are manipulating
"snapshot" and "snapshot" offsets. It was not that confusing when reading as we
are manipulating "shmem" and "shared" offsets.
As I said, the code is fully correct, that's just the wording here that sounds
weird to me in the "snapshot" context.
Except the above (which is just a Nit), 0001 LGTM.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Mon, Jul 08, 2024 at 07:22:32AM +0000, Bertrand Drouvot wrote:
Except the above (which is just a Nit), 0001 LGTM.
Looking at 0002:
It looks pretty straightforward, just one comment:
+ ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ kind_info->init_shmem_cb((void *) ptr);
I don't think we need to cast ptr to void when calling init_shmem_cb(). Looking
at some examples in the code, it does not look like we cast the argument to void
when a function has (void *) as parameter (also there is examples in 0003 where
it's not done, see next comments for 0003).
So I think removing the cast here would be more consistent.
Looking at 0003:
It looks pretty straightforward. Also for example, here:
+ fputc(PGSTAT_FILE_ENTRY_FIXED, fpout);
+ write_chunk_s(fpout, &kind);
write_chunk(fpout, ptr, info->shared_data_len);
ptr is not casted to void when calling write_chunk() while its second parameter
is a "void *".
+ ptr = ((char *) shmem) + info->shared_ctl_off +
+ info->shared_data_off;
+
+ if (!read_chunk(fpin, ptr,
Same here for read_chunk().
I think that's perfectly fine and that we should do the same in 0002 when
calling init_shmem_cb() for consistency.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Mon, Jul 08, 2024 at 07:22:32AM +0000, Bertrand Drouvot wrote:
Yeah, what I meant to say is that one could think for example that's the
PgStatShared_Archiver size while in fact it's the PgStat_ArchiverStats size.
I think it's more confusing when writing the stats. Here we are manipulating
"snapshot" and "snapshot" offsets. It was not that confusing when reading as we
are manipulating "shmem" and "shared" offsets.As I said, the code is fully correct, that's just the wording here that sounds
weird to me in the "snapshot" context.
After sleeping on it, I can see your point. If we were to do the
(shared_data_len -> stats_data_len) switch, could it make sense to
rename shared_data_off to stats_data_off to have a better symmetry?
This one is the offset of the stats data in a shmem entry, so perhaps
shared_data_off is OK, but it feels a bit inconsistent as well.
Except the above (which is just a Nit), 0001 LGTM.
Thanks, I've applied 0001 for now to improve the serialization of this code.
--
Michael
Hi,
On Tue, Jul 09, 2024 at 10:45:05AM +0900, Michael Paquier wrote:
On Mon, Jul 08, 2024 at 07:22:32AM +0000, Bertrand Drouvot wrote:
Yeah, what I meant to say is that one could think for example that's the
PgStatShared_Archiver size while in fact it's the PgStat_ArchiverStats size.
I think it's more confusing when writing the stats. Here we are manipulating
"snapshot" and "snapshot" offsets. It was not that confusing when reading as we
are manipulating "shmem" and "shared" offsets.As I said, the code is fully correct, that's just the wording here that sounds
weird to me in the "snapshot" context.After sleeping on it, I can see your point. If we were to do the
(shared_data_len -> stats_data_len) switch, could it make sense to
rename shared_data_off to stats_data_off to have a better symmetry?
This one is the offset of the stats data in a shmem entry, so perhaps
shared_data_off is OK, but it feels a bit inconsistent as well.
Agree that if we were to rename one of them then the second one should be
renamed to.
I gave a second thought on it, and I think that this is the "data" part that lead
to the confusion (as too generic), what about?
shared_data_len -> shared_stats_len
shared_data_off -> shared_stats_off
That looks ok to me even in the snapshot context (shared is fine after all
because that's where the stats come from).
Attached a patch proposal doing so.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v1-0001-Renaming-two-fields-in-PgStat_KindInfo.patchtext/x-diff; charset=us-asciiDownload
From 390fdbc5b3fb1f25df809d0541507b886f1b15ec Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Tue, 9 Jul 2024 04:52:04 +0000
Subject: [PATCH v1] Renaming two fields in PgStat_KindInfo
Renaming shared_data_off to shared_stats_off and shared_data_len to
shared_stats_len. "data" seems too generic and started to be confusing since
b68b29bc8f makes use of the "_len" one in the snapshot context. Using "stats"
instead as that's what those fields are linked to.
---
src/backend/utils/activity/pgstat.c | 56 ++++++++++++++---------------
src/include/utils/pgstat_internal.h | 12 +++----
2 files changed, 34 insertions(+), 34 deletions(-)
84.2% src/backend/utils/activity/
15.7% src/include/utils/
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 65b937a85f..bb777423fb 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -274,8 +274,8 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.accessed_across_databases = true,
.shared_size = sizeof(PgStatShared_Database),
- .shared_data_off = offsetof(PgStatShared_Database, stats),
- .shared_data_len = sizeof(((PgStatShared_Database *) 0)->stats),
+ .shared_stats_off = offsetof(PgStatShared_Database, stats),
+ .shared_stats_len = sizeof(((PgStatShared_Database *) 0)->stats),
.pending_size = sizeof(PgStat_StatDBEntry),
.flush_pending_cb = pgstat_database_flush_cb,
@@ -288,8 +288,8 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = false,
.shared_size = sizeof(PgStatShared_Relation),
- .shared_data_off = offsetof(PgStatShared_Relation, stats),
- .shared_data_len = sizeof(((PgStatShared_Relation *) 0)->stats),
+ .shared_stats_off = offsetof(PgStatShared_Relation, stats),
+ .shared_stats_len = sizeof(((PgStatShared_Relation *) 0)->stats),
.pending_size = sizeof(PgStat_TableStatus),
.flush_pending_cb = pgstat_relation_flush_cb,
@@ -302,8 +302,8 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.fixed_amount = false,
.shared_size = sizeof(PgStatShared_Function),
- .shared_data_off = offsetof(PgStatShared_Function, stats),
- .shared_data_len = sizeof(((PgStatShared_Function *) 0)->stats),
+ .shared_stats_off = offsetof(PgStatShared_Function, stats),
+ .shared_stats_len = sizeof(((PgStatShared_Function *) 0)->stats),
.pending_size = sizeof(PgStat_FunctionCounts),
.flush_pending_cb = pgstat_function_flush_cb,
@@ -317,8 +317,8 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.accessed_across_databases = true,
.shared_size = sizeof(PgStatShared_ReplSlot),
- .shared_data_off = offsetof(PgStatShared_ReplSlot, stats),
- .shared_data_len = sizeof(((PgStatShared_ReplSlot *) 0)->stats),
+ .shared_stats_off = offsetof(PgStatShared_ReplSlot, stats),
+ .shared_stats_len = sizeof(((PgStatShared_ReplSlot *) 0)->stats),
.reset_timestamp_cb = pgstat_replslot_reset_timestamp_cb,
.to_serialized_name = pgstat_replslot_to_serialized_name_cb,
@@ -333,8 +333,8 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.accessed_across_databases = true,
.shared_size = sizeof(PgStatShared_Subscription),
- .shared_data_off = offsetof(PgStatShared_Subscription, stats),
- .shared_data_len = sizeof(((PgStatShared_Subscription *) 0)->stats),
+ .shared_stats_off = offsetof(PgStatShared_Subscription, stats),
+ .shared_stats_len = sizeof(((PgStatShared_Subscription *) 0)->stats),
.pending_size = sizeof(PgStat_BackendSubEntry),
.flush_pending_cb = pgstat_subscription_flush_cb,
@@ -351,8 +351,8 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver),
.shared_ctl_off = offsetof(PgStat_ShmemControl, archiver),
- .shared_data_off = offsetof(PgStatShared_Archiver, stats),
- .shared_data_len = sizeof(((PgStatShared_Archiver *) 0)->stats),
+ .shared_stats_off = offsetof(PgStatShared_Archiver, stats),
+ .shared_stats_len = sizeof(((PgStatShared_Archiver *) 0)->stats),
.reset_all_cb = pgstat_archiver_reset_all_cb,
.snapshot_cb = pgstat_archiver_snapshot_cb,
@@ -365,8 +365,8 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter),
.shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter),
- .shared_data_off = offsetof(PgStatShared_BgWriter, stats),
- .shared_data_len = sizeof(((PgStatShared_BgWriter *) 0)->stats),
+ .shared_stats_off = offsetof(PgStatShared_BgWriter, stats),
+ .shared_stats_len = sizeof(((PgStatShared_BgWriter *) 0)->stats),
.reset_all_cb = pgstat_bgwriter_reset_all_cb,
.snapshot_cb = pgstat_bgwriter_snapshot_cb,
@@ -379,8 +379,8 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer),
.shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer),
- .shared_data_off = offsetof(PgStatShared_Checkpointer, stats),
- .shared_data_len = sizeof(((PgStatShared_Checkpointer *) 0)->stats),
+ .shared_stats_off = offsetof(PgStatShared_Checkpointer, stats),
+ .shared_stats_len = sizeof(((PgStatShared_Checkpointer *) 0)->stats),
.reset_all_cb = pgstat_checkpointer_reset_all_cb,
.snapshot_cb = pgstat_checkpointer_snapshot_cb,
@@ -393,8 +393,8 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.snapshot_ctl_off = offsetof(PgStat_Snapshot, io),
.shared_ctl_off = offsetof(PgStat_ShmemControl, io),
- .shared_data_off = offsetof(PgStatShared_IO, stats),
- .shared_data_len = sizeof(((PgStatShared_IO *) 0)->stats),
+ .shared_stats_off = offsetof(PgStatShared_IO, stats),
+ .shared_stats_len = sizeof(((PgStatShared_IO *) 0)->stats),
.reset_all_cb = pgstat_io_reset_all_cb,
.snapshot_cb = pgstat_io_snapshot_cb,
@@ -407,8 +407,8 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.snapshot_ctl_off = offsetof(PgStat_Snapshot, slru),
.shared_ctl_off = offsetof(PgStat_ShmemControl, slru),
- .shared_data_off = offsetof(PgStatShared_SLRU, stats),
- .shared_data_len = sizeof(((PgStatShared_SLRU *) 0)->stats),
+ .shared_stats_off = offsetof(PgStatShared_SLRU, stats),
+ .shared_stats_len = sizeof(((PgStatShared_SLRU *) 0)->stats),
.reset_all_cb = pgstat_slru_reset_all_cb,
.snapshot_cb = pgstat_slru_snapshot_cb,
@@ -421,8 +421,8 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.snapshot_ctl_off = offsetof(PgStat_Snapshot, wal),
.shared_ctl_off = offsetof(PgStat_ShmemControl, wal),
- .shared_data_off = offsetof(PgStatShared_Wal, stats),
- .shared_data_len = sizeof(((PgStatShared_Wal *) 0)->stats),
+ .shared_stats_off = offsetof(PgStatShared_Wal, stats),
+ .shared_stats_len = sizeof(((PgStatShared_Wal *) 0)->stats),
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
@@ -910,15 +910,15 @@ pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, Oid objoid)
* repeated accesses.
*/
if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE)
- stats_data = palloc(kind_info->shared_data_len);
+ stats_data = palloc(kind_info->shared_stats_len);
else
stats_data = MemoryContextAlloc(pgStatLocal.snapshot.context,
- kind_info->shared_data_len);
+ kind_info->shared_stats_len);
pgstat_lock_entry_shared(entry_ref, false);
memcpy(stats_data,
pgstat_get_entry_data(kind, entry_ref->shared_stats),
- kind_info->shared_data_len);
+ kind_info->shared_stats_len);
pgstat_unlock_entry(entry_ref);
if (pgstat_fetch_consistency > PGSTAT_FETCH_CONSISTENCY_NONE)
@@ -1390,7 +1390,7 @@ pgstat_write_statsfile(void)
pgstat_build_snapshot_fixed(kind);
ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
- write_chunk(fpout, ptr, info->shared_data_len);
+ write_chunk(fpout, ptr, info->shared_stats_len);
}
/*
@@ -1542,8 +1542,8 @@ pgstat_read_statsfile(void)
Assert(info->shared_ctl_off != 0);
- ptr = ((char *) shmem) + info->shared_ctl_off + info->shared_data_off;
- if (!read_chunk(fpin, ptr, info->shared_data_len))
+ ptr = ((char *) shmem) + info->shared_ctl_off + info->shared_stats_off;
+ if (!read_chunk(fpin, ptr, info->shared_stats_len))
goto error;
}
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 43d6deb03c..9246998ac6 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -217,8 +217,8 @@ typedef struct PgStat_KindInfo
* shared_size because [de-]serialization may not include in-memory state
* like lwlocks.
*/
- uint32 shared_data_off;
- uint32 shared_data_len;
+ uint32 shared_stats_off;
+ uint32 shared_stats_len;
/*
* The size of the pending data for this kind. E.g. how large
@@ -789,22 +789,22 @@ pgstat_hash_hash_key(const void *d, size_t size, void *arg)
}
/*
- * The length of the data portion of a shared memory stats entry (i.e. without
+ * The length of the stats portion of a shared memory stats entry (i.e. without
* transient data such as refcounts, lwlocks, ...).
*/
static inline size_t
pgstat_get_entry_len(PgStat_Kind kind)
{
- return pgstat_get_kind_info(kind)->shared_data_len;
+ return pgstat_get_kind_info(kind)->shared_stats_len;
}
/*
- * Returns a pointer to the data portion of a shared memory stats entry.
+ * Returns a pointer to the stats portion of a shared memory stats entry.
*/
static inline void *
pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common *entry)
{
- size_t off = pgstat_get_kind_info(kind)->shared_data_off;
+ size_t off = pgstat_get_kind_info(kind)->shared_stats_off;
Assert(off != 0 && off < PG_UINT32_MAX);
--
2.34.1
On Mon, Jul 08, 2024 at 02:07:58PM +0000, Bertrand Drouvot wrote:
It looks pretty straightforward, just one comment:
+ ptr = ((char *) ctl) + kind_info->shared_ctl_off; + kind_info->init_shmem_cb((void *) ptr);I don't think we need to cast ptr to void when calling init_shmem_cb(). Looking
at some examples in the code, it does not look like we cast the argument to void
when a function has (void *) as parameter (also there is examples in 0003 where
it's not done, see next comments for 0003).
Yep. Fine by me.
Please find attached a rebased patch set for now, to make the
CF bot happy.
--
Michael
Attachments:
v5-0001-Add-PgStat_KindInfo.init_shmem_cb.patchtext/x-diff; charset=us-asciiDownload
From 7b878d78a8ea42a509ee929802c5598d23981634 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 9 Jul 2024 15:24:40 +0900
Subject: [PATCH v5 1/7] Add PgStat_KindInfo.init_shmem_cb
This new callback gives fixed-numbered stats the possibility to take
actions based on the area of shared memory they have been allocated.
This removes from pgstat_shmem.c any knowledge specific to the types
of fixed-numbered stats, and the initializations happen in their own
files.
This is handy to make this area of the code more pluggable (as in custom
stats kinds with a fixed number of objects).
---
src/include/utils/pgstat_internal.h | 13 +++++++++++++
src/backend/utils/activity/pgstat.c | 6 ++++++
src/backend/utils/activity/pgstat_archiver.c | 8 ++++++++
src/backend/utils/activity/pgstat_bgwriter.c | 8 ++++++++
.../utils/activity/pgstat_checkpointer.c | 8 ++++++++
src/backend/utils/activity/pgstat_io.c | 9 +++++++++
src/backend/utils/activity/pgstat_shmem.c | 19 ++++++++++---------
src/backend/utils/activity/pgstat_slru.c | 8 ++++++++
src/backend/utils/activity/pgstat_wal.c | 8 ++++++++
9 files changed, 78 insertions(+), 9 deletions(-)
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 43d6deb03c..778f625ca1 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -251,6 +251,13 @@ typedef struct PgStat_KindInfo
const PgStatShared_Common *header, NameData *name);
bool (*from_serialized_name) (const NameData *name, PgStat_HashKey *key);
+ /*
+ * For fixed-numbered statistics: Initialize shared memory state.
+ *
+ * "stats" is the pointer to the allocated shared memory area.
+ */
+ void (*init_shmem_cb) (void *stats);
+
/*
* For fixed-numbered statistics: Reset All.
*/
@@ -528,6 +535,7 @@ extern void pgstat_snapshot_fixed(PgStat_Kind kind);
* Functions in pgstat_archiver.c
*/
+extern void pgstat_archiver_init_shmem_cb(void *stats);
extern void pgstat_archiver_reset_all_cb(TimestampTz ts);
extern void pgstat_archiver_snapshot_cb(void);
@@ -536,6 +544,7 @@ extern void pgstat_archiver_snapshot_cb(void);
* Functions in pgstat_bgwriter.c
*/
+extern void pgstat_bgwriter_init_shmem_cb(void *stats);
extern void pgstat_bgwriter_reset_all_cb(TimestampTz ts);
extern void pgstat_bgwriter_snapshot_cb(void);
@@ -544,6 +553,7 @@ extern void pgstat_bgwriter_snapshot_cb(void);
* Functions in pgstat_checkpointer.c
*/
+extern void pgstat_checkpointer_init_shmem_cb(void *stats);
extern void pgstat_checkpointer_reset_all_cb(TimestampTz ts);
extern void pgstat_checkpointer_snapshot_cb(void);
@@ -574,6 +584,7 @@ extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
*/
extern bool pgstat_flush_io(bool nowait);
+extern void pgstat_io_init_shmem_cb(void *stats);
extern void pgstat_io_reset_all_cb(TimestampTz ts);
extern void pgstat_io_snapshot_cb(void);
@@ -632,6 +643,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind,
*/
extern bool pgstat_slru_flush(bool nowait);
+extern void pgstat_slru_init_shmem_cb(void *stats);
extern void pgstat_slru_reset_all_cb(TimestampTz ts);
extern void pgstat_slru_snapshot_cb(void);
@@ -644,6 +656,7 @@ extern bool pgstat_flush_wal(bool nowait);
extern void pgstat_init_wal(void);
extern bool pgstat_have_pending_wal(void);
+extern void pgstat_wal_init_shmem_cb(void *stats);
extern void pgstat_wal_reset_all_cb(TimestampTz ts);
extern void pgstat_wal_snapshot_cb(void);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 65b937a85f..ef435167f7 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -354,6 +354,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.shared_data_off = offsetof(PgStatShared_Archiver, stats),
.shared_data_len = sizeof(((PgStatShared_Archiver *) 0)->stats),
+ .init_shmem_cb = pgstat_archiver_init_shmem_cb,
.reset_all_cb = pgstat_archiver_reset_all_cb,
.snapshot_cb = pgstat_archiver_snapshot_cb,
},
@@ -368,6 +369,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.shared_data_off = offsetof(PgStatShared_BgWriter, stats),
.shared_data_len = sizeof(((PgStatShared_BgWriter *) 0)->stats),
+ .init_shmem_cb = pgstat_bgwriter_init_shmem_cb,
.reset_all_cb = pgstat_bgwriter_reset_all_cb,
.snapshot_cb = pgstat_bgwriter_snapshot_cb,
},
@@ -382,6 +384,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.shared_data_off = offsetof(PgStatShared_Checkpointer, stats),
.shared_data_len = sizeof(((PgStatShared_Checkpointer *) 0)->stats),
+ .init_shmem_cb = pgstat_checkpointer_init_shmem_cb,
.reset_all_cb = pgstat_checkpointer_reset_all_cb,
.snapshot_cb = pgstat_checkpointer_snapshot_cb,
},
@@ -396,6 +399,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.shared_data_off = offsetof(PgStatShared_IO, stats),
.shared_data_len = sizeof(((PgStatShared_IO *) 0)->stats),
+ .init_shmem_cb = pgstat_io_init_shmem_cb,
.reset_all_cb = pgstat_io_reset_all_cb,
.snapshot_cb = pgstat_io_snapshot_cb,
},
@@ -410,6 +414,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.shared_data_off = offsetof(PgStatShared_SLRU, stats),
.shared_data_len = sizeof(((PgStatShared_SLRU *) 0)->stats),
+ .init_shmem_cb = pgstat_slru_init_shmem_cb,
.reset_all_cb = pgstat_slru_reset_all_cb,
.snapshot_cb = pgstat_slru_snapshot_cb,
},
@@ -424,6 +429,7 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
.shared_data_off = offsetof(PgStatShared_Wal, stats),
.shared_data_len = sizeof(((PgStatShared_Wal *) 0)->stats),
+ .init_shmem_cb = pgstat_wal_init_shmem_cb,
.reset_all_cb = pgstat_wal_reset_all_cb,
.snapshot_cb = pgstat_wal_snapshot_cb,
},
diff --git a/src/backend/utils/activity/pgstat_archiver.c b/src/backend/utils/activity/pgstat_archiver.c
index 66398b20e5..60e04711c4 100644
--- a/src/backend/utils/activity/pgstat_archiver.c
+++ b/src/backend/utils/activity/pgstat_archiver.c
@@ -62,6 +62,14 @@ pgstat_fetch_stat_archiver(void)
return &pgStatLocal.snapshot.archiver;
}
+void
+pgstat_archiver_init_shmem_cb(void *stats)
+{
+ PgStatShared_Archiver *stats_shmem = (PgStatShared_Archiver *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_archiver_reset_all_cb(TimestampTz ts)
{
diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c
index 7d2432e4fa..364a7a2024 100644
--- a/src/backend/utils/activity/pgstat_bgwriter.c
+++ b/src/backend/utils/activity/pgstat_bgwriter.c
@@ -75,6 +75,14 @@ pgstat_fetch_stat_bgwriter(void)
return &pgStatLocal.snapshot.bgwriter;
}
+void
+pgstat_bgwriter_init_shmem_cb(void *stats)
+{
+ PgStatShared_BgWriter *stats_shmem = (PgStatShared_BgWriter *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_bgwriter_reset_all_cb(TimestampTz ts)
{
diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c
index 30a8110e38..bbfc9c7e18 100644
--- a/src/backend/utils/activity/pgstat_checkpointer.c
+++ b/src/backend/utils/activity/pgstat_checkpointer.c
@@ -84,6 +84,14 @@ pgstat_fetch_stat_checkpointer(void)
return &pgStatLocal.snapshot.checkpointer;
}
+void
+pgstat_checkpointer_init_shmem_cb(void *stats)
+{
+ PgStatShared_Checkpointer *stats_shmem = (PgStatShared_Checkpointer *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_checkpointer_reset_all_cb(TimestampTz ts)
{
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 9d6e067382..8af55989ee 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -251,6 +251,15 @@ pgstat_get_io_object_name(IOObject io_object)
pg_unreachable();
}
+void
+pgstat_io_init_shmem_cb(void *stats)
+{
+ PgStatShared_IO *stat_shmem = (PgStatShared_IO *) stats;
+
+ for (int i = 0; i < BACKEND_NUM_TYPES; i++)
+ LWLockInitialize(&stat_shmem->locks[i], LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_io_reset_all_cb(TimestampTz ts)
{
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 634b967820..1c2b69d563 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -196,17 +196,18 @@ StatsShmemInit(void)
pg_atomic_init_u64(&ctl->gc_request_count, 1);
-
/* initialize fixed-numbered stats */
- LWLockInitialize(&ctl->archiver.lock, LWTRANCHE_PGSTATS_DATA);
- LWLockInitialize(&ctl->bgwriter.lock, LWTRANCHE_PGSTATS_DATA);
- LWLockInitialize(&ctl->checkpointer.lock, LWTRANCHE_PGSTATS_DATA);
- LWLockInitialize(&ctl->slru.lock, LWTRANCHE_PGSTATS_DATA);
- LWLockInitialize(&ctl->wal.lock, LWTRANCHE_PGSTATS_DATA);
+ for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ char *ptr;
- for (int i = 0; i < BACKEND_NUM_TYPES; i++)
- LWLockInitialize(&ctl->io.locks[i],
- LWTRANCHE_PGSTATS_DATA);
+ if (!kind_info->fixed_amount)
+ continue;
+
+ ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ kind_info->init_shmem_cb(ptr);
+ }
}
else
{
diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c
index 56ea1c3378..6f922a85bf 100644
--- a/src/backend/utils/activity/pgstat_slru.c
+++ b/src/backend/utils/activity/pgstat_slru.c
@@ -192,6 +192,14 @@ pgstat_slru_flush(bool nowait)
return false;
}
+void
+pgstat_slru_init_shmem_cb(void *stats)
+{
+ PgStatShared_SLRU *stats_shmem = (PgStatShared_SLRU *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_slru_reset_all_cb(TimestampTz ts)
{
diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c
index 0e374f133a..e2a3f6b865 100644
--- a/src/backend/utils/activity/pgstat_wal.c
+++ b/src/backend/utils/activity/pgstat_wal.c
@@ -163,6 +163,14 @@ pgstat_have_pending_wal(void)
PendingWalStats.wal_sync != 0;
}
+void
+pgstat_wal_init_shmem_cb(void *stats)
+{
+ PgStatShared_Wal *stats_shmem = (PgStatShared_Wal *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
void
pgstat_wal_reset_all_cb(TimestampTz ts)
{
--
2.45.2
v5-0002-Add-a-new-F-type-of-entry-in-pgstats-file.patchtext/x-diff; charset=us-asciiDownload
From f3b4d76a6c732b4b84c02f9700188b2a6a4c32a7 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 12:24:44 +0900
Subject: [PATCH v5 2/7] Add a new 'F' type of entry in pgstats file
This new entry type is used for fixed-numbered statistics, making an
upcoming change to add support for custom pluggable stats kinds more
palatable.
---
src/backend/utils/activity/pgstat.c | 44 ++++++++++++++++++-----------
1 file changed, 28 insertions(+), 16 deletions(-)
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index ef435167f7..ed32ee4b6f 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -132,6 +132,7 @@
* ---------
*/
#define PGSTAT_FILE_ENTRY_END 'E' /* end of file */
+#define PGSTAT_FILE_ENTRY_FIXED 'F' /* fixed-numbered stats entry */
#define PGSTAT_FILE_ENTRY_NAME 'N' /* stats entry identified by name */
#define PGSTAT_FILE_ENTRY_HASH 'S' /* stats entry identified by
* PgStat_HashKey */
@@ -1396,6 +1397,9 @@ pgstat_write_statsfile(void)
pgstat_build_snapshot_fixed(kind);
ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+
+ fputc(PGSTAT_FILE_ENTRY_FIXED, fpout);
+ write_chunk_s(fpout, &kind);
write_chunk(fpout, ptr, info->shared_data_len);
}
@@ -1537,22 +1541,6 @@ pgstat_read_statsfile(void)
format_id != PGSTAT_FILE_FORMAT_ID)
goto error;
- /* Read various stats structs with fixed number of objects */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
- {
- char *ptr;
- const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
-
- if (!info->fixed_amount)
- continue;
-
- Assert(info->shared_ctl_off != 0);
-
- ptr = ((char *) shmem) + info->shared_ctl_off + info->shared_data_off;
- if (!read_chunk(fpin, ptr, info->shared_data_len))
- goto error;
- }
-
/*
* We found an existing statistics file. Read it and put all the hash
* table entries into place.
@@ -1563,6 +1551,30 @@ pgstat_read_statsfile(void)
switch (t)
{
+ case PGSTAT_FILE_ENTRY_FIXED:
+ {
+ PgStat_Kind kind;
+ const PgStat_KindInfo *info;
+ char *ptr;
+
+ if (!read_chunk_s(fpin, &kind))
+ goto error;
+
+ if (!pgstat_is_kind_valid(kind))
+ goto error;
+ info = pgstat_get_kind_info(kind);
+ Assert(info->fixed_amount);
+
+ /* Load back stats into shared memory */
+ ptr = ((char *) shmem) + info->shared_ctl_off +
+ info->shared_data_off;
+
+ if (!read_chunk(fpin, ptr,
+ info->shared_data_len))
+ goto error;
+
+ break;
+ }
case PGSTAT_FILE_ENTRY_HASH:
case PGSTAT_FILE_ENTRY_NAME:
{
--
2.45.2
v5-0003-Switch-PgStat_Kind-from-enum-to-uint32.patchtext/x-diff; charset=us-asciiDownload
From 3e390bf999f7e107d2cc03ba6714b6d07c437883 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 12:28:26 +0900
Subject: [PATCH v5 3/7] Switch PgStat_Kind from enum to uint32
A follow-up patch is planned to make this counter extensible, and
keeping a trace of the kind behind a type is useful in the internal
routines used by pgstats. While on it, switch pgstat_is_kind_valid() to
use PgStat_Kind, to be more consistent with its callers.
---
src/include/pgstat.h | 35 ++++++++++++++---------------
src/backend/utils/activity/pgstat.c | 6 ++---
2 files changed, 20 insertions(+), 21 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 2136239710..2d30fadaf1 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -32,26 +32,25 @@
#define PG_STAT_TMP_DIR "pg_stat_tmp"
/* The types of statistics entries */
-typedef enum PgStat_Kind
-{
- /* use 0 for INVALID, to catch zero-initialized data */
- PGSTAT_KIND_INVALID = 0,
+#define PgStat_Kind uint32
- /* stats for variable-numbered objects */
- PGSTAT_KIND_DATABASE, /* database-wide statistics */
- PGSTAT_KIND_RELATION, /* per-table statistics */
- PGSTAT_KIND_FUNCTION, /* per-function statistics */
- PGSTAT_KIND_REPLSLOT, /* per-slot statistics */
- PGSTAT_KIND_SUBSCRIPTION, /* per-subscription statistics */
+/* use 0 for INVALID, to catch zero-initialized data */
+#define PGSTAT_KIND_INVALID 0
- /* stats for fixed-numbered objects */
- PGSTAT_KIND_ARCHIVER,
- PGSTAT_KIND_BGWRITER,
- PGSTAT_KIND_CHECKPOINTER,
- PGSTAT_KIND_IO,
- PGSTAT_KIND_SLRU,
- PGSTAT_KIND_WAL,
-} PgStat_Kind;
+/* stats for variable-numbered objects */
+#define PGSTAT_KIND_DATABASE 1 /* database-wide statistics */
+#define PGSTAT_KIND_RELATION 2 /* per-table statistics */
+#define PGSTAT_KIND_FUNCTION 3 /* per-function statistics */
+#define PGSTAT_KIND_REPLSLOT 4 /* per-slot statistics */
+#define PGSTAT_KIND_SUBSCRIPTION 5 /* per-subscription statistics */
+
+/* stats for fixed-numbered objects */
+#define PGSTAT_KIND_ARCHIVER 6
+#define PGSTAT_KIND_BGWRITER 7
+#define PGSTAT_KIND_CHECKPOINTER 8
+#define PGSTAT_KIND_IO 9
+#define PGSTAT_KIND_SLRU 10
+#define PGSTAT_KIND_WAL 11
#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index ed32ee4b6f..f572a87a61 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -182,7 +182,7 @@ static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
static void pgstat_build_snapshot_fixed(PgStat_Kind kind);
-static inline bool pgstat_is_kind_valid(int ikind);
+static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
/* ----------
@@ -1298,9 +1298,9 @@ pgstat_get_kind_from_str(char *kind_str)
}
static inline bool
-pgstat_is_kind_valid(int ikind)
+pgstat_is_kind_valid(PgStat_Kind kind)
{
- return ikind >= PGSTAT_KIND_FIRST_VALID && ikind <= PGSTAT_KIND_LAST;
+ return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
}
const PgStat_KindInfo *
--
2.45.2
v5-0004-Introduce-pluggable-APIs-for-Cumulative-Statistic.patchtext/x-diff; charset=us-asciiDownload
From 57429e5aa03efbe8ba2afc3946c69fd99d963786 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 14:03:17 +0900
Subject: [PATCH v5 4/7] Introduce pluggable APIs for Cumulative Statistics
This commit adds support in the backend for $subject, allowing
out-of-core extensions to add their own custom statistics kinds. The
stats kinds are divided into two parts for efficiency:
- The built-in stats kinds, with designated initializers.
- The custom kinds, able to use a range of IDs (128 slots available as
of this patch), with information saved in TopMemoryContext.
Custom cumulative statistics can only be loaded with
shared_preload_libraries at startup, and must allocate a unique ID
shared across all the PostgreSQL extension ecosystem with the following
wiki page:
https://wiki.postgresql.org/wiki/CustomCumulativeStats
This is able to support fixed-numbered (like WAL, archiver, bgwriter)
and variable-numbered stats kinds.
---
src/include/pgstat.h | 35 +++-
src/include/utils/pgstat_internal.h | 22 ++-
src/backend/utils/activity/pgstat.c | 231 +++++++++++++++++++---
src/backend/utils/activity/pgstat_shmem.c | 31 ++-
src/backend/utils/adt/pgstatfuncs.c | 2 +-
5 files changed, 287 insertions(+), 34 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 2d30fadaf1..8d523607a4 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -34,6 +34,10 @@
/* The types of statistics entries */
#define PgStat_Kind uint32
+/* Range of IDs allowed, for built-in and custom kinds */
+#define PGSTAT_KIND_MIN 1 /* Minimum ID allowed */
+#define PGSTAT_KIND_MAX 256 /* Maximum ID allowed */
+
/* use 0 for INVALID, to catch zero-initialized data */
#define PGSTAT_KIND_INVALID 0
@@ -52,9 +56,34 @@
#define PGSTAT_KIND_SLRU 10
#define PGSTAT_KIND_WAL 11
-#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
-#define PGSTAT_NUM_KINDS (PGSTAT_KIND_LAST + 1)
+#define PGSTAT_KIND_MIN_BUILTIN PGSTAT_KIND_DATABASE
+#define PGSTAT_KIND_MAX_BUILTIN PGSTAT_KIND_WAL
+
+/* Custom stats kinds */
+
+/* Range of IDs allowed for custom stats kinds */
+#define PGSTAT_KIND_CUSTOM_MIN 128
+#define PGSTAT_KIND_CUSTOM_MAX PGSTAT_KIND_MAX
+#define PGSTAT_KIND_CUSTOM_SIZE (PGSTAT_KIND_CUSTOM_MAX - PGSTAT_KIND_CUSTOM_MIN + 1)
+
+/*
+ * PgStat_Kind to use for extensions that require an ID, but are still in
+ * development and have not reserved their own unique kind ID yet. See:
+ * https://wiki.postgresql.org/wiki/CustomCumulativeStats
+ */
+#define PGSTAT_KIND_EXPERIMENTAL 128
+
+static inline bool
+pgstat_is_kind_builtin(PgStat_Kind kind)
+{
+ return kind > PGSTAT_KIND_INVALID && kind <= PGSTAT_KIND_MAX_BUILTIN;
+}
+
+static inline bool
+pgstat_is_kind_custom(PgStat_Kind kind)
+{
+ return kind >= PGSTAT_KIND_CUSTOM_MIN && kind <= PGSTAT_KIND_CUSTOM_MAX;
+}
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 778f625ca1..39f63362a3 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -195,7 +195,8 @@ typedef struct PgStat_KindInfo
/*
* The size of an entry in the shared stats hash table (pointed to by
- * PgStatShared_HashEntry->body).
+ * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
+ * the size of an entry in PgStat_ShmemControl->custom_data.
*/
uint32 shared_size;
@@ -446,6 +447,13 @@ typedef struct PgStat_ShmemControl
PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
+
+ /*
+ * Custom stats data with fixed-numbered objects, indexed by (PgStat_Kind
+ * - PGSTAT_KIND_CUSTOM_MIN).
+ */
+ void *custom_data[PGSTAT_KIND_CUSTOM_SIZE];
+
} PgStat_ShmemControl;
@@ -459,7 +467,7 @@ typedef struct PgStat_Snapshot
/* time at which snapshot was taken */
TimestampTz snapshot_timestamp;
- bool fixed_valid[PGSTAT_NUM_KINDS];
+ bool fixed_valid[PGSTAT_KIND_MAX_BUILTIN + 1];
PgStat_ArchiverStats archiver;
@@ -473,6 +481,14 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ /*
+ * Data in snapshot for custom fixed-numbered statistics, indexed by
+ * (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
+ * TopMemoryContext, for a size of shared_data_len.
+ */
+ bool custom_valid[PGSTAT_KIND_CUSTOM_SIZE];
+ void *custom_data[PGSTAT_KIND_CUSTOM_SIZE];
+
/* to free snapshot in bulk */
MemoryContext context;
struct pgstat_snapshot_hash *stats;
@@ -516,6 +532,8 @@ static inline void *pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common
*/
extern const PgStat_KindInfo *pgstat_get_kind_info(PgStat_Kind kind);
+extern void pgstat_register_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
#ifdef USE_ASSERT_CHECKING
extern void pgstat_assert_is_up(void);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index f572a87a61..3b85fe0fc8 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -49,8 +49,16 @@
* pgStatPending list. Pending statistics updates are flushed out by
* pgstat_report_stat().
*
+ * It is possible for external modules to define custom statistics kinds,
+ * that can use the same properties as any built-in stats kinds. Each custom
+ * stats kind needs to assign a unique ID to ensure that it does not overlap
+ * with other extensions. In order to reserve a unique stats kind ID, refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats.
+ *
* The behavior of different kinds of statistics is determined by the kind's
- * entry in pgstat_kind_infos, see PgStat_KindInfo for details.
+ * entry in pgstat_kind_builtin_infos for all the built-in statistics kinds
+ * defined, and pgstat_kind_custom_infos for custom kinds registered at
+ * startup by pgstat_register_kind(). See PgStat_KindInfo for details.
*
* The consistency of read accesses to statistics can be configured using the
* stats_fetch_consistency GUC (see config.sgml and monitoring.sgml for the
@@ -174,6 +182,8 @@ typedef struct PgStat_SnapshotEntry
static void pgstat_write_statsfile(void);
static void pgstat_read_statsfile(void);
+static void pgstat_init_snapshot_fixed(void);
+
static void pgstat_reset_after_failure(void);
static bool pgstat_flush_pending_entries(bool nowait);
@@ -251,7 +261,7 @@ static bool pgstat_is_shutdown = false;
/*
- * The different kinds of statistics.
+ * The different kinds of built-in statistics.
*
* If reasonably possible, handling specific to one kind of stats should go
* through this abstraction, rather than making more of pgstat.c aware.
@@ -263,7 +273,7 @@ static bool pgstat_is_shutdown = false;
* seem to be a great way of doing that, given the split across multiple
* files.
*/
-static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
+static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_MAX_BUILTIN + 1] = {
/* stats kinds for variable-numbered objects */
@@ -436,6 +446,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
},
};
+/*
+ * Information about custom statistics kinds.
+ *
+ * These are saved in a different array than the built-in kinds to save
+ * in clarity with the initializations.
+ *
+ * Indexed by PGSTAT_KIND_CUSTOM_MIN, of size PGSTAT_KIND_CUSTOM_SIZE.
+ */
+static const PgStat_KindInfo **pgstat_kind_custom_infos = NULL;
/* ------------------------------------------------------------
* Functions managing the state of the stats system for all backends.
@@ -586,6 +605,8 @@ pgstat_initialize(void)
pgstat_init_wal();
+ pgstat_init_snapshot_fixed();
+
/* Set up a process-exit hook to clean up */
before_shmem_exit(pgstat_shutdown_hook, 0);
@@ -829,6 +850,8 @@ pgstat_clear_snapshot(void)
memset(&pgStatLocal.snapshot.fixed_valid, 0,
sizeof(pgStatLocal.snapshot.fixed_valid));
+ memset(&pgStatLocal.snapshot.custom_valid, 0,
+ sizeof(pgStatLocal.snapshot.custom_valid));
pgStatLocal.snapshot.stats = NULL;
pgStatLocal.snapshot.mode = PGSTAT_FETCH_CONSISTENCY_NONE;
@@ -992,7 +1015,29 @@ pgstat_snapshot_fixed(PgStat_Kind kind)
else
pgstat_build_snapshot_fixed(kind);
- Assert(pgStatLocal.snapshot.fixed_valid[kind]);
+ if (pgstat_is_kind_builtin(kind))
+ Assert(pgStatLocal.snapshot.fixed_valid[kind]);
+ else if (pgstat_is_kind_custom(kind))
+ Assert(pgStatLocal.snapshot.custom_valid[kind - PGSTAT_KIND_CUSTOM_MIN]);
+}
+
+static void
+pgstat_init_snapshot_fixed(void)
+{
+ /*
+ * Initialize fixed-numbered statistics data in snapshots, only for custom
+ * stats kinds.
+ */
+ for (int kind = PGSTAT_KIND_CUSTOM_MIN; kind <= PGSTAT_KIND_CUSTOM_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->fixed_amount)
+ continue;
+
+ pgStatLocal.snapshot.custom_data[kind - PGSTAT_KIND_CUSTOM_MIN] =
+ MemoryContextAlloc(TopMemoryContext, kind_info->shared_data_len);
+ }
}
static void
@@ -1088,10 +1133,12 @@ pgstat_build_snapshot(void)
/*
* Build snapshot of all fixed-numbered stats.
*/
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ if (!kind_info)
+ continue;
if (!kind_info->fixed_amount)
{
Assert(kind_info->snapshot_cb == NULL);
@@ -1108,6 +1155,20 @@ static void
pgstat_build_snapshot_fixed(PgStat_Kind kind)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ int idx;
+ bool *valid;
+
+ /* Position in fixed_valid or custom_valid */
+ if (pgstat_is_kind_builtin(kind))
+ {
+ idx = kind;
+ valid = pgStatLocal.snapshot.fixed_valid;
+ }
+ else
+ {
+ idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+ valid = pgStatLocal.snapshot.custom_valid;
+ }
Assert(kind_info->fixed_amount);
Assert(kind_info->snapshot_cb != NULL);
@@ -1115,21 +1176,21 @@ pgstat_build_snapshot_fixed(PgStat_Kind kind)
if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE)
{
/* rebuild every time */
- pgStatLocal.snapshot.fixed_valid[kind] = false;
+ valid[idx] = false;
}
- else if (pgStatLocal.snapshot.fixed_valid[kind])
+ else if (valid[idx])
{
/* in snapshot mode we shouldn't get called again */
Assert(pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_CACHE);
return;
}
- Assert(!pgStatLocal.snapshot.fixed_valid[kind]);
+ Assert(!valid[idx]);
kind_info->snapshot_cb();
- Assert(!pgStatLocal.snapshot.fixed_valid[kind]);
- pgStatLocal.snapshot.fixed_valid[kind] = true;
+ Assert(!valid[idx]);
+ valid[idx] = true;
}
@@ -1285,30 +1346,127 @@ pgstat_flush_pending_entries(bool nowait)
PgStat_Kind
pgstat_get_kind_from_str(char *kind_str)
{
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN_BUILTIN; kind <= PGSTAT_KIND_MAX_BUILTIN; kind++)
{
- if (pg_strcasecmp(kind_str, pgstat_kind_infos[kind].name) == 0)
+ if (pg_strcasecmp(kind_str, pgstat_kind_builtin_infos[kind].name) == 0)
return kind;
}
+ /* Check the custom set of cumulative stats */
+ if (pgstat_kind_custom_infos)
+ {
+ for (int kind = 0; kind < PGSTAT_KIND_CUSTOM_SIZE; kind++)
+ {
+ if (pgstat_kind_custom_infos[kind] &&
+ pg_strcasecmp(kind_str, pgstat_kind_custom_infos[kind]->name) == 0)
+ return kind + PGSTAT_KIND_CUSTOM_MIN;
+ }
+ }
+
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid statistics kind: \"%s\"", kind_str)));
- return PGSTAT_KIND_DATABASE; /* avoid compiler warnings */
+ return PGSTAT_KIND_INVALID; /* avoid compiler warnings */
}
static inline bool
pgstat_is_kind_valid(PgStat_Kind kind)
{
- return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
+ return pgstat_is_kind_builtin(kind) || pgstat_is_kind_custom(kind);
}
const PgStat_KindInfo *
pgstat_get_kind_info(PgStat_Kind kind)
{
- Assert(pgstat_is_kind_valid(kind));
+ if (pgstat_is_kind_builtin(kind))
+ return &pgstat_kind_builtin_infos[kind];
- return &pgstat_kind_infos[kind];
+ if (pgstat_is_kind_custom(kind))
+ {
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (pgstat_kind_custom_infos == NULL ||
+ pgstat_kind_custom_infos[idx] == NULL)
+ return NULL;
+ return pgstat_kind_custom_infos[idx];
+ }
+
+ return NULL;
+}
+
+/*
+ * Register a new stats kind.
+ *
+ * PgStat_Kinds must be globally unique across all extensions. Refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats to reserve a
+ * unique ID for your extension, to avoid conflicts with other extension
+ * developers. During development, use PGSTAT_KIND_EXPERIMENTAL to avoid
+ * needlessly reserving a new ID.
+ */
+void
+pgstat_register_kind(PgStat_Kind kind, const PgStat_KindInfo *kind_info)
+{
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (kind_info->name == NULL || strlen(kind_info->name) == 0)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics name is invalid"),
+ errhint("Provide a non-empty name for the custom cumulative statistics.")));
+
+ if (!pgstat_is_kind_custom(kind))
+ ereport(ERROR, (errmsg("custom cumulative statistics ID %u is out of range", kind),
+ errhint("Provide a custom cumulative statistics ID between %u and %u.",
+ PGSTAT_KIND_CUSTOM_MIN, PGSTAT_KIND_CUSTOM_MAX)));
+
+ if (!process_shared_preload_libraries_in_progress)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics must be registered while initializing modules in \"shared_preload_libraries\".")));
+
+ /*
+ * Check some data for fixed-numbered stats.
+ */
+ if (kind_info->fixed_amount)
+ {
+ if (kind_info->shared_size == 0)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics property is invalid"),
+ errhint("Custom cumulative statistics require a shared memory size for fixed-numbered objects.")));
+ }
+
+ /*
+ * If pgstat_kind_custom_infos is not available yet, allocate it.
+ */
+ if (pgstat_kind_custom_infos == NULL)
+ {
+ pgstat_kind_custom_infos = (const PgStat_KindInfo **)
+ MemoryContextAllocZero(TopMemoryContext,
+ sizeof(PgStat_KindInfo *) * PGSTAT_KIND_CUSTOM_SIZE);
+ }
+
+ if (pgstat_kind_custom_infos[idx] != NULL &&
+ pgstat_kind_custom_infos[idx]->name != NULL)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics \"%s\" already registered with the same ID.",
+ pgstat_kind_custom_infos[idx]->name)));
+
+ /* check for existing custom stats with the same name */
+ for (int existing_kind = 0; existing_kind < PGSTAT_KIND_CUSTOM_SIZE; existing_kind++)
+ {
+ if (pgstat_kind_custom_infos[existing_kind] == NULL)
+ continue;
+ if (!pg_strcasecmp(pgstat_kind_custom_infos[existing_kind]->name, kind_info->name))
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Existing cumulative statistics with ID %u has the same name.", existing_kind + PGSTAT_KIND_CUSTOM_MIN)));
+ }
+
+ /* Register it */
+ pgstat_kind_custom_infos[idx] = kind_info;
+ ereport(LOG,
+ (errmsg("registered custom cumulative statistics \"%s\" with ID %u",
+ kind_info->name, kind)));
}
/*
@@ -1385,18 +1543,22 @@ pgstat_write_statsfile(void)
write_chunk_s(fpout, &format_id);
/* Write various stats structs for fixed number of objects */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
char *ptr;
const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
- if (!info->fixed_amount)
+ if (!info || !info->fixed_amount)
continue;
- Assert(info->snapshot_ctl_off != 0);
+ if (pgstat_is_kind_builtin(kind))
+ Assert(info->snapshot_ctl_off != 0);
pgstat_build_snapshot_fixed(kind);
- ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+ else
+ ptr = pgStatLocal.snapshot.custom_data[kind - PGSTAT_KIND_CUSTOM_MIN];
fputc(PGSTAT_FILE_ENTRY_FIXED, fpout);
write_chunk_s(fpout, &kind);
@@ -1419,6 +1581,17 @@ pgstat_write_statsfile(void)
if (ps->dropped)
continue;
+ /*
+ * This discards data related to custom stats kinds that are unknown
+ * to this process.
+ */
+ if (!pgstat_is_kind_valid(ps->key.kind))
+ {
+ elog(WARNING, "found unknown stats entry %u/%u/%u",
+ ps->key.kind, ps->key.dboid, ps->key.objoid);
+ continue;
+ }
+
shstats = (PgStatShared_Common *) dsa_get_address(pgStatLocal.dsa, ps->body);
kind_info = pgstat_get_kind_info(ps->key.kind);
@@ -1566,8 +1739,16 @@ pgstat_read_statsfile(void)
Assert(info->fixed_amount);
/* Load back stats into shared memory */
- ptr = ((char *) shmem) + info->shared_ctl_off +
- info->shared_data_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) shmem) + info->shared_ctl_off +
+ info->shared_data_off;
+ else
+ {
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ ptr = ((char *) shmem->custom_data[idx]) +
+ info->shared_data_off;
+ }
if (!read_chunk(fpin, ptr,
info->shared_data_len))
@@ -1635,7 +1816,7 @@ pgstat_read_statsfile(void)
if (found)
{
dshash_release_lock(pgStatLocal.shared_hash, p);
- elog(WARNING, "found duplicate stats entry %d/%u/%u",
+ elog(WARNING, "found duplicate stats entry %u/%u/%u",
key.kind, key.dboid, key.objoid);
goto error;
}
@@ -1693,11 +1874,11 @@ pgstat_reset_after_failure(void)
TimestampTz ts = GetCurrentTimestamp();
/* reset fixed-numbered stats */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
kind_info->reset_all_cb(ts);
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 1c2b69d563..ca3d3eecfb 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -131,6 +131,21 @@ StatsShmemSize(void)
sz = MAXALIGN(sizeof(PgStat_ShmemControl));
sz = add_size(sz, pgstat_dsa_init_size());
+ /* Add shared memory for all the custom fixed-numbered statistics */
+ for (int kind = PGSTAT_KIND_CUSTOM_MIN; kind <= PGSTAT_KIND_CUSTOM_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info)
+ continue;
+ if (!kind_info->fixed_amount)
+ continue;
+
+ Assert(kind_info->shared_size != 0);
+
+ sz += MAXALIGN(kind_info->shared_size);
+ }
+
return sz;
}
@@ -197,15 +212,25 @@ StatsShmemInit(void)
pg_atomic_init_u64(&ctl->gc_request_count, 1);
/* initialize fixed-numbered stats */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
char *ptr;
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
- ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ else
+ {
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ Assert(kind_info->shared_size != 0);
+ ctl->custom_data[idx] = ShmemAlloc(kind_info->shared_size);
+ ptr = ctl->custom_data[idx];
+ }
+
kind_info->init_shmem_cb(ptr);
}
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 3876339ee1..3221137123 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1696,7 +1696,7 @@ pg_stat_reset(PG_FUNCTION_ARGS)
* Reset some shared cluster-wide counters
*
* When adding a new reset target, ideally the name should match that in
- * pgstat_kind_infos, if relevant.
+ * pgstat_kind_builtin_infos, if relevant.
*/
Datum
pg_stat_reset_shared(PG_FUNCTION_ARGS)
--
2.45.2
v5-0005-doc-Add-section-for-Custom-Cumulative-Statistics-.patchtext/x-diff; charset=us-asciiDownload
From aca83deecd4a7ed78b2fc35dde373534cf26abc0 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 3 Jul 2024 13:49:53 +0900
Subject: [PATCH v5 5/7] doc: Add section for Custom Cumulative Statistics APIs
This provides a short description of what can be done, with a pointer to
the template in the tree.
---
doc/src/sgml/xfunc.sgml | 63 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 63 insertions(+)
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index 756a9d07fb..878722af51 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3700,6 +3700,69 @@ extern bool InjectionPointDetach(const char *name);
</para>
</sect2>
+ <sect2 id="xfunc-addin-custom-cumulative-statistics">
+ <title>Custom Cumulative Statistics</title>
+
+ <para>
+ Is is possible for add-ins written in C-language to use custom types
+ of cumulative statistics registered in the
+ <link linkend="monitoring-stats-setup">Cumulative Statistics System</link>.
+ </para>
+
+ <para>
+ First, define a <literal>PgStat_KindInfo</literal> that includes all
+ the information related to the custom type registered. For example:
+<programlisting>
+static const PgStat_KindInfo custom_stats = {
+ .name = "custom_stats",
+ .fixed_amount = false,
+ .shared_size = sizeof(PgStatShared_Custom),
+ .shared_data_off = offsetof(PgStatShared_Custom, stats),
+ .shared_data_len = sizeof(((PgStatShared_Custom *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatCustomEntry),
+}
+</programlisting>
+
+ Then, each backend that needs to use this custom type needs to register
+ it with <literal>pgstat_register_kind</literal> and a unique ID used to
+ store the entries related to this type of statistics:
+<programlisting>
+extern PgStat_Kind pgstat_add_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
+</programlisting>
+ While developing a new extension, use
+ <literal>PGSTAT_KIND_EXPERIMENTAL</literal> for
+ <parameter>kind</parameter>. When you are ready to release the extension
+ to users, reserve a kind ID at the
+ <ulink url="https://wiki.postgresql.org/wiki/CustomCumulativeStats">
+ Custom Cumulative Statistics</ulink> page.
+ </para>
+
+ <para>
+ The details of the API for <literal>PgStat_KindInfo</literal> can
+ be found in <filename>src/include/utils/pgstat_internal.h</filename>.
+ </para>
+
+ <para>
+ The type of statistics registered is associated with a name and a unique
+ ID shared across the server in shared memory. Each backend using a
+ custom type of statistics maintains a local cache storing the information
+ of each custom <literal>PgStat_KindInfo</literal>.
+ </para>
+
+ <para>
+ Place the extension module implementing the custom cumulative statistics
+ type in <xref linkend="guc-shared-preload-libraries"/> so that it will
+ be loaded early during <productname>PostgreSQL</productname> startup.
+ </para>
+
+ <para>
+ An example describing how to register and use custom statistics can be
+ found in
+ <filename>src/test/modules/injection_points/injection_stats.c</filename>.
+ </para>
+ </sect2>
+
<sect2 id="extend-cpp">
<title>Using C++ for Extensibility</title>
--
2.45.2
v5-0006-injection_points-Add-statistics-for-custom-points.patchtext/x-diff; charset=us-asciiDownload
From a26f486e0bea4fe73d3b134284415ce4054cbd04 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 13:36:06 +0900
Subject: [PATCH v5 6/7] injection_points: Add statistics for custom points
This acts as a template of what can be achieved with the pluggable
cumulative stats APIs, while being useful on its own for injection
points.
Currently, the only data gathered is the number of times an injection
point is called. This can be extended as required. All the routines
related to the stats are located in their own file, for clarity.
A TAP test is included to provide coverage for these new APIs, showing
the persistency of the data across restarts.
---
src/test/modules/injection_points/Makefile | 11 +-
.../injection_points--1.0.sql | 10 +
.../injection_points/injection_points.c | 27 +++
.../injection_points/injection_stats.c | 194 ++++++++++++++++++
.../injection_points/injection_stats.h | 23 +++
src/test/modules/injection_points/meson.build | 9 +
.../modules/injection_points/t/001_stats.pl | 48 +++++
src/tools/pgindent/typedefs.list | 2 +
8 files changed, 322 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/injection_points/injection_stats.c
create mode 100644 src/test/modules/injection_points/injection_stats.h
create mode 100644 src/test/modules/injection_points/t/001_stats.pl
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 2ffd2f77ed..7b9cd12a2a 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -1,7 +1,10 @@
# src/test/modules/injection_points/Makefile
-MODULES = injection_points
-
+MODULE_big = injection_points
+OBJS = \
+ $(WIN32RES) \
+ injection_points.o \
+ injection_stats.o
EXTENSION = injection_points
DATA = injection_points--1.0.sql
PGFILEDESC = "injection_points - facility for injection points"
@@ -11,9 +14,13 @@ REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
ISOLATION = inplace
+TAP_TESTS = 1
+
# The injection points are cluster-wide, so disable installcheck
NO_INSTALLCHECK = 1
+export enable_injection_points enable_injection_points
+
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index e275c2cf5b..747a64e812 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -64,3 +64,13 @@ CREATE FUNCTION injection_points_detach(IN point_name TEXT)
RETURNS void
AS 'MODULE_PATHNAME', 'injection_points_detach'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+--
+-- injection_points_stats_numcalls()
+--
+-- Reports statistics, if any, related to the given injection point.
+--
+CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
+RETURNS bigint
+AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index b6c8e89324..eb411b9d44 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -18,6 +18,7 @@
#include "postgres.h"
#include "fmgr.h"
+#include "injection_stats.h"
#include "miscadmin.h"
#include "nodes/pg_list.h"
#include "nodes/value.h"
@@ -170,6 +171,9 @@ injection_points_cleanup(int code, Datum arg)
char *name = strVal(lfirst(lc));
(void) InjectionPointDetach(name);
+
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
}
}
@@ -182,6 +186,8 @@ injection_error(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(ERROR, "error triggered for injection point %s", name);
}
@@ -193,6 +199,8 @@ injection_notice(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(NOTICE, "notice triggered for injection point %s", name);
}
@@ -211,6 +219,8 @@ injection_wait(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
/*
* Use the injection point name for this custom wait event. Note that
* this custom wait event name is not released, but we don't care much for
@@ -299,6 +309,10 @@ injection_points_attach(PG_FUNCTION_ARGS)
inj_list_local = lappend(inj_list_local, makeString(pstrdup(name)));
MemoryContextSwitchTo(oldctx);
}
+
+ /* Add entry for stats */
+ pgstat_create_inj(name);
+
PG_RETURN_VOID();
}
@@ -417,5 +431,18 @@ injection_points_detach(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldctx);
}
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
+
PG_RETURN_VOID();
}
+
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ pgstat_register_inj();
+}
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
new file mode 100644
index 0000000000..c37b0b33d3
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -0,0 +1,194 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.c
+ * Code for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+
+#include "common/hashfn.h"
+#include "injection_stats.h"
+#include "pgstat.h"
+#include "utils/builtins.h"
+#include "utils/pgstat_internal.h"
+
+/* Structures for statistics of injection points */
+typedef struct PgStat_StatInjEntry
+{
+ PgStat_Counter numcalls; /* number of times point has been run */
+} PgStat_StatInjEntry;
+
+typedef struct PgStatShared_InjectionPoint
+{
+ PgStatShared_Common header;
+ PgStat_StatInjEntry stats;
+} PgStatShared_InjectionPoint;
+
+static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+
+static const PgStat_KindInfo injection_stats = {
+ .name = "injection_points",
+ .fixed_amount = false, /* Bounded by the number of points */
+
+ /* Injection points are system-wide */
+ .accessed_across_databases = true,
+
+ .shared_size = sizeof(PgStatShared_InjectionPoint),
+ .shared_data_off = offsetof(PgStatShared_InjectionPoint, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPoint *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatInjEntry),
+ .flush_pending_cb = injection_stats_flush_cb,
+};
+
+/*
+ * Compute stats entry idx from point name with a 4-byte hash.
+ */
+#define PGSTAT_INJ_IDX(name) hash_bytes((const unsigned char *) name, strlen(name))
+
+#define PGSTAT_KIND_INJECTION PGSTAT_KIND_EXPERIMENTAL
+
+/* Track if stats are loaded */
+static bool inj_stats_loaded = false;
+
+/*
+ * Callback for stats handling
+ */
+static bool
+injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ PgStat_StatInjEntry *localent;
+ PgStatShared_InjectionPoint *shfuncent;
+
+ localent = (PgStat_StatInjEntry *) entry_ref->pending;
+ shfuncent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ shfuncent->stats.numcalls += localent->numcalls;
+ return true;
+}
+
+/*
+ * Support function for the SQL-callable pgstat* functions. Returns
+ * a pointer to the injection point statistics struct.
+ */
+static PgStat_StatInjEntry *
+pgstat_fetch_stat_injentry(const char *name)
+{
+ PgStat_StatInjEntry *entry = NULL;
+
+ if (!inj_stats_loaded)
+ return NULL;
+
+ /* Compile the lookup key as a hash of the point name */
+ entry = (PgStat_StatInjEntry *) pgstat_fetch_entry(PGSTAT_KIND_INJECTION,
+ InvalidOid,
+ PGSTAT_INJ_IDX(name));
+ return entry;
+}
+
+/*
+ * Workhorse to do the registration work, called in _PG_init().
+ */
+void
+pgstat_register_inj(void)
+{
+ pgstat_register_kind(PGSTAT_KIND_INJECTION, &injection_stats);
+
+ /* mark stats as loaded */
+ inj_stats_loaded = true;
+}
+
+/*
+ * Report injection point creation.
+ */
+void
+pgstat_create_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ /* initialize shared memory data */
+ memset(&shstatent->stats, 0, sizeof(shstatent->stats));
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * Report injection point drop.
+ */
+void
+pgstat_drop_inj(const char *name)
+{
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ if (!pgstat_drop_entry(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name)))
+ pgstat_request_entry_refs_gc();
+}
+
+/*
+ * Report statistics for injection point.
+ *
+ * This is simple because the set of stats to report currently is simple:
+ * track the number of times a point has been run.
+ */
+void
+pgstat_report_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+ PgStat_StatInjEntry *statent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+ statent = &shstatent->stats;
+
+ /* Update the injection point statistics */
+ statent->numcalls++;
+
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * SQL function returning the number of times an injection point
+ * has been called.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_numcalls);
+Datum
+injection_points_stats_numcalls(PG_FUNCTION_ARGS)
+{
+ char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_StatInjEntry *entry = pgstat_fetch_stat_injentry(name);
+
+ if (entry == NULL)
+ PG_RETURN_NULL();
+
+ PG_RETURN_INT64(entry->numcalls);
+}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
new file mode 100644
index 0000000000..3e99705483
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -0,0 +1,23 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.h
+ * Definitions for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.h
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#ifndef INJECTION_STATS
+#define INJECTION_STATS
+
+extern void pgstat_register_inj(void);
+extern void pgstat_create_inj(const char *name);
+extern void pgstat_drop_inj(const char *name);
+extern void pgstat_report_inj(const char *name);
+
+#endif
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index 3c23c14d81..a52fe5121e 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -6,6 +6,7 @@ endif
injection_points_sources = files(
'injection_points.c',
+ 'injection_stats.c',
)
if host_system == 'windows'
@@ -42,4 +43,12 @@ tests += {
'inplace',
],
},
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_stats.pl',
+ ],
+ },
}
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
new file mode 100644
index 0000000000..7d5a96e522
--- /dev/null
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -0,0 +1,48 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+use locale;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Test persistency of statistics generated for injection points.
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# Node initialization
+my $node = PostgreSQL::Test::Cluster->new('master');
+$node->init;
+$node->append_conf('postgresql.conf',
+ "shared_preload_libraries = 'injection_points'");
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# This should count for two calls.
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('stats-notice', 'notice');");
+$node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
+$node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
+my $numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats calls');
+
+# Restart the node cleanly, stats should still be around.
+$node->restart;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats after clean restart');
+
+# On crash the stats are gone.
+$node->stop('immediate');
+$node->start;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '', 'number of stats after clean restart');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 635e6d6e21..5206029e96 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2118,6 +2118,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_InjectionPoint
PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
@@ -2149,6 +2150,7 @@ PgStat_Snapshot
PgStat_SnapshotEntry
PgStat_StatDBEntry
PgStat_StatFuncEntry
+PgStat_StatInjEntry
PgStat_StatReplSlotEntry
PgStat_StatSubEntry
PgStat_StatTabEntry
--
2.45.2
v5-0007-injection_points-Add-example-for-fixed-numbered-s.patchtext/x-diff; charset=us-asciiDownload
From 52c75a4504605094bc47c1e5b47553de8db338a1 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 13:44:14 +0900
Subject: [PATCH v5 7/7] injection_points: Add example for fixed-numbered
statistics
This acts as a template to show what can be achieved with fixed-numbered
stats (like WAL, bgwriter, etc.) for pluggable cumulative statistics.
---
.../injection_points--1.0.sql | 11 ++
.../injection_points/injection_points.c | 3 +
.../injection_points/injection_stats.c | 155 +++++++++++++++++-
.../injection_points/injection_stats.h | 3 +
.../modules/injection_points/t/001_stats.pl | 11 +-
5 files changed, 180 insertions(+), 3 deletions(-)
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index 747a64e812..fa0b1d06ae 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -74,3 +74,14 @@ CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
RETURNS bigint
AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
LANGUAGE C STRICT;
+
+--
+-- injection_points_stats_fixed()
+--
+-- Reports fixed-numbered statistics for injection points.
+CREATE FUNCTION injection_points_stats_fixed(OUT numattach int8,
+ OUT numdetach int8,
+ OUT numrun int8)
+RETURNS record
+AS 'MODULE_PATHNAME', 'injection_points_stats_fixed'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index eb411b9d44..f65bd83cc3 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -297,6 +297,7 @@ injection_points_attach(PG_FUNCTION_ARGS)
condition.pid = MyProcPid;
}
+ pgstat_report_inj_fixed(1, 0, 0);
InjectionPointAttach(name, "injection_points", function, &condition,
sizeof(InjectionPointCondition));
@@ -342,6 +343,7 @@ injection_points_run(PG_FUNCTION_ARGS)
{
char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ pgstat_report_inj_fixed(0, 0, 1);
INJECTION_POINT(name);
PG_RETURN_VOID();
@@ -418,6 +420,7 @@ injection_points_detach(PG_FUNCTION_ARGS)
{
char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ pgstat_report_inj_fixed(0, 1, 0);
if (!InjectionPointDetach(name))
elog(ERROR, "could not detach injection point \"%s\"", name);
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
index c37b0b33d3..69560b7fdc 100644
--- a/src/test/modules/injection_points/injection_stats.c
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -17,12 +17,13 @@
#include "fmgr.h"
#include "common/hashfn.h"
+#include "funcapi.h"
#include "injection_stats.h"
#include "pgstat.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
-/* Structures for statistics of injection points */
+/* Structures for statistics of injection points, variable-size */
typedef struct PgStat_StatInjEntry
{
PgStat_Counter numcalls; /* number of times point has been run */
@@ -35,6 +36,9 @@ typedef struct PgStatShared_InjectionPoint
} PgStatShared_InjectionPoint;
static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+static void injection_stats_fixed_init_shmem_cb(void *stats);
+static void injection_stats_fixed_reset_all_cb(TimestampTz ts);
+static void injection_stats_fixed_snapshot_cb(void);
static const PgStat_KindInfo injection_stats = {
.name = "injection_points",
@@ -50,18 +54,51 @@ static const PgStat_KindInfo injection_stats = {
.flush_pending_cb = injection_stats_flush_cb,
};
+/* Structures for statistics of injection points, fixed-size */
+typedef struct PgStat_StatInjFixedEntry
+{
+ PgStat_Counter numattach; /* number of points attached */
+ PgStat_Counter numdetach; /* number of points detached */
+ PgStat_Counter numrun; /* number of points run */
+ TimestampTz stat_reset_timestamp;
+} PgStat_StatInjFixedEntry;
+
+typedef struct PgStatShared_InjectionPointFixed
+{
+ LWLock lock; /* protects all the counters */
+ uint32 changecount;
+ PgStat_StatInjFixedEntry stats;
+ PgStat_StatInjFixedEntry reset_offset;
+} PgStatShared_InjectionPointFixed;
+
+static const PgStat_KindInfo injection_stats_fixed = {
+ .name = "injection_points_fixed",
+ .fixed_amount = true,
+
+ .shared_size = sizeof(PgStat_StatInjFixedEntry),
+ .shared_data_off = offsetof(PgStatShared_InjectionPointFixed, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPointFixed *) 0)->stats),
+
+ .init_shmem_cb = injection_stats_fixed_init_shmem_cb,
+ .reset_all_cb = injection_stats_fixed_reset_all_cb,
+ .snapshot_cb = injection_stats_fixed_snapshot_cb,
+};
+
/*
* Compute stats entry idx from point name with a 4-byte hash.
*/
#define PGSTAT_INJ_IDX(name) hash_bytes((const unsigned char *) name, strlen(name))
#define PGSTAT_KIND_INJECTION PGSTAT_KIND_EXPERIMENTAL
+#define PGSTAT_KIND_INJECTION_FIXED (PGSTAT_KIND_EXPERIMENTAL + 1)
+/* Position of fixed-numbered data in internal structures */
+#define PGSTAT_KIND_INJECTION_IDX (PGSTAT_KIND_INJECTION_FIXED - PGSTAT_KIND_CUSTOM_MIN)
/* Track if stats are loaded */
static bool inj_stats_loaded = false;
/*
- * Callback for stats handling
+ * Callbacks for stats handling
*/
static bool
injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
@@ -79,6 +116,59 @@ injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
return true;
}
+static void
+injection_stats_fixed_init_shmem_cb(void *stats)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+static void
+injection_stats_fixed_reset_all_cb(TimestampTz ts)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *)
+ pgStatLocal.shmem->custom_data[PGSTAT_KIND_INJECTION_IDX];
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ pgstat_copy_changecounted_stats(&stats_shmem->reset_offset,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+ stats_shmem->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&stats_shmem->lock);
+}
+
+static void
+injection_stats_fixed_snapshot_cb(void)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *)
+ pgStatLocal.shmem->custom_data[PGSTAT_KIND_INJECTION_IDX];
+ PgStat_StatInjFixedEntry *stat_snap = (PgStat_StatInjFixedEntry *)
+ pgStatLocal.snapshot.custom_data[PGSTAT_KIND_INJECTION_IDX];
+ PgStat_StatInjFixedEntry *reset_offset = &stats_shmem->reset_offset;
+ PgStat_StatInjFixedEntry reset;
+
+ pgstat_copy_changecounted_stats(stat_snap,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+
+ LWLockAcquire(&stats_shmem->lock, LW_SHARED);
+ memcpy(&reset, reset_offset, sizeof(stats_shmem->stats));
+ LWLockRelease(&stats_shmem->lock);
+
+ /* compensate by reset offsets */
+#define FIXED_COMP(fld) stat_snap->fld -= reset.fld;
+ FIXED_COMP(numattach);
+ FIXED_COMP(numdetach);
+ FIXED_COMP(numrun);
+#undef FIXED_COMP
+}
+
/*
* Support function for the SQL-callable pgstat* functions. Returns
* a pointer to the injection point statistics struct.
@@ -105,6 +195,7 @@ void
pgstat_register_inj(void)
{
pgstat_register_kind(PGSTAT_KIND_INJECTION, &injection_stats);
+ pgstat_register_kind(PGSTAT_KIND_INJECTION_FIXED, &injection_stats_fixed);
/* mark stats as loaded */
inj_stats_loaded = true;
@@ -176,6 +267,30 @@ pgstat_report_inj(const char *name)
pgstat_unlock_entry(entry_ref);
}
+/*
+ * Report fixed number of statistics for an injection point.
+ */
+void
+pgstat_report_inj_fixed(uint32 numattach,
+ uint32 numdetach,
+ uint32 numrun)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ stats_shmem = (PgStatShared_InjectionPointFixed *)
+ pgStatLocal.shmem->custom_data[PGSTAT_KIND_INJECTION_IDX];
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numattach += numattach;
+ stats_shmem->stats.numdetach += numdetach;
+ stats_shmem->stats.numrun += numrun;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+}
+
/*
* SQL function returning the number of times an injection point
* has been called.
@@ -192,3 +307,39 @@ injection_points_stats_numcalls(PG_FUNCTION_ARGS)
PG_RETURN_INT64(entry->numcalls);
}
+
+/*
+ * SQL function returning fixed-numbered statistics for injection points.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_fixed);
+Datum
+injection_points_stats_fixed(PG_FUNCTION_ARGS)
+{
+ TupleDesc tupdesc;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ PgStat_StatInjFixedEntry *stats;
+
+ pgstat_snapshot_fixed(PGSTAT_KIND_INJECTION_FIXED);
+ stats = pgStatLocal.snapshot.custom_data[PGSTAT_KIND_INJECTION_IDX];
+
+ /* Initialise attributes information in the tuple descriptor */
+ tupdesc = CreateTemplateTupleDesc(3);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "numattach",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "numdetach",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "numrun",
+ INT8OID, -1, 0);
+ BlessTupleDesc(tupdesc);
+
+ values[0] = Int64GetDatum(stats->numattach);
+ values[1] = Int64GetDatum(stats->numdetach);
+ values[2] = Int64GetDatum(stats->numrun);
+ nulls[0] = false;
+ nulls[1] = false;
+ nulls[2] = false;
+
+ /* Returns the record as Datum */
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
index 3e99705483..cf68b25f7b 100644
--- a/src/test/modules/injection_points/injection_stats.h
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -19,5 +19,8 @@ extern void pgstat_register_inj(void);
extern void pgstat_create_inj(const char *name);
extern void pgstat_drop_inj(const char *name);
extern void pgstat_report_inj(const char *name);
+extern void pgstat_report_inj_fixed(uint32 numattach,
+ uint32 numdetach,
+ uint32 numrun);
#endif
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
index 7d5a96e522..e3c69b94ca 100644
--- a/src/test/modules/injection_points/t/001_stats.pl
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -31,18 +31,27 @@ $node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
my $numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
is($numcalls, '2', 'number of stats calls');
+my $fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '1|0|2', 'number of fixed stats');
# Restart the node cleanly, stats should still be around.
$node->restart;
$numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
is($numcalls, '2', 'number of stats after clean restart');
+$fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '1|0|2', 'number of fixed stats after clean restart');
# On crash the stats are gone.
$node->stop('immediate');
$node->start;
$numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
-is($numcalls, '', 'number of stats after clean restart');
+is($numcalls, '', 'number of stats after crash');
+$fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '0|0|0', 'number of fixed stats after crash');
done_testing();
--
2.45.2
On Tue, Jul 09, 2024 at 05:23:03AM +0000, Bertrand Drouvot wrote:
I gave a second thought on it, and I think that this is the "data" part that lead
to the confusion (as too generic), what about?shared_data_len -> shared_stats_len
shared_data_off -> shared_stats_offThat looks ok to me even in the snapshot context (shared is fine after all
because that's where the stats come from).
I'd tend to prefer the original suggestion because of the snapshot
context, actually, as the fixed-numbered stats in a snapshot are a
copy of what's in shmem, and that's not shared at all.
The rename is not the most important part, still if others have an
opinion, feel free.
--
Michael
Hi,
On Tue, Jul 09, 2024 at 03:54:37PM +0900, Michael Paquier wrote:
On Mon, Jul 08, 2024 at 02:07:58PM +0000, Bertrand Drouvot wrote:
It looks pretty straightforward, just one comment:
+ ptr = ((char *) ctl) + kind_info->shared_ctl_off; + kind_info->init_shmem_cb((void *) ptr);I don't think we need to cast ptr to void when calling init_shmem_cb(). Looking
at some examples in the code, it does not look like we cast the argument to void
when a function has (void *) as parameter (also there is examples in 0003 where
it's not done, see next comments for 0003).Yep. Fine by me.
Thanks!
Please find attached a rebased patch set for now, to make the
CF bot happy.
v5-0001 LGTM.
As far v5-0002:
+ goto error;
+ info = pgstat_get_kind_info(kind);
Nit: add an empty line between the two?
Except this Nit, v5-0002 LGTM.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Wed, Jul 10, 2024 at 08:28:56AM +0000, Bertrand Drouvot wrote:
Hi,
On Tue, Jul 09, 2024 at 03:54:37PM +0900, Michael Paquier wrote:
On Mon, Jul 08, 2024 at 02:07:58PM +0000, Bertrand Drouvot wrote:
It looks pretty straightforward, just one comment:
+ ptr = ((char *) ctl) + kind_info->shared_ctl_off; + kind_info->init_shmem_cb((void *) ptr);I don't think we need to cast ptr to void when calling init_shmem_cb(). Looking
at some examples in the code, it does not look like we cast the argument to void
when a function has (void *) as parameter (also there is examples in 0003 where
it's not done, see next comments for 0003).Yep. Fine by me.
Thanks!
Please find attached a rebased patch set for now, to make the
CF bot happy.v5-0001 LGTM.
As far v5-0002:
+ goto error; + info = pgstat_get_kind_info(kind);Nit: add an empty line between the two?
Except this Nit, v5-0002 LGTM.
Oh, and also due to this change in 0002:
switch (t)
{
+ case PGSTAT_FILE_ENTRY_FIXED:
+ {
Then this comment:
/*
* We found an existing statistics file. Read it and put all the hash
* table entries into place.
*/
for (;;)
{
int t = fgetc(fpin);
switch (t)
{
case PGSTAT_FILE_ENTRY_FIXED:
{
is not correct anymore (as we're not reading the stats only into the hash table
anymore).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Wed, Jul 10, 2024 at 09:00:31AM +0000, Bertrand Drouvot wrote:
On Wed, Jul 10, 2024 at 08:28:56AM +0000, Bertrand Drouvot wrote:
v5-0001 LGTM.
Thanks. I've applied this refactoring piece.
/*
* We found an existing statistics file. Read it and put all the hash
* table entries into place.
*/
Indeed. Reworded that slightly and applied it as well.
So we are down to the remaining parts of the patch, and this is going
to need a consensus about a few things because this impacts the
developer experience when implementing one's own custom stats:
- Are folks OK with the point of fixing the kind IDs in time like
RMGRs with a control in the wiki? Or should a more artistic approach
be used like what I am mentioning at the bottom of [1]/messages/by-id/ZoshTO9K7O7Z1wrX@paquier.xyz -- Michael. The patch
allows a range of IDs to be used, to make the access to the stats
faster even if some area of memory may not be used.
- The fixed-numbered custom stats kinds are stored in an array in
PgStat_Snapshot and PgStat_ShmemControl, so as we have something
consistent with the built-in kinds. This makes the tracking of the
validity of the data in the snapshots split into parts of the
structure for builtin and custom kinds. Perhaps there are better
ideas than that? The built-in fixed-numbered kinds have no
redirection.
- The handling of both built-in and custom kinds touches some areas of
pgstat.c and pgstat_shmem.c, which is the minimal I could come up
with.
Attached is a rebased patch set with the remaining pieces.
[1]: /messages/by-id/ZoshTO9K7O7Z1wrX@paquier.xyz -- Michael
--
Michael
Attachments:
v6-0001-Switch-PgStat_Kind-from-enum-to-uint32.patchtext/x-diff; charset=us-asciiDownload
From 2cb5aa81d0d93a910934105756d893d2c0586ea2 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 12:28:26 +0900
Subject: [PATCH v6 1/5] Switch PgStat_Kind from enum to uint32
A follow-up patch is planned to make this counter extensible, and
keeping a trace of the kind behind a type is useful in the internal
routines used by pgstats. While on it, switch pgstat_is_kind_valid() to
use PgStat_Kind, to be more consistent with its callers.
---
src/include/pgstat.h | 35 ++++++++++++++---------------
src/backend/utils/activity/pgstat.c | 6 ++---
2 files changed, 20 insertions(+), 21 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 6b99bb8aad..cc97274708 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -32,26 +32,25 @@
#define PG_STAT_TMP_DIR "pg_stat_tmp"
/* The types of statistics entries */
-typedef enum PgStat_Kind
-{
- /* use 0 for INVALID, to catch zero-initialized data */
- PGSTAT_KIND_INVALID = 0,
+#define PgStat_Kind uint32
- /* stats for variable-numbered objects */
- PGSTAT_KIND_DATABASE, /* database-wide statistics */
- PGSTAT_KIND_RELATION, /* per-table statistics */
- PGSTAT_KIND_FUNCTION, /* per-function statistics */
- PGSTAT_KIND_REPLSLOT, /* per-slot statistics */
- PGSTAT_KIND_SUBSCRIPTION, /* per-subscription statistics */
+/* use 0 for INVALID, to catch zero-initialized data */
+#define PGSTAT_KIND_INVALID 0
- /* stats for fixed-numbered objects */
- PGSTAT_KIND_ARCHIVER,
- PGSTAT_KIND_BGWRITER,
- PGSTAT_KIND_CHECKPOINTER,
- PGSTAT_KIND_IO,
- PGSTAT_KIND_SLRU,
- PGSTAT_KIND_WAL,
-} PgStat_Kind;
+/* stats for variable-numbered objects */
+#define PGSTAT_KIND_DATABASE 1 /* database-wide statistics */
+#define PGSTAT_KIND_RELATION 2 /* per-table statistics */
+#define PGSTAT_KIND_FUNCTION 3 /* per-function statistics */
+#define PGSTAT_KIND_REPLSLOT 4 /* per-slot statistics */
+#define PGSTAT_KIND_SUBSCRIPTION 5 /* per-subscription statistics */
+
+/* stats for fixed-numbered objects */
+#define PGSTAT_KIND_ARCHIVER 6
+#define PGSTAT_KIND_BGWRITER 7
+#define PGSTAT_KIND_CHECKPOINTER 8
+#define PGSTAT_KIND_IO 9
+#define PGSTAT_KIND_SLRU 10
+#define PGSTAT_KIND_WAL 11
#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index c0ec9e8195..805712d414 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -182,7 +182,7 @@ static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
static void pgstat_build_snapshot_fixed(PgStat_Kind kind);
-static inline bool pgstat_is_kind_valid(int ikind);
+static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
/* ----------
@@ -1298,9 +1298,9 @@ pgstat_get_kind_from_str(char *kind_str)
}
static inline bool
-pgstat_is_kind_valid(int ikind)
+pgstat_is_kind_valid(PgStat_Kind kind)
{
- return ikind >= PGSTAT_KIND_FIRST_VALID && ikind <= PGSTAT_KIND_LAST;
+ return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
}
const PgStat_KindInfo *
--
2.45.2
v6-0002-Introduce-pluggable-APIs-for-Cumulative-Statistic.patchtext/x-diff; charset=us-asciiDownload
From 0869c45608b7acb73afa4afc7bf2e481c239d705 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 14:03:17 +0900
Subject: [PATCH v6 2/5] Introduce pluggable APIs for Cumulative Statistics
This commit adds support in the backend for $subject, allowing
out-of-core extensions to add their own custom statistics kinds. The
stats kinds are divided into two parts for efficiency:
- The built-in stats kinds, with designated initializers.
- The custom kinds, able to use a range of IDs (128 slots available as
of this patch), with information saved in TopMemoryContext.
Custom cumulative statistics can only be loaded with
shared_preload_libraries at startup, and must allocate a unique ID
shared across all the PostgreSQL extension ecosystem with the following
wiki page:
https://wiki.postgresql.org/wiki/CustomCumulativeStats
This is able to support fixed-numbered (like WAL, archiver, bgwriter)
and variable-numbered stats kinds.
---
src/include/pgstat.h | 35 +++-
src/include/utils/pgstat_internal.h | 22 ++-
src/backend/utils/activity/pgstat.c | 231 +++++++++++++++++++---
src/backend/utils/activity/pgstat_shmem.c | 31 ++-
src/backend/utils/adt/pgstatfuncs.c | 2 +-
5 files changed, 287 insertions(+), 34 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index cc97274708..21c3705ce4 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -34,6 +34,10 @@
/* The types of statistics entries */
#define PgStat_Kind uint32
+/* Range of IDs allowed, for built-in and custom kinds */
+#define PGSTAT_KIND_MIN 1 /* Minimum ID allowed */
+#define PGSTAT_KIND_MAX 256 /* Maximum ID allowed */
+
/* use 0 for INVALID, to catch zero-initialized data */
#define PGSTAT_KIND_INVALID 0
@@ -52,9 +56,34 @@
#define PGSTAT_KIND_SLRU 10
#define PGSTAT_KIND_WAL 11
-#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
-#define PGSTAT_NUM_KINDS (PGSTAT_KIND_LAST + 1)
+#define PGSTAT_KIND_MIN_BUILTIN PGSTAT_KIND_DATABASE
+#define PGSTAT_KIND_MAX_BUILTIN PGSTAT_KIND_WAL
+
+/* Custom stats kinds */
+
+/* Range of IDs allowed for custom stats kinds */
+#define PGSTAT_KIND_CUSTOM_MIN 128
+#define PGSTAT_KIND_CUSTOM_MAX PGSTAT_KIND_MAX
+#define PGSTAT_KIND_CUSTOM_SIZE (PGSTAT_KIND_CUSTOM_MAX - PGSTAT_KIND_CUSTOM_MIN + 1)
+
+/*
+ * PgStat_Kind to use for extensions that require an ID, but are still in
+ * development and have not reserved their own unique kind ID yet. See:
+ * https://wiki.postgresql.org/wiki/CustomCumulativeStats
+ */
+#define PGSTAT_KIND_EXPERIMENTAL 128
+
+static inline bool
+pgstat_is_kind_builtin(PgStat_Kind kind)
+{
+ return kind > PGSTAT_KIND_INVALID && kind <= PGSTAT_KIND_MAX_BUILTIN;
+}
+
+static inline bool
+pgstat_is_kind_custom(PgStat_Kind kind)
+{
+ return kind >= PGSTAT_KIND_CUSTOM_MIN && kind <= PGSTAT_KIND_CUSTOM_MAX;
+}
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 778f625ca1..39f63362a3 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -195,7 +195,8 @@ typedef struct PgStat_KindInfo
/*
* The size of an entry in the shared stats hash table (pointed to by
- * PgStatShared_HashEntry->body).
+ * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
+ * the size of an entry in PgStat_ShmemControl->custom_data.
*/
uint32 shared_size;
@@ -446,6 +447,13 @@ typedef struct PgStat_ShmemControl
PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
+
+ /*
+ * Custom stats data with fixed-numbered objects, indexed by (PgStat_Kind
+ * - PGSTAT_KIND_CUSTOM_MIN).
+ */
+ void *custom_data[PGSTAT_KIND_CUSTOM_SIZE];
+
} PgStat_ShmemControl;
@@ -459,7 +467,7 @@ typedef struct PgStat_Snapshot
/* time at which snapshot was taken */
TimestampTz snapshot_timestamp;
- bool fixed_valid[PGSTAT_NUM_KINDS];
+ bool fixed_valid[PGSTAT_KIND_MAX_BUILTIN + 1];
PgStat_ArchiverStats archiver;
@@ -473,6 +481,14 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ /*
+ * Data in snapshot for custom fixed-numbered statistics, indexed by
+ * (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
+ * TopMemoryContext, for a size of shared_data_len.
+ */
+ bool custom_valid[PGSTAT_KIND_CUSTOM_SIZE];
+ void *custom_data[PGSTAT_KIND_CUSTOM_SIZE];
+
/* to free snapshot in bulk */
MemoryContext context;
struct pgstat_snapshot_hash *stats;
@@ -516,6 +532,8 @@ static inline void *pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common
*/
extern const PgStat_KindInfo *pgstat_get_kind_info(PgStat_Kind kind);
+extern void pgstat_register_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
#ifdef USE_ASSERT_CHECKING
extern void pgstat_assert_is_up(void);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 805712d414..aceb4bed00 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -49,8 +49,16 @@
* pgStatPending list. Pending statistics updates are flushed out by
* pgstat_report_stat().
*
+ * It is possible for external modules to define custom statistics kinds,
+ * that can use the same properties as any built-in stats kinds. Each custom
+ * stats kind needs to assign a unique ID to ensure that it does not overlap
+ * with other extensions. In order to reserve a unique stats kind ID, refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats.
+ *
* The behavior of different kinds of statistics is determined by the kind's
- * entry in pgstat_kind_infos, see PgStat_KindInfo for details.
+ * entry in pgstat_kind_builtin_infos for all the built-in statistics kinds
+ * defined, and pgstat_kind_custom_infos for custom kinds registered at
+ * startup by pgstat_register_kind(). See PgStat_KindInfo for details.
*
* The consistency of read accesses to statistics can be configured using the
* stats_fetch_consistency GUC (see config.sgml and monitoring.sgml for the
@@ -174,6 +182,8 @@ typedef struct PgStat_SnapshotEntry
static void pgstat_write_statsfile(void);
static void pgstat_read_statsfile(void);
+static void pgstat_init_snapshot_fixed(void);
+
static void pgstat_reset_after_failure(void);
static bool pgstat_flush_pending_entries(bool nowait);
@@ -251,7 +261,7 @@ static bool pgstat_is_shutdown = false;
/*
- * The different kinds of statistics.
+ * The different kinds of built-in statistics.
*
* If reasonably possible, handling specific to one kind of stats should go
* through this abstraction, rather than making more of pgstat.c aware.
@@ -263,7 +273,7 @@ static bool pgstat_is_shutdown = false;
* seem to be a great way of doing that, given the split across multiple
* files.
*/
-static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
+static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_MAX_BUILTIN + 1] = {
/* stats kinds for variable-numbered objects */
@@ -436,6 +446,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
},
};
+/*
+ * Information about custom statistics kinds.
+ *
+ * These are saved in a different array than the built-in kinds to save
+ * in clarity with the initializations.
+ *
+ * Indexed by PGSTAT_KIND_CUSTOM_MIN, of size PGSTAT_KIND_CUSTOM_SIZE.
+ */
+static const PgStat_KindInfo **pgstat_kind_custom_infos = NULL;
/* ------------------------------------------------------------
* Functions managing the state of the stats system for all backends.
@@ -586,6 +605,8 @@ pgstat_initialize(void)
pgstat_init_wal();
+ pgstat_init_snapshot_fixed();
+
/* Set up a process-exit hook to clean up */
before_shmem_exit(pgstat_shutdown_hook, 0);
@@ -829,6 +850,8 @@ pgstat_clear_snapshot(void)
memset(&pgStatLocal.snapshot.fixed_valid, 0,
sizeof(pgStatLocal.snapshot.fixed_valid));
+ memset(&pgStatLocal.snapshot.custom_valid, 0,
+ sizeof(pgStatLocal.snapshot.custom_valid));
pgStatLocal.snapshot.stats = NULL;
pgStatLocal.snapshot.mode = PGSTAT_FETCH_CONSISTENCY_NONE;
@@ -992,7 +1015,29 @@ pgstat_snapshot_fixed(PgStat_Kind kind)
else
pgstat_build_snapshot_fixed(kind);
- Assert(pgStatLocal.snapshot.fixed_valid[kind]);
+ if (pgstat_is_kind_builtin(kind))
+ Assert(pgStatLocal.snapshot.fixed_valid[kind]);
+ else if (pgstat_is_kind_custom(kind))
+ Assert(pgStatLocal.snapshot.custom_valid[kind - PGSTAT_KIND_CUSTOM_MIN]);
+}
+
+static void
+pgstat_init_snapshot_fixed(void)
+{
+ /*
+ * Initialize fixed-numbered statistics data in snapshots, only for custom
+ * stats kinds.
+ */
+ for (int kind = PGSTAT_KIND_CUSTOM_MIN; kind <= PGSTAT_KIND_CUSTOM_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->fixed_amount)
+ continue;
+
+ pgStatLocal.snapshot.custom_data[kind - PGSTAT_KIND_CUSTOM_MIN] =
+ MemoryContextAlloc(TopMemoryContext, kind_info->shared_data_len);
+ }
}
static void
@@ -1088,10 +1133,12 @@ pgstat_build_snapshot(void)
/*
* Build snapshot of all fixed-numbered stats.
*/
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ if (!kind_info)
+ continue;
if (!kind_info->fixed_amount)
{
Assert(kind_info->snapshot_cb == NULL);
@@ -1108,6 +1155,20 @@ static void
pgstat_build_snapshot_fixed(PgStat_Kind kind)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ int idx;
+ bool *valid;
+
+ /* Position in fixed_valid or custom_valid */
+ if (pgstat_is_kind_builtin(kind))
+ {
+ idx = kind;
+ valid = pgStatLocal.snapshot.fixed_valid;
+ }
+ else
+ {
+ idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+ valid = pgStatLocal.snapshot.custom_valid;
+ }
Assert(kind_info->fixed_amount);
Assert(kind_info->snapshot_cb != NULL);
@@ -1115,21 +1176,21 @@ pgstat_build_snapshot_fixed(PgStat_Kind kind)
if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE)
{
/* rebuild every time */
- pgStatLocal.snapshot.fixed_valid[kind] = false;
+ valid[idx] = false;
}
- else if (pgStatLocal.snapshot.fixed_valid[kind])
+ else if (valid[idx])
{
/* in snapshot mode we shouldn't get called again */
Assert(pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_CACHE);
return;
}
- Assert(!pgStatLocal.snapshot.fixed_valid[kind]);
+ Assert(!valid[idx]);
kind_info->snapshot_cb();
- Assert(!pgStatLocal.snapshot.fixed_valid[kind]);
- pgStatLocal.snapshot.fixed_valid[kind] = true;
+ Assert(!valid[idx]);
+ valid[idx] = true;
}
@@ -1285,30 +1346,127 @@ pgstat_flush_pending_entries(bool nowait)
PgStat_Kind
pgstat_get_kind_from_str(char *kind_str)
{
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN_BUILTIN; kind <= PGSTAT_KIND_MAX_BUILTIN; kind++)
{
- if (pg_strcasecmp(kind_str, pgstat_kind_infos[kind].name) == 0)
+ if (pg_strcasecmp(kind_str, pgstat_kind_builtin_infos[kind].name) == 0)
return kind;
}
+ /* Check the custom set of cumulative stats */
+ if (pgstat_kind_custom_infos)
+ {
+ for (int kind = 0; kind < PGSTAT_KIND_CUSTOM_SIZE; kind++)
+ {
+ if (pgstat_kind_custom_infos[kind] &&
+ pg_strcasecmp(kind_str, pgstat_kind_custom_infos[kind]->name) == 0)
+ return kind + PGSTAT_KIND_CUSTOM_MIN;
+ }
+ }
+
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid statistics kind: \"%s\"", kind_str)));
- return PGSTAT_KIND_DATABASE; /* avoid compiler warnings */
+ return PGSTAT_KIND_INVALID; /* avoid compiler warnings */
}
static inline bool
pgstat_is_kind_valid(PgStat_Kind kind)
{
- return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
+ return pgstat_is_kind_builtin(kind) || pgstat_is_kind_custom(kind);
}
const PgStat_KindInfo *
pgstat_get_kind_info(PgStat_Kind kind)
{
- Assert(pgstat_is_kind_valid(kind));
+ if (pgstat_is_kind_builtin(kind))
+ return &pgstat_kind_builtin_infos[kind];
- return &pgstat_kind_infos[kind];
+ if (pgstat_is_kind_custom(kind))
+ {
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (pgstat_kind_custom_infos == NULL ||
+ pgstat_kind_custom_infos[idx] == NULL)
+ return NULL;
+ return pgstat_kind_custom_infos[idx];
+ }
+
+ return NULL;
+}
+
+/*
+ * Register a new stats kind.
+ *
+ * PgStat_Kinds must be globally unique across all extensions. Refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats to reserve a
+ * unique ID for your extension, to avoid conflicts with other extension
+ * developers. During development, use PGSTAT_KIND_EXPERIMENTAL to avoid
+ * needlessly reserving a new ID.
+ */
+void
+pgstat_register_kind(PgStat_Kind kind, const PgStat_KindInfo *kind_info)
+{
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (kind_info->name == NULL || strlen(kind_info->name) == 0)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics name is invalid"),
+ errhint("Provide a non-empty name for the custom cumulative statistics.")));
+
+ if (!pgstat_is_kind_custom(kind))
+ ereport(ERROR, (errmsg("custom cumulative statistics ID %u is out of range", kind),
+ errhint("Provide a custom cumulative statistics ID between %u and %u.",
+ PGSTAT_KIND_CUSTOM_MIN, PGSTAT_KIND_CUSTOM_MAX)));
+
+ if (!process_shared_preload_libraries_in_progress)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics must be registered while initializing modules in \"shared_preload_libraries\".")));
+
+ /*
+ * Check some data for fixed-numbered stats.
+ */
+ if (kind_info->fixed_amount)
+ {
+ if (kind_info->shared_size == 0)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics property is invalid"),
+ errhint("Custom cumulative statistics require a shared memory size for fixed-numbered objects.")));
+ }
+
+ /*
+ * If pgstat_kind_custom_infos is not available yet, allocate it.
+ */
+ if (pgstat_kind_custom_infos == NULL)
+ {
+ pgstat_kind_custom_infos = (const PgStat_KindInfo **)
+ MemoryContextAllocZero(TopMemoryContext,
+ sizeof(PgStat_KindInfo *) * PGSTAT_KIND_CUSTOM_SIZE);
+ }
+
+ if (pgstat_kind_custom_infos[idx] != NULL &&
+ pgstat_kind_custom_infos[idx]->name != NULL)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics \"%s\" already registered with the same ID.",
+ pgstat_kind_custom_infos[idx]->name)));
+
+ /* check for existing custom stats with the same name */
+ for (int existing_kind = 0; existing_kind < PGSTAT_KIND_CUSTOM_SIZE; existing_kind++)
+ {
+ if (pgstat_kind_custom_infos[existing_kind] == NULL)
+ continue;
+ if (!pg_strcasecmp(pgstat_kind_custom_infos[existing_kind]->name, kind_info->name))
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Existing cumulative statistics with ID %u has the same name.", existing_kind + PGSTAT_KIND_CUSTOM_MIN)));
+ }
+
+ /* Register it */
+ pgstat_kind_custom_infos[idx] = kind_info;
+ ereport(LOG,
+ (errmsg("registered custom cumulative statistics \"%s\" with ID %u",
+ kind_info->name, kind)));
}
/*
@@ -1385,18 +1543,22 @@ pgstat_write_statsfile(void)
write_chunk_s(fpout, &format_id);
/* Write various stats structs for fixed number of objects */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
char *ptr;
const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
- if (!info->fixed_amount)
+ if (!info || !info->fixed_amount)
continue;
- Assert(info->snapshot_ctl_off != 0);
+ if (pgstat_is_kind_builtin(kind))
+ Assert(info->snapshot_ctl_off != 0);
pgstat_build_snapshot_fixed(kind);
- ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+ else
+ ptr = pgStatLocal.snapshot.custom_data[kind - PGSTAT_KIND_CUSTOM_MIN];
fputc(PGSTAT_FILE_ENTRY_FIXED, fpout);
write_chunk_s(fpout, &kind);
@@ -1419,6 +1581,17 @@ pgstat_write_statsfile(void)
if (ps->dropped)
continue;
+ /*
+ * This discards data related to custom stats kinds that are unknown
+ * to this process.
+ */
+ if (!pgstat_is_kind_valid(ps->key.kind))
+ {
+ elog(WARNING, "found unknown stats entry %u/%u/%u",
+ ps->key.kind, ps->key.dboid, ps->key.objoid);
+ continue;
+ }
+
shstats = (PgStatShared_Common *) dsa_get_address(pgStatLocal.dsa, ps->body);
kind_info = pgstat_get_kind_info(ps->key.kind);
@@ -1570,8 +1743,16 @@ pgstat_read_statsfile(void)
goto error;
/* Load back stats into shared memory */
- ptr = ((char *) shmem) + info->shared_ctl_off +
- info->shared_data_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) shmem) + info->shared_ctl_off +
+ info->shared_data_off;
+ else
+ {
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ ptr = ((char *) shmem->custom_data[idx]) +
+ info->shared_data_off;
+ }
if (!read_chunk(fpin, ptr, info->shared_data_len))
goto error;
@@ -1638,7 +1819,7 @@ pgstat_read_statsfile(void)
if (found)
{
dshash_release_lock(pgStatLocal.shared_hash, p);
- elog(WARNING, "found duplicate stats entry %d/%u/%u",
+ elog(WARNING, "found duplicate stats entry %u/%u/%u",
key.kind, key.dboid, key.objoid);
goto error;
}
@@ -1696,11 +1877,11 @@ pgstat_reset_after_failure(void)
TimestampTz ts = GetCurrentTimestamp();
/* reset fixed-numbered stats */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
kind_info->reset_all_cb(ts);
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 1c2b69d563..ca3d3eecfb 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -131,6 +131,21 @@ StatsShmemSize(void)
sz = MAXALIGN(sizeof(PgStat_ShmemControl));
sz = add_size(sz, pgstat_dsa_init_size());
+ /* Add shared memory for all the custom fixed-numbered statistics */
+ for (int kind = PGSTAT_KIND_CUSTOM_MIN; kind <= PGSTAT_KIND_CUSTOM_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info)
+ continue;
+ if (!kind_info->fixed_amount)
+ continue;
+
+ Assert(kind_info->shared_size != 0);
+
+ sz += MAXALIGN(kind_info->shared_size);
+ }
+
return sz;
}
@@ -197,15 +212,25 @@ StatsShmemInit(void)
pg_atomic_init_u64(&ctl->gc_request_count, 1);
/* initialize fixed-numbered stats */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
char *ptr;
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
- ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ else
+ {
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ Assert(kind_info->shared_size != 0);
+ ctl->custom_data[idx] = ShmemAlloc(kind_info->shared_size);
+ ptr = ctl->custom_data[idx];
+ }
+
kind_info->init_shmem_cb(ptr);
}
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 3876339ee1..3221137123 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1696,7 +1696,7 @@ pg_stat_reset(PG_FUNCTION_ARGS)
* Reset some shared cluster-wide counters
*
* When adding a new reset target, ideally the name should match that in
- * pgstat_kind_infos, if relevant.
+ * pgstat_kind_builtin_infos, if relevant.
*/
Datum
pg_stat_reset_shared(PG_FUNCTION_ARGS)
--
2.45.2
v6-0003-doc-Add-section-for-Custom-Cumulative-Statistics-.patchtext/x-diff; charset=us-asciiDownload
From 6081492b5d0e603496dcf6edfe667bd431d8b86f Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 3 Jul 2024 13:49:53 +0900
Subject: [PATCH v6 3/5] doc: Add section for Custom Cumulative Statistics APIs
This provides a short description of what can be done, with a pointer to
the template in the tree.
---
doc/src/sgml/xfunc.sgml | 63 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 63 insertions(+)
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index 756a9d07fb..878722af51 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3700,6 +3700,69 @@ extern bool InjectionPointDetach(const char *name);
</para>
</sect2>
+ <sect2 id="xfunc-addin-custom-cumulative-statistics">
+ <title>Custom Cumulative Statistics</title>
+
+ <para>
+ Is is possible for add-ins written in C-language to use custom types
+ of cumulative statistics registered in the
+ <link linkend="monitoring-stats-setup">Cumulative Statistics System</link>.
+ </para>
+
+ <para>
+ First, define a <literal>PgStat_KindInfo</literal> that includes all
+ the information related to the custom type registered. For example:
+<programlisting>
+static const PgStat_KindInfo custom_stats = {
+ .name = "custom_stats",
+ .fixed_amount = false,
+ .shared_size = sizeof(PgStatShared_Custom),
+ .shared_data_off = offsetof(PgStatShared_Custom, stats),
+ .shared_data_len = sizeof(((PgStatShared_Custom *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatCustomEntry),
+}
+</programlisting>
+
+ Then, each backend that needs to use this custom type needs to register
+ it with <literal>pgstat_register_kind</literal> and a unique ID used to
+ store the entries related to this type of statistics:
+<programlisting>
+extern PgStat_Kind pgstat_add_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
+</programlisting>
+ While developing a new extension, use
+ <literal>PGSTAT_KIND_EXPERIMENTAL</literal> for
+ <parameter>kind</parameter>. When you are ready to release the extension
+ to users, reserve a kind ID at the
+ <ulink url="https://wiki.postgresql.org/wiki/CustomCumulativeStats">
+ Custom Cumulative Statistics</ulink> page.
+ </para>
+
+ <para>
+ The details of the API for <literal>PgStat_KindInfo</literal> can
+ be found in <filename>src/include/utils/pgstat_internal.h</filename>.
+ </para>
+
+ <para>
+ The type of statistics registered is associated with a name and a unique
+ ID shared across the server in shared memory. Each backend using a
+ custom type of statistics maintains a local cache storing the information
+ of each custom <literal>PgStat_KindInfo</literal>.
+ </para>
+
+ <para>
+ Place the extension module implementing the custom cumulative statistics
+ type in <xref linkend="guc-shared-preload-libraries"/> so that it will
+ be loaded early during <productname>PostgreSQL</productname> startup.
+ </para>
+
+ <para>
+ An example describing how to register and use custom statistics can be
+ found in
+ <filename>src/test/modules/injection_points/injection_stats.c</filename>.
+ </para>
+ </sect2>
+
<sect2 id="extend-cpp">
<title>Using C++ for Extensibility</title>
--
2.45.2
v6-0004-injection_points-Add-statistics-for-custom-points.patchtext/x-diff; charset=us-asciiDownload
From ae6acc8de61ebd9e1c040b46c859b4d075f70e2a Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 13:36:06 +0900
Subject: [PATCH v6 4/5] injection_points: Add statistics for custom points
This acts as a template of what can be achieved with the pluggable
cumulative stats APIs, while being useful on its own for injection
points.
Currently, the only data gathered is the number of times an injection
point is called. This can be extended as required. All the routines
related to the stats are located in their own file, for clarity.
A TAP test is included to provide coverage for these new APIs, showing
the persistency of the data across restarts.
---
src/test/modules/injection_points/Makefile | 11 +-
.../injection_points--1.0.sql | 10 +
.../injection_points/injection_points.c | 27 +++
.../injection_points/injection_stats.c | 194 ++++++++++++++++++
.../injection_points/injection_stats.h | 23 +++
src/test/modules/injection_points/meson.build | 9 +
.../modules/injection_points/t/001_stats.pl | 48 +++++
src/tools/pgindent/typedefs.list | 2 +
8 files changed, 322 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/injection_points/injection_stats.c
create mode 100644 src/test/modules/injection_points/injection_stats.h
create mode 100644 src/test/modules/injection_points/t/001_stats.pl
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 2ffd2f77ed..7b9cd12a2a 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -1,7 +1,10 @@
# src/test/modules/injection_points/Makefile
-MODULES = injection_points
-
+MODULE_big = injection_points
+OBJS = \
+ $(WIN32RES) \
+ injection_points.o \
+ injection_stats.o
EXTENSION = injection_points
DATA = injection_points--1.0.sql
PGFILEDESC = "injection_points - facility for injection points"
@@ -11,9 +14,13 @@ REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
ISOLATION = inplace
+TAP_TESTS = 1
+
# The injection points are cluster-wide, so disable installcheck
NO_INSTALLCHECK = 1
+export enable_injection_points enable_injection_points
+
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index e275c2cf5b..747a64e812 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -64,3 +64,13 @@ CREATE FUNCTION injection_points_detach(IN point_name TEXT)
RETURNS void
AS 'MODULE_PATHNAME', 'injection_points_detach'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+--
+-- injection_points_stats_numcalls()
+--
+-- Reports statistics, if any, related to the given injection point.
+--
+CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
+RETURNS bigint
+AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index b6c8e89324..eb411b9d44 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -18,6 +18,7 @@
#include "postgres.h"
#include "fmgr.h"
+#include "injection_stats.h"
#include "miscadmin.h"
#include "nodes/pg_list.h"
#include "nodes/value.h"
@@ -170,6 +171,9 @@ injection_points_cleanup(int code, Datum arg)
char *name = strVal(lfirst(lc));
(void) InjectionPointDetach(name);
+
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
}
}
@@ -182,6 +186,8 @@ injection_error(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(ERROR, "error triggered for injection point %s", name);
}
@@ -193,6 +199,8 @@ injection_notice(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(NOTICE, "notice triggered for injection point %s", name);
}
@@ -211,6 +219,8 @@ injection_wait(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
/*
* Use the injection point name for this custom wait event. Note that
* this custom wait event name is not released, but we don't care much for
@@ -299,6 +309,10 @@ injection_points_attach(PG_FUNCTION_ARGS)
inj_list_local = lappend(inj_list_local, makeString(pstrdup(name)));
MemoryContextSwitchTo(oldctx);
}
+
+ /* Add entry for stats */
+ pgstat_create_inj(name);
+
PG_RETURN_VOID();
}
@@ -417,5 +431,18 @@ injection_points_detach(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldctx);
}
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
+
PG_RETURN_VOID();
}
+
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ pgstat_register_inj();
+}
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
new file mode 100644
index 0000000000..c37b0b33d3
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -0,0 +1,194 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.c
+ * Code for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+
+#include "common/hashfn.h"
+#include "injection_stats.h"
+#include "pgstat.h"
+#include "utils/builtins.h"
+#include "utils/pgstat_internal.h"
+
+/* Structures for statistics of injection points */
+typedef struct PgStat_StatInjEntry
+{
+ PgStat_Counter numcalls; /* number of times point has been run */
+} PgStat_StatInjEntry;
+
+typedef struct PgStatShared_InjectionPoint
+{
+ PgStatShared_Common header;
+ PgStat_StatInjEntry stats;
+} PgStatShared_InjectionPoint;
+
+static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+
+static const PgStat_KindInfo injection_stats = {
+ .name = "injection_points",
+ .fixed_amount = false, /* Bounded by the number of points */
+
+ /* Injection points are system-wide */
+ .accessed_across_databases = true,
+
+ .shared_size = sizeof(PgStatShared_InjectionPoint),
+ .shared_data_off = offsetof(PgStatShared_InjectionPoint, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPoint *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatInjEntry),
+ .flush_pending_cb = injection_stats_flush_cb,
+};
+
+/*
+ * Compute stats entry idx from point name with a 4-byte hash.
+ */
+#define PGSTAT_INJ_IDX(name) hash_bytes((const unsigned char *) name, strlen(name))
+
+#define PGSTAT_KIND_INJECTION PGSTAT_KIND_EXPERIMENTAL
+
+/* Track if stats are loaded */
+static bool inj_stats_loaded = false;
+
+/*
+ * Callback for stats handling
+ */
+static bool
+injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ PgStat_StatInjEntry *localent;
+ PgStatShared_InjectionPoint *shfuncent;
+
+ localent = (PgStat_StatInjEntry *) entry_ref->pending;
+ shfuncent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ shfuncent->stats.numcalls += localent->numcalls;
+ return true;
+}
+
+/*
+ * Support function for the SQL-callable pgstat* functions. Returns
+ * a pointer to the injection point statistics struct.
+ */
+static PgStat_StatInjEntry *
+pgstat_fetch_stat_injentry(const char *name)
+{
+ PgStat_StatInjEntry *entry = NULL;
+
+ if (!inj_stats_loaded)
+ return NULL;
+
+ /* Compile the lookup key as a hash of the point name */
+ entry = (PgStat_StatInjEntry *) pgstat_fetch_entry(PGSTAT_KIND_INJECTION,
+ InvalidOid,
+ PGSTAT_INJ_IDX(name));
+ return entry;
+}
+
+/*
+ * Workhorse to do the registration work, called in _PG_init().
+ */
+void
+pgstat_register_inj(void)
+{
+ pgstat_register_kind(PGSTAT_KIND_INJECTION, &injection_stats);
+
+ /* mark stats as loaded */
+ inj_stats_loaded = true;
+}
+
+/*
+ * Report injection point creation.
+ */
+void
+pgstat_create_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ /* initialize shared memory data */
+ memset(&shstatent->stats, 0, sizeof(shstatent->stats));
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * Report injection point drop.
+ */
+void
+pgstat_drop_inj(const char *name)
+{
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ if (!pgstat_drop_entry(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name)))
+ pgstat_request_entry_refs_gc();
+}
+
+/*
+ * Report statistics for injection point.
+ *
+ * This is simple because the set of stats to report currently is simple:
+ * track the number of times a point has been run.
+ */
+void
+pgstat_report_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+ PgStat_StatInjEntry *statent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+ statent = &shstatent->stats;
+
+ /* Update the injection point statistics */
+ statent->numcalls++;
+
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * SQL function returning the number of times an injection point
+ * has been called.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_numcalls);
+Datum
+injection_points_stats_numcalls(PG_FUNCTION_ARGS)
+{
+ char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_StatInjEntry *entry = pgstat_fetch_stat_injentry(name);
+
+ if (entry == NULL)
+ PG_RETURN_NULL();
+
+ PG_RETURN_INT64(entry->numcalls);
+}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
new file mode 100644
index 0000000000..3e99705483
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -0,0 +1,23 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.h
+ * Definitions for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.h
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#ifndef INJECTION_STATS
+#define INJECTION_STATS
+
+extern void pgstat_register_inj(void);
+extern void pgstat_create_inj(const char *name);
+extern void pgstat_drop_inj(const char *name);
+extern void pgstat_report_inj(const char *name);
+
+#endif
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index 3c23c14d81..a52fe5121e 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -6,6 +6,7 @@ endif
injection_points_sources = files(
'injection_points.c',
+ 'injection_stats.c',
)
if host_system == 'windows'
@@ -42,4 +43,12 @@ tests += {
'inplace',
],
},
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_stats.pl',
+ ],
+ },
}
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
new file mode 100644
index 0000000000..7d5a96e522
--- /dev/null
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -0,0 +1,48 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+use locale;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Test persistency of statistics generated for injection points.
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# Node initialization
+my $node = PostgreSQL::Test::Cluster->new('master');
+$node->init;
+$node->append_conf('postgresql.conf',
+ "shared_preload_libraries = 'injection_points'");
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# This should count for two calls.
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('stats-notice', 'notice');");
+$node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
+$node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
+my $numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats calls');
+
+# Restart the node cleanly, stats should still be around.
+$node->restart;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats after clean restart');
+
+# On crash the stats are gone.
+$node->stop('immediate');
+$node->start;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '', 'number of stats after clean restart');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 635e6d6e21..5206029e96 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2118,6 +2118,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_InjectionPoint
PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
@@ -2149,6 +2150,7 @@ PgStat_Snapshot
PgStat_SnapshotEntry
PgStat_StatDBEntry
PgStat_StatFuncEntry
+PgStat_StatInjEntry
PgStat_StatReplSlotEntry
PgStat_StatSubEntry
PgStat_StatTabEntry
--
2.45.2
v6-0005-injection_points-Add-example-for-fixed-numbered-s.patchtext/x-diff; charset=us-asciiDownload
From f3a4855aa5356c7d21f656f37d8799b91ba85b52 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 13:44:14 +0900
Subject: [PATCH v6 5/5] injection_points: Add example for fixed-numbered
statistics
This acts as a template to show what can be achieved with fixed-numbered
stats (like WAL, bgwriter, etc.) for pluggable cumulative statistics.
---
.../injection_points--1.0.sql | 11 ++
.../injection_points/injection_points.c | 3 +
.../injection_points/injection_stats.c | 155 +++++++++++++++++-
.../injection_points/injection_stats.h | 3 +
.../modules/injection_points/t/001_stats.pl | 11 +-
5 files changed, 180 insertions(+), 3 deletions(-)
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index 747a64e812..fa0b1d06ae 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -74,3 +74,14 @@ CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
RETURNS bigint
AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
LANGUAGE C STRICT;
+
+--
+-- injection_points_stats_fixed()
+--
+-- Reports fixed-numbered statistics for injection points.
+CREATE FUNCTION injection_points_stats_fixed(OUT numattach int8,
+ OUT numdetach int8,
+ OUT numrun int8)
+RETURNS record
+AS 'MODULE_PATHNAME', 'injection_points_stats_fixed'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index eb411b9d44..f65bd83cc3 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -297,6 +297,7 @@ injection_points_attach(PG_FUNCTION_ARGS)
condition.pid = MyProcPid;
}
+ pgstat_report_inj_fixed(1, 0, 0);
InjectionPointAttach(name, "injection_points", function, &condition,
sizeof(InjectionPointCondition));
@@ -342,6 +343,7 @@ injection_points_run(PG_FUNCTION_ARGS)
{
char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ pgstat_report_inj_fixed(0, 0, 1);
INJECTION_POINT(name);
PG_RETURN_VOID();
@@ -418,6 +420,7 @@ injection_points_detach(PG_FUNCTION_ARGS)
{
char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ pgstat_report_inj_fixed(0, 1, 0);
if (!InjectionPointDetach(name))
elog(ERROR, "could not detach injection point \"%s\"", name);
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
index c37b0b33d3..69560b7fdc 100644
--- a/src/test/modules/injection_points/injection_stats.c
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -17,12 +17,13 @@
#include "fmgr.h"
#include "common/hashfn.h"
+#include "funcapi.h"
#include "injection_stats.h"
#include "pgstat.h"
#include "utils/builtins.h"
#include "utils/pgstat_internal.h"
-/* Structures for statistics of injection points */
+/* Structures for statistics of injection points, variable-size */
typedef struct PgStat_StatInjEntry
{
PgStat_Counter numcalls; /* number of times point has been run */
@@ -35,6 +36,9 @@ typedef struct PgStatShared_InjectionPoint
} PgStatShared_InjectionPoint;
static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+static void injection_stats_fixed_init_shmem_cb(void *stats);
+static void injection_stats_fixed_reset_all_cb(TimestampTz ts);
+static void injection_stats_fixed_snapshot_cb(void);
static const PgStat_KindInfo injection_stats = {
.name = "injection_points",
@@ -50,18 +54,51 @@ static const PgStat_KindInfo injection_stats = {
.flush_pending_cb = injection_stats_flush_cb,
};
+/* Structures for statistics of injection points, fixed-size */
+typedef struct PgStat_StatInjFixedEntry
+{
+ PgStat_Counter numattach; /* number of points attached */
+ PgStat_Counter numdetach; /* number of points detached */
+ PgStat_Counter numrun; /* number of points run */
+ TimestampTz stat_reset_timestamp;
+} PgStat_StatInjFixedEntry;
+
+typedef struct PgStatShared_InjectionPointFixed
+{
+ LWLock lock; /* protects all the counters */
+ uint32 changecount;
+ PgStat_StatInjFixedEntry stats;
+ PgStat_StatInjFixedEntry reset_offset;
+} PgStatShared_InjectionPointFixed;
+
+static const PgStat_KindInfo injection_stats_fixed = {
+ .name = "injection_points_fixed",
+ .fixed_amount = true,
+
+ .shared_size = sizeof(PgStat_StatInjFixedEntry),
+ .shared_data_off = offsetof(PgStatShared_InjectionPointFixed, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPointFixed *) 0)->stats),
+
+ .init_shmem_cb = injection_stats_fixed_init_shmem_cb,
+ .reset_all_cb = injection_stats_fixed_reset_all_cb,
+ .snapshot_cb = injection_stats_fixed_snapshot_cb,
+};
+
/*
* Compute stats entry idx from point name with a 4-byte hash.
*/
#define PGSTAT_INJ_IDX(name) hash_bytes((const unsigned char *) name, strlen(name))
#define PGSTAT_KIND_INJECTION PGSTAT_KIND_EXPERIMENTAL
+#define PGSTAT_KIND_INJECTION_FIXED (PGSTAT_KIND_EXPERIMENTAL + 1)
+/* Position of fixed-numbered data in internal structures */
+#define PGSTAT_KIND_INJECTION_IDX (PGSTAT_KIND_INJECTION_FIXED - PGSTAT_KIND_CUSTOM_MIN)
/* Track if stats are loaded */
static bool inj_stats_loaded = false;
/*
- * Callback for stats handling
+ * Callbacks for stats handling
*/
static bool
injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
@@ -79,6 +116,59 @@ injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
return true;
}
+static void
+injection_stats_fixed_init_shmem_cb(void *stats)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+static void
+injection_stats_fixed_reset_all_cb(TimestampTz ts)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *)
+ pgStatLocal.shmem->custom_data[PGSTAT_KIND_INJECTION_IDX];
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ pgstat_copy_changecounted_stats(&stats_shmem->reset_offset,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+ stats_shmem->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&stats_shmem->lock);
+}
+
+static void
+injection_stats_fixed_snapshot_cb(void)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *)
+ pgStatLocal.shmem->custom_data[PGSTAT_KIND_INJECTION_IDX];
+ PgStat_StatInjFixedEntry *stat_snap = (PgStat_StatInjFixedEntry *)
+ pgStatLocal.snapshot.custom_data[PGSTAT_KIND_INJECTION_IDX];
+ PgStat_StatInjFixedEntry *reset_offset = &stats_shmem->reset_offset;
+ PgStat_StatInjFixedEntry reset;
+
+ pgstat_copy_changecounted_stats(stat_snap,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+
+ LWLockAcquire(&stats_shmem->lock, LW_SHARED);
+ memcpy(&reset, reset_offset, sizeof(stats_shmem->stats));
+ LWLockRelease(&stats_shmem->lock);
+
+ /* compensate by reset offsets */
+#define FIXED_COMP(fld) stat_snap->fld -= reset.fld;
+ FIXED_COMP(numattach);
+ FIXED_COMP(numdetach);
+ FIXED_COMP(numrun);
+#undef FIXED_COMP
+}
+
/*
* Support function for the SQL-callable pgstat* functions. Returns
* a pointer to the injection point statistics struct.
@@ -105,6 +195,7 @@ void
pgstat_register_inj(void)
{
pgstat_register_kind(PGSTAT_KIND_INJECTION, &injection_stats);
+ pgstat_register_kind(PGSTAT_KIND_INJECTION_FIXED, &injection_stats_fixed);
/* mark stats as loaded */
inj_stats_loaded = true;
@@ -176,6 +267,30 @@ pgstat_report_inj(const char *name)
pgstat_unlock_entry(entry_ref);
}
+/*
+ * Report fixed number of statistics for an injection point.
+ */
+void
+pgstat_report_inj_fixed(uint32 numattach,
+ uint32 numdetach,
+ uint32 numrun)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ stats_shmem = (PgStatShared_InjectionPointFixed *)
+ pgStatLocal.shmem->custom_data[PGSTAT_KIND_INJECTION_IDX];
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numattach += numattach;
+ stats_shmem->stats.numdetach += numdetach;
+ stats_shmem->stats.numrun += numrun;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+}
+
/*
* SQL function returning the number of times an injection point
* has been called.
@@ -192,3 +307,39 @@ injection_points_stats_numcalls(PG_FUNCTION_ARGS)
PG_RETURN_INT64(entry->numcalls);
}
+
+/*
+ * SQL function returning fixed-numbered statistics for injection points.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_fixed);
+Datum
+injection_points_stats_fixed(PG_FUNCTION_ARGS)
+{
+ TupleDesc tupdesc;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ PgStat_StatInjFixedEntry *stats;
+
+ pgstat_snapshot_fixed(PGSTAT_KIND_INJECTION_FIXED);
+ stats = pgStatLocal.snapshot.custom_data[PGSTAT_KIND_INJECTION_IDX];
+
+ /* Initialise attributes information in the tuple descriptor */
+ tupdesc = CreateTemplateTupleDesc(3);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "numattach",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "numdetach",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "numrun",
+ INT8OID, -1, 0);
+ BlessTupleDesc(tupdesc);
+
+ values[0] = Int64GetDatum(stats->numattach);
+ values[1] = Int64GetDatum(stats->numdetach);
+ values[2] = Int64GetDatum(stats->numrun);
+ nulls[0] = false;
+ nulls[1] = false;
+ nulls[2] = false;
+
+ /* Returns the record as Datum */
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
index 3e99705483..cf68b25f7b 100644
--- a/src/test/modules/injection_points/injection_stats.h
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -19,5 +19,8 @@ extern void pgstat_register_inj(void);
extern void pgstat_create_inj(const char *name);
extern void pgstat_drop_inj(const char *name);
extern void pgstat_report_inj(const char *name);
+extern void pgstat_report_inj_fixed(uint32 numattach,
+ uint32 numdetach,
+ uint32 numrun);
#endif
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
index 7d5a96e522..e3c69b94ca 100644
--- a/src/test/modules/injection_points/t/001_stats.pl
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -31,18 +31,27 @@ $node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
my $numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
is($numcalls, '2', 'number of stats calls');
+my $fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '1|0|2', 'number of fixed stats');
# Restart the node cleanly, stats should still be around.
$node->restart;
$numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
is($numcalls, '2', 'number of stats after clean restart');
+$fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '1|0|2', 'number of fixed stats after clean restart');
# On crash the stats are gone.
$node->stop('immediate');
$node->start;
$numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
-is($numcalls, '', 'number of stats after clean restart');
+is($numcalls, '', 'number of stats after crash');
+$fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '0|0|0', 'number of fixed stats after crash');
done_testing();
--
2.45.2
Hi,
On Fri, Jul 05, 2024 at 09:35:19AM +0900, Michael Paquier wrote:
On Thu, Jul 04, 2024 at 11:30:17AM +0000, Bertrand Drouvot wrote:
/*
* Reads in existing statistics file into the shared stats hash.This comment above pgstat_read_statsfile() is not correct, fixed stats
are not going to the hash (was there before your patch though).Good catch. Let's adjust that separately.
Please find attached a patch to do so (attached as .txt to not perturb the
cfbot).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v1-0001-Fix-comment-on-top-of-pgstat_read_statsfile.txttext/plain; charset=us-asciiDownload
From 67fdfa3eeef3e089a7b99b97f47f6c1e64b501cf Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Thu, 11 Jul 2024 12:26:05 +0000
Subject: [PATCH v1] Fix comment on top of pgstat_read_statsfile()
The stats are read from the file and then loaded into the shared stats hash (
for non fixed amount stats) or into the fixed amount stats. Previous comment
was mentioning only the shared stats hash case.
---
src/backend/utils/activity/pgstat.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
100.0% src/backend/utils/activity/
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index c0ec9e8195..a7b8ecc1b6 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -1495,10 +1495,11 @@ read_chunk(FILE *fpin, void *ptr, size_t len)
#define read_chunk_s(fpin, ptr) read_chunk(fpin, ptr, sizeof(*ptr))
/*
- * Reads in existing statistics file into the shared stats hash.
+ * Reads in existing statistics file into the shared stats hash (for non fixed
+ * amount stats) or into the fixed amount stats.
*
- * This function is called in the only process that is accessing the shared
- * stats so locking is not required.
+ * This function is called in the only process that is accessing the stats
+ * so locking is not required.
*/
static void
pgstat_read_statsfile(void)
--
2.34.1
On Thu, Jul 11, 2024 at 01:29:08PM +0000, Bertrand Drouvot wrote:
Please find attached a patch to do so (attached as .txt to not perturb the
cfbot).
+ * Reads in existing statistics file into the shared stats hash (for non fixed
+ * amount stats) or into the fixed amount stats.
Thanks. I have applied a simplified version of that, not mentioning
the details of what happens depending on the kinds of stats dealt
with.
--
Michael
On Thu, Jul 11, 2024 at 04:42:22PM GMT, Michael Paquier wrote:
So we are down to the remaining parts of the patch, and this is going
to need a consensus about a few things because this impacts the
developer experience when implementing one's own custom stats:
- Are folks OK with the point of fixing the kind IDs in time like
RMGRs with a control in the wiki? Or should a more artistic approach
be used like what I am mentioning at the bottom of [1]. The patch
allows a range of IDs to be used, to make the access to the stats
faster even if some area of memory may not be used.
I think it's fine. Although this solution feels a bit uncomfortable,
after thinking back and forth I don't see any significantly better
option. Worth noting that since the main goal is to maintain uniqueness,
fixing the kind IDs could be accomplished in more than one way, with
varying amount of control over the list of custom IDs:
* One coud say "lets keep it in wiki and let the community organize
itself somehow", and it's done.
* Another way would be to keep it in wiki, and introduce some
maintenance rules, e.g. once per release someone is going to cleanup
the list from old unmaintained extensions, correct errors if needed,
etc. Not sure if such cleanup would be needed, but it's not impossible
to image.
* Even more closed option would be to keep the kind IDs in some separate
git repository, and let committers add new records on demand,
expressed via some request form.
As far as I understand the current proposal is about the first option,
on one side of the spectrum.
- The fixed-numbered custom stats kinds are stored in an array in
PgStat_Snapshot and PgStat_ShmemControl, so as we have something
consistent with the built-in kinds. This makes the tracking of the
validity of the data in the snapshots split into parts of the
structure for builtin and custom kinds. Perhaps there are better
ideas than that? The built-in fixed-numbered kinds have no
redirection.
Are you talking about this pattern?
if (pgstat_is_kind_builtin(kind))
ptr = // get something from a snapshot/shmem by offset
else
ptr = // get something from a custom_data by kind
Maybe it would be possible to hide it behind some macros or an inlinable
function with the offset and kind as arguments (and one of them will not
be used)?
On Fri, Jul 12, 2024 at 03:44:26PM +0200, Dmitry Dolgov wrote:
I think it's fine. Although this solution feels a bit uncomfortable,
after thinking back and forth I don't see any significantly better
option. Worth noting that since the main goal is to maintain uniqueness,
fixing the kind IDs could be accomplished in more than one way, with
varying amount of control over the list of custom IDs:* One coud say "lets keep it in wiki and let the community organize
itself somehow", and it's done.
* Another way would be to keep it in wiki, and introduce some
maintenance rules, e.g. once per release someone is going to cleanup
the list from old unmaintained extensions, correct errors if needed,
etc. Not sure if such cleanup would be needed, but it's not impossible
to image.
* Even more closed option would be to keep the kind IDs in some separate
git repository, and let committers add new records on demand,
expressed via some request form.
RMGRs have been taking the wiki page approach to control the source of
truth, that still sounds like the simplest option to me. I'm OK to be
outvoted, but this simplifies the read/write pgstats paths a lot, and
these would get more complicated if we add more options because of new
entry types (more things like serialized names I cannot think of,
etc). Extra point is that this makes future entensibility a bit
easier to work on.
As far as I understand the current proposal is about the first option,
on one side of the spectrum.
Yes.
- The fixed-numbered custom stats kinds are stored in an array in
PgStat_Snapshot and PgStat_ShmemControl, so as we have something
consistent with the built-in kinds. This makes the tracking of the
validity of the data in the snapshots split into parts of the
structure for builtin and custom kinds. Perhaps there are better
ideas than that? The built-in fixed-numbered kinds have no
redirection.Are you talking about this pattern?
if (pgstat_is_kind_builtin(kind))
ptr = // get something from a snapshot/shmem by offset
else
ptr = // get something from a custom_data by kindMaybe it would be possible to hide it behind some macros or an inlinable
function with the offset and kind as arguments (and one of them will not
be used)?
Kind of. All the code paths calling pgstat_is_kind_builtin() in the
patch manipulate different areas of the snapshot and/or the shmem
control structures, so a macro makes little sense.
Perhaps we should have a few more inline functions like
pgstat_get_entry_len() to retrieve the parts of the custom data in the
snapshot and shmem control structures for fixed-numbered stats. That
would limit what extensions need to know about
pgStatLocal.shmem->custom_data[] and
pgStatLocal.snapshot.custom_data[], which is easy to use incorrectly.
They don't need to know about pgStatLocal at all, either.
Thinking over the weekend on this patch, splitting injection_stats.c
into two separate files to act as two templates for the variable and
fixed-numbered cases would be more friendly to developers, as well.
--
Michael
On Tue, Jul 16, 2024 at 10:27:25AM +0900, Michael Paquier wrote:
Perhaps we should have a few more inline functions like
pgstat_get_entry_len() to retrieve the parts of the custom data in the
snapshot and shmem control structures for fixed-numbered stats. That
would limit what extensions need to know about
pgStatLocal.shmem->custom_data[] and
pgStatLocal.snapshot.custom_data[], which is easy to use incorrectly.
They don't need to know about pgStatLocal at all, either.Thinking over the weekend on this patch, splitting injection_stats.c
into two separate files to act as two templates for the variable and
fixed-numbered cases would be more friendly to developers, as well.
I've been toying a bit with these two ideas, and the result is
actually neater:
- The example for fixed-numbered stats is now in its own new file,
called injection_stats_fixed.c.
- Stats in the dshash are at the same location, injection_stats.c.
- pgstat_internal.h gains two inline routines called
pgstat_get_custom_shmem_data and pgstat_get_custom_snapshot_data that
hide completely the snapshot structure for extensions when it comes to
custom fixed-numbered stats, see the new injection_stats_fixed.c that
uses them.
--
Michael
Attachments:
v7-0005-injection_points-Add-statistics-for-custom-points.patchtext/x-diff; charset=us-asciiDownload
From 63ea23477e13322587bf100ddc85ffecfbe1a627 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 18 Jul 2024 14:22:54 +0900
Subject: [PATCH v7 5/6] injection_points: Add statistics for custom points
This acts as a template of what can be achieved with the pluggable
cumulative stats APIs, while being useful on its own for injection
points.
Currently, the only data gathered is the number of times an injection
point is called. This can be extended as required. All the routines
related to the stats are located in their own file, for clarity.
A TAP test is included to provide coverage for these new APIs, showing
the persistency of the data across restarts.
---
src/test/modules/injection_points/Makefile | 11 +-
.../injection_points--1.0.sql | 10 +
.../injection_points/injection_points.c | 27 +++
.../injection_points/injection_stats.c | 197 ++++++++++++++++++
.../injection_points/injection_stats.h | 23 ++
src/test/modules/injection_points/meson.build | 9 +
.../modules/injection_points/t/001_stats.pl | 48 +++++
src/tools/pgindent/typedefs.list | 2 +
8 files changed, 325 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/injection_points/injection_stats.c
create mode 100644 src/test/modules/injection_points/injection_stats.h
create mode 100644 src/test/modules/injection_points/t/001_stats.pl
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 2ffd2f77ed..7b9cd12a2a 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -1,7 +1,10 @@
# src/test/modules/injection_points/Makefile
-MODULES = injection_points
-
+MODULE_big = injection_points
+OBJS = \
+ $(WIN32RES) \
+ injection_points.o \
+ injection_stats.o
EXTENSION = injection_points
DATA = injection_points--1.0.sql
PGFILEDESC = "injection_points - facility for injection points"
@@ -11,9 +14,13 @@ REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
ISOLATION = inplace
+TAP_TESTS = 1
+
# The injection points are cluster-wide, so disable installcheck
NO_INSTALLCHECK = 1
+export enable_injection_points enable_injection_points
+
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index 0f280419a5..b98d571889 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -74,3 +74,13 @@ CREATE FUNCTION injection_points_detach(IN point_name TEXT)
RETURNS void
AS 'MODULE_PATHNAME', 'injection_points_detach'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+--
+-- injection_points_stats_numcalls()
+--
+-- Reports statistics, if any, related to the given injection point.
+--
+CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
+RETURNS bigint
+AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index 15f9d0233c..acec4e95b0 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -18,6 +18,7 @@
#include "postgres.h"
#include "fmgr.h"
+#include "injection_stats.h"
#include "miscadmin.h"
#include "nodes/pg_list.h"
#include "nodes/value.h"
@@ -170,6 +171,9 @@ injection_points_cleanup(int code, Datum arg)
char *name = strVal(lfirst(lc));
(void) InjectionPointDetach(name);
+
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
}
}
@@ -182,6 +186,8 @@ injection_error(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(ERROR, "error triggered for injection point %s", name);
}
@@ -193,6 +199,8 @@ injection_notice(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(NOTICE, "notice triggered for injection point %s", name);
}
@@ -211,6 +219,8 @@ injection_wait(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
/*
* Use the injection point name for this custom wait event. Note that
* this custom wait event name is not released, but we don't care much for
@@ -299,6 +309,10 @@ injection_points_attach(PG_FUNCTION_ARGS)
inj_list_local = lappend(inj_list_local, makeString(pstrdup(name)));
MemoryContextSwitchTo(oldctx);
}
+
+ /* Add entry for stats */
+ pgstat_create_inj(name);
+
PG_RETURN_VOID();
}
@@ -431,5 +445,18 @@ injection_points_detach(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldctx);
}
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
+
PG_RETURN_VOID();
}
+
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ pgstat_register_inj();
+}
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
new file mode 100644
index 0000000000..78042074ff
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -0,0 +1,197 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.c
+ * Code for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+
+#include "common/hashfn.h"
+#include "injection_stats.h"
+#include "pgstat.h"
+#include "utils/builtins.h"
+#include "utils/pgstat_internal.h"
+
+/* Structures for statistics of injection points */
+typedef struct PgStat_StatInjEntry
+{
+ PgStat_Counter numcalls; /* number of times point has been run */
+} PgStat_StatInjEntry;
+
+typedef struct PgStatShared_InjectionPoint
+{
+ PgStatShared_Common header;
+ PgStat_StatInjEntry stats;
+} PgStatShared_InjectionPoint;
+
+static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+
+static const PgStat_KindInfo injection_stats = {
+ .name = "injection_points",
+ .fixed_amount = false, /* Bounded by the number of points */
+
+ /* Injection points are system-wide */
+ .accessed_across_databases = true,
+
+ .shared_size = sizeof(PgStatShared_InjectionPoint),
+ .shared_data_off = offsetof(PgStatShared_InjectionPoint, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPoint *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatInjEntry),
+ .flush_pending_cb = injection_stats_flush_cb,
+};
+
+/*
+ * Compute stats entry idx from point name with a 4-byte hash.
+ */
+#define PGSTAT_INJ_IDX(name) hash_bytes((const unsigned char *) name, strlen(name))
+
+/*
+ * Kind ID reserved for statistics of injection points.
+ */
+#define PGSTAT_KIND_INJECTION 129
+
+/* Track if stats are loaded */
+static bool inj_stats_loaded = false;
+
+/*
+ * Callback for stats handling
+ */
+static bool
+injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ PgStat_StatInjEntry *localent;
+ PgStatShared_InjectionPoint *shfuncent;
+
+ localent = (PgStat_StatInjEntry *) entry_ref->pending;
+ shfuncent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ shfuncent->stats.numcalls += localent->numcalls;
+ return true;
+}
+
+/*
+ * Support function for the SQL-callable pgstat* functions. Returns
+ * a pointer to the injection point statistics struct.
+ */
+static PgStat_StatInjEntry *
+pgstat_fetch_stat_injentry(const char *name)
+{
+ PgStat_StatInjEntry *entry = NULL;
+
+ if (!inj_stats_loaded)
+ return NULL;
+
+ /* Compile the lookup key as a hash of the point name */
+ entry = (PgStat_StatInjEntry *) pgstat_fetch_entry(PGSTAT_KIND_INJECTION,
+ InvalidOid,
+ PGSTAT_INJ_IDX(name));
+ return entry;
+}
+
+/*
+ * Workhorse to do the registration work, called in _PG_init().
+ */
+void
+pgstat_register_inj(void)
+{
+ pgstat_register_kind(PGSTAT_KIND_INJECTION, &injection_stats);
+
+ /* mark stats as loaded */
+ inj_stats_loaded = true;
+}
+
+/*
+ * Report injection point creation.
+ */
+void
+pgstat_create_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ /* initialize shared memory data */
+ memset(&shstatent->stats, 0, sizeof(shstatent->stats));
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * Report injection point drop.
+ */
+void
+pgstat_drop_inj(const char *name)
+{
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ if (!pgstat_drop_entry(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name)))
+ pgstat_request_entry_refs_gc();
+}
+
+/*
+ * Report statistics for injection point.
+ *
+ * This is simple because the set of stats to report currently is simple:
+ * track the number of times a point has been run.
+ */
+void
+pgstat_report_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+ PgStat_StatInjEntry *statent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+ statent = &shstatent->stats;
+
+ /* Update the injection point statistics */
+ statent->numcalls++;
+
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * SQL function returning the number of times an injection point
+ * has been called.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_numcalls);
+Datum
+injection_points_stats_numcalls(PG_FUNCTION_ARGS)
+{
+ char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_StatInjEntry *entry = pgstat_fetch_stat_injentry(name);
+
+ if (entry == NULL)
+ PG_RETURN_NULL();
+
+ PG_RETURN_INT64(entry->numcalls);
+}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
new file mode 100644
index 0000000000..3e99705483
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -0,0 +1,23 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.h
+ * Definitions for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.h
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#ifndef INJECTION_STATS
+#define INJECTION_STATS
+
+extern void pgstat_register_inj(void);
+extern void pgstat_create_inj(const char *name);
+extern void pgstat_drop_inj(const char *name);
+extern void pgstat_report_inj(const char *name);
+
+#endif
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index 3c23c14d81..a52fe5121e 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -6,6 +6,7 @@ endif
injection_points_sources = files(
'injection_points.c',
+ 'injection_stats.c',
)
if host_system == 'windows'
@@ -42,4 +43,12 @@ tests += {
'inplace',
],
},
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_stats.pl',
+ ],
+ },
}
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
new file mode 100644
index 0000000000..7d5a96e522
--- /dev/null
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -0,0 +1,48 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+use locale;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Test persistency of statistics generated for injection points.
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# Node initialization
+my $node = PostgreSQL::Test::Cluster->new('master');
+$node->init;
+$node->append_conf('postgresql.conf',
+ "shared_preload_libraries = 'injection_points'");
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# This should count for two calls.
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('stats-notice', 'notice');");
+$node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
+$node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
+my $numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats calls');
+
+# Restart the node cleanly, stats should still be around.
+$node->restart;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats after clean restart');
+
+# On crash the stats are gone.
+$node->stop('immediate');
+$node->start;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '', 'number of stats after clean restart');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b4d7f9217c..c0625b8830 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2119,6 +2119,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_InjectionPoint
PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
@@ -2150,6 +2151,7 @@ PgStat_Snapshot
PgStat_SnapshotEntry
PgStat_StatDBEntry
PgStat_StatFuncEntry
+PgStat_StatInjEntry
PgStat_StatReplSlotEntry
PgStat_StatSubEntry
PgStat_StatTabEntry
--
2.45.2
v7-0006-injection_points-Add-example-for-fixed-numbered-s.patchtext/x-diff; charset=us-asciiDownload
From cd82e799d2bd2d05d217b2eaec646e0298677302 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 18 Jul 2024 14:44:33 +0900
Subject: [PATCH v7 6/6] injection_points: Add example for fixed-numbered
statistics
This acts as a template to show what can be achieved with fixed-numbered
stats (like WAL, bgwriter, etc.) for pluggable cumulative statistics.
---
src/test/modules/injection_points/Makefile | 3 +-
.../injection_points--1.0.sql | 11 +
.../injection_points/injection_points.c | 4 +
.../injection_points/injection_stats.h | 7 +
.../injection_points/injection_stats_fixed.c | 192 ++++++++++++++++++
src/test/modules/injection_points/meson.build | 1 +
.../modules/injection_points/t/001_stats.pl | 11 +-
7 files changed, 227 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/injection_points/injection_stats_fixed.c
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 7b9cd12a2a..ed28cd13a8 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -4,7 +4,8 @@ MODULE_big = injection_points
OBJS = \
$(WIN32RES) \
injection_points.o \
- injection_stats.o
+ injection_stats.o \
+ injection_stats_fixed.o
EXTENSION = injection_points
DATA = injection_points--1.0.sql
PGFILEDESC = "injection_points - facility for injection points"
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index b98d571889..1b2a4938a9 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -84,3 +84,14 @@ CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
RETURNS bigint
AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
LANGUAGE C STRICT;
+
+--
+-- injection_points_stats_fixed()
+--
+-- Reports fixed-numbered statistics for injection points.
+CREATE FUNCTION injection_points_stats_fixed(OUT numattach int8,
+ OUT numdetach int8,
+ OUT numrun int8)
+RETURNS record
+AS 'MODULE_PATHNAME', 'injection_points_stats_fixed'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index acec4e95b0..dc02be1bbf 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -297,6 +297,7 @@ injection_points_attach(PG_FUNCTION_ARGS)
condition.pid = MyProcPid;
}
+ pgstat_report_inj_fixed(1, 0, 0);
InjectionPointAttach(name, "injection_points", function, &condition,
sizeof(InjectionPointCondition));
@@ -342,6 +343,7 @@ injection_points_run(PG_FUNCTION_ARGS)
{
char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ pgstat_report_inj_fixed(0, 0, 1);
INJECTION_POINT(name);
PG_RETURN_VOID();
@@ -432,6 +434,7 @@ injection_points_detach(PG_FUNCTION_ARGS)
{
char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ pgstat_report_inj_fixed(0, 1, 0);
if (!InjectionPointDetach(name))
elog(ERROR, "could not detach injection point \"%s\"", name);
@@ -459,4 +462,5 @@ _PG_init(void)
return;
pgstat_register_inj();
+ pgstat_register_inj_fixed();
}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
index 3e99705483..d519f29f83 100644
--- a/src/test/modules/injection_points/injection_stats.h
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -15,9 +15,16 @@
#ifndef INJECTION_STATS
#define INJECTION_STATS
+/* injection_stats.c */
extern void pgstat_register_inj(void);
extern void pgstat_create_inj(const char *name);
extern void pgstat_drop_inj(const char *name);
extern void pgstat_report_inj(const char *name);
+/* injection_stats_fixed.c */
+extern void pgstat_register_inj_fixed(void);
+extern void pgstat_report_inj_fixed(uint32 numattach,
+ uint32 numdetach,
+ uint32 numrun);
+
#endif
diff --git a/src/test/modules/injection_points/injection_stats_fixed.c b/src/test/modules/injection_points/injection_stats_fixed.c
new file mode 100644
index 0000000000..75639328a8
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats_fixed.c
@@ -0,0 +1,192 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats_fixed.c
+ * Code for fixed-numbered statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats_fixed.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+
+#include "common/hashfn.h"
+#include "funcapi.h"
+#include "injection_stats.h"
+#include "pgstat.h"
+#include "utils/builtins.h"
+#include "utils/pgstat_internal.h"
+
+/* Structures for statistics of injection points, fixed-size */
+typedef struct PgStat_StatInjFixedEntry
+{
+ PgStat_Counter numattach; /* number of points attached */
+ PgStat_Counter numdetach; /* number of points detached */
+ PgStat_Counter numrun; /* number of points run */
+ TimestampTz stat_reset_timestamp;
+} PgStat_StatInjFixedEntry;
+
+typedef struct PgStatShared_InjectionPointFixed
+{
+ LWLock lock; /* protects all the counters */
+ uint32 changecount;
+ PgStat_StatInjFixedEntry stats;
+ PgStat_StatInjFixedEntry reset_offset;
+} PgStatShared_InjectionPointFixed;
+
+/* Callbacks for fixed-numbered stats */
+static void injection_stats_fixed_init_shmem_cb(void *stats);
+static void injection_stats_fixed_reset_all_cb(TimestampTz ts);
+static void injection_stats_fixed_snapshot_cb(void);
+
+static const PgStat_KindInfo injection_stats_fixed = {
+ .name = "injection_points_fixed",
+ .fixed_amount = true,
+
+ .shared_size = sizeof(PgStat_StatInjFixedEntry),
+ .shared_data_off = offsetof(PgStatShared_InjectionPointFixed, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPointFixed *) 0)->stats),
+
+ .init_shmem_cb = injection_stats_fixed_init_shmem_cb,
+ .reset_all_cb = injection_stats_fixed_reset_all_cb,
+ .snapshot_cb = injection_stats_fixed_snapshot_cb,
+};
+
+/*
+ * Kind ID reserved for statistics of injection points.
+ */
+#define PGSTAT_KIND_INJECTION_FIXED 130
+
+/* Track if fixed-numbered stats are loaded */
+static bool inj_fixed_loaded = false;
+
+static void
+injection_stats_fixed_init_shmem_cb(void *stats)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+static void
+injection_stats_fixed_reset_all_cb(TimestampTz ts)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ pgstat_get_custom_shmem_data(PGSTAT_KIND_INJECTION_FIXED);
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ pgstat_copy_changecounted_stats(&stats_shmem->reset_offset,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+ stats_shmem->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&stats_shmem->lock);
+}
+
+static void
+injection_stats_fixed_snapshot_cb(void)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ pgstat_get_custom_shmem_data(PGSTAT_KIND_INJECTION_FIXED);
+ PgStat_StatInjFixedEntry *stat_snap =
+ pgstat_get_custom_snapshot_data(PGSTAT_KIND_INJECTION_FIXED);
+ PgStat_StatInjFixedEntry *reset_offset = &stats_shmem->reset_offset;
+ PgStat_StatInjFixedEntry reset;
+
+ pgstat_copy_changecounted_stats(stat_snap,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+
+ LWLockAcquire(&stats_shmem->lock, LW_SHARED);
+ memcpy(&reset, reset_offset, sizeof(stats_shmem->stats));
+ LWLockRelease(&stats_shmem->lock);
+
+ /* compensate by reset offsets */
+#define FIXED_COMP(fld) stat_snap->fld -= reset.fld;
+ FIXED_COMP(numattach);
+ FIXED_COMP(numdetach);
+ FIXED_COMP(numrun);
+#undef FIXED_COMP
+}
+
+/*
+ * Workhorse to do the registration work, called in _PG_init().
+ */
+void
+pgstat_register_inj_fixed(void)
+{
+ pgstat_register_kind(PGSTAT_KIND_INJECTION_FIXED, &injection_stats_fixed);
+
+ /* mark stats as loaded */
+ inj_fixed_loaded = true;
+}
+
+/*
+ * Report fixed number of statistics for an injection point.
+ */
+void
+pgstat_report_inj_fixed(uint32 numattach,
+ uint32 numdetach,
+ uint32 numrun)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem;
+
+ /* leave if disabled */
+ if (!inj_fixed_loaded)
+ return;
+
+ stats_shmem = pgstat_get_custom_shmem_data(PGSTAT_KIND_INJECTION_FIXED);
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numattach += numattach;
+ stats_shmem->stats.numdetach += numdetach;
+ stats_shmem->stats.numrun += numrun;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+}
+
+/*
+ * SQL function returning fixed-numbered statistics for injection points.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_fixed);
+Datum
+injection_points_stats_fixed(PG_FUNCTION_ARGS)
+{
+ TupleDesc tupdesc;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ PgStat_StatInjFixedEntry *stats;
+
+ if (!inj_fixed_loaded)
+ PG_RETURN_NULL();
+
+ pgstat_snapshot_fixed(PGSTAT_KIND_INJECTION_FIXED);
+ stats = pgstat_get_custom_snapshot_data(PGSTAT_KIND_INJECTION_FIXED);
+
+ /* Initialise attributes information in the tuple descriptor */
+ tupdesc = CreateTemplateTupleDesc(3);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "numattach",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "numdetach",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "numrun",
+ INT8OID, -1, 0);
+ BlessTupleDesc(tupdesc);
+
+ values[0] = Int64GetDatum(stats->numattach);
+ values[1] = Int64GetDatum(stats->numdetach);
+ values[2] = Int64GetDatum(stats->numrun);
+ nulls[0] = false;
+ nulls[1] = false;
+ nulls[2] = false;
+
+ /* Returns the record as Datum */
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index a52fe5121e..c9e357f644 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -7,6 +7,7 @@ endif
injection_points_sources = files(
'injection_points.c',
'injection_stats.c',
+ 'injection_stats_fixed.c',
)
if host_system == 'windows'
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
index 7d5a96e522..e3c69b94ca 100644
--- a/src/test/modules/injection_points/t/001_stats.pl
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -31,18 +31,27 @@ $node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
my $numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
is($numcalls, '2', 'number of stats calls');
+my $fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '1|0|2', 'number of fixed stats');
# Restart the node cleanly, stats should still be around.
$node->restart;
$numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
is($numcalls, '2', 'number of stats after clean restart');
+$fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '1|0|2', 'number of fixed stats after clean restart');
# On crash the stats are gone.
$node->stop('immediate');
$node->start;
$numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
-is($numcalls, '', 'number of stats after clean restart');
+is($numcalls, '', 'number of stats after crash');
+$fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '0|0|0', 'number of fixed stats after crash');
done_testing();
--
2.45.2
v7-0001-Switch-PgStat_Kind-from-enum-to-uint32.patchtext/x-diff; charset=us-asciiDownload
From 54aa07f04489aca5dc2fed02121639d5fb453abd Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 12:28:26 +0900
Subject: [PATCH v7 1/6] Switch PgStat_Kind from enum to uint32
A follow-up patch is planned to make this counter extensible, and
keeping a trace of the kind behind a type is useful in the internal
routines used by pgstats. While on it, switch pgstat_is_kind_valid() to
use PgStat_Kind, to be more consistent with its callers.
---
src/include/pgstat.h | 35 ++++++++++++++---------------
src/backend/utils/activity/pgstat.c | 6 ++---
2 files changed, 20 insertions(+), 21 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 6b99bb8aad..cc97274708 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -32,26 +32,25 @@
#define PG_STAT_TMP_DIR "pg_stat_tmp"
/* The types of statistics entries */
-typedef enum PgStat_Kind
-{
- /* use 0 for INVALID, to catch zero-initialized data */
- PGSTAT_KIND_INVALID = 0,
+#define PgStat_Kind uint32
- /* stats for variable-numbered objects */
- PGSTAT_KIND_DATABASE, /* database-wide statistics */
- PGSTAT_KIND_RELATION, /* per-table statistics */
- PGSTAT_KIND_FUNCTION, /* per-function statistics */
- PGSTAT_KIND_REPLSLOT, /* per-slot statistics */
- PGSTAT_KIND_SUBSCRIPTION, /* per-subscription statistics */
+/* use 0 for INVALID, to catch zero-initialized data */
+#define PGSTAT_KIND_INVALID 0
- /* stats for fixed-numbered objects */
- PGSTAT_KIND_ARCHIVER,
- PGSTAT_KIND_BGWRITER,
- PGSTAT_KIND_CHECKPOINTER,
- PGSTAT_KIND_IO,
- PGSTAT_KIND_SLRU,
- PGSTAT_KIND_WAL,
-} PgStat_Kind;
+/* stats for variable-numbered objects */
+#define PGSTAT_KIND_DATABASE 1 /* database-wide statistics */
+#define PGSTAT_KIND_RELATION 2 /* per-table statistics */
+#define PGSTAT_KIND_FUNCTION 3 /* per-function statistics */
+#define PGSTAT_KIND_REPLSLOT 4 /* per-slot statistics */
+#define PGSTAT_KIND_SUBSCRIPTION 5 /* per-subscription statistics */
+
+/* stats for fixed-numbered objects */
+#define PGSTAT_KIND_ARCHIVER 6
+#define PGSTAT_KIND_BGWRITER 7
+#define PGSTAT_KIND_CHECKPOINTER 8
+#define PGSTAT_KIND_IO 9
+#define PGSTAT_KIND_SLRU 10
+#define PGSTAT_KIND_WAL 11
#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 2e22bf2707..e3058b32c0 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -182,7 +182,7 @@ static void pgstat_prep_snapshot(void);
static void pgstat_build_snapshot(void);
static void pgstat_build_snapshot_fixed(PgStat_Kind kind);
-static inline bool pgstat_is_kind_valid(int ikind);
+static inline bool pgstat_is_kind_valid(PgStat_Kind kind);
/* ----------
@@ -1298,9 +1298,9 @@ pgstat_get_kind_from_str(char *kind_str)
}
static inline bool
-pgstat_is_kind_valid(int ikind)
+pgstat_is_kind_valid(PgStat_Kind kind)
{
- return ikind >= PGSTAT_KIND_FIRST_VALID && ikind <= PGSTAT_KIND_LAST;
+ return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
}
const PgStat_KindInfo *
--
2.45.2
v7-0002-Introduce-pluggable-APIs-for-Cumulative-Statistic.patchtext/x-diff; charset=us-asciiDownload
From 07c4be73bb31c06b1c44b2457ab5425f2426cc8c Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 8 Jul 2024 14:03:17 +0900
Subject: [PATCH v7 2/6] Introduce pluggable APIs for Cumulative Statistics
This commit adds support in the backend for $subject, allowing
out-of-core extensions to add their own custom statistics kinds. The
stats kinds are divided into two parts for efficiency:
- The built-in stats kinds, with designated initializers.
- The custom kinds, able to use a range of IDs (128 slots available as
of this patch), with information saved in TopMemoryContext.
Custom cumulative statistics can only be loaded with
shared_preload_libraries at startup, and must allocate a unique ID
shared across all the PostgreSQL extension ecosystem with the following
wiki page:
https://wiki.postgresql.org/wiki/CustomCumulativeStats
This is able to support fixed-numbered (like WAL, archiver, bgwriter)
and variable-numbered stats kinds.
---
src/include/pgstat.h | 35 +++-
src/include/utils/pgstat_internal.h | 22 ++-
src/backend/utils/activity/pgstat.c | 231 +++++++++++++++++++---
src/backend/utils/activity/pgstat_shmem.c | 31 ++-
src/backend/utils/adt/pgstatfuncs.c | 2 +-
5 files changed, 287 insertions(+), 34 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index cc97274708..21c3705ce4 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -34,6 +34,10 @@
/* The types of statistics entries */
#define PgStat_Kind uint32
+/* Range of IDs allowed, for built-in and custom kinds */
+#define PGSTAT_KIND_MIN 1 /* Minimum ID allowed */
+#define PGSTAT_KIND_MAX 256 /* Maximum ID allowed */
+
/* use 0 for INVALID, to catch zero-initialized data */
#define PGSTAT_KIND_INVALID 0
@@ -52,9 +56,34 @@
#define PGSTAT_KIND_SLRU 10
#define PGSTAT_KIND_WAL 11
-#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
-#define PGSTAT_NUM_KINDS (PGSTAT_KIND_LAST + 1)
+#define PGSTAT_KIND_MIN_BUILTIN PGSTAT_KIND_DATABASE
+#define PGSTAT_KIND_MAX_BUILTIN PGSTAT_KIND_WAL
+
+/* Custom stats kinds */
+
+/* Range of IDs allowed for custom stats kinds */
+#define PGSTAT_KIND_CUSTOM_MIN 128
+#define PGSTAT_KIND_CUSTOM_MAX PGSTAT_KIND_MAX
+#define PGSTAT_KIND_CUSTOM_SIZE (PGSTAT_KIND_CUSTOM_MAX - PGSTAT_KIND_CUSTOM_MIN + 1)
+
+/*
+ * PgStat_Kind to use for extensions that require an ID, but are still in
+ * development and have not reserved their own unique kind ID yet. See:
+ * https://wiki.postgresql.org/wiki/CustomCumulativeStats
+ */
+#define PGSTAT_KIND_EXPERIMENTAL 128
+
+static inline bool
+pgstat_is_kind_builtin(PgStat_Kind kind)
+{
+ return kind > PGSTAT_KIND_INVALID && kind <= PGSTAT_KIND_MAX_BUILTIN;
+}
+
+static inline bool
+pgstat_is_kind_custom(PgStat_Kind kind)
+{
+ return kind >= PGSTAT_KIND_CUSTOM_MIN && kind <= PGSTAT_KIND_CUSTOM_MAX;
+}
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 778f625ca1..39f63362a3 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -195,7 +195,8 @@ typedef struct PgStat_KindInfo
/*
* The size of an entry in the shared stats hash table (pointed to by
- * PgStatShared_HashEntry->body).
+ * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
+ * the size of an entry in PgStat_ShmemControl->custom_data.
*/
uint32 shared_size;
@@ -446,6 +447,13 @@ typedef struct PgStat_ShmemControl
PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
+
+ /*
+ * Custom stats data with fixed-numbered objects, indexed by (PgStat_Kind
+ * - PGSTAT_KIND_CUSTOM_MIN).
+ */
+ void *custom_data[PGSTAT_KIND_CUSTOM_SIZE];
+
} PgStat_ShmemControl;
@@ -459,7 +467,7 @@ typedef struct PgStat_Snapshot
/* time at which snapshot was taken */
TimestampTz snapshot_timestamp;
- bool fixed_valid[PGSTAT_NUM_KINDS];
+ bool fixed_valid[PGSTAT_KIND_MAX_BUILTIN + 1];
PgStat_ArchiverStats archiver;
@@ -473,6 +481,14 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ /*
+ * Data in snapshot for custom fixed-numbered statistics, indexed by
+ * (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
+ * TopMemoryContext, for a size of shared_data_len.
+ */
+ bool custom_valid[PGSTAT_KIND_CUSTOM_SIZE];
+ void *custom_data[PGSTAT_KIND_CUSTOM_SIZE];
+
/* to free snapshot in bulk */
MemoryContext context;
struct pgstat_snapshot_hash *stats;
@@ -516,6 +532,8 @@ static inline void *pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common
*/
extern const PgStat_KindInfo *pgstat_get_kind_info(PgStat_Kind kind);
+extern void pgstat_register_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
#ifdef USE_ASSERT_CHECKING
extern void pgstat_assert_is_up(void);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index e3058b32c0..bb70b81e49 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -49,8 +49,16 @@
* pgStatPending list. Pending statistics updates are flushed out by
* pgstat_report_stat().
*
+ * It is possible for external modules to define custom statistics kinds,
+ * that can use the same properties as any built-in stats kinds. Each custom
+ * stats kind needs to assign a unique ID to ensure that it does not overlap
+ * with other extensions. In order to reserve a unique stats kind ID, refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats.
+ *
* The behavior of different kinds of statistics is determined by the kind's
- * entry in pgstat_kind_infos, see PgStat_KindInfo for details.
+ * entry in pgstat_kind_builtin_infos for all the built-in statistics kinds
+ * defined, and pgstat_kind_custom_infos for custom kinds registered at
+ * startup by pgstat_register_kind(). See PgStat_KindInfo for details.
*
* The consistency of read accesses to statistics can be configured using the
* stats_fetch_consistency GUC (see config.sgml and monitoring.sgml for the
@@ -174,6 +182,8 @@ typedef struct PgStat_SnapshotEntry
static void pgstat_write_statsfile(void);
static void pgstat_read_statsfile(void);
+static void pgstat_init_snapshot_fixed(void);
+
static void pgstat_reset_after_failure(void);
static bool pgstat_flush_pending_entries(bool nowait);
@@ -251,7 +261,7 @@ static bool pgstat_is_shutdown = false;
/*
- * The different kinds of statistics.
+ * The different kinds of built-in statistics.
*
* If reasonably possible, handling specific to one kind of stats should go
* through this abstraction, rather than making more of pgstat.c aware.
@@ -263,7 +273,7 @@ static bool pgstat_is_shutdown = false;
* seem to be a great way of doing that, given the split across multiple
* files.
*/
-static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
+static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_MAX_BUILTIN + 1] = {
/* stats kinds for variable-numbered objects */
@@ -436,6 +446,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
},
};
+/*
+ * Information about custom statistics kinds.
+ *
+ * These are saved in a different array than the built-in kinds to save
+ * in clarity with the initializations.
+ *
+ * Indexed by PGSTAT_KIND_CUSTOM_MIN, of size PGSTAT_KIND_CUSTOM_SIZE.
+ */
+static const PgStat_KindInfo **pgstat_kind_custom_infos = NULL;
/* ------------------------------------------------------------
* Functions managing the state of the stats system for all backends.
@@ -586,6 +605,8 @@ pgstat_initialize(void)
pgstat_init_wal();
+ pgstat_init_snapshot_fixed();
+
/* Set up a process-exit hook to clean up */
before_shmem_exit(pgstat_shutdown_hook, 0);
@@ -829,6 +850,8 @@ pgstat_clear_snapshot(void)
memset(&pgStatLocal.snapshot.fixed_valid, 0,
sizeof(pgStatLocal.snapshot.fixed_valid));
+ memset(&pgStatLocal.snapshot.custom_valid, 0,
+ sizeof(pgStatLocal.snapshot.custom_valid));
pgStatLocal.snapshot.stats = NULL;
pgStatLocal.snapshot.mode = PGSTAT_FETCH_CONSISTENCY_NONE;
@@ -992,7 +1015,29 @@ pgstat_snapshot_fixed(PgStat_Kind kind)
else
pgstat_build_snapshot_fixed(kind);
- Assert(pgStatLocal.snapshot.fixed_valid[kind]);
+ if (pgstat_is_kind_builtin(kind))
+ Assert(pgStatLocal.snapshot.fixed_valid[kind]);
+ else if (pgstat_is_kind_custom(kind))
+ Assert(pgStatLocal.snapshot.custom_valid[kind - PGSTAT_KIND_CUSTOM_MIN]);
+}
+
+static void
+pgstat_init_snapshot_fixed(void)
+{
+ /*
+ * Initialize fixed-numbered statistics data in snapshots, only for custom
+ * stats kinds.
+ */
+ for (int kind = PGSTAT_KIND_CUSTOM_MIN; kind <= PGSTAT_KIND_CUSTOM_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->fixed_amount)
+ continue;
+
+ pgStatLocal.snapshot.custom_data[kind - PGSTAT_KIND_CUSTOM_MIN] =
+ MemoryContextAlloc(TopMemoryContext, kind_info->shared_data_len);
+ }
}
static void
@@ -1088,10 +1133,12 @@ pgstat_build_snapshot(void)
/*
* Build snapshot of all fixed-numbered stats.
*/
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ if (!kind_info)
+ continue;
if (!kind_info->fixed_amount)
{
Assert(kind_info->snapshot_cb == NULL);
@@ -1108,6 +1155,20 @@ static void
pgstat_build_snapshot_fixed(PgStat_Kind kind)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ int idx;
+ bool *valid;
+
+ /* Position in fixed_valid or custom_valid */
+ if (pgstat_is_kind_builtin(kind))
+ {
+ idx = kind;
+ valid = pgStatLocal.snapshot.fixed_valid;
+ }
+ else
+ {
+ idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+ valid = pgStatLocal.snapshot.custom_valid;
+ }
Assert(kind_info->fixed_amount);
Assert(kind_info->snapshot_cb != NULL);
@@ -1115,21 +1176,21 @@ pgstat_build_snapshot_fixed(PgStat_Kind kind)
if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE)
{
/* rebuild every time */
- pgStatLocal.snapshot.fixed_valid[kind] = false;
+ valid[idx] = false;
}
- else if (pgStatLocal.snapshot.fixed_valid[kind])
+ else if (valid[idx])
{
/* in snapshot mode we shouldn't get called again */
Assert(pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_CACHE);
return;
}
- Assert(!pgStatLocal.snapshot.fixed_valid[kind]);
+ Assert(!valid[idx]);
kind_info->snapshot_cb();
- Assert(!pgStatLocal.snapshot.fixed_valid[kind]);
- pgStatLocal.snapshot.fixed_valid[kind] = true;
+ Assert(!valid[idx]);
+ valid[idx] = true;
}
@@ -1285,30 +1346,127 @@ pgstat_flush_pending_entries(bool nowait)
PgStat_Kind
pgstat_get_kind_from_str(char *kind_str)
{
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN_BUILTIN; kind <= PGSTAT_KIND_MAX_BUILTIN; kind++)
{
- if (pg_strcasecmp(kind_str, pgstat_kind_infos[kind].name) == 0)
+ if (pg_strcasecmp(kind_str, pgstat_kind_builtin_infos[kind].name) == 0)
return kind;
}
+ /* Check the custom set of cumulative stats */
+ if (pgstat_kind_custom_infos)
+ {
+ for (int kind = 0; kind < PGSTAT_KIND_CUSTOM_SIZE; kind++)
+ {
+ if (pgstat_kind_custom_infos[kind] &&
+ pg_strcasecmp(kind_str, pgstat_kind_custom_infos[kind]->name) == 0)
+ return kind + PGSTAT_KIND_CUSTOM_MIN;
+ }
+ }
+
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid statistics kind: \"%s\"", kind_str)));
- return PGSTAT_KIND_DATABASE; /* avoid compiler warnings */
+ return PGSTAT_KIND_INVALID; /* avoid compiler warnings */
}
static inline bool
pgstat_is_kind_valid(PgStat_Kind kind)
{
- return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
+ return pgstat_is_kind_builtin(kind) || pgstat_is_kind_custom(kind);
}
const PgStat_KindInfo *
pgstat_get_kind_info(PgStat_Kind kind)
{
- Assert(pgstat_is_kind_valid(kind));
+ if (pgstat_is_kind_builtin(kind))
+ return &pgstat_kind_builtin_infos[kind];
- return &pgstat_kind_infos[kind];
+ if (pgstat_is_kind_custom(kind))
+ {
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (pgstat_kind_custom_infos == NULL ||
+ pgstat_kind_custom_infos[idx] == NULL)
+ return NULL;
+ return pgstat_kind_custom_infos[idx];
+ }
+
+ return NULL;
+}
+
+/*
+ * Register a new stats kind.
+ *
+ * PgStat_Kinds must be globally unique across all extensions. Refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats to reserve a
+ * unique ID for your extension, to avoid conflicts with other extension
+ * developers. During development, use PGSTAT_KIND_EXPERIMENTAL to avoid
+ * needlessly reserving a new ID.
+ */
+void
+pgstat_register_kind(PgStat_Kind kind, const PgStat_KindInfo *kind_info)
+{
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (kind_info->name == NULL || strlen(kind_info->name) == 0)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics name is invalid"),
+ errhint("Provide a non-empty name for the custom cumulative statistics.")));
+
+ if (!pgstat_is_kind_custom(kind))
+ ereport(ERROR, (errmsg("custom cumulative statistics ID %u is out of range", kind),
+ errhint("Provide a custom cumulative statistics ID between %u and %u.",
+ PGSTAT_KIND_CUSTOM_MIN, PGSTAT_KIND_CUSTOM_MAX)));
+
+ if (!process_shared_preload_libraries_in_progress)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics must be registered while initializing modules in \"shared_preload_libraries\".")));
+
+ /*
+ * Check some data for fixed-numbered stats.
+ */
+ if (kind_info->fixed_amount)
+ {
+ if (kind_info->shared_size == 0)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics property is invalid"),
+ errhint("Custom cumulative statistics require a shared memory size for fixed-numbered objects.")));
+ }
+
+ /*
+ * If pgstat_kind_custom_infos is not available yet, allocate it.
+ */
+ if (pgstat_kind_custom_infos == NULL)
+ {
+ pgstat_kind_custom_infos = (const PgStat_KindInfo **)
+ MemoryContextAllocZero(TopMemoryContext,
+ sizeof(PgStat_KindInfo *) * PGSTAT_KIND_CUSTOM_SIZE);
+ }
+
+ if (pgstat_kind_custom_infos[idx] != NULL &&
+ pgstat_kind_custom_infos[idx]->name != NULL)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics \"%s\" already registered with the same ID.",
+ pgstat_kind_custom_infos[idx]->name)));
+
+ /* check for existing custom stats with the same name */
+ for (int existing_kind = 0; existing_kind < PGSTAT_KIND_CUSTOM_SIZE; existing_kind++)
+ {
+ if (pgstat_kind_custom_infos[existing_kind] == NULL)
+ continue;
+ if (!pg_strcasecmp(pgstat_kind_custom_infos[existing_kind]->name, kind_info->name))
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Existing cumulative statistics with ID %u has the same name.", existing_kind + PGSTAT_KIND_CUSTOM_MIN)));
+ }
+
+ /* Register it */
+ pgstat_kind_custom_infos[idx] = kind_info;
+ ereport(LOG,
+ (errmsg("registered custom cumulative statistics \"%s\" with ID %u",
+ kind_info->name, kind)));
}
/*
@@ -1388,18 +1546,22 @@ pgstat_write_statsfile(void)
write_chunk_s(fpout, &format_id);
/* Write various stats structs for fixed number of objects */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
char *ptr;
const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
- if (!info->fixed_amount)
+ if (!info || !info->fixed_amount)
continue;
- Assert(info->snapshot_ctl_off != 0);
+ if (pgstat_is_kind_builtin(kind))
+ Assert(info->snapshot_ctl_off != 0);
pgstat_build_snapshot_fixed(kind);
- ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+ else
+ ptr = pgStatLocal.snapshot.custom_data[kind - PGSTAT_KIND_CUSTOM_MIN];
fputc(PGSTAT_FILE_ENTRY_FIXED, fpout);
write_chunk_s(fpout, &kind);
@@ -1422,6 +1584,17 @@ pgstat_write_statsfile(void)
if (ps->dropped)
continue;
+ /*
+ * This discards data related to custom stats kinds that are unknown
+ * to this process.
+ */
+ if (!pgstat_is_kind_valid(ps->key.kind))
+ {
+ elog(WARNING, "found unknown stats entry %u/%u/%u",
+ ps->key.kind, ps->key.dboid, ps->key.objoid);
+ continue;
+ }
+
shstats = (PgStatShared_Common *) dsa_get_address(pgStatLocal.dsa, ps->body);
kind_info = pgstat_get_kind_info(ps->key.kind);
@@ -1570,8 +1743,16 @@ pgstat_read_statsfile(void)
goto error;
/* Load back stats into shared memory */
- ptr = ((char *) shmem) + info->shared_ctl_off +
- info->shared_data_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) shmem) + info->shared_ctl_off +
+ info->shared_data_off;
+ else
+ {
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ ptr = ((char *) shmem->custom_data[idx]) +
+ info->shared_data_off;
+ }
if (!read_chunk(fpin, ptr, info->shared_data_len))
goto error;
@@ -1638,7 +1819,7 @@ pgstat_read_statsfile(void)
if (found)
{
dshash_release_lock(pgStatLocal.shared_hash, p);
- elog(WARNING, "found duplicate stats entry %d/%u/%u",
+ elog(WARNING, "found duplicate stats entry %u/%u/%u",
key.kind, key.dboid, key.objoid);
goto error;
}
@@ -1696,11 +1877,11 @@ pgstat_reset_after_failure(void)
TimestampTz ts = GetCurrentTimestamp();
/* reset fixed-numbered stats */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
kind_info->reset_all_cb(ts);
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 1c2b69d563..ca3d3eecfb 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -131,6 +131,21 @@ StatsShmemSize(void)
sz = MAXALIGN(sizeof(PgStat_ShmemControl));
sz = add_size(sz, pgstat_dsa_init_size());
+ /* Add shared memory for all the custom fixed-numbered statistics */
+ for (int kind = PGSTAT_KIND_CUSTOM_MIN; kind <= PGSTAT_KIND_CUSTOM_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info)
+ continue;
+ if (!kind_info->fixed_amount)
+ continue;
+
+ Assert(kind_info->shared_size != 0);
+
+ sz += MAXALIGN(kind_info->shared_size);
+ }
+
return sz;
}
@@ -197,15 +212,25 @@ StatsShmemInit(void)
pg_atomic_init_u64(&ctl->gc_request_count, 1);
/* initialize fixed-numbered stats */
- for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (int kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
char *ptr;
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
- ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ else
+ {
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ Assert(kind_info->shared_size != 0);
+ ctl->custom_data[idx] = ShmemAlloc(kind_info->shared_size);
+ ptr = ctl->custom_data[idx];
+ }
+
kind_info->init_shmem_cb(ptr);
}
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 3876339ee1..3221137123 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1696,7 +1696,7 @@ pg_stat_reset(PG_FUNCTION_ARGS)
* Reset some shared cluster-wide counters
*
* When adding a new reset target, ideally the name should match that in
- * pgstat_kind_infos, if relevant.
+ * pgstat_kind_builtin_infos, if relevant.
*/
Datum
pg_stat_reset_shared(PG_FUNCTION_ARGS)
--
2.45.2
v7-0003-Add-helper-routines-to-retrieve-custom-data-for-f.patchtext/x-diff; charset=us-asciiDownload
From e475faaa8943356379923433f80125efbe68b9d0 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 18 Jul 2024 14:10:35 +0900
Subject: [PATCH v7 3/6] Add helper routines to retrieve custom data for
fixed-numbered stats
This is useful for extensions to get snapshot and shmem data for custom
statistics, so as these don't need to know about the current snapshots.
---
src/include/utils/pgstat_internal.h | 32 +++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 39f63362a3..6bbd49388c 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -525,6 +525,8 @@ static inline int pgstat_cmp_hash_key(const void *a, const void *b, size_t size,
static inline uint32 pgstat_hash_hash_key(const void *d, size_t size, void *arg);
static inline size_t pgstat_get_entry_len(PgStat_Kind kind);
static inline void *pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common *entry);
+static inline void *pgstat_get_custom_shmem_data(PgStat_Kind kind);
+static inline void *pgstat_get_custom_snapshot_data(PgStat_Kind kind);
/*
@@ -842,4 +844,34 @@ pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common *entry)
return ((char *) (entry)) + off;
}
+/*
+ * Returns a pointer to the shared memory area of custom stats for
+ * fixed-numbered statistics.
+ */
+static inline void *
+pgstat_get_custom_shmem_data(PgStat_Kind kind)
+{
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ Assert(pgstat_is_kind_custom(kind));
+ Assert(pgstat_get_kind_info(kind)->fixed_amount);
+
+ return pgStatLocal.shmem->custom_data[idx];
+}
+
+/*
+ * Returns a pointer to the portion of custom data for fixed-numbered
+ * statistics in the current snapshot.
+ */
+static inline void *
+pgstat_get_custom_snapshot_data(PgStat_Kind kind)
+{
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ Assert(pgstat_is_kind_custom(kind));
+ Assert(pgstat_get_kind_info(kind)->fixed_amount);
+
+ return pgStatLocal.snapshot.custom_data[idx];
+}
+
#endif /* PGSTAT_INTERNAL_H */
--
2.45.2
v7-0004-doc-Add-section-for-Custom-Cumulative-Statistics-.patchtext/x-diff; charset=us-asciiDownload
From 75e499b54c7c6ab79dadffe6eacb943df0a55038 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 18 Jul 2024 14:14:58 +0900
Subject: [PATCH v7 4/6] doc: Add section for Custom Cumulative Statistics APIs
This provides a short description of what can be done, with a pointer to
the template in the tree.
---
doc/src/sgml/xfunc.sgml | 62 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index 7e92e89846..839c78c102 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3703,6 +3703,68 @@ extern bool InjectionPointDetach(const char *name);
</para>
</sect2>
+ <sect2 id="xfunc-addin-custom-cumulative-statistics">
+ <title>Custom Cumulative Statistics</title>
+
+ <para>
+ Is is possible for add-ins written in C-language to use custom types
+ of cumulative statistics registered in the
+ <link linkend="monitoring-stats-setup">Cumulative Statistics System</link>.
+ </para>
+
+ <para>
+ First, define a <literal>PgStat_KindInfo</literal> that includes all
+ the information related to the custom type registered. For example:
+<programlisting>
+static const PgStat_KindInfo custom_stats = {
+ .name = "custom_stats",
+ .fixed_amount = false,
+ .shared_size = sizeof(PgStatShared_Custom),
+ .shared_data_off = offsetof(PgStatShared_Custom, stats),
+ .shared_data_len = sizeof(((PgStatShared_Custom *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatCustomEntry),
+}
+</programlisting>
+
+ Then, each backend that needs to use this custom type needs to register
+ it with <literal>pgstat_register_kind</literal> and a unique ID used to
+ store the entries related to this type of statistics:
+<programlisting>
+extern PgStat_Kind pgstat_add_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
+</programlisting>
+ While developing a new extension, use
+ <literal>PGSTAT_KIND_EXPERIMENTAL</literal> for
+ <parameter>kind</parameter>. When you are ready to release the extension
+ to users, reserve a kind ID at the
+ <ulink url="https://wiki.postgresql.org/wiki/CustomCumulativeStats">
+ Custom Cumulative Statistics</ulink> page.
+ </para>
+
+ <para>
+ The details of the API for <literal>PgStat_KindInfo</literal> can
+ be found in <filename>src/include/utils/pgstat_internal.h</filename>.
+ </para>
+
+ <para>
+ The type of statistics registered is associated with a name and a unique
+ ID shared across the server in shared memory. Each backend using a
+ custom type of statistics maintains a local cache storing the information
+ of each custom <literal>PgStat_KindInfo</literal>.
+ </para>
+
+ <para>
+ Place the extension module implementing the custom cumulative statistics
+ type in <xref linkend="guc-shared-preload-libraries"/> so that it will
+ be loaded early during <productname>PostgreSQL</productname> startup.
+ </para>
+
+ <para>
+ An example describing how to register and use custom statistics can be
+ found in <filename>src/test/modules/injection_points</filename>.
+ </para>
+ </sect2>
+
<sect2 id="extend-cpp">
<title>Using C++ for Extensibility</title>
--
2.45.2
On Thu, Jul 18, 2024 at 02:56:20PM GMT, Michael Paquier wrote:
On Tue, Jul 16, 2024 at 10:27:25AM +0900, Michael Paquier wrote:Perhaps we should have a few more inline functions like
pgstat_get_entry_len() to retrieve the parts of the custom data in the
snapshot and shmem control structures for fixed-numbered stats. That
would limit what extensions need to know about
pgStatLocal.shmem->custom_data[] and
pgStatLocal.snapshot.custom_data[], which is easy to use incorrectly.
They don't need to know about pgStatLocal at all, either.Thinking over the weekend on this patch, splitting injection_stats.c
into two separate files to act as two templates for the variable and
fixed-numbered cases would be more friendly to developers, as well.I've been toying a bit with these two ideas, and the result is
actually neater:
- The example for fixed-numbered stats is now in its own new file,
called injection_stats_fixed.c.
- Stats in the dshash are at the same location, injection_stats.c.
- pgstat_internal.h gains two inline routines called
pgstat_get_custom_shmem_data and pgstat_get_custom_snapshot_data that
hide completely the snapshot structure for extensions when it comes to
custom fixed-numbered stats, see the new injection_stats_fixed.c that
uses them.
Agree, looks good. I've tried to quickly sketch out such a fixed
statistic for some another extension, everything was fine and pretty
straightforward. One question, why don't you use
pgstat_get_custom_shmem_data & pgstat_get_custom_snapshot_data outside
of the injection points code? There seems to be a couple of possible
places in pgstats itself.
On Sat, Jul 27, 2024 at 03:49:42PM +0200, Dmitry Dolgov wrote:
Agree, looks good. I've tried to quickly sketch out such a fixed
statistic for some another extension, everything was fine and pretty
straightforward.
That's my hope. Thanks a lot for the feedback.
One question, why don't you use
pgstat_get_custom_shmem_data & pgstat_get_custom_snapshot_data outside
of the injection points code? There seems to be a couple of possible
places in pgstats itself.
Because these two helper routines are only able to fetch the fixed
data area in the snapshot and the control shmem structures for the
custom kinds, not the in-core ones. We could, but the current code is
OK as well. My point was just to ease the pluggability effort.
I would like to apply this new infrastructure stuff and move on to the
problems related to the scability of pg_stat_statements. So, are
there any objections with all that?
--
Michael
On Sun, Jul 28, 2024 at 10:20:45PM GMT, Michael Paquier wrote:
I would like to apply this new infrastructure stuff and move on to the
problems related to the scability of pg_stat_statements. So, are
there any objections with all that?
So far I've got nothing against :)
On Sun, Jul 28, 2024 at 10:03:56PM +0200, Dmitry Dolgov wrote:
So far I've got nothing against :)
I've looked again at the first patch of this series, and applied the
first one. Another last-minute edit I have done is to use more
consistently PgStat_Kind in the loops for the stats kinds across all
the pgstats code.
Attached is a rebased set of the rest, with 0001 now introducing the
pluggable core part.
--
Michael
Attachments:
v3-0001-Introduce-pluggable-APIs-for-Cumulative-Statistic.patchtext/x-diff; charset=us-asciiDownload
From af5a2456f90bf02b993c7c718e45a4cd6d863c79 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 2 Aug 2024 05:42:52 +0900
Subject: [PATCH v3 1/5] Introduce pluggable APIs for Cumulative Statistics
This commit adds support in the backend for $subject, allowing
out-of-core extensions to add their own custom statistics kinds. The
stats kinds are divided into two parts for efficiency:
- The built-in stats kinds, with designated initializers.
- The custom kinds, able to use a range of IDs (128 slots available as
of this patch), with information saved in TopMemoryContext.
Custom cumulative statistics can only be loaded with
shared_preload_libraries at startup, and must allocate a unique ID
shared across all the PostgreSQL extension ecosystem with the following
wiki page:
https://wiki.postgresql.org/wiki/CustomCumulativeStats
This is able to support fixed-numbered (like WAL, archiver, bgwriter)
and variable-numbered stats kinds.
---
src/include/pgstat.h | 35 +++-
src/include/utils/pgstat_internal.h | 22 ++-
src/backend/utils/activity/pgstat.c | 231 +++++++++++++++++++---
src/backend/utils/activity/pgstat_shmem.c | 31 ++-
src/backend/utils/adt/pgstatfuncs.c | 2 +-
5 files changed, 287 insertions(+), 34 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index f84e9fdeca..cd93a0109b 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -35,6 +35,10 @@
/* The types of statistics entries */
#define PgStat_Kind uint32
+/* Range of IDs allowed, for built-in and custom kinds */
+#define PGSTAT_KIND_MIN 1 /* Minimum ID allowed */
+#define PGSTAT_KIND_MAX 256 /* Maximum ID allowed */
+
/* use 0 for INVALID, to catch zero-initialized data */
#define PGSTAT_KIND_INVALID 0
@@ -53,9 +57,34 @@
#define PGSTAT_KIND_SLRU 10
#define PGSTAT_KIND_WAL 11
-#define PGSTAT_KIND_FIRST_VALID PGSTAT_KIND_DATABASE
-#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL
-#define PGSTAT_NUM_KINDS (PGSTAT_KIND_LAST + 1)
+#define PGSTAT_KIND_MIN_BUILTIN PGSTAT_KIND_DATABASE
+#define PGSTAT_KIND_MAX_BUILTIN PGSTAT_KIND_WAL
+
+/* Custom stats kinds */
+
+/* Range of IDs allowed for custom stats kinds */
+#define PGSTAT_KIND_CUSTOM_MIN 128
+#define PGSTAT_KIND_CUSTOM_MAX PGSTAT_KIND_MAX
+#define PGSTAT_KIND_CUSTOM_SIZE (PGSTAT_KIND_CUSTOM_MAX - PGSTAT_KIND_CUSTOM_MIN + 1)
+
+/*
+ * PgStat_Kind to use for extensions that require an ID, but are still in
+ * development and have not reserved their own unique kind ID yet. See:
+ * https://wiki.postgresql.org/wiki/CustomCumulativeStats
+ */
+#define PGSTAT_KIND_EXPERIMENTAL 128
+
+static inline bool
+pgstat_is_kind_builtin(PgStat_Kind kind)
+{
+ return kind > PGSTAT_KIND_INVALID && kind <= PGSTAT_KIND_MAX_BUILTIN;
+}
+
+static inline bool
+pgstat_is_kind_custom(PgStat_Kind kind)
+{
+ return kind >= PGSTAT_KIND_CUSTOM_MIN && kind <= PGSTAT_KIND_CUSTOM_MAX;
+}
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 778f625ca1..39f63362a3 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -195,7 +195,8 @@ typedef struct PgStat_KindInfo
/*
* The size of an entry in the shared stats hash table (pointed to by
- * PgStatShared_HashEntry->body).
+ * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is
+ * the size of an entry in PgStat_ShmemControl->custom_data.
*/
uint32 shared_size;
@@ -446,6 +447,13 @@ typedef struct PgStat_ShmemControl
PgStatShared_IO io;
PgStatShared_SLRU slru;
PgStatShared_Wal wal;
+
+ /*
+ * Custom stats data with fixed-numbered objects, indexed by (PgStat_Kind
+ * - PGSTAT_KIND_CUSTOM_MIN).
+ */
+ void *custom_data[PGSTAT_KIND_CUSTOM_SIZE];
+
} PgStat_ShmemControl;
@@ -459,7 +467,7 @@ typedef struct PgStat_Snapshot
/* time at which snapshot was taken */
TimestampTz snapshot_timestamp;
- bool fixed_valid[PGSTAT_NUM_KINDS];
+ bool fixed_valid[PGSTAT_KIND_MAX_BUILTIN + 1];
PgStat_ArchiverStats archiver;
@@ -473,6 +481,14 @@ typedef struct PgStat_Snapshot
PgStat_WalStats wal;
+ /*
+ * Data in snapshot for custom fixed-numbered statistics, indexed by
+ * (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in
+ * TopMemoryContext, for a size of shared_data_len.
+ */
+ bool custom_valid[PGSTAT_KIND_CUSTOM_SIZE];
+ void *custom_data[PGSTAT_KIND_CUSTOM_SIZE];
+
/* to free snapshot in bulk */
MemoryContext context;
struct pgstat_snapshot_hash *stats;
@@ -516,6 +532,8 @@ static inline void *pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common
*/
extern const PgStat_KindInfo *pgstat_get_kind_info(PgStat_Kind kind);
+extern void pgstat_register_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
#ifdef USE_ASSERT_CHECKING
extern void pgstat_assert_is_up(void);
diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c
index 143478aca0..9bc8baca9c 100644
--- a/src/backend/utils/activity/pgstat.c
+++ b/src/backend/utils/activity/pgstat.c
@@ -49,8 +49,16 @@
* pgStatPending list. Pending statistics updates are flushed out by
* pgstat_report_stat().
*
+ * It is possible for external modules to define custom statistics kinds,
+ * that can use the same properties as any built-in stats kinds. Each custom
+ * stats kind needs to assign a unique ID to ensure that it does not overlap
+ * with other extensions. In order to reserve a unique stats kind ID, refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats.
+ *
* The behavior of different kinds of statistics is determined by the kind's
- * entry in pgstat_kind_infos, see PgStat_KindInfo for details.
+ * entry in pgstat_kind_builtin_infos for all the built-in statistics kinds
+ * defined, and pgstat_kind_custom_infos for custom kinds registered at
+ * startup by pgstat_register_kind(). See PgStat_KindInfo for details.
*
* The consistency of read accesses to statistics can be configured using the
* stats_fetch_consistency GUC (see config.sgml and monitoring.sgml for the
@@ -175,6 +183,8 @@ typedef struct PgStat_SnapshotEntry
static void pgstat_write_statsfile(XLogRecPtr redo);
static void pgstat_read_statsfile(XLogRecPtr redo);
+static void pgstat_init_snapshot_fixed(void);
+
static void pgstat_reset_after_failure(void);
static bool pgstat_flush_pending_entries(bool nowait);
@@ -252,7 +262,7 @@ static bool pgstat_is_shutdown = false;
/*
- * The different kinds of statistics.
+ * The different kinds of built-in statistics.
*
* If reasonably possible, handling specific to one kind of stats should go
* through this abstraction, rather than making more of pgstat.c aware.
@@ -264,7 +274,7 @@ static bool pgstat_is_shutdown = false;
* seem to be a great way of doing that, given the split across multiple
* files.
*/
-static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
+static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_MAX_BUILTIN + 1] = {
/* stats kinds for variable-numbered objects */
@@ -437,6 +447,15 @@ static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = {
},
};
+/*
+ * Information about custom statistics kinds.
+ *
+ * These are saved in a different array than the built-in kinds to save
+ * in clarity with the initializations.
+ *
+ * Indexed by PGSTAT_KIND_CUSTOM_MIN, of size PGSTAT_KIND_CUSTOM_SIZE.
+ */
+static const PgStat_KindInfo **pgstat_kind_custom_infos = NULL;
/* ------------------------------------------------------------
* Functions managing the state of the stats system for all backends.
@@ -587,6 +606,8 @@ pgstat_initialize(void)
pgstat_init_wal();
+ pgstat_init_snapshot_fixed();
+
/* Set up a process-exit hook to clean up */
before_shmem_exit(pgstat_shutdown_hook, 0);
@@ -830,6 +851,8 @@ pgstat_clear_snapshot(void)
memset(&pgStatLocal.snapshot.fixed_valid, 0,
sizeof(pgStatLocal.snapshot.fixed_valid));
+ memset(&pgStatLocal.snapshot.custom_valid, 0,
+ sizeof(pgStatLocal.snapshot.custom_valid));
pgStatLocal.snapshot.stats = NULL;
pgStatLocal.snapshot.mode = PGSTAT_FETCH_CONSISTENCY_NONE;
@@ -993,7 +1016,29 @@ pgstat_snapshot_fixed(PgStat_Kind kind)
else
pgstat_build_snapshot_fixed(kind);
- Assert(pgStatLocal.snapshot.fixed_valid[kind]);
+ if (pgstat_is_kind_builtin(kind))
+ Assert(pgStatLocal.snapshot.fixed_valid[kind]);
+ else if (pgstat_is_kind_custom(kind))
+ Assert(pgStatLocal.snapshot.custom_valid[kind - PGSTAT_KIND_CUSTOM_MIN]);
+}
+
+static void
+pgstat_init_snapshot_fixed(void)
+{
+ /*
+ * Initialize fixed-numbered statistics data in snapshots, only for custom
+ * stats kinds.
+ */
+ for (int kind = PGSTAT_KIND_CUSTOM_MIN; kind <= PGSTAT_KIND_CUSTOM_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info || !kind_info->fixed_amount)
+ continue;
+
+ pgStatLocal.snapshot.custom_data[kind - PGSTAT_KIND_CUSTOM_MIN] =
+ MemoryContextAlloc(TopMemoryContext, kind_info->shared_data_len);
+ }
}
static void
@@ -1089,10 +1134,12 @@ pgstat_build_snapshot(void)
/*
* Build snapshot of all fixed-numbered stats.
*/
- for (PgStat_Kind kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ if (!kind_info)
+ continue;
if (!kind_info->fixed_amount)
{
Assert(kind_info->snapshot_cb == NULL);
@@ -1109,6 +1156,20 @@ static void
pgstat_build_snapshot_fixed(PgStat_Kind kind)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+ int idx;
+ bool *valid;
+
+ /* Position in fixed_valid or custom_valid */
+ if (pgstat_is_kind_builtin(kind))
+ {
+ idx = kind;
+ valid = pgStatLocal.snapshot.fixed_valid;
+ }
+ else
+ {
+ idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+ valid = pgStatLocal.snapshot.custom_valid;
+ }
Assert(kind_info->fixed_amount);
Assert(kind_info->snapshot_cb != NULL);
@@ -1116,21 +1177,21 @@ pgstat_build_snapshot_fixed(PgStat_Kind kind)
if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE)
{
/* rebuild every time */
- pgStatLocal.snapshot.fixed_valid[kind] = false;
+ valid[idx] = false;
}
- else if (pgStatLocal.snapshot.fixed_valid[kind])
+ else if (valid[idx])
{
/* in snapshot mode we shouldn't get called again */
Assert(pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_CACHE);
return;
}
- Assert(!pgStatLocal.snapshot.fixed_valid[kind]);
+ Assert(!valid[idx]);
kind_info->snapshot_cb();
- Assert(!pgStatLocal.snapshot.fixed_valid[kind]);
- pgStatLocal.snapshot.fixed_valid[kind] = true;
+ Assert(!valid[idx]);
+ valid[idx] = true;
}
@@ -1286,30 +1347,127 @@ pgstat_flush_pending_entries(bool nowait)
PgStat_Kind
pgstat_get_kind_from_str(char *kind_str)
{
- for (PgStat_Kind kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN_BUILTIN; kind <= PGSTAT_KIND_MAX_BUILTIN; kind++)
{
- if (pg_strcasecmp(kind_str, pgstat_kind_infos[kind].name) == 0)
+ if (pg_strcasecmp(kind_str, pgstat_kind_builtin_infos[kind].name) == 0)
return kind;
}
+ /* Check the custom set of cumulative stats */
+ if (pgstat_kind_custom_infos)
+ {
+ for (int kind = 0; kind < PGSTAT_KIND_CUSTOM_SIZE; kind++)
+ {
+ if (pgstat_kind_custom_infos[kind] &&
+ pg_strcasecmp(kind_str, pgstat_kind_custom_infos[kind]->name) == 0)
+ return kind + PGSTAT_KIND_CUSTOM_MIN;
+ }
+ }
+
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid statistics kind: \"%s\"", kind_str)));
- return PGSTAT_KIND_DATABASE; /* avoid compiler warnings */
+ return PGSTAT_KIND_INVALID; /* avoid compiler warnings */
}
static inline bool
pgstat_is_kind_valid(PgStat_Kind kind)
{
- return kind >= PGSTAT_KIND_FIRST_VALID && kind <= PGSTAT_KIND_LAST;
+ return pgstat_is_kind_builtin(kind) || pgstat_is_kind_custom(kind);
}
const PgStat_KindInfo *
pgstat_get_kind_info(PgStat_Kind kind)
{
- Assert(pgstat_is_kind_valid(kind));
+ if (pgstat_is_kind_builtin(kind))
+ return &pgstat_kind_builtin_infos[kind];
- return &pgstat_kind_infos[kind];
+ if (pgstat_is_kind_custom(kind))
+ {
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (pgstat_kind_custom_infos == NULL ||
+ pgstat_kind_custom_infos[idx] == NULL)
+ return NULL;
+ return pgstat_kind_custom_infos[idx];
+ }
+
+ return NULL;
+}
+
+/*
+ * Register a new stats kind.
+ *
+ * PgStat_Kinds must be globally unique across all extensions. Refer
+ * to https://wiki.postgresql.org/wiki/CustomCumulativeStats to reserve a
+ * unique ID for your extension, to avoid conflicts with other extension
+ * developers. During development, use PGSTAT_KIND_EXPERIMENTAL to avoid
+ * needlessly reserving a new ID.
+ */
+void
+pgstat_register_kind(PgStat_Kind kind, const PgStat_KindInfo *kind_info)
+{
+ uint32 idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ if (kind_info->name == NULL || strlen(kind_info->name) == 0)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics name is invalid"),
+ errhint("Provide a non-empty name for the custom cumulative statistics.")));
+
+ if (!pgstat_is_kind_custom(kind))
+ ereport(ERROR, (errmsg("custom cumulative statistics ID %u is out of range", kind),
+ errhint("Provide a custom cumulative statistics ID between %u and %u.",
+ PGSTAT_KIND_CUSTOM_MIN, PGSTAT_KIND_CUSTOM_MAX)));
+
+ if (!process_shared_preload_libraries_in_progress)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics must be registered while initializing modules in \"shared_preload_libraries\".")));
+
+ /*
+ * Check some data for fixed-numbered stats.
+ */
+ if (kind_info->fixed_amount)
+ {
+ if (kind_info->shared_size == 0)
+ ereport(ERROR,
+ (errmsg("custom cumulative statistics property is invalid"),
+ errhint("Custom cumulative statistics require a shared memory size for fixed-numbered objects.")));
+ }
+
+ /*
+ * If pgstat_kind_custom_infos is not available yet, allocate it.
+ */
+ if (pgstat_kind_custom_infos == NULL)
+ {
+ pgstat_kind_custom_infos = (const PgStat_KindInfo **)
+ MemoryContextAllocZero(TopMemoryContext,
+ sizeof(PgStat_KindInfo *) * PGSTAT_KIND_CUSTOM_SIZE);
+ }
+
+ if (pgstat_kind_custom_infos[idx] != NULL &&
+ pgstat_kind_custom_infos[idx]->name != NULL)
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Custom cumulative statistics \"%s\" already registered with the same ID.",
+ pgstat_kind_custom_infos[idx]->name)));
+
+ /* check for existing custom stats with the same name */
+ for (int existing_kind = 0; existing_kind < PGSTAT_KIND_CUSTOM_SIZE; existing_kind++)
+ {
+ if (pgstat_kind_custom_infos[existing_kind] == NULL)
+ continue;
+ if (!pg_strcasecmp(pgstat_kind_custom_infos[existing_kind]->name, kind_info->name))
+ ereport(ERROR,
+ (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind),
+ errdetail("Existing cumulative statistics with ID %u has the same name.", existing_kind + PGSTAT_KIND_CUSTOM_MIN)));
+ }
+
+ /* Register it */
+ pgstat_kind_custom_infos[idx] = kind_info;
+ ereport(LOG,
+ (errmsg("registered custom cumulative statistics \"%s\" with ID %u",
+ kind_info->name, kind)));
}
/*
@@ -1393,18 +1551,22 @@ pgstat_write_statsfile(XLogRecPtr redo)
write_chunk_s(fpout, &redo);
/* Write various stats structs for fixed number of objects */
- for (PgStat_Kind kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
char *ptr;
const PgStat_KindInfo *info = pgstat_get_kind_info(kind);
- if (!info->fixed_amount)
+ if (!info || !info->fixed_amount)
continue;
- Assert(info->snapshot_ctl_off != 0);
+ if (pgstat_is_kind_builtin(kind))
+ Assert(info->snapshot_ctl_off != 0);
pgstat_build_snapshot_fixed(kind);
- ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) &pgStatLocal.snapshot) + info->snapshot_ctl_off;
+ else
+ ptr = pgStatLocal.snapshot.custom_data[kind - PGSTAT_KIND_CUSTOM_MIN];
fputc(PGSTAT_FILE_ENTRY_FIXED, fpout);
write_chunk_s(fpout, &kind);
@@ -1427,6 +1589,17 @@ pgstat_write_statsfile(XLogRecPtr redo)
if (ps->dropped)
continue;
+ /*
+ * This discards data related to custom stats kinds that are unknown
+ * to this process.
+ */
+ if (!pgstat_is_kind_valid(ps->key.kind))
+ {
+ elog(WARNING, "found unknown stats entry %u/%u/%u",
+ ps->key.kind, ps->key.dboid, ps->key.objoid);
+ continue;
+ }
+
shstats = (PgStatShared_Common *) dsa_get_address(pgStatLocal.dsa, ps->body);
kind_info = pgstat_get_kind_info(ps->key.kind);
@@ -1613,8 +1786,16 @@ pgstat_read_statsfile(XLogRecPtr redo)
}
/* Load back stats into shared memory */
- ptr = ((char *) shmem) + info->shared_ctl_off +
- info->shared_data_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) shmem) + info->shared_ctl_off +
+ info->shared_data_off;
+ else
+ {
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ ptr = ((char *) shmem->custom_data[idx]) +
+ info->shared_data_off;
+ }
if (!read_chunk(fpin, ptr, info->shared_data_len))
{
@@ -1711,7 +1892,7 @@ pgstat_read_statsfile(XLogRecPtr redo)
if (found)
{
dshash_release_lock(pgStatLocal.shared_hash, p);
- elog(WARNING, "found duplicate stats entry %d/%u/%u of type %c",
+ elog(WARNING, "found duplicate stats entry %u/%u/%u of type %c",
key.kind, key.dboid, key.objoid, t);
goto error;
}
@@ -1777,11 +1958,11 @@ pgstat_reset_after_failure(void)
TimestampTz ts = GetCurrentTimestamp();
/* reset fixed-numbered stats */
- for (PgStat_Kind kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
kind_info->reset_all_cb(ts);
diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c
index 2d5f7d46de..c0ac9e4bbc 100644
--- a/src/backend/utils/activity/pgstat_shmem.c
+++ b/src/backend/utils/activity/pgstat_shmem.c
@@ -131,6 +131,21 @@ StatsShmemSize(void)
sz = MAXALIGN(sizeof(PgStat_ShmemControl));
sz = add_size(sz, pgstat_dsa_init_size());
+ /* Add shared memory for all the custom fixed-numbered statistics */
+ for (int kind = PGSTAT_KIND_CUSTOM_MIN; kind <= PGSTAT_KIND_CUSTOM_MAX; kind++)
+ {
+ const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
+
+ if (!kind_info)
+ continue;
+ if (!kind_info->fixed_amount)
+ continue;
+
+ Assert(kind_info->shared_size != 0);
+
+ sz += MAXALIGN(kind_info->shared_size);
+ }
+
return sz;
}
@@ -197,15 +212,25 @@ StatsShmemInit(void)
pg_atomic_init_u64(&ctl->gc_request_count, 1);
/* initialize fixed-numbered stats */
- for (PgStat_Kind kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; kind++)
+ for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++)
{
const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind);
char *ptr;
- if (!kind_info->fixed_amount)
+ if (!kind_info || !kind_info->fixed_amount)
continue;
- ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ if (pgstat_is_kind_builtin(kind))
+ ptr = ((char *) ctl) + kind_info->shared_ctl_off;
+ else
+ {
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ Assert(kind_info->shared_size != 0);
+ ctl->custom_data[idx] = ShmemAlloc(kind_info->shared_size);
+ ptr = ctl->custom_data[idx];
+ }
+
kind_info->init_shmem_cb(ptr);
}
}
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 3876339ee1..3221137123 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1696,7 +1696,7 @@ pg_stat_reset(PG_FUNCTION_ARGS)
* Reset some shared cluster-wide counters
*
* When adding a new reset target, ideally the name should match that in
- * pgstat_kind_infos, if relevant.
+ * pgstat_kind_builtin_infos, if relevant.
*/
Datum
pg_stat_reset_shared(PG_FUNCTION_ARGS)
--
2.45.2
v3-0002-Add-helper-routines-to-retrieve-custom-data-for-f.patchtext/x-diff; charset=us-asciiDownload
From 5bcbe74bdb930ce1bd46791419e5fdd195b5f0d4 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 18 Jul 2024 14:10:35 +0900
Subject: [PATCH v3 2/5] Add helper routines to retrieve custom data for
fixed-numbered stats
This is useful for extensions to get snapshot and shmem data for custom
statistics, so as these don't need to know about the current snapshots.
---
src/include/utils/pgstat_internal.h | 32 +++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 39f63362a3..6bbd49388c 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -525,6 +525,8 @@ static inline int pgstat_cmp_hash_key(const void *a, const void *b, size_t size,
static inline uint32 pgstat_hash_hash_key(const void *d, size_t size, void *arg);
static inline size_t pgstat_get_entry_len(PgStat_Kind kind);
static inline void *pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common *entry);
+static inline void *pgstat_get_custom_shmem_data(PgStat_Kind kind);
+static inline void *pgstat_get_custom_snapshot_data(PgStat_Kind kind);
/*
@@ -842,4 +844,34 @@ pgstat_get_entry_data(PgStat_Kind kind, PgStatShared_Common *entry)
return ((char *) (entry)) + off;
}
+/*
+ * Returns a pointer to the shared memory area of custom stats for
+ * fixed-numbered statistics.
+ */
+static inline void *
+pgstat_get_custom_shmem_data(PgStat_Kind kind)
+{
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ Assert(pgstat_is_kind_custom(kind));
+ Assert(pgstat_get_kind_info(kind)->fixed_amount);
+
+ return pgStatLocal.shmem->custom_data[idx];
+}
+
+/*
+ * Returns a pointer to the portion of custom data for fixed-numbered
+ * statistics in the current snapshot.
+ */
+static inline void *
+pgstat_get_custom_snapshot_data(PgStat_Kind kind)
+{
+ int idx = kind - PGSTAT_KIND_CUSTOM_MIN;
+
+ Assert(pgstat_is_kind_custom(kind));
+ Assert(pgstat_get_kind_info(kind)->fixed_amount);
+
+ return pgStatLocal.snapshot.custom_data[idx];
+}
+
#endif /* PGSTAT_INTERNAL_H */
--
2.45.2
v3-0003-doc-Add-section-for-Custom-Cumulative-Statistics-.patchtext/x-diff; charset=us-asciiDownload
From 99d3799031bd7d8f6f19880c356337fe14208e9a Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 18 Jul 2024 14:14:58 +0900
Subject: [PATCH v3 3/5] doc: Add section for Custom Cumulative Statistics APIs
This provides a short description of what can be done, with a pointer to
the template in the tree.
---
doc/src/sgml/xfunc.sgml | 62 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index bf76490cbc..fbdbdbd883 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3864,6 +3864,68 @@ extern bool InjectionPointDetach(const char *name);
</para>
</sect2>
+ <sect2 id="xfunc-addin-custom-cumulative-statistics">
+ <title>Custom Cumulative Statistics</title>
+
+ <para>
+ Is is possible for add-ins written in C-language to use custom types
+ of cumulative statistics registered in the
+ <link linkend="monitoring-stats-setup">Cumulative Statistics System</link>.
+ </para>
+
+ <para>
+ First, define a <literal>PgStat_KindInfo</literal> that includes all
+ the information related to the custom type registered. For example:
+<programlisting>
+static const PgStat_KindInfo custom_stats = {
+ .name = "custom_stats",
+ .fixed_amount = false,
+ .shared_size = sizeof(PgStatShared_Custom),
+ .shared_data_off = offsetof(PgStatShared_Custom, stats),
+ .shared_data_len = sizeof(((PgStatShared_Custom *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatCustomEntry),
+}
+</programlisting>
+
+ Then, each backend that needs to use this custom type needs to register
+ it with <literal>pgstat_register_kind</literal> and a unique ID used to
+ store the entries related to this type of statistics:
+<programlisting>
+extern PgStat_Kind pgstat_add_kind(PgStat_Kind kind,
+ const PgStat_KindInfo *kind_info);
+</programlisting>
+ While developing a new extension, use
+ <literal>PGSTAT_KIND_EXPERIMENTAL</literal> for
+ <parameter>kind</parameter>. When you are ready to release the extension
+ to users, reserve a kind ID at the
+ <ulink url="https://wiki.postgresql.org/wiki/CustomCumulativeStats">
+ Custom Cumulative Statistics</ulink> page.
+ </para>
+
+ <para>
+ The details of the API for <literal>PgStat_KindInfo</literal> can
+ be found in <filename>src/include/utils/pgstat_internal.h</filename>.
+ </para>
+
+ <para>
+ The type of statistics registered is associated with a name and a unique
+ ID shared across the server in shared memory. Each backend using a
+ custom type of statistics maintains a local cache storing the information
+ of each custom <literal>PgStat_KindInfo</literal>.
+ </para>
+
+ <para>
+ Place the extension module implementing the custom cumulative statistics
+ type in <xref linkend="guc-shared-preload-libraries"/> so that it will
+ be loaded early during <productname>PostgreSQL</productname> startup.
+ </para>
+
+ <para>
+ An example describing how to register and use custom statistics can be
+ found in <filename>src/test/modules/injection_points</filename>.
+ </para>
+ </sect2>
+
<sect2 id="extend-cpp">
<title>Using C++ for Extensibility</title>
--
2.45.2
v3-0004-injection_points-Add-statistics-for-custom-points.patchtext/x-diff; charset=us-asciiDownload
From 47dacf574ece0f3b348c02e2a78ef602c3f14a23 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 18 Jul 2024 14:22:54 +0900
Subject: [PATCH v3 4/5] injection_points: Add statistics for custom points
This acts as a template of what can be achieved with the pluggable
cumulative stats APIs, while being useful on its own for injection
points.
Currently, the only data gathered is the number of times an injection
point is called. This can be extended as required. All the routines
related to the stats are located in their own file, for clarity.
A TAP test is included to provide coverage for these new APIs, showing
the persistency of the data across restarts.
---
src/test/modules/injection_points/Makefile | 11 +-
.../injection_points--1.0.sql | 10 +
.../injection_points/injection_points.c | 27 +++
.../injection_points/injection_stats.c | 197 ++++++++++++++++++
.../injection_points/injection_stats.h | 23 ++
src/test/modules/injection_points/meson.build | 9 +
.../modules/injection_points/t/001_stats.pl | 48 +++++
src/tools/pgindent/typedefs.list | 2 +
8 files changed, 325 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/injection_points/injection_stats.c
create mode 100644 src/test/modules/injection_points/injection_stats.h
create mode 100644 src/test/modules/injection_points/t/001_stats.pl
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 2ffd2f77ed..7b9cd12a2a 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -1,7 +1,10 @@
# src/test/modules/injection_points/Makefile
-MODULES = injection_points
-
+MODULE_big = injection_points
+OBJS = \
+ $(WIN32RES) \
+ injection_points.o \
+ injection_stats.o
EXTENSION = injection_points
DATA = injection_points--1.0.sql
PGFILEDESC = "injection_points - facility for injection points"
@@ -11,9 +14,13 @@ REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
ISOLATION = inplace
+TAP_TESTS = 1
+
# The injection points are cluster-wide, so disable installcheck
NO_INSTALLCHECK = 1
+export enable_injection_points enable_injection_points
+
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index 0f280419a5..b98d571889 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -74,3 +74,13 @@ CREATE FUNCTION injection_points_detach(IN point_name TEXT)
RETURNS void
AS 'MODULE_PATHNAME', 'injection_points_detach'
LANGUAGE C STRICT PARALLEL UNSAFE;
+
+--
+-- injection_points_stats_numcalls()
+--
+-- Reports statistics, if any, related to the given injection point.
+--
+CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
+RETURNS bigint
+AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index 15f9d0233c..acec4e95b0 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -18,6 +18,7 @@
#include "postgres.h"
#include "fmgr.h"
+#include "injection_stats.h"
#include "miscadmin.h"
#include "nodes/pg_list.h"
#include "nodes/value.h"
@@ -170,6 +171,9 @@ injection_points_cleanup(int code, Datum arg)
char *name = strVal(lfirst(lc));
(void) InjectionPointDetach(name);
+
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
}
}
@@ -182,6 +186,8 @@ injection_error(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(ERROR, "error triggered for injection point %s", name);
}
@@ -193,6 +199,8 @@ injection_notice(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
elog(NOTICE, "notice triggered for injection point %s", name);
}
@@ -211,6 +219,8 @@ injection_wait(const char *name, const void *private_data)
if (!injection_point_allowed(condition))
return;
+ pgstat_report_inj(name);
+
/*
* Use the injection point name for this custom wait event. Note that
* this custom wait event name is not released, but we don't care much for
@@ -299,6 +309,10 @@ injection_points_attach(PG_FUNCTION_ARGS)
inj_list_local = lappend(inj_list_local, makeString(pstrdup(name)));
MemoryContextSwitchTo(oldctx);
}
+
+ /* Add entry for stats */
+ pgstat_create_inj(name);
+
PG_RETURN_VOID();
}
@@ -431,5 +445,18 @@ injection_points_detach(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldctx);
}
+ /* Remove stats entry */
+ pgstat_drop_inj(name);
+
PG_RETURN_VOID();
}
+
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ pgstat_register_inj();
+}
diff --git a/src/test/modules/injection_points/injection_stats.c b/src/test/modules/injection_points/injection_stats.c
new file mode 100644
index 0000000000..78042074ff
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.c
@@ -0,0 +1,197 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.c
+ * Code for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+
+#include "common/hashfn.h"
+#include "injection_stats.h"
+#include "pgstat.h"
+#include "utils/builtins.h"
+#include "utils/pgstat_internal.h"
+
+/* Structures for statistics of injection points */
+typedef struct PgStat_StatInjEntry
+{
+ PgStat_Counter numcalls; /* number of times point has been run */
+} PgStat_StatInjEntry;
+
+typedef struct PgStatShared_InjectionPoint
+{
+ PgStatShared_Common header;
+ PgStat_StatInjEntry stats;
+} PgStatShared_InjectionPoint;
+
+static bool injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait);
+
+static const PgStat_KindInfo injection_stats = {
+ .name = "injection_points",
+ .fixed_amount = false, /* Bounded by the number of points */
+
+ /* Injection points are system-wide */
+ .accessed_across_databases = true,
+
+ .shared_size = sizeof(PgStatShared_InjectionPoint),
+ .shared_data_off = offsetof(PgStatShared_InjectionPoint, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPoint *) 0)->stats),
+ .pending_size = sizeof(PgStat_StatInjEntry),
+ .flush_pending_cb = injection_stats_flush_cb,
+};
+
+/*
+ * Compute stats entry idx from point name with a 4-byte hash.
+ */
+#define PGSTAT_INJ_IDX(name) hash_bytes((const unsigned char *) name, strlen(name))
+
+/*
+ * Kind ID reserved for statistics of injection points.
+ */
+#define PGSTAT_KIND_INJECTION 129
+
+/* Track if stats are loaded */
+static bool inj_stats_loaded = false;
+
+/*
+ * Callback for stats handling
+ */
+static bool
+injection_stats_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
+{
+ PgStat_StatInjEntry *localent;
+ PgStatShared_InjectionPoint *shfuncent;
+
+ localent = (PgStat_StatInjEntry *) entry_ref->pending;
+ shfuncent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ if (!pgstat_lock_entry(entry_ref, nowait))
+ return false;
+
+ shfuncent->stats.numcalls += localent->numcalls;
+ return true;
+}
+
+/*
+ * Support function for the SQL-callable pgstat* functions. Returns
+ * a pointer to the injection point statistics struct.
+ */
+static PgStat_StatInjEntry *
+pgstat_fetch_stat_injentry(const char *name)
+{
+ PgStat_StatInjEntry *entry = NULL;
+
+ if (!inj_stats_loaded)
+ return NULL;
+
+ /* Compile the lookup key as a hash of the point name */
+ entry = (PgStat_StatInjEntry *) pgstat_fetch_entry(PGSTAT_KIND_INJECTION,
+ InvalidOid,
+ PGSTAT_INJ_IDX(name));
+ return entry;
+}
+
+/*
+ * Workhorse to do the registration work, called in _PG_init().
+ */
+void
+pgstat_register_inj(void)
+{
+ pgstat_register_kind(PGSTAT_KIND_INJECTION, &injection_stats);
+
+ /* mark stats as loaded */
+ inj_stats_loaded = true;
+}
+
+/*
+ * Report injection point creation.
+ */
+void
+pgstat_create_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+
+ /* initialize shared memory data */
+ memset(&shstatent->stats, 0, sizeof(shstatent->stats));
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * Report injection point drop.
+ */
+void
+pgstat_drop_inj(const char *name)
+{
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ if (!pgstat_drop_entry(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name)))
+ pgstat_request_entry_refs_gc();
+}
+
+/*
+ * Report statistics for injection point.
+ *
+ * This is simple because the set of stats to report currently is simple:
+ * track the number of times a point has been run.
+ */
+void
+pgstat_report_inj(const char *name)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_InjectionPoint *shstatent;
+ PgStat_StatInjEntry *statent;
+
+ /* leave if disabled */
+ if (!inj_stats_loaded)
+ return;
+
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_INJECTION, InvalidOid,
+ PGSTAT_INJ_IDX(name), false);
+
+ shstatent = (PgStatShared_InjectionPoint *) entry_ref->shared_stats;
+ statent = &shstatent->stats;
+
+ /* Update the injection point statistics */
+ statent->numcalls++;
+
+ pgstat_unlock_entry(entry_ref);
+}
+
+/*
+ * SQL function returning the number of times an injection point
+ * has been called.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_numcalls);
+Datum
+injection_points_stats_numcalls(PG_FUNCTION_ARGS)
+{
+ char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ PgStat_StatInjEntry *entry = pgstat_fetch_stat_injentry(name);
+
+ if (entry == NULL)
+ PG_RETURN_NULL();
+
+ PG_RETURN_INT64(entry->numcalls);
+}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
new file mode 100644
index 0000000000..3e99705483
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -0,0 +1,23 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats.h
+ * Definitions for statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats.h
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#ifndef INJECTION_STATS
+#define INJECTION_STATS
+
+extern void pgstat_register_inj(void);
+extern void pgstat_create_inj(const char *name);
+extern void pgstat_drop_inj(const char *name);
+extern void pgstat_report_inj(const char *name);
+
+#endif
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index 3c23c14d81..a52fe5121e 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -6,6 +6,7 @@ endif
injection_points_sources = files(
'injection_points.c',
+ 'injection_stats.c',
)
if host_system == 'windows'
@@ -42,4 +43,12 @@ tests += {
'inplace',
],
},
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_stats.pl',
+ ],
+ },
}
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
new file mode 100644
index 0000000000..7d5a96e522
--- /dev/null
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -0,0 +1,48 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+use locale;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Test persistency of statistics generated for injection points.
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# Node initialization
+my $node = PostgreSQL::Test::Cluster->new('master');
+$node->init;
+$node->append_conf('postgresql.conf',
+ "shared_preload_libraries = 'injection_points'");
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# This should count for two calls.
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('stats-notice', 'notice');");
+$node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
+$node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
+my $numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats calls');
+
+# Restart the node cleanly, stats should still be around.
+$node->restart;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '2', 'number of stats after clean restart');
+
+# On crash the stats are gone.
+$node->stop('immediate');
+$node->start;
+$numcalls = $node->safe_psql('postgres',
+ "SELECT injection_points_stats_numcalls('stats-notice');");
+is($numcalls, '', 'number of stats after clean restart');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8de9978ad8..48febe4698 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2119,6 +2119,7 @@ PgStatShared_Common
PgStatShared_Database
PgStatShared_Function
PgStatShared_HashEntry
+PgStatShared_InjectionPoint
PgStatShared_IO
PgStatShared_Relation
PgStatShared_ReplSlot
@@ -2150,6 +2151,7 @@ PgStat_Snapshot
PgStat_SnapshotEntry
PgStat_StatDBEntry
PgStat_StatFuncEntry
+PgStat_StatInjEntry
PgStat_StatReplSlotEntry
PgStat_StatSubEntry
PgStat_StatTabEntry
--
2.45.2
v3-0005-injection_points-Add-example-for-fixed-numbered-s.patchtext/x-diff; charset=us-asciiDownload
From 29774effafad952be4a6e9d6aafc8b0638dbe90d Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 18 Jul 2024 14:44:33 +0900
Subject: [PATCH v3 5/5] injection_points: Add example for fixed-numbered
statistics
This acts as a template to show what can be achieved with fixed-numbered
stats (like WAL, bgwriter, etc.) for pluggable cumulative statistics.
---
src/test/modules/injection_points/Makefile | 3 +-
.../injection_points--1.0.sql | 11 +
.../injection_points/injection_points.c | 4 +
.../injection_points/injection_stats.h | 7 +
.../injection_points/injection_stats_fixed.c | 192 ++++++++++++++++++
src/test/modules/injection_points/meson.build | 1 +
.../modules/injection_points/t/001_stats.pl | 11 +-
7 files changed, 227 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/injection_points/injection_stats_fixed.c
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 7b9cd12a2a..ed28cd13a8 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -4,7 +4,8 @@ MODULE_big = injection_points
OBJS = \
$(WIN32RES) \
injection_points.o \
- injection_stats.o
+ injection_stats.o \
+ injection_stats_fixed.o
EXTENSION = injection_points
DATA = injection_points--1.0.sql
PGFILEDESC = "injection_points - facility for injection points"
diff --git a/src/test/modules/injection_points/injection_points--1.0.sql b/src/test/modules/injection_points/injection_points--1.0.sql
index b98d571889..1b2a4938a9 100644
--- a/src/test/modules/injection_points/injection_points--1.0.sql
+++ b/src/test/modules/injection_points/injection_points--1.0.sql
@@ -84,3 +84,14 @@ CREATE FUNCTION injection_points_stats_numcalls(IN point_name TEXT)
RETURNS bigint
AS 'MODULE_PATHNAME', 'injection_points_stats_numcalls'
LANGUAGE C STRICT;
+
+--
+-- injection_points_stats_fixed()
+--
+-- Reports fixed-numbered statistics for injection points.
+CREATE FUNCTION injection_points_stats_fixed(OUT numattach int8,
+ OUT numdetach int8,
+ OUT numrun int8)
+RETURNS record
+AS 'MODULE_PATHNAME', 'injection_points_stats_fixed'
+LANGUAGE C STRICT;
diff --git a/src/test/modules/injection_points/injection_points.c b/src/test/modules/injection_points/injection_points.c
index acec4e95b0..dc02be1bbf 100644
--- a/src/test/modules/injection_points/injection_points.c
+++ b/src/test/modules/injection_points/injection_points.c
@@ -297,6 +297,7 @@ injection_points_attach(PG_FUNCTION_ARGS)
condition.pid = MyProcPid;
}
+ pgstat_report_inj_fixed(1, 0, 0);
InjectionPointAttach(name, "injection_points", function, &condition,
sizeof(InjectionPointCondition));
@@ -342,6 +343,7 @@ injection_points_run(PG_FUNCTION_ARGS)
{
char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ pgstat_report_inj_fixed(0, 0, 1);
INJECTION_POINT(name);
PG_RETURN_VOID();
@@ -432,6 +434,7 @@ injection_points_detach(PG_FUNCTION_ARGS)
{
char *name = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ pgstat_report_inj_fixed(0, 1, 0);
if (!InjectionPointDetach(name))
elog(ERROR, "could not detach injection point \"%s\"", name);
@@ -459,4 +462,5 @@ _PG_init(void)
return;
pgstat_register_inj();
+ pgstat_register_inj_fixed();
}
diff --git a/src/test/modules/injection_points/injection_stats.h b/src/test/modules/injection_points/injection_stats.h
index 3e99705483..d519f29f83 100644
--- a/src/test/modules/injection_points/injection_stats.h
+++ b/src/test/modules/injection_points/injection_stats.h
@@ -15,9 +15,16 @@
#ifndef INJECTION_STATS
#define INJECTION_STATS
+/* injection_stats.c */
extern void pgstat_register_inj(void);
extern void pgstat_create_inj(const char *name);
extern void pgstat_drop_inj(const char *name);
extern void pgstat_report_inj(const char *name);
+/* injection_stats_fixed.c */
+extern void pgstat_register_inj_fixed(void);
+extern void pgstat_report_inj_fixed(uint32 numattach,
+ uint32 numdetach,
+ uint32 numrun);
+
#endif
diff --git a/src/test/modules/injection_points/injection_stats_fixed.c b/src/test/modules/injection_points/injection_stats_fixed.c
new file mode 100644
index 0000000000..75639328a8
--- /dev/null
+++ b/src/test/modules/injection_points/injection_stats_fixed.c
@@ -0,0 +1,192 @@
+/*--------------------------------------------------------------------------
+ *
+ * injection_stats_fixed.c
+ * Code for fixed-numbered statistics of injection points.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/test/modules/injection_points/injection_stats_fixed.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+
+#include "common/hashfn.h"
+#include "funcapi.h"
+#include "injection_stats.h"
+#include "pgstat.h"
+#include "utils/builtins.h"
+#include "utils/pgstat_internal.h"
+
+/* Structures for statistics of injection points, fixed-size */
+typedef struct PgStat_StatInjFixedEntry
+{
+ PgStat_Counter numattach; /* number of points attached */
+ PgStat_Counter numdetach; /* number of points detached */
+ PgStat_Counter numrun; /* number of points run */
+ TimestampTz stat_reset_timestamp;
+} PgStat_StatInjFixedEntry;
+
+typedef struct PgStatShared_InjectionPointFixed
+{
+ LWLock lock; /* protects all the counters */
+ uint32 changecount;
+ PgStat_StatInjFixedEntry stats;
+ PgStat_StatInjFixedEntry reset_offset;
+} PgStatShared_InjectionPointFixed;
+
+/* Callbacks for fixed-numbered stats */
+static void injection_stats_fixed_init_shmem_cb(void *stats);
+static void injection_stats_fixed_reset_all_cb(TimestampTz ts);
+static void injection_stats_fixed_snapshot_cb(void);
+
+static const PgStat_KindInfo injection_stats_fixed = {
+ .name = "injection_points_fixed",
+ .fixed_amount = true,
+
+ .shared_size = sizeof(PgStat_StatInjFixedEntry),
+ .shared_data_off = offsetof(PgStatShared_InjectionPointFixed, stats),
+ .shared_data_len = sizeof(((PgStatShared_InjectionPointFixed *) 0)->stats),
+
+ .init_shmem_cb = injection_stats_fixed_init_shmem_cb,
+ .reset_all_cb = injection_stats_fixed_reset_all_cb,
+ .snapshot_cb = injection_stats_fixed_snapshot_cb,
+};
+
+/*
+ * Kind ID reserved for statistics of injection points.
+ */
+#define PGSTAT_KIND_INJECTION_FIXED 130
+
+/* Track if fixed-numbered stats are loaded */
+static bool inj_fixed_loaded = false;
+
+static void
+injection_stats_fixed_init_shmem_cb(void *stats)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ (PgStatShared_InjectionPointFixed *) stats;
+
+ LWLockInitialize(&stats_shmem->lock, LWTRANCHE_PGSTATS_DATA);
+}
+
+static void
+injection_stats_fixed_reset_all_cb(TimestampTz ts)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ pgstat_get_custom_shmem_data(PGSTAT_KIND_INJECTION_FIXED);
+
+ LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE);
+ pgstat_copy_changecounted_stats(&stats_shmem->reset_offset,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+ stats_shmem->stats.stat_reset_timestamp = ts;
+ LWLockRelease(&stats_shmem->lock);
+}
+
+static void
+injection_stats_fixed_snapshot_cb(void)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem =
+ pgstat_get_custom_shmem_data(PGSTAT_KIND_INJECTION_FIXED);
+ PgStat_StatInjFixedEntry *stat_snap =
+ pgstat_get_custom_snapshot_data(PGSTAT_KIND_INJECTION_FIXED);
+ PgStat_StatInjFixedEntry *reset_offset = &stats_shmem->reset_offset;
+ PgStat_StatInjFixedEntry reset;
+
+ pgstat_copy_changecounted_stats(stat_snap,
+ &stats_shmem->stats,
+ sizeof(stats_shmem->stats),
+ &stats_shmem->changecount);
+
+ LWLockAcquire(&stats_shmem->lock, LW_SHARED);
+ memcpy(&reset, reset_offset, sizeof(stats_shmem->stats));
+ LWLockRelease(&stats_shmem->lock);
+
+ /* compensate by reset offsets */
+#define FIXED_COMP(fld) stat_snap->fld -= reset.fld;
+ FIXED_COMP(numattach);
+ FIXED_COMP(numdetach);
+ FIXED_COMP(numrun);
+#undef FIXED_COMP
+}
+
+/*
+ * Workhorse to do the registration work, called in _PG_init().
+ */
+void
+pgstat_register_inj_fixed(void)
+{
+ pgstat_register_kind(PGSTAT_KIND_INJECTION_FIXED, &injection_stats_fixed);
+
+ /* mark stats as loaded */
+ inj_fixed_loaded = true;
+}
+
+/*
+ * Report fixed number of statistics for an injection point.
+ */
+void
+pgstat_report_inj_fixed(uint32 numattach,
+ uint32 numdetach,
+ uint32 numrun)
+{
+ PgStatShared_InjectionPointFixed *stats_shmem;
+
+ /* leave if disabled */
+ if (!inj_fixed_loaded)
+ return;
+
+ stats_shmem = pgstat_get_custom_shmem_data(PGSTAT_KIND_INJECTION_FIXED);
+
+ pgstat_begin_changecount_write(&stats_shmem->changecount);
+ stats_shmem->stats.numattach += numattach;
+ stats_shmem->stats.numdetach += numdetach;
+ stats_shmem->stats.numrun += numrun;
+ pgstat_end_changecount_write(&stats_shmem->changecount);
+}
+
+/*
+ * SQL function returning fixed-numbered statistics for injection points.
+ */
+PG_FUNCTION_INFO_V1(injection_points_stats_fixed);
+Datum
+injection_points_stats_fixed(PG_FUNCTION_ARGS)
+{
+ TupleDesc tupdesc;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ PgStat_StatInjFixedEntry *stats;
+
+ if (!inj_fixed_loaded)
+ PG_RETURN_NULL();
+
+ pgstat_snapshot_fixed(PGSTAT_KIND_INJECTION_FIXED);
+ stats = pgstat_get_custom_snapshot_data(PGSTAT_KIND_INJECTION_FIXED);
+
+ /* Initialise attributes information in the tuple descriptor */
+ tupdesc = CreateTemplateTupleDesc(3);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "numattach",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "numdetach",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "numrun",
+ INT8OID, -1, 0);
+ BlessTupleDesc(tupdesc);
+
+ values[0] = Int64GetDatum(stats->numattach);
+ values[1] = Int64GetDatum(stats->numdetach);
+ values[2] = Int64GetDatum(stats->numrun);
+ nulls[0] = false;
+ nulls[1] = false;
+ nulls[2] = false;
+
+ /* Returns the record as Datum */
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index a52fe5121e..c9e357f644 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -7,6 +7,7 @@ endif
injection_points_sources = files(
'injection_points.c',
'injection_stats.c',
+ 'injection_stats_fixed.c',
)
if host_system == 'windows'
diff --git a/src/test/modules/injection_points/t/001_stats.pl b/src/test/modules/injection_points/t/001_stats.pl
index 7d5a96e522..e3c69b94ca 100644
--- a/src/test/modules/injection_points/t/001_stats.pl
+++ b/src/test/modules/injection_points/t/001_stats.pl
@@ -31,18 +31,27 @@ $node->safe_psql('postgres', "SELECT injection_points_run('stats-notice');");
my $numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
is($numcalls, '2', 'number of stats calls');
+my $fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '1|0|2', 'number of fixed stats');
# Restart the node cleanly, stats should still be around.
$node->restart;
$numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
is($numcalls, '2', 'number of stats after clean restart');
+$fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '1|0|2', 'number of fixed stats after clean restart');
# On crash the stats are gone.
$node->stop('immediate');
$node->start;
$numcalls = $node->safe_psql('postgres',
"SELECT injection_points_stats_numcalls('stats-notice');");
-is($numcalls, '', 'number of stats after clean restart');
+is($numcalls, '', 'number of stats after crash');
+$fixedstats = $node->safe_psql('postgres',
+ "SELECT * FROM injection_points_stats_fixed();");
+is($fixedstats, '0|0|0', 'number of fixed stats after crash');
done_testing();
--
2.45.2
On Fri, Aug 02, 2024 at 05:53:31AM +0900, Michael Paquier wrote:
Attached is a rebased set of the rest, with 0001 now introducing the
pluggable core part.
So, I have been able to spend a few more days on all that while
travelling across three continents, and I have applied the core patch
followed by the template parts after more polishing. The core part
has been tweaked a bit more in terms of variable and structure names,
to bring the builtin and custom stats parts more consistent with each
other. There were also a bunch of loops that did not use the
PgStat_Kind, but an int with an index on the custom_data arrays. I
have uniformized the whole.
I am keeping an eye on the buildfarm and it is currently green. My
machines don't seem to have run the new tests with injection points
yet, the CI on the CF app is not reporting any failure caused by that,
and my CI runs have all been stable.
--
Michael