Get memory contexts of an arbitrary backend process
Hi,
After commit 3e98c0bafb28de, we can display the usage of the
memory contexts using pg_backend_memory_contexts system
view.
However, its target is limited to the process attached to
the current session.
As discussed in the thread[1]/messages/by-id/72a656e0f71d0860161e0b3f67e4d771@oss.nttdata.com, it'll be useful to make it
possible to get the memory contexts of an arbitrary backend
process.
Attached PoC patch makes pg_get_backend_memory_contexts()
display meory contexts of the specified PID of the process.
=# -- PID of the target process is 17051
=# SELECT * FROM pg_get_backend_memory_contexts(17051) ;
name | ident | parent | level |
total_bytes | total_nblocks | free_bytes | free_chunks | used_bytes
-----------------------+-------+------------------+-------+-------------+---------------+------------+-------------+------------
TopMemoryContext | | | 0 |
68720 | 5 | 16816 | 16 | 51904
RowDescriptionContext | | TopMemoryContext | 1 |
8192 | 1 | 6880 | 0 | 1312
MessageContext | | TopMemoryContext | 1 |
65536 | 4 | 19912 | 1 | 45624
...
It doesn't display contexts of all the backends but only
the contexts of specified process.
I think it would be enough because I suppose this function
is used after investigations using ps command or other OS
level utilities.
The rough idea of implementation is like below:
1. send a signal to the specified process
2. signaled process dumps its memory contexts to a file
3. read the dumped file and display it to the user
Any thoughts?
[1]: /messages/by-id/72a656e0f71d0860161e0b3f67e4d771@oss.nttdata.com
/messages/by-id/72a656e0f71d0860161e0b3f67e4d771@oss.nttdata.com
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
0001-Enabled-pg_get_backend_memory_contexts-to-collect-ar.patchtext/x-diff; name=0001-Enabled-pg_get_backend_memory_contexts-to-collect-ar.patchDownload
From 7decfda337bbc422fece4c736a719b6fcfdc5cf3 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 31 Aug 2020 18:20:34 +0900
Subject: [PATCH] Enabled pg_get_backend_memory_contexts() to collect arbitrary
backend process's memory contexts.
Previously, pg_get_backend_memory_contexts() could only get the
memory contexts of the process which kicked it. This patch enables
to get memory contexts of the arbitrary process which PID is
specified by the argument.
---
doc/src/sgml/func.sgml | 13 +
src/backend/catalog/system_views.sql | 4 +-
src/backend/replication/basebackup.c | 3 +
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/tcop/postgres.c | 5 +
src/backend/utils/adt/mcxtfuncs.c | 315 ++++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/bin/initdb/initdb.c | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 2 +-
src/bin/pg_rewind/filemap.c | 3 +
src/include/catalog/pg_proc.dat | 7 +-
src/include/miscadmin.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/mcxtfuncs.h | 21 ++
src/test/regress/expected/rules.out | 2 +-
15 files changed, 364 insertions(+), 21 deletions(-)
create mode 100644 src/include/utils/mcxtfuncs.h
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index b9f591296a..cc9a458334 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -21062,6 +21062,19 @@ SELECT * FROM pg_ls_dir('.') WITH ORDINALITY AS t(ls,n);
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_backend_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_backend_memory_contexts</function> ( <type>integer</type> )
+ <returnvalue>setof records</returnvalue>
+ </para>
+ <para>
+ Returns all the memory contexts of the specified process ID.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index a2d61302f9..88fb837ecd 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -555,10 +555,10 @@ REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
CREATE VIEW pg_backend_memory_contexts AS
- SELECT * FROM pg_get_backend_memory_contexts();
+ SELECT * FROM pg_get_backend_memory_contexts(-1);
REVOKE ALL ON pg_backend_memory_contexts FROM PUBLIC;
-REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts FROM PUBLIC;
-- Statistics views
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index 6064384e32..f69d851b6b 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -184,6 +184,9 @@ static const char *const excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 4fa385b0ec..5966729897 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -28,6 +28,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/mcxtfuncs.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -567,6 +568,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_DUMP_MEMORY))
+ HandleProcSignalDumpMemory();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index c9424f167c..d30426cf0f 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -76,6 +76,7 @@
#include "tcop/utility.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/mcxtfuncs.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
#include "utils/timeout.h"
@@ -539,6 +540,10 @@ ProcessClientReadInterrupt(bool blocked)
/* Process notify interrupts, if any */
if (notifyInterruptPending)
ProcessNotifyInterrupt();
+
+ /* Process memory contexts dump interrupts, if any */
+ if (ProcSignalDumpMemoryPending)
+ ProcessDumpMemoryInterrupt();
}
else if (ProcDiePending)
{
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 50e1b07ff0..12f7ecc116 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,34 +15,44 @@
#include "postgres.h"
+#include <unistd.h>
+
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "storage/latch.h"
+#include "storage/procsignal.h"
#include "utils/builtins.h"
+#include "utils/mcxtfuncs.h"
/* ----------
* The max bytes for showing identifiers of MemoryContext.
* ----------
*/
-#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MEMORY_CONTEXT_DISPLAY_SIZE 1024
+
+#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
/*
* PutMemoryContextsStatsTupleStore
* One recursion level for pg_get_backend_memory_contexts.
+ *
+ * Note: When fpout is not NULL, ferror() check must be done
+ * by the caller.
*/
static void
PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
TupleDesc tupdesc, MemoryContext context,
- const char *parent, int level)
+ const char *parent, int level, FILE *fpout)
{
-#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
-
Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
MemoryContextCounters stat;
MemoryContext child;
const char *name;
const char *ident;
+ int idlen;
AssertArg(MemoryContextIsValid(context));
@@ -73,22 +83,23 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
if (ident)
{
- int idlen = strlen(ident);
- char clipped_ident[MEMORY_CONTEXT_IDENT_DISPLAY_SIZE];
-
+ idlen = strlen(ident);
/*
* Some identifiers such as SQL query string can be very long,
* truncate oversize identifiers.
*/
- if (idlen >= MEMORY_CONTEXT_IDENT_DISPLAY_SIZE)
- idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_IDENT_DISPLAY_SIZE - 1);
+ if (idlen >= MEMORY_CONTEXT_DISPLAY_SIZE)
+ idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_DISPLAY_SIZE - 1);
memcpy(clipped_ident, ident, idlen);
clipped_ident[idlen] = '\0';
values[1] = CStringGetTextDatum(clipped_ident);
}
else
+ {
+ idlen = 0;
nulls[1] = true;
+ }
if (parent)
values[2] = CStringGetTextDatum(parent);
@@ -101,12 +112,41 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
values[6] = Int64GetDatum(stat.freespace);
values[7] = Int64GetDatum(stat.freechunks);
values[8] = Int64GetDatum(stat.totalspace - stat.freespace);
- tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ if(fpout == NULL)
+ /* pg_get_backend_memory_contexts() is called from local process */
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ else
+ {
+ int rc;
+ int parent_len = strlen(parent);
+ int name_len = strlen(name);
+
+ /*
+ * write out the current memory context information.
+ * Since some elements of values are reusable, we write it out.
+ */
+ fputc('D', fpout);
+ rc = fwrite(values, sizeof(values), 1, fpout);
+ rc = fwrite(nulls, sizeof(nulls), 1, fpout);
+
+ /* write out information which is not resuable from serialized values */
+ rc = fwrite(&name_len, sizeof(int), 1, fpout);
+ rc = fwrite(name, name_len, 1, fpout);
+ rc = fwrite(&idlen, sizeof(int), 1, fpout);
+ rc = fwrite(clipped_ident, idlen, 1, fpout);
+ rc = fwrite(&level, sizeof(int), 1, fpout);
+ rc = fwrite(&parent_len, sizeof(int), 1, fpout);
+ rc = fwrite(parent, parent_len, 1, fpout);
+ (void) rc; /* we'll check for error with ferror */
+
+ }
for (child = context->firstchild; child != NULL; child = child->nextchild)
{
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- child, name, level + 1);
+ child, name, level + 1, fpout);
}
}
@@ -117,6 +157,8 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
{
+ int pid = PG_GETARG_INT32(0);
+
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -147,11 +189,258 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldcontext);
- PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- TopMemoryContext, NULL, 0);
+ if (pid == -1)
+ {
+ /*
+ * Since pid -1 indicates target is the local process, simply
+ * traverse memory contexts.
+ */
+ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
+ TopMemoryContext, "", 0, NULL);
+ }
+ else
+ {
+ /*
+ * Send signal for dumping memory contexts to the target process,
+ * and read the dumped file.
+ */
+ FILE *fpin;
+ char dumpfile[MAXPGPATH];
+
+ SendProcSignal(pid, PROCSIG_DUMP_MEMORY, InvalidBackendId);
+
+ snprintf(dumpfile, sizeof(dumpfile), "pg_memusage/%d", pid);
+
+ while (true)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ pg_usleep(10000L);
+
+ if ((fpin = AllocateFile(dumpfile, PG_BINARY_R)) == NULL)
+ {
+ if (errno != ENOENT)
+ ereport(LOG, (errcode_for_file_access(),
+ errmsg("could not open temporary memory dump file \"%s\": %m",
+ dumpfile)));
+ }
+ else
+ {
+ elog(DEBUG1, "Succeeded opening temporary memory dump file \"%s\": %m",
+ dumpfile);
+ break;
+ }
+ }
+
+ /* read dumped files */
+ while (true)
+ {
+ Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char name[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char parent[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
+ int level;
+ int name_len;
+ int parent_len;
+ int idlen;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ switch (fgetc(fpin))
+ {
+ /* 'D' A memory context information follows. */
+ case 'D':
+ if (fread(values, 1, sizeof(values), fpin) != sizeof(values))
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ if (fread(nulls, 1, sizeof(nulls), fpin) != sizeof(nulls))
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ if (fread(&name_len, 1, sizeof(int), fpin) != sizeof(int))
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ if (fread(name, 1, name_len, fpin) != name_len)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ name[name_len] = '\0';
+ if (fread(&idlen, 1, sizeof(int), fpin) != sizeof(int))
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ if (fread(clipped_ident, 1, idlen, fpin) != idlen)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ clipped_ident[idlen] = '\0';
+ if (fread(&level, 1, sizeof(int), fpin) != sizeof(int))
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ if (fread(&parent_len, 1, sizeof(int), fpin) != sizeof(int))
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ if (fread(parent, 1, parent_len, fpin) != parent_len)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ parent[parent_len] = '\0';
+
+ /*
+ * Overwrite some elements of values which are not available
+ * from serialized values.
+ */
+ if (!nulls[0])
+ values[0] = CStringGetTextDatum(name);
+
+ if (!nulls[1])
+ values[1] = CStringGetTextDatum(clipped_ident);
+
+ if (!nulls[2])
+ values[2] = CStringGetTextDatum(parent);
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ break;
+
+ case 'E':
+ goto done;
+
+ default:
+ ereport(WARNING,
+ (errmsg("corrupted memory dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ }
+done:
+ FreeFile(fpin);
+ unlink(dumpfile);
+ }
/* clean up and return the tuplestore */
tuplestore_donestoring(tupstore);
return (Datum) 0;
}
+
+/*
+ * dump_memory_contexts
+ * Dumping local memory contexts to a file.
+ * This function does not delete the file as it is intended to be read by
+ * another process.
+ */
+static void
+dump_memory_contexts(void)
+{
+ FILE *fpout;
+ char tmpfile[MAXPGPATH];
+ char dumpfile[MAXPGPATH];
+
+ snprintf(tmpfile, sizeof(tmpfile), "pg_memusage/%d.tmp", MyProcPid);
+ snprintf(dumpfile, sizeof(dumpfile), "pg_memusage/%d", MyProcPid);
+
+ /*
+ * Open a temp file to dump the current memory context.
+ */
+ fpout = AllocateFile(tmpfile, PG_BINARY_W);
+ if (fpout == NULL)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write temporary memory context file \"%s\": %m",
+ tmpfile)));
+ return;
+ }
+
+ PutMemoryContextsStatsTupleStore(NULL, NULL,
+ TopMemoryContext, "", 0, fpout);
+
+ /* No more output to be done. */
+ fputc('E', fpout);
+
+ if (ferror(fpout))
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write temporary memory context dump file \"%s\": %m",
+ tmpfile)));
+ FreeFile(fpout);
+ unlink(tmpfile);
+ }
+ else if (FreeFile(fpout) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not close temporary memory context file \"%s\": %m",
+ tmpfile)));
+ unlink(tmpfile);
+ }
+ else if (rename(tmpfile, dumpfile) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not rename dump file \"%s\" to \"%s\": %m",
+ tmpfile, dumpfile)));
+ unlink(tmpfile);
+ }
+}
+
+/*
+ * ProcessDumpMemoryInterrupt
+ *
+ * The portion of memory dump interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessDumpMemoryInterrupt(void)
+{
+ ProcSignalDumpMemoryPending = false;
+ dump_memory_contexts();
+}
+
+/*
+ * Handle receipt of an interrupt indicating a dump memory context.
+ *
+ * Signal handler portion of interrupt handling.
+ */
+void
+HandleProcSignalDumpMemory(void)
+{
+ ProcSignalDumpMemoryPending = true;
+
+ /* make sure the event is processed in due course */
+ SetLatch(MyLatch);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 6ab8216839..463337f661 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -33,6 +33,7 @@ volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t ProcSignalDumpMemoryPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 786672b1b6..c3130dc6f9 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -220,7 +220,8 @@ static const char *const subdirs[] = {
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
- "pg_logical/mappings"
+ "pg_logical/mappings",
+ "pg_memusage"
};
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index f674a7c94e..683c2d5408 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -124,7 +124,7 @@ is_deeply(
# Contents of these directories should not be copied.
foreach my $dirname (
- qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans)
+ qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans pg_memusage)
)
{
is_deeply(
diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c
index 1879257b66..23471007d8 100644
--- a/src/bin/pg_rewind/filemap.c
+++ b/src/bin/pg_rewind/filemap.c
@@ -85,6 +85,9 @@ static const char *excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 1dd325e0e6..b6859b23e8 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7812,9 +7812,10 @@
{ oid => '2282', descr => 'information about all memory contexts of local backend',
proname => 'pg_get_backend_memory_contexts', prorows => '100', proretset => 't',
provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
- proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'int4',
+ proallargtypes => '{int4,text,text,text,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
# non-persistent series generator
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..812032bb15 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -83,6 +83,7 @@ extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t ProcSignalDumpMemoryPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 5cb39697f3..5db92a9a52 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_DUMP_MEMORY, /* request dumping memory context interrupt */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/mcxtfuncs.h b/src/include/utils/mcxtfuncs.h
new file mode 100644
index 0000000000..cd74bb6700
--- /dev/null
+++ b/src/include/utils/mcxtfuncs.h
@@ -0,0 +1,21 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcxtfuncs.h
+ * Declarations for showing backend memory context.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/mcxtfuncs.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MCXT_H
+#define MCXT_H
+
+extern void ProcessDumpMemoryInterrupt(void);
+extern void HandleProcSignalDumpMemory(void);
+
+#endif /* MCXT_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 2a18dc423e..990b54d777 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1333,7 +1333,7 @@ pg_backend_memory_contexts| SELECT pg_get_backend_memory_contexts.name,
pg_get_backend_memory_contexts.free_bytes,
pg_get_backend_memory_contexts.free_chunks,
pg_get_backend_memory_contexts.used_bytes
- FROM pg_get_backend_memory_contexts() pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
+ FROM pg_get_backend_memory_contexts('-1'::integer) pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
pg_config| SELECT pg_config.name,
pg_config.setting
FROM pg_config() pg_config(name, setting);
--
2.18.1
Hi,
On Mon, Aug 31, 2020 at 8:22 PM torikoshia <torikoshia@oss.nttdata.com> wrote:
As discussed in the thread[1], it'll be useful to make it
possible to get the memory contexts of an arbitrary backend
process.
+1
Attached PoC patch makes pg_get_backend_memory_contexts()
display meory contexts of the specified PID of the process.
Thanks, it's a very good patch for discussion.
It doesn't display contexts of all the backends but only
the contexts of specified process.
or we can "SELECT (pg_get_backend_memory_contexts(pid)).* FROM
pg_stat_activity WHERE ...",
so I don't think it's a big deal.
The rough idea of implementation is like below:
1. send a signal to the specified process
2. signaled process dumps its memory contexts to a file
3. read the dumped file and display it to the user
I agree with the overview of the idea.
Here are some comments and questions.
- Currently, "the signal transmission for dumping memory information"
and "the read & output of dump information "
are on the same interface, but I think it would be better to separate them.
How about providing the following three types of functions for users?
- send a signal to specified pid
- check the status of the signal sent and received
- read the dumped information
- How about managing the status of signal send/receive and dump
operations on a shared hash or others ?
Sending and receiving signals, dumping memory information, and
referencing dump information all work asynchronously.
Therefore, it would be good to have management information to check
the status of each process.
A simple idea is that ..
- send a signal to dump to a PID, it first record following
information into the shared hash.
pid (specified pid)
loc (dump location, currently might be ASAP)
recv (did the pid process receive a signal? first false)
dumped (did the pid process dump a mem information? first false)
- specified process receive the signal, update the status in the
shared hash, then dumped at specified location.
- specified process finish dump mem information, update the status
in the shared hash.
- Does it allow one process to output multiple dump files?
It appears to be a specification to overwrite at present, but I
thought it would be good to be able to generate
multiple dump files in different phases (e.g., planning phase and
execution phase) in the future.
- How is the dump file cleaned up?
Best regards,
--
Tatsuhito Kasahara
kasahara.tatsuhito _at_ gmail.com
Hi
po 31. 8. 2020 v 17:03 odesílatel Kasahara Tatsuhito <
kasahara.tatsuhito@gmail.com> napsal:
Hi,
On Mon, Aug 31, 2020 at 8:22 PM torikoshia <torikoshia@oss.nttdata.com>
wrote:As discussed in the thread[1], it'll be useful to make it
possible to get the memory contexts of an arbitrary backend
process.+1
Attached PoC patch makes pg_get_backend_memory_contexts()
display meory contexts of the specified PID of the process.Thanks, it's a very good patch for discussion.
It doesn't display contexts of all the backends but only
the contexts of specified process.or we can "SELECT (pg_get_backend_memory_contexts(pid)).* FROM
pg_stat_activity WHERE ...",
so I don't think it's a big deal.The rough idea of implementation is like below:
1. send a signal to the specified process
2. signaled process dumps its memory contexts to a file
3. read the dumped file and display it to the userI agree with the overview of the idea.
Here are some comments and questions.- Currently, "the signal transmission for dumping memory information"
and "the read & output of dump information "
are on the same interface, but I think it would be better to separate
them.
How about providing the following three types of functions for users?
- send a signal to specified pid
- check the status of the signal sent and received
- read the dumped information
- How about managing the status of signal send/receive and dump
operations on a shared hash or others ?
Sending and receiving signals, dumping memory information, and
referencing dump information all work asynchronously.
Therefore, it would be good to have management information to check
the status of each process.
A simple idea is that ..
- send a signal to dump to a PID, it first record following
information into the shared hash.
pid (specified pid)
loc (dump location, currently might be ASAP)
recv (did the pid process receive a signal? first false)
dumped (did the pid process dump a mem information? first false)
- specified process receive the signal, update the status in the
shared hash, then dumped at specified location.
- specified process finish dump mem information, update the status
in the shared hash.
- Does it allow one process to output multiple dump files?
It appears to be a specification to overwrite at present, but I
thought it would be good to be able to generate
multiple dump files in different phases (e.g., planning phase and
execution phase) in the future.
- How is the dump file cleaned up?
For a very long time there has been similar discussion about taking
session query and session execution plans from other sessions.
I am not sure how necessary information is in the memory dump, but I am
sure so taking the current execution plan and complete text of the current
query is pretty necessary information.
but can be great so this infrastructure can be used for any debugging
purpose.
Regards
Pavel
Show quoted text
Best regards,
--
Tatsuhito Kasahara
kasahara.tatsuhito _at_ gmail.com
Hi,
On 2020-08-31 20:22:18 +0900, torikoshia wrote:
After commit 3e98c0bafb28de, we can display the usage of the
memory contexts using pg_backend_memory_contexts system
view.However, its target is limited to the process attached to
the current session.As discussed in the thread[1], it'll be useful to make it
possible to get the memory contexts of an arbitrary backend
process.Attached PoC patch makes pg_get_backend_memory_contexts()
display meory contexts of the specified PID of the process.
Awesome!
It doesn't display contexts of all the backends but only
the contexts of specified process.
I think it would be enough because I suppose this function
is used after investigations using ps command or other OS
level utilities.
It can be used as a building block if all are needed. Getting the
infrastructure right is the big thing here, I think. Adding more
detailed views on top of that data later is easier.
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index a2d61302f9..88fb837ecd 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -555,10 +555,10 @@ REVOKE ALL ON pg_shmem_allocations FROM PUBLIC; REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;CREATE VIEW pg_backend_memory_contexts AS - SELECT * FROM pg_get_backend_memory_contexts(); + SELECT * FROM pg_get_backend_memory_contexts(-1);
-1 is odd. Why not use NULL or even 0?
+ else + { + int rc; + int parent_len = strlen(parent); + int name_len = strlen(name); + + /* + * write out the current memory context information. + * Since some elements of values are reusable, we write it out.
Not sure what the second comment line here is supposed to mean?
+ */ + fputc('D', fpout); + rc = fwrite(values, sizeof(values), 1, fpout); + rc = fwrite(nulls, sizeof(nulls), 1, fpout); + + /* write out information which is not resuable from serialized values */
s/resuable/reusable/
+ rc = fwrite(&name_len, sizeof(int), 1, fpout); + rc = fwrite(name, name_len, 1, fpout); + rc = fwrite(&idlen, sizeof(int), 1, fpout); + rc = fwrite(clipped_ident, idlen, 1, fpout); + rc = fwrite(&level, sizeof(int), 1, fpout); + rc = fwrite(&parent_len, sizeof(int), 1, fpout); + rc = fwrite(parent, parent_len, 1, fpout); + (void) rc; /* we'll check for error with ferror */ + + }
This format is not descriptive. How about serializing to json or
something? Or at least having field names?
Alternatively, build the same tuple we build for the SRF, and serialize
that. Then there's basically no conversion needed.
@@ -117,6 +157,8 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
{
+ int pid = PG_GETARG_INT32(0);
+
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -147,11 +189,258 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)MemoryContextSwitchTo(oldcontext);
- PutMemoryContextsStatsTupleStore(tupstore, tupdesc, - TopMemoryContext, NULL, 0); + if (pid == -1) + { + /* + * Since pid -1 indicates target is the local process, simply + * traverse memory contexts. + */ + PutMemoryContextsStatsTupleStore(tupstore, tupdesc, + TopMemoryContext, "", 0, NULL); + } + else + { + /* + * Send signal for dumping memory contexts to the target process, + * and read the dumped file. + */ + FILE *fpin; + char dumpfile[MAXPGPATH]; + + SendProcSignal(pid, PROCSIG_DUMP_MEMORY, InvalidBackendId); + + snprintf(dumpfile, sizeof(dumpfile), "pg_memusage/%d", pid); + + while (true) + { + CHECK_FOR_INTERRUPTS(); + + pg_usleep(10000L); +
Need better signalling back/forth here.
+/* + * dump_memory_contexts + * Dumping local memory contexts to a file. + * This function does not delete the file as it is intended to be read by + * another process. + */ +static void +dump_memory_contexts(void) +{ + FILE *fpout; + char tmpfile[MAXPGPATH]; + char dumpfile[MAXPGPATH]; + + snprintf(tmpfile, sizeof(tmpfile), "pg_memusage/%d.tmp", MyProcPid); + snprintf(dumpfile, sizeof(dumpfile), "pg_memusage/%d", MyProcPid); + + /* + * Open a temp file to dump the current memory context. + */ + fpout = AllocateFile(tmpfile, PG_BINARY_W); + if (fpout == NULL) + { + ereport(LOG, + (errcode_for_file_access(), + errmsg("could not write temporary memory context file \"%s\": %m", + tmpfile))); + return; + }
Probably should be opened with O_CREAT | O_TRUNC?
Greetings,
Andres Freund
On 2020-09-01 03:29, Pavel Stehule wrote:
Hi
po 31. 8. 2020 v 17:03 odesílatel Kasahara Tatsuhito
<kasahara.tatsuhito@gmail.com> napsal:Hi,
On Mon, Aug 31, 2020 at 8:22 PM torikoshia
<torikoshia@oss.nttdata.com> wrote:As discussed in the thread[1], it'll be useful to make it
possible to get the memory contexts of an arbitrary backend
process.+1
Attached PoC patch makes pg_get_backend_memory_contexts()
display meory contexts of the specified PID of the process.Thanks, it's a very good patch for discussion.
It doesn't display contexts of all the backends but only
the contexts of specified process.or we can "SELECT (pg_get_backend_memory_contexts(pid)).* FROM
pg_stat_activity WHERE ...",
so I don't think it's a big deal.The rough idea of implementation is like below:
1. send a signal to the specified process
2. signaled process dumps its memory contexts to a file
3. read the dumped file and display it to the userI agree with the overview of the idea.
Here are some comments and questions.
Thanks for the comments!
- Currently, "the signal transmission for dumping memory
information"
and "the read & output of dump information "
are on the same interface, but I think it would be better to
separate them.
How about providing the following three types of functions for
users?
- send a signal to specified pid
- check the status of the signal sent and received
- read the dumped information
Is this for future extensibility to make it possible to get
other information like the current execution plan which was
suggested by Pavel?
If so, I agree with considering extensibility, but I'm not
sure it's necessary whether providing these types of
functions for 'users'.
- How about managing the status of signal send/receive and dump
operations on a shared hash or others ?
Sending and receiving signals, dumping memory information, and
referencing dump information all work asynchronously.
Therefore, it would be good to have management information to
check
the status of each process.
A simple idea is that ..
- send a signal to dump to a PID, it first record following
information into the shared hash.
pid (specified pid)
loc (dump location, currently might be ASAP)
recv (did the pid process receive a signal? first false)
dumped (did the pid process dump a mem information? first
false)
- specified process receive the signal, update the status in the
shared hash, then dumped at specified location.
- specified process finish dump mem information, update the
status
in the shared hash.
Adding management information on shared memory seems necessary
when we want to have more controls over dumping like 'dump
location' or any other information such as 'current execution
plan'.
I'm going to consider this.
- Does it allow one process to output multiple dump files?
It appears to be a specification to overwrite at present, but I
thought it would be good to be able to generate
multiple dump files in different phases (e.g., planning phase and
execution phase) in the future.
- How is the dump file cleaned up?For a very long time there has been similar discussion about taking
session query and session execution plans from other sessions.I am not sure how necessary information is in the memory dump, but I
am sure so taking the current execution plan and complete text of the
current query is pretty necessary information.but can be great so this infrastructure can be used for any debugging
purpose.
Thanks!
It would be good if some part of this effort can be an infrastructure
of other debugging.
It may be hard, but I will keep your comment in mind.
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Show quoted text
Regards
Pavel
Best regards,
--
Tatsuhito Kasahara
kasahara.tatsuhito _at_ gmail.com [1]Links:
------
[1] http://gmail.com
Thanks for reviewing!
I'm going to modify the patch according to your comments.
On 2020-09-01 10:54, Andres Freund wrote:
Hi,
On 2020-08-31 20:22:18 +0900, torikoshia wrote:
After commit 3e98c0bafb28de, we can display the usage of the
memory contexts using pg_backend_memory_contexts system
view.However, its target is limited to the process attached to
the current session.As discussed in the thread[1], it'll be useful to make it
possible to get the memory contexts of an arbitrary backend
process.Attached PoC patch makes pg_get_backend_memory_contexts()
display meory contexts of the specified PID of the process.Awesome!
It doesn't display contexts of all the backends but only
the contexts of specified process.
I think it would be enough because I suppose this function
is used after investigations using ps command or other OS
level utilities.It can be used as a building block if all are needed. Getting the
infrastructure right is the big thing here, I think. Adding more
detailed views on top of that data later is easier.diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index a2d61302f9..88fb837ecd 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -555,10 +555,10 @@ REVOKE ALL ON pg_shmem_allocations FROM PUBLIC; REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;CREATE VIEW pg_backend_memory_contexts AS - SELECT * FROM pg_get_backend_memory_contexts(); + SELECT * FROM pg_get_backend_memory_contexts(-1);-1 is odd. Why not use NULL or even 0?
+ else + { + int rc; + int parent_len = strlen(parent); + int name_len = strlen(name); + + /* + * write out the current memory context information. + * Since some elements of values are reusable, we write it out.Not sure what the second comment line here is supposed to mean?
+ */ + fputc('D', fpout); + rc = fwrite(values, sizeof(values), 1, fpout); + rc = fwrite(nulls, sizeof(nulls), 1, fpout); + + /* write out information which is not resuable from serialized values */s/resuable/reusable/
+ rc = fwrite(&name_len, sizeof(int), 1, fpout); + rc = fwrite(name, name_len, 1, fpout); + rc = fwrite(&idlen, sizeof(int), 1, fpout); + rc = fwrite(clipped_ident, idlen, 1, fpout); + rc = fwrite(&level, sizeof(int), 1, fpout); + rc = fwrite(&parent_len, sizeof(int), 1, fpout); + rc = fwrite(parent, parent_len, 1, fpout); + (void) rc; /* we'll check for error with ferror */ + + }This format is not descriptive. How about serializing to json or
something? Or at least having field names?Alternatively, build the same tuple we build for the SRF, and serialize
that. Then there's basically no conversion needed.@@ -117,6 +157,8 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate
*tupstore,
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
{
+ int pid = PG_GETARG_INT32(0);
+
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -147,11 +189,258 @@
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)MemoryContextSwitchTo(oldcontext);
- PutMemoryContextsStatsTupleStore(tupstore, tupdesc, - TopMemoryContext, NULL, 0); + if (pid == -1) + { + /* + * Since pid -1 indicates target is the local process, simply + * traverse memory contexts. + */ + PutMemoryContextsStatsTupleStore(tupstore, tupdesc, + TopMemoryContext, "", 0, NULL); + } + else + { + /* + * Send signal for dumping memory contexts to the target process, + * and read the dumped file. + */ + FILE *fpin; + char dumpfile[MAXPGPATH]; + + SendProcSignal(pid, PROCSIG_DUMP_MEMORY, InvalidBackendId); + + snprintf(dumpfile, sizeof(dumpfile), "pg_memusage/%d", pid); + + while (true) + { + CHECK_FOR_INTERRUPTS(); + + pg_usleep(10000L); +Need better signalling back/forth here.
Do you mean I should also send another signal from the dumped
process to the caller of the pg_get_backend_memory_contexts()
when it finishes dumping?
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Show quoted text
+/* + * dump_memory_contexts + * Dumping local memory contexts to a file. + * This function does not delete the file as it is intended to be read by + * another process. + */ +static void +dump_memory_contexts(void) +{ + FILE *fpout; + char tmpfile[MAXPGPATH]; + char dumpfile[MAXPGPATH]; + + snprintf(tmpfile, sizeof(tmpfile), "pg_memusage/%d.tmp", MyProcPid); + snprintf(dumpfile, sizeof(dumpfile), "pg_memusage/%d", MyProcPid); + + /* + * Open a temp file to dump the current memory context. + */ + fpout = AllocateFile(tmpfile, PG_BINARY_W); + if (fpout == NULL) + { + ereport(LOG, + (errcode_for_file_access(), + errmsg("could not write temporary memory context file \"%s\": %m", + tmpfile))); + return; + }Probably should be opened with O_CREAT | O_TRUNC?
Greetings,
Andres Freund
Hi,
On Thu, Sep 3, 2020 at 3:38 PM torikoshia <torikoshia@oss.nttdata.com> wrote:
- Currently, "the signal transmission for dumping memory
information"
and "the read & output of dump information "
are on the same interface, but I think it would be better to
separate them.
How about providing the following three types of functions for
users?
- send a signal to specified pid
- check the status of the signal sent and received
- read the dumped informationIs this for future extensibility to make it possible to get
other information like the current execution plan which was
suggested by Pavel?
Yes, but it's not only for future expansion, but also for the
usability and the stability of this feature.
For example, if you want to read one dumped file multiple times and analyze it,
you will want the ability to just read the dump.
Moreover, when it takes a long time from the receive the signal to the
dump output,
or the dump output itself takes a long time, users can investigate
where it takes time
if each process is separated.
If so, I agree with considering extensibility, but I'm not
sure it's necessary whether providing these types of
functions for 'users'.
Of course, it is possible and may be necessary to provide a wrapped
sequence of processes
from sending a signal to reading dump files.
But IMO, some users would like to perform the signal transmission,
state management and
dump file reading processes separately.
Best regards,
--
Tatsuhito Kasahara
kasahara.tatsuhito _at_ gmail.com
Kasahara Tatsuhito <kasahara.tatsuhito@gmail.com> writes:
Yes, but it's not only for future expansion, but also for the
usability and the stability of this feature.
For example, if you want to read one dumped file multiple times and analyze it,
you will want the ability to just read the dump.
If we design it to make that possible, how are we going to prevent disk
space leaks from never-cleaned-up dump files?
regards, tom lane
On Fri, Sep 4, 2020 at 2:40 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Kasahara Tatsuhito <kasahara.tatsuhito@gmail.com> writes:
Yes, but it's not only for future expansion, but also for the
usability and the stability of this feature.
For example, if you want to read one dumped file multiple times and analyze it,
you will want the ability to just read the dump.If we design it to make that possible, how are we going to prevent disk
space leaks from never-cleaned-up dump files?
In my thought, with features such as a view that allows us to see a
list of dumped files,
it would be better to have a function that simply deletes the dump
files associated with a specific PID,
or to delete all dump files.
Some files may be dumped with unexpected delays, so I think the
cleaning feature will be necessary.
( Also, as the pgsql_tmp file, it might better to delete dump files
when PostgreSQL start.)
Or should we try to delete the dump file as soon as we can read it?
Best regards,
--
Tatsuhito Kasahara
kasahara.tatsuhito _at_ gmail.com
On Fri, Sep 04, 2020 at 11:47:30AM +0900, Kasahara Tatsuhito wrote:
On Fri, Sep 4, 2020 at 2:40 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Kasahara Tatsuhito <kasahara.tatsuhito@gmail.com> writes:
Yes, but it's not only for future expansion, but also for the
usability and the stability of this feature.
For example, if you want to read one dumped file multiple times and analyze it,
you will want the ability to just read the dump.If we design it to make that possible, how are we going to prevent disk
space leaks from never-cleaned-up dump files?In my thought, with features such as a view that allows us to see a
list of dumped files,
it would be better to have a function that simply deletes the dump
files associated with a specific PID,
or to delete all dump files.
Some files may be dumped with unexpected delays, so I think the
cleaning feature will be necessary.
( Also, as the pgsql_tmp file, it might better to delete dump files
when PostgreSQL start.)Or should we try to delete the dump file as soon as we can read it?
IMO making the cleanup a responsibility of the users (e.g. by exposing
the list of dumped files through a view and expecting users to delete
them in some way) is rather fragile.
I don't quite see what's the point of designing it this way. It was
suggested this improves stability and usability of this feature, but
surely making it unnecessarily complex contradicts both points?
IMHO if the user needs to process the dump repeatedly, what's preventing
him/her from storing it in a file, or something like that? At that point
it's clear it's up to them to remove the file. So I suggest to keep the
feature as simple as possible - hand the dump over and delete.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2020-09-04 21:46, Tomas Vondra wrote:
On Fri, Sep 04, 2020 at 11:47:30AM +0900, Kasahara Tatsuhito wrote:
On Fri, Sep 4, 2020 at 2:40 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Kasahara Tatsuhito <kasahara.tatsuhito@gmail.com> writes:
Yes, but it's not only for future expansion, but also for the
usability and the stability of this feature.
For example, if you want to read one dumped file multiple times and analyze it,
you will want the ability to just read the dump.If we design it to make that possible, how are we going to prevent
disk
space leaks from never-cleaned-up dump files?In my thought, with features such as a view that allows us to see a
list of dumped files,
it would be better to have a function that simply deletes the dump
files associated with a specific PID,
or to delete all dump files.
Some files may be dumped with unexpected delays, so I think the
cleaning feature will be necessary.
( Also, as the pgsql_tmp file, it might better to delete dump files
when PostgreSQL start.)Or should we try to delete the dump file as soon as we can read it?
IMO making the cleanup a responsibility of the users (e.g. by exposing
the list of dumped files through a view and expecting users to delete
them in some way) is rather fragile.I don't quite see what's the point of designing it this way. It was
suggested this improves stability and usability of this feature, but
surely making it unnecessarily complex contradicts both points?IMHO if the user needs to process the dump repeatedly, what's
preventing
him/her from storing it in a file, or something like that? At that
point
it's clear it's up to them to remove the file. So I suggest to keep the
feature as simple as possible - hand the dump over and delete.
+1.
If there are no other objections, I'm going to accept this
suggestion.
Regards
Hi,
On Thu, Sep 10, 2020 at 8:53 PM torikoshia <torikoshia@oss.nttdata.com> wrote:
On 2020-09-04 21:46, Tomas Vondra wrote:
On Fri, Sep 04, 2020 at 11:47:30AM +0900, Kasahara Tatsuhito wrote:
On Fri, Sep 4, 2020 at 2:40 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Kasahara Tatsuhito <kasahara.tatsuhito@gmail.com> writes:
Yes, but it's not only for future expansion, but also for the
usability and the stability of this feature.
For example, if you want to read one dumped file multiple times and analyze it,
you will want the ability to just read the dump.If we design it to make that possible, how are we going to prevent
disk
space leaks from never-cleaned-up dump files?In my thought, with features such as a view that allows us to see a
list of dumped files,
it would be better to have a function that simply deletes the dump
files associated with a specific PID,
or to delete all dump files.
Some files may be dumped with unexpected delays, so I think the
cleaning feature will be necessary.
( Also, as the pgsql_tmp file, it might better to delete dump files
when PostgreSQL start.)Or should we try to delete the dump file as soon as we can read it?
IMO making the cleanup a responsibility of the users (e.g. by exposing
the list of dumped files through a view and expecting users to delete
them in some way) is rather fragile.I don't quite see what's the point of designing it this way. It was
suggested this improves stability and usability of this feature, but
surely making it unnecessarily complex contradicts both points?IMHO if the user needs to process the dump repeatedly, what's
preventing
him/her from storing it in a file, or something like that? At that
point
it's clear it's up to them to remove the file. So I suggest to keep the
feature as simple as possible - hand the dump over and delete.
Yeah, it might be better to avoid making the user responsible for removal.
I think it's fine to have an interface to delete in an emergency, but
I agree that
users shouldn't be made aware of the existence or deletion of dump
files, basically.
+1.
If there are no other objections, I'm going to accept this
suggestion.
So +1
Best regards,
--
Tatsuhito Kasahara
kasahara.tatsuhito _at_ gmail.com
On Thu, Sep 10, 2020 at 09:11:21PM +0900, Kasahara Tatsuhito wrote:
I think it's fine to have an interface to delete in an emergency, but
I agree that
users shouldn't be made aware of the existence or deletion of dump
files, basically.
Per the CF bot, the number of tests needs to be tweaked, because we
test each entry filtered out with is_deeply(), meaning that the number
of tests needs to be updated to reflect that if the filtered list is
changed:
t/010_pg_basebackup.pl ... 104/109 # Looks like you planned 109 tests but ran 110.
t/010_pg_basebackup.pl ... Dubious, test returned 255 (wstat 65280, 0xff00)
All 109 subtests passed
Simple enough to fix.
--
Michael
Hi,
Thanks for all your comments, I updated the patch.
On Tue, Sep 1, 2020 at 12:03 AM Kasahara Tatsuhito
<kasahara.tatsuhito@gmail.com> wrote:
- How about managing the status of signal send/receive and dump
operations on a shared hash or others ?
Sending and receiving signals, dumping memory information, and
referencing dump information all work asynchronously.
Therefore, it would be good to have management information to
check
the status of each process.
A simple idea is that ..
- send a signal to dump to a PID, it first record following
information into the shared hash.
pid (specified pid)
loc (dump location, currently might be ASAP)
recv (did the pid process receive a signal? first false)
dumped (did the pid process dump a mem information? first
false)
- specified process receive the signal, update the status in the
shared hash, then dumped at specified location.
- specified process finish dump mem information, update the
status
in the shared hash.
I added a shared hash table consisted of minimal members
mainly for managing whether the file is dumped or not.
Some members like 'loc' seem useful in the future, but I
haven't added them since it's not essential at this point.
On 2020-09-01 10:54, Andres Freund wrote:
CREATE VIEW pg_backend_memory_contexts AS - SELECT * FROM pg_get_backend_memory_contexts(); + SELECT * FROM pg_get_backend_memory_contexts(-1);-1 is odd. Why not use NULL or even 0?
Changed it from -1 to NULL.
+ rc = fwrite(&name_len, sizeof(int), 1, fpout); + rc = fwrite(name, name_len, 1, fpout); + rc = fwrite(&idlen, sizeof(int), 1, fpout); + rc = fwrite(clipped_ident, idlen, 1, fpout); + rc = fwrite(&level, sizeof(int), 1, fpout); + rc = fwrite(&parent_len, sizeof(int), 1, fpout); + rc = fwrite(parent, parent_len, 1, fpout); + (void) rc; /* we'll check for error with ferror */ + + }This format is not descriptive. How about serializing to json or
something? Or at least having field names?
Added field names for each value.
+ while (true) + { + CHECK_FOR_INTERRUPTS(); + + pg_usleep(10000L); +Need better signalling back/forth here.
Added a shared hash table that has a flag for managing whether the file
is dumped or not.
I modified it to use this flag.
+ /* + * Open a temp file to dump the current memory context. + */ + fpout = AllocateFile(tmpfile, PG_BINARY_W); + if (fpout == NULL) + { + ereport(LOG, + (errcode_for_file_access(), + errmsg("could not write temporary memory context file \"%s\": %m", + tmpfile))); + return; + }Probably should be opened with O_CREAT | O_TRUNC?
AllocateFile() calls fopen() and AFAIU fopen() with mode "w" corresponds
to open() with "O_WRONLY | O_CREAT | O_TRUNC".
Do you mean I should not use fopen() here?
On 2020-09-24 13:01, Michael Paquier wrote:
On Thu, Sep 10, 2020 at 09:11:21PM +0900, Kasahara Tatsuhito wrote:
I think it's fine to have an interface to delete in an emergency, but
I agree that
users shouldn't be made aware of the existence or deletion of dump
files, basically.Per the CF bot, the number of tests needs to be tweaked, because we
test each entry filtered out with is_deeply(), meaning that the number
of tests needs to be updated to reflect that if the filtered list is
changed:
t/010_pg_basebackup.pl ... 104/109 # Looks like you planned 109 tests
but ran 110.
t/010_pg_basebackup.pl ... Dubious, test returned 255 (wstat 65280,
0xff00)
All 109 subtests passedSimple enough to fix.
Incremented the number of tests.
Any thoughts?
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
0002-Enabled-pg_get_backend_memory_contexts-to-collect-ar.patchtext/x-diff; name=0002-Enabled-pg_get_backend_memory_contexts-to-collect-ar.patchDownload
From 4d3ff254a634895e8c23c83bb63f519a14785f06 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Fri, 25 Sep 2020 11:34:38 +0900
Subject: [PATCH] Enabled pg_get_backend_memory_contexts() to collect
arbitrary backend process's memory contexts.
Previsouly, pg_get_backend_memory_contexts() could only get the
local memory contexts. This patch enables to get memory contexts
of the arbitrary process which PID is specified by the argument.
---
src/backend/access/transam/xlog.c | 7 +
src/backend/catalog/system_views.sql | 4 +-
src/backend/replication/basebackup.c | 3 +
src/backend/storage/ipc/ipci.c | 2 +
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/tcop/postgres.c | 5 +
src/backend/utils/adt/mcxtfuncs.c | 381 ++++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/bin/initdb/initdb.c | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 4 +-
src/bin/pg_rewind/filemap.c | 3 +
src/include/catalog/pg_proc.dat | 10 +-
src/include/miscadmin.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/mcxtfuncs.h | 44 +++
src/test/regress/expected/rules.out | 2 +-
16 files changed, 448 insertions(+), 27 deletions(-)
create mode 100644 src/include/utils/mcxtfuncs.h
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 61754312e2..f6ead80d2d 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -77,6 +77,7 @@
#include "utils/ps_status.h"
#include "utils/relmapper.h"
#include "utils/pg_rusage.h"
+#include "utils/mcxtfuncs.h"
#include "utils/snapmgr.h"
#include "utils/timestamp.h"
@@ -6985,6 +6986,12 @@ StartupXLOG(void)
*/
pgstat_reset_all();
+ /*
+ * Reset dumped files in pg_memusage, because target processes do
+ * not exist any more.
+ */
+ pg_memusage_reset();
+
/*
* If there was a backup label file, it's done its job and the info
* has now been propagated into pg_control. We must get rid of the
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index ed4f3f142d..adfde0ef5c 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -555,10 +555,10 @@ REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
CREATE VIEW pg_backend_memory_contexts AS
- SELECT * FROM pg_get_backend_memory_contexts();
+ SELECT * FROM pg_get_backend_memory_contexts(NULL);
REVOKE ALL ON pg_backend_memory_contexts FROM PUBLIC;
-REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts FROM PUBLIC;
-- Statistics views
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index b89df01fa7..3edb591952 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -184,6 +184,9 @@ static const char *const excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 96c2aaabbd..92f21ad2bf 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -45,6 +45,7 @@
#include "storage/procsignal.h"
#include "storage/sinvaladt.h"
#include "storage/spin.h"
+#include "utils/mcxtfuncs.h"
#include "utils/snapmgr.h"
/* GUCs */
@@ -267,6 +268,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ McxtDumpShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index ffe67acea1..ce6c67d9f2 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -28,6 +28,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/mcxtfuncs.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -567,6 +568,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_DUMP_MEMORY))
+ HandleProcSignalDumpMemory();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 411cfadbff..e8f4175c48 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -75,6 +75,7 @@
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
#include "utils/lsyscache.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -539,6 +540,10 @@ ProcessClientReadInterrupt(bool blocked)
/* Process notify interrupts, if any */
if (notifyInterruptPending)
ProcessNotifyInterrupt();
+
+ /* Process memory contexts dump interrupts, if any */
+ if (ProcSignalDumpMemoryPending)
+ ProcessDumpMemoryInterrupt();
}
else if (ProcDiePending)
{
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 50e1b07ff0..eb2efac304 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,30 +15,45 @@
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "common/logging.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "storage/shmem.h"
#include "utils/builtins.h"
+#include "utils/mcxtfuncs.h"
+
+/* The max bytes for showing names and identifiers of MemoryContext. */
+#define MEMORY_CONTEXT_DISPLAY_SIZE 1024
+
+/* Number of columns in pg_backend_memory_contexts view */
+#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
+
+/* Hash for managing the status of memory context dump. */
+static HTAB *mcxtdumpHash = NULL;
-/* ----------
- * The max bytes for showing identifiers of MemoryContext.
- * ----------
- */
-#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
/*
* PutMemoryContextsStatsTupleStore
* One recursion level for pg_get_backend_memory_contexts.
+ *
+ * Note: When fpout is not NULL, ferror() check must be done by the caller.
*/
static void
PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
TupleDesc tupdesc, MemoryContext context,
- const char *parent, int level)
+ const char *parent, int level, FILE *fpout)
{
-#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
-
Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
MemoryContextCounters stat;
MemoryContext child;
const char *name;
@@ -74,14 +89,12 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
if (ident)
{
int idlen = strlen(ident);
- char clipped_ident[MEMORY_CONTEXT_IDENT_DISPLAY_SIZE];
-
/*
* Some identifiers such as SQL query string can be very long,
* truncate oversize identifiers.
*/
- if (idlen >= MEMORY_CONTEXT_IDENT_DISPLAY_SIZE)
- idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_IDENT_DISPLAY_SIZE - 1);
+ if (idlen >= MEMORY_CONTEXT_DISPLAY_SIZE)
+ idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_DISPLAY_SIZE - 1);
memcpy(clipped_ident, ident, idlen);
clipped_ident[idlen] = '\0';
@@ -101,12 +114,44 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
values[6] = Int64GetDatum(stat.freespace);
values[7] = Int64GetDatum(stat.freechunks);
values[8] = Int64GetDatum(stat.totalspace - stat.freespace);
- tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ if(fpout == NULL)
+ /*
+ * Since pg_get_backend_memory_contexts() is called from local process,
+ * simply put tuples.
+ */
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ else
+ {
+ /*
+ * Write out the current memory context information in the form of
+ * "key: value" pairs to the file specified by the caller of
+ * pg_get_backend_memory_contexts().
+ */
+
+ /*
+ * Make each memory context information starts with 'D'.
+ * This is checked by the caller when reading the file.
+ */
+ fputc('D', fpout);
+
+ fprintf(fpout,
+ "name: %s, ident: %s, parent: %s, level: %d, total_bytes: %lu, total_nblocks: %lu, free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name,
+ ident ? clipped_ident : "none",
+ parent ? parent : "none", level,
+ stat.totalspace,
+ stat.nblocks,
+ stat.freespace,
+ stat.freechunks,
+ stat.totalspace - stat.freespace);
+ }
for (child = context->firstchild; child != NULL; child = child->nextchild)
{
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- child, name, level + 1);
+ child, name, level + 1, fpout);
}
}
@@ -117,6 +162,8 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
{
+ int dst_pid = PG_ARGISNULL(0) ? -1 : PG_GETARG_INT32(0);
+
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -147,11 +194,313 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldcontext);
- PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- TopMemoryContext, NULL, 0);
+ /*
+ * If the target is local process, simply look into memory contexts
+ * recursively.
+ */
+ if (dst_pid == -1 || dst_pid == MyProcPid)
+ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
+ TopMemoryContext, "", 0, NULL);
+
+ /*
+ * Send signal for dumping memory contexts to the target process,
+ * and read the dumped file.
+ */
+ else
+ {
+ FILE *fpin;
+ char tmpfile[MAXPGPATH];
+ char dumpfile[MAXPGPATH];
+ bool found;
+ mcxtdumpEntry *entry;
+ struct stat stat_tmp;
+ PGPROC *proc;
+ int format_id;
+
+ snprintf(tmpfile, sizeof(tmpfile), "%s/%d.tmp", PG_MEMUSAGE_DIR, dst_pid);
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, dst_pid);
+
+ /*
+ * Since we allow only one session can request to dump memory context at
+ * the same time, check whether the dump files already exist.
+ */
+ while (stat(dumpfile, &stat_tmp) == 0 || stat(tmpfile, &stat_tmp) == 0)
+ {
+ pg_usleep(1000000L);
+ }
+
+ entry = (mcxtdumpEntry *) hash_search(mcxtdumpHash, &dst_pid, HASH_ENTER, &found);
+
+ if (!found)
+ {
+ entry->is_dumped = false;
+ entry->src_pid = MyProcPid;
+ }
+
+ /* Check whether the target process is PostgreSQL backend process */
+ /* TODO: Check also whether backend or not. */
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", dst_pid)));
+ return (Datum) 1;
+ }
+
+ SendProcSignal(dst_pid, PROCSIG_DUMP_MEMORY, InvalidBackendId);
+
+ /* Wait until target process finished dumping file. */
+ while (!entry->is_dumped)
+ {
+ CHECK_FOR_INTERRUPTS();
+ pg_usleep(10000L);
+ }
+
+ if ((fpin = AllocateFile(dumpfile, "r")) == NULL)
+ {
+ if (errno != ENOENT)
+ ereport(LOG, (errcode_for_file_access(),
+ errmsg("could not open memory context dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ /* Verify it's of the expected format. */
+ if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ format_id != PG_MEMCONTEXT_FILE_FORMAT_ID)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"", dumpfile)));
+ goto done;
+ }
+
+ while (true)
+ {
+ Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char name[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char parent[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
+ int level;
+ Size total_bytes;
+ Size total_nblocks;
+ Size free_bytes;
+ Size free_chunks;
+ Size used_bytes;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ switch (fgetc(fpin))
+ {
+ /* 'D' A memory context information follows. */
+ case 'D':
+ if (fscanf(fpin, "name: %1023[^,], ident: %1023[^,], parent: %1023[^,], level: %d, total_bytes: %lu, total_nblocks: %lu, free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name, clipped_ident, parent, &level, &total_bytes, &total_nblocks,
+ &free_bytes, &free_chunks, &used_bytes)
+ != PG_GET_BACKEND_MEMORY_CONTEXTS_COLS)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+
+ values[0] = CStringGetTextDatum(name);
+
+ if (strcmp(clipped_ident, "none"))
+ values[1] = CStringGetTextDatum(clipped_ident);
+ else
+ nulls[1] = true;
+
+ if (strcmp(parent, "none"))
+ values[2] = CStringGetTextDatum(parent);
+ else
+ nulls[2] = true;
+
+ values[3] = Int32GetDatum(level);
+ values[4] = Int64GetDatum(total_bytes);
+ values[5] = Int64GetDatum(total_nblocks);
+ values[6] = Int64GetDatum(free_bytes);
+ values[7] = Int64GetDatum(free_chunks);
+ values[8] = Int64GetDatum(used_bytes);
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ break;
+
+ case 'E':
+ goto done;
+
+ default:
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ }
+done:
+ FreeFile(fpin);
+ unlink(dumpfile);
+
+ if (hash_search(mcxtdumpHash, &dst_pid, HASH_REMOVE, NULL) == NULL)
+ elog(ERROR, "hash table corrupted");
+ else
+ elog(INFO, "successed deleting hash entry");
+ }
/* clean up and return the tuplestore */
tuplestore_donestoring(tupstore);
return (Datum) 0;
}
+
+/*
+ * dump_memory_contexts
+ * Dumping local memory contexts to a file.
+ * This function does not delete dumped file, as it is intended to be read
+ * by another process.
+ */
+static void
+dump_memory_contexts(void)
+{
+ FILE *fpout;
+ char tmpfile[MAXPGPATH];
+ char dumpfile[MAXPGPATH];
+ mcxtdumpEntry *entry;
+ int format_id;
+
+ snprintf(tmpfile, sizeof(tmpfile), "%s/%d.tmp", PG_MEMUSAGE_DIR, MyProcPid);
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, MyProcPid);
+
+ entry = (mcxtdumpEntry *) hash_search(mcxtdumpHash, &MyProcPid, HASH_FIND, NULL);
+
+ Assert(entry);
+
+ fpout = AllocateFile(tmpfile, "w");
+
+ if (fpout == NULL)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write temporary memory context file \"%s\": %m",
+ tmpfile)));
+ return;
+ }
+
+ format_id = PG_MEMCONTEXT_FILE_FORMAT_ID;
+ fwrite(&format_id, sizeof(format_id), 1, fpout);
+
+ /* Look into each memory context from TopMemoryContext recursively. */
+ PutMemoryContextsStatsTupleStore(NULL, NULL,
+ TopMemoryContext, NULL, 0, fpout);
+
+ /* No more output to be done. Close the tmp file and rename it. */
+ fputc('E', fpout);
+
+ if (ferror(fpout))
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write temporary memory context dump file \"%s\": %m",
+ tmpfile)));
+ FreeFile(fpout);
+ unlink(tmpfile);
+ }
+ else if (FreeFile(fpout) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not close temporary memory context dump file \"%s\": %m",
+ tmpfile)));
+ unlink(tmpfile);
+ }
+ else if (rename(tmpfile, dumpfile) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not rename dump file \"%s\" to \"%s\": %m",
+ tmpfile, dumpfile)));
+ unlink(tmpfile);
+ }
+
+ entry->is_dumped = true;
+}
+
+/*
+ * ProcessDumpMemoryInterrupt
+ * The portion of memory context dump interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessDumpMemoryInterrupt(void)
+{
+ ProcSignalDumpMemoryPending = false;
+ dump_memory_contexts();
+}
+
+/*
+ * HandleProcSignalDumpMemory
+ * Handle receipt of an interrupt indicating a memory context dump.
+ * Signal handler portion of interrupt handling.
+ */
+void
+HandleProcSignalDumpMemory(void)
+{
+ ProcSignalDumpMemoryPending = true;
+
+ /* make sure the event is processed in due course */
+ SetLatch(MyLatch);
+}
+
+/*
+ * McxtDumpShmemInit
+ * Initialize mcxtdump hash table
+ */
+void
+McxtDumpShmemInit(void)
+{
+ HASHCTL info;
+
+ MemSet(&info, 0, sizeof(info));
+ info.keysize = sizeof(pid_t);
+ info.entrysize = sizeof(mcxtdumpEntry);
+
+ mcxtdumpHash = ShmemInitHash("mcxtdump hash",
+ SHMEM_MEMCONTEXT_SIZE,
+ SHMEM_MEMCONTEXT_SIZE,
+ &info,
+ HASH_ELEM | HASH_BLOBS);
+}
+
+/*
+ * pg_memusage_reset
+ * Remove the memory context dump files.
+ */
+void
+pg_memusage_reset(void)
+{
+ DIR *dir;
+ struct dirent *dumpfile;
+
+ dir = AllocateDir(PG_MEMUSAGE_DIR);
+ while ((dumpfile = ReadDir(dir, PG_MEMUSAGE_DIR)) != NULL)
+ {
+ char dumpfilepath[32];
+
+ if (strcmp(dumpfile->d_name, ".") == 0 || strcmp(dumpfile->d_name, "..") == 0)
+ continue;
+
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, dumpfile->d_name);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ FreeDir(dir);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 6ab8216839..463337f661 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -33,6 +33,7 @@ volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t ProcSignalDumpMemoryPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 118b282d1c..953a7c682d 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -221,7 +221,8 @@ static const char *const subdirs[] = {
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
- "pg_logical/mappings"
+ "pg_logical/mappings",
+ "pg_memusage"
};
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index f674a7c94e..340a80fc11 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -6,7 +6,7 @@ use File::Basename qw(basename dirname);
use File::Path qw(rmtree);
use PostgresNode;
use TestLib;
-use Test::More tests => 109;
+use Test::More tests => 110;
program_help_ok('pg_basebackup');
program_version_ok('pg_basebackup');
@@ -124,7 +124,7 @@ is_deeply(
# Contents of these directories should not be copied.
foreach my $dirname (
- qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans)
+ qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans pg_memusage)
)
{
is_deeply(
diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c
index 1abc257177..ff3338e9be 100644
--- a/src/bin/pg_rewind/filemap.c
+++ b/src/bin/pg_rewind/filemap.c
@@ -85,6 +85,9 @@ static const char *excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index f48f5fb4d9..e7eeb1e5d6 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7815,11 +7815,11 @@
# memory context of local backend
{ oid => '2282', descr => 'information about all memory contexts of local backend',
- proname => 'pg_get_backend_memory_contexts', prorows => '100', proretset => 't',
- provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
- proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ proname => 'pg_get_backend_memory_contexts', prorows => '100', proisstrict => 'f',
+ proretset => 't', provolatile => 'v', proparallel => 'r', prorettype => 'record',
+ proargtypes => 'int4', proallargtypes => '{int4,text,text,text,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
# non-persistent series generator
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..812032bb15 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -83,6 +83,7 @@ extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t ProcSignalDumpMemoryPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 5cb39697f3..5db92a9a52 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_DUMP_MEMORY, /* request dumping memory context interrupt */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/mcxtfuncs.h b/src/include/utils/mcxtfuncs.h
new file mode 100644
index 0000000000..75af92a2cf
--- /dev/null
+++ b/src/include/utils/mcxtfuncs.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcxtfuncs.h
+ * Declarations for showing backend memory context.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mcxtfuncs.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MCXT_H
+#define MCXT_H
+
+/* Directory to store dumped memory files */
+#define PG_MEMUSAGE_DIR "pg_memusage"
+
+#define PG_MEMCONTEXT_FILE_FORMAT_ID 0x01B5BC9E
+
+/*
+ * Size of the shmem hash table size(not a hard limit).
+ *
+ * Although it may be better to increase this number in the future (e.g.,
+ * adding views for all the backend process of memory contexts), currently
+ * small number would be enough.
+ */
+#define SHMEM_MEMCONTEXT_SIZE 64
+
+typedef struct mcxtdumpEntry
+{
+ pid_t dst_pid; /* pid of the signal receiver */
+ pid_t src_pid; /* pid of the signal sender */
+ bool is_dumped; /* is dumped to a file? */
+} mcxtdumpEntry;
+
+extern void ProcessDumpMemoryInterrupt(void);
+extern void HandleProcSignalDumpMemory(void);
+extern void McxtDumpShmemInit(void);
+extern void pg_memusage_reset(void);
+
+#endif /* MCXT_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 2a18dc423e..ca1cb0e786 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1333,7 +1333,7 @@ pg_backend_memory_contexts| SELECT pg_get_backend_memory_contexts.name,
pg_get_backend_memory_contexts.free_bytes,
pg_get_backend_memory_contexts.free_chunks,
pg_get_backend_memory_contexts.used_bytes
- FROM pg_get_backend_memory_contexts() pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
+ FROM pg_get_backend_memory_contexts(NULL::integer) pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
pg_config| SELECT pg_config.name,
pg_config.setting
FROM pg_config() pg_config(name, setting);
--
2.18.1
Hi,
On Fri, Sep 25, 2020 at 4:28 PM torikoshia <torikoshia@oss.nttdata.com> wrote:
Thanks for all your comments, I updated the patch.
Thanks for updating the patch.
I did a brief test and code review.
I added a shared hash table consisted of minimal members
mainly for managing whether the file is dumped or not.
Some members like 'loc' seem useful in the future, but I
haven't added them since it's not essential at this point.
Yes, that would be good.
+ /*
+ * Since we allow only one session can request to dump
memory context at
+ * the same time, check whether the dump files already exist.
+ */
+ while (stat(dumpfile, &stat_tmp) == 0 || stat(tmpfile, &stat_tmp) == 0)
+ {
+ pg_usleep(1000000L);
+ }
If pg_get_backend_memory_contexts() is executed by two or more
sessions at the same time, it cannot be run exclusively in this way.
Currently it seems to cause a crash when do it so.
This is easy to reproduce and can be done as follows.
[session-1]
BEGIN;
LOCK TABKE t1;
[Session-2]
BEGIN;
LOCK TABLE t1; <- waiting
[Session-3]
select * FROM pg_get_backend_memory_contexts(<pid of session-2>);
[Session-4]
select * FROM pg_get_backend_memory_contexts(<pid of session-2>);
If you issue commit or abort at session-1, you will get SEGV.
Instead of checking for the existence of the file, it might be better
to use a hash (mcxtdumpHash) entry with LWLock.
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server
process", dst_pid)));
+ return (Datum) 1;
+ }
Shouldn't it clear the hash entry before return?
+ /* Wait until target process finished dumping file. */
+ while (!entry->is_dumped)
+ {
+ CHECK_FOR_INTERRUPTS();
+ pg_usleep(10000L);
+ }
If the target is killed and exit before dumping the memory
information, you're in an infinite loop here.
So how about making sure that any process that has to stop before
doing a memory dump changes the status of the hash (entry->is_dumped)
before stopping and the caller detects it?
I'm not sure it's best or not, but you might want to use something
like the on_shmem_exit callback.
In the current design, if the caller stops processing before reading
the dumped file, you will have an orphaned file.
It looks like the following.
[session-1]
BEGIN;
LOCK TABKE t1;
[Session-2]
BEGIN;
LOCK TABLE t1; <- waiting
[Session-3]
select * FROM pg_get_backend_memory_contexts(<pid of session-2>);
If you cancel or terminate the session-3, then issue commit or abort
at session-1, you will get orphaned files in pg_memusage.
So if you allow only one session can request to dump file, it could
call pg_memusage_reset() before send the signal in this function.
Best regards,
--
Tatsuhito Kasahara
kasahara.tatsuhito _at_ gmail.com
On Thu, Oct 1, 2020 at 4:06 PM Kasahara Tatsuhito
<kasahara.tatsuhito@gmail.com> wrote:
Hi,On Fri, Sep 25, 2020 at 4:28 PM torikoshia <torikoshia@oss.nttdata.com>
wrote:Thanks for all your comments, I updated the patch.
Thanks for updating the patch.
I did a brief test and code review.
Thanks for your tests and review!
I added a shared hash table consisted of minimal members
mainly for managing whether the file is dumped or not.
Some members like 'loc' seem useful in the future, but I
haven't added them since it's not essential at this point.Yes, that would be good.
+ /* + * Since we allow only one session can request to dump memory context at + * the same time, check whether the dump files already exist. + */ + while (stat(dumpfile, &stat_tmp) == 0 || stat(tmpfile, &stat_tmp) == 0) + { + pg_usleep(1000000L); + }If pg_get_backend_memory_contexts() is executed by two or more
sessions at the same time, it cannot be run exclusively in this way.
Currently it seems to cause a crash when do it so.
This is easy to reproduce and can be done as follows.[session-1]
BEGIN;
LOCK TABKE t1;
[Session-2]
BEGIN;
LOCK TABLE t1; <- waiting
[Session-3]
select * FROM pg_get_backend_memory_contexts(<pid of session-2>);
[Session-4]
select * FROM pg_get_backend_memory_contexts(<pid of session-2>);If you issue commit or abort at session-1, you will get SEGV.
Instead of checking for the existence of the file, it might be better
to use a hash (mcxtdumpHash) entry with LWLock.
Thanks!
Added a LWLock and changed the way from checking the file existence
to finding the hash entry.
+ if (proc == NULL) + { + ereport(WARNING, + (errmsg("PID %d is not a PostgreSQL server process", dst_pid))); + return (Datum) 1; + }Shouldn't it clear the hash entry before return?
Yeah. added codes for removing the entry.
+ /* Wait until target process finished dumping file. */ + while (!entry->is_dumped) + { + CHECK_FOR_INTERRUPTS(); + pg_usleep(10000L); + }If the target is killed and exit before dumping the memory
information, you're in an infinite loop here.
So how about making sure that any process that has to stop before
doing a memory dump changes the status of the hash (entry->is_dumped)
before stopping and the caller detects it?
I'm not sure it's best or not, but you might want to use something
like the on_shmem_exit callback.
Thanks for your idea!
Added a callback to change the status of the hash table entry.
Although I think it's necessary to remove this callback when it finished
processing memory dumping, on_shmem_exit() does not seem to have such
a function.
I used before_shmem_exit() since it has a corresponding function to
remove registered callback.
If it's inappropriate, I'm going to add a function removing the
registered callback of on_shmem_exit().
In the current design, if the caller stops processing before reading
the dumped file, you will have an orphaned file.
It looks like the following.[session-1]
BEGIN;
LOCK TABKE t1;
[Session-2]
BEGIN;
LOCK TABLE t1; <- waiting
[Session-3]
select * FROM pg_get_backend_memory_contexts(<pid of session-2>);If you cancel or terminate the session-3, then issue commit or abort
at session-1, you will get orphaned files in pg_memusage.So if you allow only one session can request to dump file, it could
call pg_memusage_reset() before send the signal in this function.
Although I'm going to allow only one session per one target process,
I'd like to allow running multiple pg_get_backend_memory_contexts()
which target process is different.
Instead of calling pg_memusage_reset(), I added a callback for
cleaning up orphaned files and the elements of the hash table
using before_shmem_exit() through PG_ENSURE_ERROR_CLEANUP() and
PG_END_ENSURE_ERROR_CLEANUP().
I chose PG_ENSURE_ERROR_CLEANUP() and PG_END_ENSURE_ERROR_CLEANUP()
here since it can handle not only termination but also cancellation.
Any thoughts?
--
Atsushi Torikoshi
Attachments:
0003-Enabled-pg_get_backend_memory_contexts-to-collect.patchtext/x-diff; name=0003-Enabled-pg_get_backend_memory_contexts-to-collect.patchDownload
From 4d3ff254a634895e8c23c83bb63f519a14785f06 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Thu, 22 Oct 2020 20:24:19 +0900
Subject: [PATCH] Enabled pg_get_backend_memory_contexts() to collect arbitrary
backend process's memory contexts.
Previsouly, pg_get_backend_memory_contexts() could only get the
local memory contexts. This patch enables to get memory contexts
of the arbitrary process which PID is specified by the argument.
---
src/backend/access/transam/xlog.c | 7 +
src/backend/catalog/system_views.sql | 4 +-
src/backend/replication/basebackup.c | 3 +
src/backend/storage/ipc/ipci.c | 2 +
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/tcop/postgres.c | 5 +
src/backend/utils/adt/mcxtfuncs.c | 566 ++++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/bin/initdb/initdb.c | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 4 +-
src/bin/pg_rewind/filemap.c | 3 +
src/include/catalog/pg_proc.dat | 10 +-
src/include/miscadmin.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/mcxtfuncs.h | 51 ++
src/test/regress/expected/rules.out | 2 +-
17 files changed, 642 insertions(+), 26 deletions(-)
create mode 100644 src/include/utils/mcxtfuncs.h
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 52a67b1170..820b66da62 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -73,6 +73,7 @@
#include "storage/sync.h"
#include "utils/builtins.h"
#include "utils/guc.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/relmapper.h"
@@ -6987,6 +6988,12 @@ StartupXLOG(void)
*/
pgstat_reset_all();
+ /*
+ * Reset dumped files in pg_memusage, because target processes do
+ * not exist any more.
+ */
+ pg_memusage_reset(0);
+
/*
* If there was a backup label file, it's done its job and the info
* has now been propagated into pg_control. We must get rid of the
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 85cd147e21..3f177f9688 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -555,10 +555,10 @@ REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
CREATE VIEW pg_backend_memory_contexts AS
- SELECT * FROM pg_get_backend_memory_contexts();
+ SELECT * FROM pg_get_backend_memory_contexts(NULL);
REVOKE ALL ON pg_backend_memory_contexts FROM PUBLIC;
-REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts FROM PUBLIC;
-- Statistics views
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index b89df01fa7..3edb591952 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -184,6 +184,9 @@ static const char *const excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 96c2aaabbd..92f21ad2bf 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -45,6 +45,7 @@
#include "storage/procsignal.h"
#include "storage/sinvaladt.h"
#include "storage/spin.h"
+#include "utils/mcxtfuncs.h"
#include "utils/snapmgr.h"
/* GUCs */
@@ -267,6 +268,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ McxtDumpShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index ffe67acea1..ce6c67d9f2 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -28,6 +28,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/mcxtfuncs.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -567,6 +568,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_DUMP_MEMORY))
+ HandleProcSignalDumpMemory();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 774292fd94..6036713f11 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+McxtDumpHashLock 48
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 411cfadbff..e8f4175c48 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -75,6 +75,7 @@
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
#include "utils/lsyscache.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -539,6 +540,10 @@ ProcessClientReadInterrupt(bool blocked)
/* Process notify interrupts, if any */
if (notifyInterruptPending)
ProcessNotifyInterrupt();
+
+ /* Process memory contexts dump interrupts, if any */
+ if (ProcSignalDumpMemoryPending)
+ ProcessDumpMemoryInterrupt();
}
else if (ProcDiePending)
{
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 50e1b07ff0..5b7441f4dd 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,30 +15,82 @@
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "common/logging.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "storage/shmem.h"
#include "utils/builtins.h"
+#include "utils/mcxtfuncs.h"
+
+/* The max bytes for showing names and identifiers of MemoryContext. */
+#define MEMORY_CONTEXT_DISPLAY_SIZE 1024
+
+/* Number of columns in pg_backend_memory_contexts view */
+#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
+
+/* Hash for managing the status of memory context dump. */
+static HTAB *mcxtdumpHash = NULL;
+
+/*
+ * McxtDumpKill
+ * Called when target process of dumping memory exits.
+ *
+ * This function just changes the dump_status and actual cleanup is
+ * done by the caller of pg_get_backend_memory_contexts().
+ */
+static void
+McxtDumpKill(int code, Datum arg)
+{
+ mcxtdumpEntry *entry = (mcxtdumpEntry *) DatumGetPointer(arg);
+
+ entry->dump_status = MCXTDUMPSTATUS_ERROR;
+}
-/* ----------
- * The max bytes for showing identifiers of MemoryContext.
- * ----------
+/*
+ * McxtReqKill
+ * Cleanup function.
+ *
+ * Called when the caller of pg_get_backend_memory_contexts()
+ * exits.
*/
-#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+static void
+McxtReqKill(int code, Datum arg)
+{
+ int dst_pid = DatumGetInt32(arg);;
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ if (hash_search(mcxtdumpHash, &dst_pid, HASH_REMOVE, NULL) == NULL)
+ elog(ERROR, "hash table corrupted");
+
+ LWLockRelease(McxtDumpHashLock);
+
+ pg_memusage_reset(dst_pid);
+}
/*
* PutMemoryContextsStatsTupleStore
* One recursion level for pg_get_backend_memory_contexts.
+ *
+ * Note: When fpout is not NULL, ferror() check must be done by the caller.
*/
static void
PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
TupleDesc tupdesc, MemoryContext context,
- const char *parent, int level)
+ const char *parent, int level, FILE *fpout)
{
-#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
-
Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
MemoryContextCounters stat;
MemoryContext child;
const char *name;
@@ -74,14 +126,12 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
if (ident)
{
int idlen = strlen(ident);
- char clipped_ident[MEMORY_CONTEXT_IDENT_DISPLAY_SIZE];
-
/*
* Some identifiers such as SQL query string can be very long,
* truncate oversize identifiers.
*/
- if (idlen >= MEMORY_CONTEXT_IDENT_DISPLAY_SIZE)
- idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_IDENT_DISPLAY_SIZE - 1);
+ if (idlen >= MEMORY_CONTEXT_DISPLAY_SIZE)
+ idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_DISPLAY_SIZE - 1);
memcpy(clipped_ident, ident, idlen);
clipped_ident[idlen] = '\0';
@@ -101,15 +151,200 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
values[6] = Int64GetDatum(stat.freespace);
values[7] = Int64GetDatum(stat.freechunks);
values[8] = Int64GetDatum(stat.totalspace - stat.freespace);
- tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Since pg_get_backend_memory_contexts() is called from local process,
+ * simply put tuples.
+ */
+ if(fpout == NULL)
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Write out the current memory context information in the form of
+ * "key: value" pairs to the file specified by the caller of
+ * pg_get_backend_memory_contexts().
+ */
+ else
+ {
+ /*
+ * Make each memory context information starts with 'D'.
+ * This is checked by the caller when reading the file.
+ */
+ fputc('D', fpout);
+
+ fprintf(fpout,
+ "name: %s, ident: %s, parent: %s, level: %d, total_bytes: %lu, \
+ total_nblocks: %lu, free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name,
+ ident ? clipped_ident : "none",
+ parent ? parent : "none", level,
+ stat.totalspace,
+ stat.nblocks,
+ stat.freespace,
+ stat.freechunks,
+ stat.totalspace - stat.freespace);
+ }
for (child = context->firstchild; child != NULL; child = child->nextchild)
{
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- child, name, level + 1);
+ child, name, level + 1, fpout);
}
}
+/*
+ * AddEntryToMcxtdumpHash
+ * add an entry to McxtdumpHash for specified PID.
+ */
+static mcxtdumpEntry *
+AddEntryToMcxtdumpHash(int pid)
+{
+ mcxtdumpEntry *entry;
+ bool found;
+
+ /*
+ * We only allow one session per target process to request a memory
+ * dump at a time.
+ * If mcxtdumpHash has corresponding entry, wait until it has removed.
+ */
+ while (true)
+ {
+ LWLockAcquire(McxtDumpHashLock, LW_SHARED);
+ entry = (mcxtdumpEntry *) hash_search(mcxtdumpHash, &pid,
+ HASH_ENTER, &found);
+
+ if (!found)
+ {
+ LWLockRelease(McxtDumpHashLock);
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ entry->dump_status = MCXTDUMPSTATUS_NOTYET;
+ entry->src_pid = MyProcPid;
+
+ LWLockRelease(McxtDumpHashLock);
+
+ return entry;
+ }
+ else
+ {
+ ereport(INFO,
+ (errmsg("PID %d is looked up by another process", pid)));
+
+ LWLockRelease(McxtDumpHashLock);
+
+ pg_usleep(1000000L);
+ }
+ }
+}
+
+/*
+ * PutDumpedValuesOnTuplestore
+ * Read specified memory context dump file and put its values
+ * on the tuple store.
+ */
+static void
+PutDumpedValuesOnTuplestore(char *dumpfile, Tuplestorestate *tupstore,
+ TupleDesc tupdesc, int pid)
+{
+ int format_id;
+ FILE *fpin;
+
+ if ((fpin = AllocateFile(dumpfile, "r")) == NULL)
+ {
+ if (errno != ENOENT)
+ ereport(LOG, (errcode_for_file_access(),
+ errmsg("could not open memory context dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ /* Verify it's of the expected format. */
+ if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ format_id != PG_MEMCONTEXT_FILE_FORMAT_ID)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"", dumpfile)));
+ goto done;
+ }
+
+ /* Read dumped file and put values on tuple store. */
+ while (true)
+ {
+ Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char name[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char parent[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
+ int level;
+ Size total_bytes;
+ Size total_nblocks;
+ Size free_bytes;
+ Size free_chunks;
+ Size used_bytes;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ switch (fgetc(fpin))
+ {
+ /* 'D' A memory context information follows. */
+ case 'D':
+ if (fscanf(fpin, "name: %1023[^,], ident: %1023[^,], parent: %1023[^,], \
+ level: %d, total_bytes: %lu, total_nblocks: %lu, \
+ free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name, clipped_ident, parent, &level, &total_bytes, &total_nblocks,
+ &free_bytes, &free_chunks, &used_bytes)
+ != PG_GET_BACKEND_MEMORY_CONTEXTS_COLS)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+
+ values[0] = CStringGetTextDatum(name);
+
+ if (strcmp(clipped_ident, "none"))
+ values[1] = CStringGetTextDatum(clipped_ident);
+ else
+ nulls[1] = true;
+
+ if (strcmp(parent, "none"))
+ values[2] = CStringGetTextDatum(parent);
+ else
+ nulls[2] = true;
+
+ values[3] = Int32GetDatum(level);
+ values[4] = Int64GetDatum(total_bytes);
+ values[5] = Int64GetDatum(total_nblocks);
+ values[6] = Int64GetDatum(free_bytes);
+ values[7] = Int64GetDatum(free_chunks);
+ values[8] = Int64GetDatum(used_bytes);
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ break;
+
+ case 'E':
+ goto done;
+
+ default:
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ }
+done:
+ FreeFile(fpin);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ if (hash_search(mcxtdumpHash, &pid, HASH_REMOVE, NULL) == NULL)
+ elog(ERROR, "hash table corrupted");
+
+ LWLockRelease(McxtDumpHashLock);
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -117,6 +352,8 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
{
+ int dst_pid = PG_ARGISNULL(0) ? -1 : PG_GETARG_INT32(0);
+
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -147,11 +384,310 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldcontext);
- PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- TopMemoryContext, NULL, 0);
+ /*
+ * If the target is local process, simply look into memory contexts
+ * recursively.
+ */
+ if (dst_pid == -1 || dst_pid == MyProcPid)
+ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
+ TopMemoryContext, "", 0, NULL);
+
+ /*
+ * Send signal for dumping memory contexts to the target process,
+ * and read the dumped file.
+ */
+ else
+ {
+ char tmpfile[MAXPGPATH];
+ char dumpfile[MAXPGPATH];
+ struct stat stat_tmp;
+ mcxtdumpEntry *entry;
+ PGPROC *proc;
+
+ snprintf(tmpfile, sizeof(tmpfile), "%s/%d.tmp", PG_MEMUSAGE_DIR, dst_pid);
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, dst_pid);
+
+ entry = AddEntryToMcxtdumpHash(dst_pid);
+
+ /* Check whether the target process is PostgreSQL backend process. */
+ /* TODO: Check also whether backend or not. */
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", dst_pid)));
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ if (hash_search(mcxtdumpHash, &dst_pid, HASH_REMOVE, NULL) == NULL)
+ elog(WARNING, "hash table corrupted");
+
+ LWLockRelease(McxtDumpHashLock);
+
+ return (Datum) 1;
+ }
+
+ /* The ENSURE stuff ensures we clean up shared memory and files on failure */
+ PG_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid));
+ {
+ SendProcSignal(dst_pid, PROCSIG_DUMP_MEMORY, InvalidBackendId);
+
+ /* Wait until target process finished dumping file. */
+ while (entry->dump_status == MCXTDUMPSTATUS_NOTYET)
+ {
+ CHECK_FOR_INTERRUPTS();
+ pg_usleep(10000L);
+ }
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid));
+
+
+ if (entry->dump_status == MCXTDUMPSTATUS_ERROR)
+ {
+ ereport(WARNING,
+ (errmsg("Failed to get memory context from PID %d", dst_pid)));
+
+ if (stat(tmpfile, &stat_tmp) == 0)
+ unlink(tmpfile);
+ if (stat(dumpfile, &stat_tmp) == 0)
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ if (hash_search(mcxtdumpHash, &dst_pid, HASH_REMOVE, NULL) == NULL)
+ elog(ERROR, "hash table corrupted");
+
+ LWLockRelease(McxtDumpHashLock);
+
+ /* clean up and return the tuplestore */
+ tuplestore_donestoring(tupstore);
+
+
+ return (Datum) 0;
+ }
+
+ /* Read values from the dumped file and put them on tuplestore. */
+ PutDumpedValuesOnTuplestore(dumpfile, tupstore, tupdesc, dst_pid);
+ }
/* clean up and return the tuplestore */
tuplestore_donestoring(tupstore);
return (Datum) 0;
}
+
+/*
+ * dump_memory_contexts
+ * Dump local memory contexts to a file.
+ *
+ * This function does not delete dumped file, as it is intended to be read
+ * by another process.
+ */
+static void
+dump_memory_contexts(void)
+{
+ FILE *fpout;
+ char tmpfile[MAXPGPATH];
+ char dumpfile[MAXPGPATH];
+ mcxtdumpEntry *entry;
+ int format_id;
+
+ snprintf(tmpfile, sizeof(tmpfile), "%s/%d.tmp", PG_MEMUSAGE_DIR, MyProcPid);
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, MyProcPid);
+
+ LWLockAcquire(McxtDumpHashLock, LW_SHARED);
+
+ entry = (mcxtdumpEntry *) hash_search(mcxtdumpHash, &MyProcPid, HASH_FIND, NULL);
+
+ LWLockRelease(McxtDumpHashLock);
+
+ /* The process that requested to dump seems already exited. */
+ if (entry == NULL)
+ return;
+
+ before_shmem_exit(McxtDumpKill, PointerGetDatum(entry));
+
+ fpout = AllocateFile(tmpfile, "w");
+
+ if (fpout == NULL)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write temporary memory context file \"%s\": %m",
+ tmpfile)));
+
+ entry->dump_status = MCXTDUMPSTATUS_ERROR;
+
+ return;
+ }
+
+ format_id = PG_MEMCONTEXT_FILE_FORMAT_ID;
+ fwrite(&format_id, sizeof(format_id), 1, fpout);
+
+ /* Look into each memory context from TopMemoryContext recursively. */
+ PutMemoryContextsStatsTupleStore(NULL, NULL,
+ TopMemoryContext, NULL, 0, fpout);
+
+ /*
+ * Make dump file ends with 'D'.
+ * This is checked by the caller when reading the file.
+ */
+ fputc('E', fpout);
+
+ CHECK_FOR_INTERRUPTS();
+
+ if (ferror(fpout))
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write temporary memory context dump file \"%s\": %m",
+ tmpfile)));
+ FreeFile(fpout);
+ unlink(tmpfile);
+ }
+
+ /* No more output to be done. Close the tmp file and rename it. */
+ else if (FreeFile(fpout) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not close temporary memory context dump file \"%s\": %m",
+ tmpfile)));
+ unlink(tmpfile);
+ }
+ else if (rename(tmpfile, dumpfile) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not rename dump file \"%s\" to \"%s\": %m",
+ tmpfile, dumpfile)));
+ unlink(tmpfile);
+ }
+
+ entry->dump_status = MCXTDUMPSTATUS_DONE;
+
+ cancel_before_shmem_exit(McxtDumpKill, (Datum) PointerGetDatum(entry));
+}
+
+/*
+ * ProcessDumpMemoryInterrupt
+ * The portion of memory context dump interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessDumpMemoryInterrupt(void)
+{
+ ProcSignalDumpMemoryPending = false;
+ dump_memory_contexts();
+}
+
+/*
+ * HandleProcSignalDumpMemory
+ * Handle receipt of an interrupt indicating a memory context dump.
+ * Signal handler portion of interrupt handling.
+ */
+void
+HandleProcSignalDumpMemory(void)
+{
+ ProcSignalDumpMemoryPending = true;
+
+ /* make sure the event is processed in due course */
+ SetLatch(MyLatch);
+}
+
+/*
+ * McxtDumpShmemInit
+ * Initialize mcxtdump hash table.
+ */
+void
+McxtDumpShmemInit(void)
+{
+ HASHCTL info;
+
+ MemSet(&info, 0, sizeof(info));
+ info.keysize = sizeof(pid_t);
+ info.entrysize = sizeof(mcxtdumpEntry);
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ mcxtdumpHash = ShmemInitHash("mcxtdump hash",
+ SHMEM_MEMCONTEXT_SIZE,
+ SHMEM_MEMCONTEXT_SIZE,
+ &info,
+ HASH_ELEM | HASH_BLOBS);
+
+ LWLockRelease(McxtDumpHashLock);
+}
+
+/*
+ * pg_memusage_reset
+ * Remove the memory context dump files.
+ */
+void
+pg_memusage_reset(int pid)
+{
+ DIR *dir;
+ struct dirent *dumpfile;
+
+ if (pid == 0)
+ {
+ dir = AllocateDir(PG_MEMUSAGE_DIR);
+ while ((dumpfile = ReadDir(dir, PG_MEMUSAGE_DIR)) != NULL)
+ {
+ char dumpfilepath[32];
+
+ if (strcmp(dumpfile->d_name, ".") == 0 || strcmp(dumpfile->d_name, "..") == 0)
+ continue;
+
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, dumpfile->d_name);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ FreeDir(dir);
+ }
+ else
+ {
+ char str_pid[12];
+ char dumpfilepath[32];
+ struct stat stat_tmp;
+
+ pg_ltoa(pid, str_pid);
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, str_pid);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (stat(dumpfilepath, &stat_tmp) == 0)
+ {
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ sprintf(dumpfilepath, "%s/%s.tmp", PG_MEMUSAGE_DIR, str_pid);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (stat(dumpfilepath, &stat_tmp) == 0)
+ {
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ }
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 6ab8216839..463337f661 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -33,6 +33,7 @@ volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t ProcSignalDumpMemoryPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index ee3bfa82f4..52cdb26272 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -221,7 +221,8 @@ static const char *const subdirs[] = {
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
- "pg_logical/mappings"
+ "pg_logical/mappings",
+ "pg_memusage"
};
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index f674a7c94e..340a80fc11 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -6,7 +6,7 @@ use File::Basename qw(basename dirname);
use File::Path qw(rmtree);
use PostgresNode;
use TestLib;
-use Test::More tests => 109;
+use Test::More tests => 110;
program_help_ok('pg_basebackup');
program_version_ok('pg_basebackup');
@@ -124,7 +124,7 @@ is_deeply(
# Contents of these directories should not be copied.
foreach my $dirname (
- qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans)
+ qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans pg_memusage)
)
{
is_deeply(
diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c
index 1abc257177..ff3338e9be 100644
--- a/src/bin/pg_rewind/filemap.c
+++ b/src/bin/pg_rewind/filemap.c
@@ -85,6 +85,9 @@ static const char *excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index bbcac69d48..93fd542055 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7835,11 +7835,11 @@
# memory context of local backend
{ oid => '2282', descr => 'information about all memory contexts of local backend',
- proname => 'pg_get_backend_memory_contexts', prorows => '100', proretset => 't',
- provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
- proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ proname => 'pg_get_backend_memory_contexts', prorows => '100', proisstrict => 'f',
+ proretset => 't', provolatile => 'v', proparallel => 'r', prorettype => 'record',
+ proargtypes => 'int4', proallargtypes => '{int4,text,text,text,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
# non-persistent series generator
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..812032bb15 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -83,6 +83,7 @@ extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t ProcSignalDumpMemoryPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 5cb39697f3..5db92a9a52 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_DUMP_MEMORY, /* request dumping memory context interrupt */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/mcxtfuncs.h b/src/include/utils/mcxtfuncs.h
new file mode 100644
index 0000000000..85acae9c40
--- /dev/null
+++ b/src/include/utils/mcxtfuncs.h
@@ -0,0 +1,51 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcxtfuncs.h
+ * Declarations for showing backend memory context.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mcxtfuncs.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MCXT_H
+#define MCXT_H
+
+/* Directory to store dumped memory files */
+#define PG_MEMUSAGE_DIR "pg_memusage"
+
+#define PG_MEMCONTEXT_FILE_FORMAT_ID 0x01B5BC9E
+
+/*
+ * Size of the shmem hash table size(not a hard limit).
+ *
+ * Although it may be better to increase this number in the future (e.g.,
+ * adding views for all the backend process of memory contexts), currently
+ * small number would be enough.
+ */
+#define SHMEM_MEMCONTEXT_SIZE 64
+
+typedef enum McxtDumpStatus
+{
+ MCXTDUMPSTATUS_NOTYET,
+ MCXTDUMPSTATUS_DONE,
+ MCXTDUMPSTATUS_ERROR
+} McxtDumpStatus;
+
+typedef struct mcxtdumpEntry
+{
+ pid_t dst_pid; /* pid of the signal receiver */
+ pid_t src_pid; /* pid of the signal sender */
+ McxtDumpStatus dump_status; /* dump status */
+} mcxtdumpEntry;
+
+extern void ProcessDumpMemoryInterrupt(void);
+extern void HandleProcSignalDumpMemory(void);
+extern void McxtDumpShmemInit(void);
+extern void pg_memusage_reset(int);
+
+#endif /* MCXT_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 492cdcf74c..d637b9bbce 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1333,7 +1333,7 @@ pg_backend_memory_contexts| SELECT pg_get_backend_memory_contexts.name,
pg_get_backend_memory_contexts.free_bytes,
pg_get_backend_memory_contexts.free_chunks,
pg_get_backend_memory_contexts.used_bytes
- FROM pg_get_backend_memory_contexts() pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
+ FROM pg_get_backend_memory_contexts(NULL::integer) pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
pg_config| SELECT pg_config.name,
pg_config.setting
FROM pg_config() pg_config(name, setting);
--
2.18.1
Wait...
Attachments: 0003-Enabled-pg_get_backend_memory_contexts-to-collect.patch
For a moment I thought that the number is patch number but the
predecessors are 0002-Enabled..collect.patch and 0001-(same
name). It's not mandatory but we usually do as the follows and it's
the way of git.
v1-0001-Enabled...collect.patch
v2-0001-Enabled...collect.patch
The vn is added by -v option for git-format-patch.
At Thu, 22 Oct 2020 21:32:00 +0900, torikoshia <torikoshia@oss.nttdata.com> wrote in
I added a shared hash table consisted of minimal members
mainly for managing whether the file is dumped or not.
Some members like 'loc' seem useful in the future, but I
haven't added them since it's not essential at this point.Yes, that would be good. + /* + * Since we allow only one session can request to dump memory context at + * the same time, check whether the dump files already exist. + */ + while (stat(dumpfile, &stat_tmp) == 0 || stat(tmpfile, &stat_tmp) == 0) + { + pg_usleep(1000000L); + } If pg_get_backend_memory_contexts() is executed by two or more sessions at the same time, it cannot be run exclusively in this way. Currently it seems to cause a crash when do it so. This is easy to reproduce and can be done as follows. [session-1] BEGIN; LOCK TABKE t1; [Session-2] BEGIN; LOCK TABLE t1; <- waiting [Session-3] select * FROM pg_get_backend_memory_contexts(<pid of session-2>); [Session-4] select * FROM pg_get_backend_memory_contexts(<pid of session-2>); If you issue commit or abort at session-1, you will get SEGV. Instead of checking for the existence of the file, it might be better to use a hash (mcxtdumpHash) entry with LWLock.Thanks!
Added a LWLock and changed the way from checking the file existence
to finding the hash entry.
+ if (proc == NULL) + { + ereport(WARNING, + (errmsg("PID %d is not a PostgreSQL server process", dst_pid))); + return (Datum) 1; + } Shouldn't it clear the hash entry before return?Yeah. added codes for removing the entry.
+ entry = AddEntryToMcxtdumpHash(dst_pid);
+
+ /* Check whether the target process is PostgreSQL backend process. */
+ /* TODO: Check also whether backend or not. */
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", dst_pid)));
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ if (hash_search(mcxtdumpHash, &dst_pid, HASH_REMOVE, NULL) == NULL)
+ elog(WARNING, "hash table corrupted");
+
+ LWLockRelease(McxtDumpHashLock);
+
+ return (Datum) 1;
+ }
Why do you enter a useles entry then remove it immedately?
+ PG_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid));
+ {
+ SendProcSignal(dst_pid, PROCSIG_DUMP_MEMORY, InvalidBackendId);
"PROCSIG_DUMP_MEMORY" is somewhat misleading. Hwo about
"PROCSIG_DUMP_MEMCXT" or "PROCSIG_DUMP_MEMORY_CONTEXT"?
I thought that the hash table would prevent multiple reqestors from
making a request at once, but the patch doesn't seem to do that.
+ /* Wait until target process finished dumping file. */
+ while (entry->dump_status == MCXTDUMPSTATUS_NOTYET)
This needs LWLock. And this could read the entry after reused by
another backend if the dumper process is gone. That isn't likely to
happen, but theoretically the other backend may set it to
MCXTDUMPSTATUS_NOTYET inbetween two successive check on the member.
+ /*
+ * Make dump file ends with 'D'.
+ * This is checked by the caller when reading the file.
+ */
+ fputc('E', fpout);
Which is right?
+ fputc('E', fpout);
+
+ CHECK_FOR_INTERRUPTS();
This means that the process accepts another request and rewrite the
file even while the first requester is reading it. And, the file can
be removed by the first requestor before the second requestor can read
it.
+ mcxtdumpHash = ShmemInitHash("mcxtdump hash",
+ SHMEM_MEMCONTEXT_SIZE,
+ SHMEM_MEMCONTEXT_SIZE,
The space needed for this hash doesn't seem to be secured. The hash is
sized to 64 entries, so pg_get_backend_memory_contexts() may fail with
ERROR "out of shared memory".
The hash is used only to check if the dump file is completed and if
ended with error. If we need only those, an shared byte array with the
length of max_backend is sufficient.
+ PG_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid));
+ {
+ SendProcSignal(dst_pid, PROCSIG_DUMP_MEMORY, InvalidBackendId);
+
+ /* Wait until target process finished dumping file. */
+ while (entry->dump_status == MCXTDUMPSTATUS_NOTYET)
+ {
+ CHECK_FOR_INTERRUPTS();
+ pg_usleep(10000L);
+ }
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid));
+
+ if (entry->dump_status == MCXTDUMPSTATUS_ERROR)
+ {
..
+ if (stat(tmpfile, &stat_tmp) == 0)
+ unlink(tmpfile);
+ if (stat(dumpfile, &stat_tmp) == 0)
+ unlink(dumpfile);
...
+ return (Datum) 0;
+ }
+
+ /* Read values from the dumped file and put them on tuplestore. */
+ PutDumpedValuesOnTuplestore(dumpfile, tupstore, tupdesc, dst_pid);
This means that if the function gets sigint before the dumper creates
the file, the dumper can leave a dump file?
+ /* Wait until target process finished dumping file. */ + while (!entry->is_dumped) + { + CHECK_FOR_INTERRUPTS(); + pg_usleep(10000L); + } If the target is killed and exit before dumping the memory information, you're in an infinite loop here. So how about making sure that any process that has to stop before doing a memory dump changes the status of the hash (entry->is_dumped) before stopping and the caller detects it? I'm not sure it's best or not, but you might want to use something like the on_shmem_exit callback.Thanks for your idea!
Added a callback to change the status of the hash table entry.Although I think it's necessary to remove this callback when it
finished
processing memory dumping, on_shmem_exit() does not seem to have such
a function.
I used before_shmem_exit() since it has a corresponding function to
remove registered callback.
If it's inappropriate, I'm going to add a function removing the
registered callback of on_shmem_exit().
This seems to leave a file for the pid.
In the current design, if the caller stops processing before reading
the dumped file, you will have an orphaned file.
It looks like the following.
[session-1]
BEGIN;
LOCK TABKE t1;
[Session-2]
BEGIN;
LOCK TABLE t1; <- waiting
[Session-3]
select * FROM pg_get_backend_memory_contexts(<pid of session-2>);
If you cancel or terminate the session-3, then issue commit or abort
at session-1, you will get orphaned files in pg_memusage.
So if you allow only one session can request to dump file, it could
call pg_memusage_reset() before send the signal in this function.Although I'm going to allow only one session per one target process,
I'd like to allow running multiple pg_get_backend_memory_contexts()
which target process is different.Instead of calling pg_memusage_reset(), I added a callback for
cleaning up orphaned files and the elements of the hash table
using before_shmem_exit() through PG_ENSURE_ERROR_CLEANUP() and
PG_END_ENSURE_ERROR_CLEANUP().I chose PG_ENSURE_ERROR_CLEANUP() and PG_END_ENSURE_ERROR_CLEANUP()
here since it can handle not only termination but also cancellation.Any thoughts?
+/*
+ * pg_memusage_reset
+ * Remove the memory context dump files.
+ */
+void
+pg_memusage_reset(int pid)
The function name seem to represents somewhat different from what it
does.
I think we might need to step-back to basic design of this feature
since this patch seems to have unhandled corner cases that are
difficult to find.
- Dump file lifecycle or state-transition of the dumper
Currently the lifecycle of a dump file, or the state-transition of
the dumper process doesn't seem to be well defined.
The file is create only by the dumper.
If the requestor reads the completed file, the reader removes it.
If the dumper receives a cancel request, the dumper removes it if
any.
Of course dumper removes it if it fails to complete the file.
- Better way of signalling between the requestor and the dumper
I think there's no doubt about request signal.
About the complete signal, currently the requestor polls on a flag
in a hash entry. I'm wondering if we can get rid of polling. The
problem on doing that is the lack of a means for a dumper to know
the requestor. We need to store requestor pid or backend id in the
shared hash entry.
By the way, about shared hash entry, it uses about 70kB for only 64
entries so it seems inefficient than a shared array that has
MaxBackends entries. If we used a following array on shared memory,
struct hoge
{
BackendId requestor[MAX_BACKENDS];
int status[MAX_BACKENDS];
LWLock lock;
};
This array has the size of 24 * MaxBackends + 16. 24kB for 1000
backends. It could be on dsm since this feature is not used
commonly.
- The way to cancel a request already made. (or management of the dump
state transition.)
Cancellation seems to contain some race conditions. But basically
that could be done by sending a request signal after setting the
hoge.requestor above to some special value, not needing the third
type of signal. The special value should be different from the
initial state, which signals that the process is accepting a new
request.
As the whole, that would looks like the folloing?
------------------------------------------------------------
Successful request.
Requestor dumper state
[idle] initial
[request] -------------------> requestor pid/backendid
-signal->
[dumping]
<-signal-[done]
[read]
[done] --------------------> initial
------------------------------------------------------------
On failure, the dumper signals with setting state to initial.
[request] -------------------> requestor pid/backendid
-signal->
[dumping]
[failed] initial
<-signal-
(some other requestor might come meantime.)
<sees that the requestor is not me>
[failed]
------------------------------------------------------------
If the requestor wants to cancel the request, it sets the state to
'cancel' then signal.
Requestor dumper state
[idle] initial
[request] -------------------> cancel
<if canceled. clean up>
[dumping]
<if canceled. clean up>
<-signal-[done]
-signal-><try to clean up>
Other aspects to cnosider?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On 2020-10-23 13:46, Kyotaro Horiguchi wrote:
Wait...
Attachments:
0003-Enabled-pg_get_backend_memory_contexts-to-collect.patchFor a moment I thought that the number is patch number but the
predecessors are 0002-Enabled..collect.patch and 0001-(same
name). It's not mandatory but we usually do as the follows and it's
the way of git.v1-0001-Enabled...collect.patch
v2-0001-Enabled...collect.patchThe vn is added by -v option for git-format-patch.
Sorry for the confusion. I'll follow that way next time.
At Thu, 22 Oct 2020 21:32:00 +0900, torikoshia
<torikoshia@oss.nttdata.com> wrote inI added a shared hash table consisted of minimal members
mainly for managing whether the file is dumped or not.
Some members like 'loc' seem useful in the future, but I
haven't added them since it's not essential at this point.Yes, that would be good. + /* + * Since we allow only one session can request to dump memory context at + * the same time, check whether the dump files already exist. + */ + while (stat(dumpfile, &stat_tmp) == 0 || stat(tmpfile, &stat_tmp) == 0) + { + pg_usleep(1000000L); + } If pg_get_backend_memory_contexts() is executed by two or more sessions at the same time, it cannot be run exclusively in this way. Currently it seems to cause a crash when do it so. This is easy to reproduce and can be done as follows. [session-1] BEGIN; LOCK TABKE t1; [Session-2] BEGIN; LOCK TABLE t1; <- waiting [Session-3] select * FROM pg_get_backend_memory_contexts(<pid of session-2>); [Session-4] select * FROM pg_get_backend_memory_contexts(<pid of session-2>); If you issue commit or abort at session-1, you will get SEGV. Instead of checking for the existence of the file, it might be better to use a hash (mcxtdumpHash) entry with LWLock.Thanks!
Added a LWLock and changed the way from checking the file existence
to finding the hash entry.+ if (proc == NULL) + { + ereport(WARNING, + (errmsg("PID %d is not a PostgreSQL server process", dst_pid))); + return (Datum) 1; + } Shouldn't it clear the hash entry before return?Yeah. added codes for removing the entry.
+ entry = AddEntryToMcxtdumpHash(dst_pid); + + /* Check whether the target process is PostgreSQL backend process. */ + /* TODO: Check also whether backend or not. */ + proc = BackendPidGetProc(dst_pid); + + if (proc == NULL) + { + ereport(WARNING, + (errmsg("PID %d is not a PostgreSQL server process", dst_pid))); + + LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE); + + if (hash_search(mcxtdumpHash, &dst_pid, HASH_REMOVE, NULL) == NULL) + elog(WARNING, "hash table corrupted"); + + LWLockRelease(McxtDumpHashLock); + + return (Datum) 1; + }Why do you enter a useles entry then remove it immedately?
Do you mean I should check the process existence first
since it enables us to skip entering hash entries?
+ PG_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid)); + { + SendProcSignal(dst_pid, PROCSIG_DUMP_MEMORY, InvalidBackendId);"PROCSIG_DUMP_MEMORY" is somewhat misleading. Hwo about
"PROCSIG_DUMP_MEMCXT" or "PROCSIG_DUMP_MEMORY_CONTEXT"?
I'll go with "PROCSIG_DUMP_MEMCXT".
I thought that the hash table would prevent multiple reqestors from
making a request at once, but the patch doesn't seem to do that.+ /* Wait until target process finished dumping file. */ + while (entry->dump_status == MCXTDUMPSTATUS_NOTYET)This needs LWLock. And this could read the entry after reused by
another backend if the dumper process is gone. That isn't likely to
happen, but theoretically the other backend may set it to
MCXTDUMPSTATUS_NOTYET inbetween two successive check on the member.
Thanks for your notification.
I'll use an LWLock.
+ /* + * Make dump file ends with 'D'. + * This is checked by the caller when reading the file. + */ + fputc('E', fpout);Which is right?
Sorry, the comment was wrong..
+ fputc('E', fpout); + + CHECK_FOR_INTERRUPTS();This means that the process accepts another request and rewrite the
file even while the first requester is reading it. And, the file can
be removed by the first requestor before the second requestor can read
it.
I added CHECK_FOR_INTERRUPTS() here to make the dump cancellation
possible, however, considering your indication, it needs to think
about a way to handle only the dump cancellation.
+ mcxtdumpHash = ShmemInitHash("mcxtdump hash", + SHMEM_MEMCONTEXT_SIZE, + SHMEM_MEMCONTEXT_SIZE, . The space needed for this hash doesn't seem to be secured. The hash is sized to 64 entries, so pg_get_backend_memory_contexts() may fail with ERROR "out of shared memory".The hash is used only to check if the dump file is completed and if
ended with error. If we need only those, an shared byte array with the
length of max_backend is sufficient.
Although there was a comment that controlling dump location may be a
good idea, but it also seems possible to include the location
information
in the struct Hoge you suggested below.
On Tue, Sep 1, 2020 at 12:03 AM Kasahara Tatsuhito <[hidden email]>
wrote:
| - send a signal to dump to a PID, it first record following
information into the shared hash.
| pid (specified pid)
| loc (dump location, currently might be ASAP)
+ PG_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid)); + { + SendProcSignal(dst_pid, PROCSIG_DUMP_MEMORY, InvalidBackendId); + + /* Wait until target process finished dumping file. */ + while (entry->dump_status == MCXTDUMPSTATUS_NOTYET) + { + CHECK_FOR_INTERRUPTS(); + pg_usleep(10000L); + } + } + PG_END_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid)); + + if (entry->dump_status == MCXTDUMPSTATUS_ERROR) + { .. + if (stat(tmpfile, &stat_tmp) == 0) + unlink(tmpfile); + if (stat(dumpfile, &stat_tmp) == 0) + unlink(dumpfile); ... + return (Datum) 0; + } + + /* Read values from the dumped file and put them on tuplestore. */ + PutDumpedValuesOnTuplestore(dumpfile, tupstore, tupdesc, dst_pid);This means that if the function gets sigint before the dumper creates
the file, the dumper can leave a dump file?
In that case, the requestor removes the corresponding hash entry in the
callback McxtReqKill, then the dumper who can not find the hash entry
does not dump a file.
However, I've now noticed that when the requestor gets sigint just
after the dumper check, the dump file remains.
In the current design, only the requestor can remove the dump file,
but it seems necessary to allow the dumper to remove it.
+ /* Wait until target process finished dumping file. */ + while (!entry->is_dumped) + { + CHECK_FOR_INTERRUPTS(); + pg_usleep(10000L); + } If the target is killed and exit before dumping the memory information, you're in an infinite loop here. So how about making sure that any process that has to stop before doing a memory dump changes the status of the hash (entry->is_dumped) before stopping and the caller detects it? I'm not sure it's best or not, but you might want to use something like the on_shmem_exit callback.Thanks for your idea!
Added a callback to change the status of the hash table entry.Although I think it's necessary to remove this callback when it
finished
processing memory dumping, on_shmem_exit() does not seem to have such
a function.
I used before_shmem_exit() since it has a corresponding function to
remove registered callback.
If it's inappropriate, I'm going to add a function removing the
registered callback of on_shmem_exit().This seems to leave a file for the pid.
As mentioned above, there can be a chance to remain files.
In the current design, if the caller stops processing before reading
the dumped file, you will have an orphaned file.
It looks like the following.
[session-1]
BEGIN;
LOCK TABKE t1;
[Session-2]
BEGIN;
LOCK TABLE t1; <- waiting
[Session-3]
select * FROM pg_get_backend_memory_contexts(<pid of session-2>);
If you cancel or terminate the session-3, then issue commit or abort
at session-1, you will get orphaned files in pg_memusage.
So if you allow only one session can request to dump file, it could
call pg_memusage_reset() before send the signal in this function.Although I'm going to allow only one session per one target process,
I'd like to allow running multiple pg_get_backend_memory_contexts()
which target process is different.Instead of calling pg_memusage_reset(), I added a callback for
cleaning up orphaned files and the elements of the hash table
using before_shmem_exit() through PG_ENSURE_ERROR_CLEANUP() and
PG_END_ENSURE_ERROR_CLEANUP().I chose PG_ENSURE_ERROR_CLEANUP() and PG_END_ENSURE_ERROR_CLEANUP()
here since it can handle not only termination but also cancellation.Any thoughts?
+/* + * pg_memusage_reset + * Remove the memory context dump files. + */ +void +pg_memusage_reset(int pid)The function name seem to represents somewhat different from what it
does.
Yeah, It looks like it actually reset the memory.
I'll rename it to remove_memcxt_file or something.
I think we might need to step-back to basic design of this feature
since this patch seems to have unhandled corner cases that are
difficult to find.
Agreed and thanks for writing it down below.
- Dump file lifecycle or state-transition of the dumper
Currently the lifecycle of a dump file, or the state-transition of
the dumper process doesn't seem to be well defined.The file is create only by the dumper.
If the requestor reads the completed file, the reader removes it.If the dumper receives a cancel request, the dumper removes it if
any.
Of course dumper removes it if it fails to complete the file.
- Better way of signalling between the requestor and the dumper
I think there's no doubt about request signal.
About the complete signal, currently the requestor polls on a flag
in a hash entry. I'm wondering if we can get rid of polling. The
problem on doing that is the lack of a means for a dumper to know
the requestor. We need to store requestor pid or backend id in the
shared hash entry.
Agreed to get rid of polling.
BTW, it seems common to use a latch instead of pg_usleep() to wait until
signals arrive as far as I read latch.h.
I'm now thinking about using a latch here and it would make polling
removed.
By the way, about shared hash entry, it uses about 70kB for only 64
entries so it seems inefficient than a shared array that has
MaxBackends entries. If we used a following array on shared memory,struct hoge
{
BackendId requestor[MAX_BACKENDS];
int status[MAX_BACKENDS];
LWLock lock;
};
If the requestor's id is 5 and dumper's id is 10,
is this struct used like below?
- hoge.requestor[10] = 5
- Both status[5] and status[10] change like "request", "idle" or "done"
This array has the size of 24 * MaxBackends + 16. 24kB for 1000
backends. It could be on dsm since this feature is not used
commonly.
Sorry but I'm not sure this calculation.
Do you mean 16.24kB for 1000 backends?
Regarding dsm, do you imagine using hoge on dsm?
- The way to cancel a request already made. (or management of the dump
state transition.)Cancellation seems to contain some race conditions. But basically
that could be done by sending a request signal after setting the
hoge.requestor above to some special value, not needing the third
I imagined hoge.requestor was set to requestor's backendid.
Isn't it hoge.status?
type of signal. The special value should be different from the
initial state, which signals that the process is accepting a new
request.As the whole, that would looks like the folloing?
------------------------------------------------------------
Successful request.Requestor dumper state
[idle] initial
[request] -------------------> requestor pid/backendid
-signal->
[dumping]
<-signal-[done]
[read]
[done] --------------------> initial------------------------------------------------------------
On failure, the dumper signals with setting state to initial.
[request] -------------------> requestor pid/backendid
-signal->
[dumping]
[failed] initial
<-signal-
(some other requestor might come meantime.)
<sees that the requestor is not me>
Is this "some other requestor come case" relevant to the dumper failure?
Regards,
--
Atsushi Torikoshi
Show quoted text
[failed]
------------------------------------------------------------
If the requestor wants to cancel the request, it sets the state to
'cancel' then signal.Requestor dumper state
[idle] initial
[request] -------------------> cancel
<if canceled. clean up>
[dumping]
<if canceled. clean up>
<-signal-[done]-signal-><try to clean up>
Other aspects to cnosider?
regards.
Hi,
I noticed that this patch fails on the cfbot.
For this, I changed the status to: 'Waiting on Author'.
Cheers,
//Georgios
The new status of this patch is: Waiting on Author
On 2020-10-28 15:32, torikoshia wrote:
On 2020-10-23 13:46, Kyotaro Horiguchi wrote:
I think we might need to step-back to basic design of this feature
since this patch seems to have unhandled corner cases that are
difficult to find.
I've written out the basic design below and attached
corresponding patch.
# Communication flow between the dumper and the requester
- (1) When REQUESTING memory context dumping, the dumper adds an entry
to the shared memory. The entry manages the dump state and it is set to
'REQUESTING'.
- (2) The dumper sends the signal to the dumper and wait on the latch.
- (3) The dumper looks into the corresponding shared memory entry and
changes its state to 'DUMPING'.
- (4) When the dumper completes dumping, it changes the state to
'DONE' and set the latch.
- (5) The dumper reads the dump file and shows it to the user.
Finally, the dumper removes the dump file and reset the shared memory
entry.
# Query cancellation
- When the requestor cancels dumping, e.g. signaling using ctrl-C, the
requestor changes the status of the shared memory entry to 'CANCELING'.
- The dumper checks the status when it tries to change the state to
'DONE' at (4), and if the state is 'CANCELING', it removes the dump file
and reset the shared memory entry.
# Cleanup dump file and the shared memory entry
- In the normal case, the dumper removes the dump file and resets the
shared memory entry as described in (5).
- When something like query cancellation or process termination
happens on the dumper after (1) and before (3), in other words, the
state is 'REQUESTING', the requestor does the cleanup.
- When something happens on the dumper or the requestor after (3) and
before (4), in other words, the state is 'DUMPING', the dumper does the
cleanup. Specifically, if the requestor cancels the query, it just
changes the state to 'CANCELING' and the dumper notices it and cleans up
things later. OTOH, when the dumper fails to dump, it cleans up the dump
file and deletes the entry on the shared memory.
- When something happens on the requestor after (4), i.e., the state
is 'DONE', the requestor does the cleanup.
- In the case of receiving SIGKILL or power failure, all dump files
are removed in the crash recovery process.
Although there was a suggestion that shared memory hash
table should be changed to more efficient structures,
I haven't done it in this patch.
I think it can be treated separately, I'm going to work
on that later.
On 2020-11-11 00:07, Georgios Kokolatos wrote:
Hi,
I noticed that this patch fails on the cfbot.
For this, I changed the status to: 'Waiting on Author'.Cheers,
//GeorgiosThe new status of this patch is: Waiting on Author
Thanks for your notification and updated the patch.
Changed the status to: 'Waiting on Author'.
Regards,
--
Atsushi Torikoshi
Attachments:
v4-0001-Enabled-pg_get_backend_memory_contexts-to-collect.patchtext/x-diff; name=v4-0001-Enabled-pg_get_backend_memory_contexts-to-collect.patchDownload
From c6d06b11d16961acd59bfa022af52cb5fc668b3e Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 16 Nov 2020 11:49:03 +0900
Subject: [PATCH v4] Enabled pg_get_backend_memory_contexts() to collect
arbitrary backend process's memory contexts.
Previsouly, pg_get_backend_memory_contexts() could only get the
local memory contexts. This patch enables to get memory contexts
of the arbitrary backend process which PID is specified by the
argument.
---
src/backend/access/transam/xlog.c | 7 +
src/backend/catalog/system_views.sql | 4 +-
src/backend/postmaster/pgstat.c | 3 +
src/backend/replication/basebackup.c | 3 +
src/backend/storage/ipc/ipci.c | 2 +
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/tcop/postgres.c | 5 +
src/backend/utils/adt/mcxtfuncs.c | 615 ++++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/bin/initdb/initdb.c | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 4 +-
src/bin/pg_rewind/filemap.c | 3 +
src/include/catalog/pg_proc.dat | 11 +-
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 3 +-
src/include/storage/procsignal.h | 1 +
src/include/utils/mcxtfuncs.h | 52 ++
src/test/regress/expected/rules.out | 2 +-
19 files changed, 697 insertions(+), 28 deletions(-)
create mode 100644 src/include/utils/mcxtfuncs.h
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index a1078a7cfc..f628fa8b53 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -73,6 +73,7 @@
#include "storage/sync.h"
#include "utils/builtins.h"
#include "utils/guc.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/relmapper.h"
@@ -6986,6 +6987,12 @@ StartupXLOG(void)
*/
pgstat_reset_all();
+ /*
+ * Reset dumped files in pg_memusage, because target processes do
+ * not exist any more.
+ */
+ RemoveMemcxtFile(0);
+
/*
* If there was a backup label file, it's done its job and the info
* has now been propagated into pg_control. We must get rid of the
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2e4aa1c4b6..06b0bd16b5 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -555,10 +555,10 @@ REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
CREATE VIEW pg_backend_memory_contexts AS
- SELECT * FROM pg_get_backend_memory_contexts();
+ SELECT * FROM pg_get_backend_memory_contexts(NULL);
REVOKE ALL ON pg_backend_memory_contexts FROM PUBLIC;
-REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts FROM PUBLIC;
-- Statistics views
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index e76e627c6b..225354354a 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4024,6 +4024,9 @@ pgstat_get_wait_ipc(WaitEventIPC w)
case WAIT_EVENT_XACT_GROUP_UPDATE:
event_name = "XactGroupUpdate";
break;
+ case WAIT_EVENT_MEMORY_CONTEXT_DUMP:
+ event_name = "MemoryContextDump";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index b89df01fa7..3edb591952 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -184,6 +184,9 @@ static const char *const excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 96c2aaabbd..92f21ad2bf 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -45,6 +45,7 @@
#include "storage/procsignal.h"
#include "storage/sinvaladt.h"
#include "storage/spin.h"
+#include "utils/mcxtfuncs.h"
#include "utils/snapmgr.h"
/* GUCs */
@@ -267,6 +268,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ McxtDumpShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index ffe67acea1..a1e8890642 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -28,6 +28,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/mcxtfuncs.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -567,6 +568,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_DUMP_MEMCXT))
+ HandleProcSignalDumpMemory();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 774292fd94..6036713f11 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+McxtDumpHashLock 48
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 411cfadbff..e8f4175c48 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -75,6 +75,7 @@
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
#include "utils/lsyscache.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -539,6 +540,10 @@ ProcessClientReadInterrupt(bool blocked)
/* Process notify interrupts, if any */
if (notifyInterruptPending)
ProcessNotifyInterrupt();
+
+ /* Process memory contexts dump interrupts, if any */
+ if (ProcSignalDumpMemoryPending)
+ ProcessDumpMemoryInterrupt();
}
else if (ProcDiePending)
{
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 50e1b07ff0..564224ea3d 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,30 +15,92 @@
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "common/logging.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "pgstat.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "storage/shmem.h"
#include "utils/builtins.h"
+#include "utils/mcxtfuncs.h"
+
+/* The max bytes for showing names and identifiers of MemoryContext. */
+#define MEMORY_CONTEXT_DISPLAY_SIZE 1024
+
+/* Number of columns in pg_backend_memory_contexts view */
+#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
-/* ----------
- * The max bytes for showing identifiers of MemoryContext.
- * ----------
+/* Hash for managing the status of memory context dump. */
+static HTAB *mcxtdumpHash = NULL;
+
+
+/*
+ * McxtReqKill
+ * Cleanup function for memory context dump requestor.
+ *
+ * Called when the caller of pg_get_backend_memory_contexts()
+ * exits.
*/
-#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+static void
+McxtReqKill(int code, Datum arg)
+{
+ mcxtdumpEntry *entry;
+ int dump_status;
+ int dst_pid = DatumGetInt32(arg);;
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ entry = (mcxtdumpEntry *) hash_search(mcxtdumpHash, &dst_pid, HASH_FIND, NULL);
+
+ if (entry == NULL)
+ elog(ERROR, "hash table corrupted");
+
+ dump_status = entry->dump_status;
+
+ if (dump_status == MCXTDUMPSTATUS_REQUESTING)
+ {
+ elog(DEBUG2, "removing %d entry at MCXTDUMPSTATUS_REQUESTING." , dst_pid);
+ hash_search(mcxtdumpHash, &dst_pid, HASH_REMOVE, NULL);
+ }
+
+ else if (dump_status == MCXTDUMPSTATUS_DUMPING)
+ {
+ entry->dump_status = MCXTDUMPSTATUS_CANCELING;
+ elog(DEBUG2, "status changed from MCXTDUMPSTATUS_DUMPING to MCXTDUMPSTATUS_CANCELING.");
+ }
+ else if (dump_status == MCXTDUMPSTATUS_DONE)
+ {
+ hash_search(mcxtdumpHash, &dst_pid, HASH_REMOVE, NULL);
+
+ /* for debug */
+ elog(DEBUG2, "removing dump file of PID %d at MCXTDUMPSTATUS_DONE.", dst_pid);
+ RemoveMemcxtFile(dst_pid);
+ }
+ LWLockRelease(McxtDumpHashLock);
+}
/*
* PutMemoryContextsStatsTupleStore
* One recursion level for pg_get_backend_memory_contexts.
+ *
+ * Note: When fpout is not NULL, ferror() check must be done by the caller.
*/
static void
PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
TupleDesc tupdesc, MemoryContext context,
- const char *parent, int level)
+ const char *parent, int level, FILE *fpout)
{
-#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
-
Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
MemoryContextCounters stat;
MemoryContext child;
const char *name;
@@ -74,14 +136,12 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
if (ident)
{
int idlen = strlen(ident);
- char clipped_ident[MEMORY_CONTEXT_IDENT_DISPLAY_SIZE];
-
/*
* Some identifiers such as SQL query string can be very long,
* truncate oversize identifiers.
*/
- if (idlen >= MEMORY_CONTEXT_IDENT_DISPLAY_SIZE)
- idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_IDENT_DISPLAY_SIZE - 1);
+ if (idlen >= MEMORY_CONTEXT_DISPLAY_SIZE)
+ idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_DISPLAY_SIZE - 1);
memcpy(clipped_ident, ident, idlen);
clipped_ident[idlen] = '\0';
@@ -101,13 +161,198 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
values[6] = Int64GetDatum(stat.freespace);
values[7] = Int64GetDatum(stat.freechunks);
values[8] = Int64GetDatum(stat.totalspace - stat.freespace);
- tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Since pg_get_backend_memory_contexts() is called from local process,
+ * simply put tuples.
+ */
+ if(fpout == NULL)
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Write out the current memory context information in the form of
+ * "key: value" pairs to the file specified by the requestor.
+ */
+ else
+ {
+ /*
+ * Make each memory context information starts with 'D'.
+ * This is checked by the requestor when reading the file.
+ */
+ fputc('D', fpout);
+
+ fprintf(fpout,
+ "name: %s, ident: %s, parent: %s, level: %d, total_bytes: %lu, \
+ total_nblocks: %lu, free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name,
+ ident ? clipped_ident : "none",
+ parent ? parent : "none", level,
+ stat.totalspace,
+ stat.nblocks,
+ stat.freespace,
+ stat.freechunks,
+ stat.totalspace - stat.freespace);
+ }
for (child = context->firstchild; child != NULL; child = child->nextchild)
{
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- child, name, level + 1);
+ child, name, level + 1, fpout);
+ }
+}
+
+/*
+ * AddEntryToMcxtdumpHash
+ * Add an entry to McxtdumpHash for specified PID.
+ */
+static mcxtdumpEntry *
+AddEntryToMcxtdumpHash(int pid)
+{
+ mcxtdumpEntry *entry;
+ bool found;
+
+ /*
+ * We only allow one session per target process to request a memory
+ * dump at a time.
+ * If mcxtdumpHash has corresponding entry, wait until it has removed.
+ */
+ while (true)
+ {
+ LWLockAcquire(McxtDumpHashLock, LW_SHARED);
+ entry = (mcxtdumpEntry *) hash_search(mcxtdumpHash, &pid,
+ HASH_ENTER, &found);
+
+ if (!found)
+ {
+ /* Need exclusive lock to make a new hashtable entry */
+ LWLockRelease(McxtDumpHashLock);
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ entry->dump_status = MCXTDUMPSTATUS_REQUESTING;
+ entry->src_pid = MyProcPid;
+
+ LWLockRelease(McxtDumpHashLock);
+
+ return entry;
+ }
+ else
+ {
+ ereport(INFO,
+ (errmsg("PID %d is looked up by another process", pid)));
+
+ LWLockRelease(McxtDumpHashLock);
+
+ pg_usleep(5000000L);
+ }
+ }
+}
+
+/*
+ * PutDumpedValuesOnTuplestore
+ * Read specified memory context dump file and put its values
+ * on the tuple store.
+ */
+static void
+PutDumpedValuesOnTuplestore(char *dumpfile, Tuplestorestate *tupstore,
+ TupleDesc tupdesc, int pid)
+{
+ FILE *fpin;
+ int format_id;
+
+ if ((fpin = AllocateFile(dumpfile, "r")) == NULL)
+ {
+ if (errno != ENOENT)
+ ereport(LOG, (errcode_for_file_access(),
+ errmsg("could not open memory context dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ /* Verify it's of the expected format. */
+ if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ format_id != PG_MEMCONTEXT_FILE_FORMAT_ID)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"", dumpfile)));
+ goto done;
+ }
+
+ /* Read dump file and put values on tuple store. */
+ while (true)
+ {
+ Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char name[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char parent[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
+ int level;
+ Size total_bytes;
+ Size total_nblocks;
+ Size free_bytes;
+ Size free_chunks;
+ Size used_bytes;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ switch (fgetc(fpin))
+ {
+ /* 'D' A memory context information follows. */
+ case 'D':
+ if (fscanf(fpin, "name: %1023[^,], ident: %1023[^,], parent: %1023[^,], \
+ level: %d, total_bytes: %lu, total_nblocks: %lu, \
+ free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name, clipped_ident, parent, &level, &total_bytes, &total_nblocks,
+ &free_bytes, &free_chunks, &used_bytes)
+ != PG_GET_BACKEND_MEMORY_CONTEXTS_COLS)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+
+ values[0] = CStringGetTextDatum(name);
+
+ if (strcmp(clipped_ident, "none"))
+ values[1] = CStringGetTextDatum(clipped_ident);
+ else
+ nulls[1] = true;
+
+ if (strcmp(parent, "none"))
+ values[2] = CStringGetTextDatum(parent);
+ else
+ nulls[2] = true;
+
+ values[3] = Int32GetDatum(level);
+ values[4] = Int64GetDatum(total_bytes);
+ values[5] = Int64GetDatum(total_nblocks);
+ values[6] = Int64GetDatum(free_bytes);
+ values[7] = Int64GetDatum(free_chunks);
+ values[8] = Int64GetDatum(used_bytes);
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ break;
+
+ case 'E':
+ goto done;
+
+ default:
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
}
+done:
+ FreeFile(fpin);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ if (hash_search(mcxtdumpHash, &pid, HASH_REMOVE, NULL) == NULL)
+ elog(ERROR, "hash table corrupted");
+
+ LWLockRelease(McxtDumpHashLock);
}
/*
@@ -117,6 +362,8 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
{
+ int dst_pid = PG_ARGISNULL(0) ? -1 : PG_GETARG_INT32(0);
+
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -147,11 +394,349 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldcontext);
- PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- TopMemoryContext, NULL, 0);
+ /*
+ * If the target is local process, simply look into memory contexts
+ * recursively.
+ */
+ if (dst_pid == -1 || dst_pid == MyProcPid)
+ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
+ TopMemoryContext, "", 0, NULL);
+
+ /*
+ * Target process is not local.
+ * Send signal for dumping memory contexts to the target process,
+ * and read the dump file.
+ */
+ else
+ {
+ char dumpfile[MAXPGPATH];
+ mcxtdumpEntry *entry;
+ PGPROC *proc;
+
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, dst_pid);
+
+ /*
+ * Check whether the target process is PostgreSQL backend process.
+ *
+ * If the target process dies after this point and before sending signal,
+ * users are expected to cancel the request.
+ */
+
+ /* TODO: Check also whether backend or not. */
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", dst_pid)));
+
+ return (Datum) 1;
+ }
+
+ /*
+ * The ENSURE stuff ensures we clean up the shared memory entry and files
+ * on failure.
+ */
+ PG_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid));
+ {
+ entry = AddEntryToMcxtdumpHash(dst_pid);
+
+ SendProcSignal(dst_pid, PROCSIG_DUMP_MEMCXT, InvalidBackendId);
+
+ /* Wait until target process finishes dumping file. */
+ for (;;)
+ {
+ /* Check for dump cancel request. */
+ CHECK_FOR_INTERRUPTS();
+
+ /* Must reset the latch before testing state. */
+ ResetLatch(MyLatch);
+
+ /* Check whether the dump has completed. */
+ LWLockAcquire(McxtDumpHashLock, LW_SHARED);
+ entry = (mcxtdumpEntry *) hash_search(mcxtdumpHash, &dst_pid, HASH_FIND, NULL);
+
+ if (entry == NULL)
+ {
+ /*
+ * Dumper seems to cleanup the enry because of failures or
+ * cancellation.
+ * Since the dumper has already removed the dump file, the
+ * requestor can simply exit.
+ */
+ LWLockRelease(McxtDumpHashLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+
+ if (entry->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /* Request has canceled. Exit without dumping. */
+ LWLockRelease(McxtDumpHashLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+
+ else if (entry->dump_status == MCXTDUMPSTATUS_DONE)
+ {
+ /* Dumping has completed. */
+ LWLockRelease(McxtDumpHashLock);
+ break;
+ }
+
+ LWLockRelease(McxtDumpHashLock);
+
+ /*
+ * The state is either the dumper is in the middle of a dump,
+ * or the request hasn't been reached yet.
+ */
+ Assert(entry->dump_status == MCXTDUMPSTATUS_REQUESTING ||
+ entry->dump_status == MCXTDUMPSTATUS_DUMPING);
+
+ /*
+ * Wait. We expect to get a latch signal back from the dumper,
+ * but use a timeout to enable cancellation.
+ */
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ 1000L, WAIT_EVENT_MEMORY_CONTEXT_DUMP);
+ }
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid));
+
+ /* Read values from the dump file and put them on tuplestore. */
+ PutDumpedValuesOnTuplestore(dumpfile, tupstore, tupdesc, dst_pid);
+ }
/* clean up and return the tuplestore */
tuplestore_donestoring(tupstore);
return (Datum) 0;
}
+
+/*
+ * dump_memory_contexts
+ * Dump local memory contexts to a file.
+ */
+static void
+dump_memory_contexts(void)
+{
+ FILE *fpout;
+ char dumpfile[MAXPGPATH];
+ int format_id;
+ pid_t src_pid;
+ PGPROC *src_proc;
+ mcxtdumpEntry *entry;
+
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, MyProcPid);
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+ entry = (mcxtdumpEntry *) hash_search(mcxtdumpHash, &MyProcPid, HASH_FIND, NULL);
+
+ if (entry == NULL)
+ {
+ /*
+ * The dump request seems to have canceled already.
+ * Exit without dumping.
+ */
+ LWLockRelease(McxtDumpHashLock);
+ return;
+ }
+
+ entry->dump_status = MCXTDUMPSTATUS_DUMPING;
+ src_pid = entry->src_pid;
+
+ LWLockRelease(McxtDumpHashLock);
+
+ fpout = AllocateFile(dumpfile, "w");
+
+ if (fpout == NULL)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write memory context file \"%s\": %m",
+ dumpfile)));
+ FreeFile(fpout);
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ if (hash_search(mcxtdumpHash, &MyProcPid, HASH_REMOVE, NULL) == NULL)
+ elog(ERROR, "hash table corrupted");
+
+ LWLockRelease(McxtDumpHashLock);
+
+ return;
+ }
+
+ format_id = PG_MEMCONTEXT_FILE_FORMAT_ID;
+ fwrite(&format_id, sizeof(format_id), 1, fpout);
+
+ /* Look into each memory context from TopMemoryContext recursively. */
+ PutMemoryContextsStatsTupleStore(NULL, NULL,
+ TopMemoryContext, NULL, 0, fpout);
+
+ /*
+ * Make dump file ends with 'E'.
+ * This is checked by the requestor later.
+ */
+ fputc('E', fpout);
+
+ if (ferror(fpout))
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write dump file \"%s\": %m",
+ dumpfile)));
+ FreeFile(fpout);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ if (hash_search(mcxtdumpHash, &MyProcPid, HASH_REMOVE, NULL) == NULL)
+ elog(ERROR, "hash table corrupted");
+
+ LWLockRelease(McxtDumpHashLock);
+
+ return;
+ }
+
+ /* No more output to be done. Close file. */
+ else if (FreeFile(fpout) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not close dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+ entry = (mcxtdumpEntry *) hash_search(mcxtdumpHash, &MyProcPid, HASH_FIND, NULL);
+
+ /* During dumping, the requestor canceled the request. */
+ if (entry->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ unlink(dumpfile);
+
+ if (hash_search(mcxtdumpHash, &MyProcPid, HASH_REMOVE, NULL) == NULL)
+ elog(ERROR, "hash table corrupted");
+
+ LWLockRelease(McxtDumpHashLock);
+
+ return;
+ }
+
+ /* Dump has succeeded, notify it to the request. */
+ entry->dump_status = MCXTDUMPSTATUS_DONE;
+ LWLockRelease(McxtDumpHashLock);
+ src_proc = BackendPidGetProc(src_pid);
+ SetLatch(&(src_proc->procLatch));
+
+ return;
+}
+
+/*
+ * ProcessDumpMemoryInterrupt
+ * The portion of memory context dump interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessDumpMemoryInterrupt(void)
+{
+ ProcSignalDumpMemoryPending = false;
+ dump_memory_contexts();
+}
+
+/*
+ * HandleProcSignalDumpMemory
+ * Handle receipt of an interrupt indicating a memory context dump.
+ * Signal handler portion of interrupt handling.
+ */
+void
+HandleProcSignalDumpMemory(void)
+{
+ ProcSignalDumpMemoryPending = true;
+}
+
+/*
+ * McxtDumpShmemInit
+ * Initialize mcxtdump hash table.
+ */
+void
+McxtDumpShmemInit(void)
+{
+ HASHCTL info;
+
+ MemSet(&info, 0, sizeof(info));
+ info.keysize = sizeof(pid_t);
+ info.entrysize = sizeof(mcxtdumpEntry);
+
+ LWLockAcquire(McxtDumpHashLock, LW_EXCLUSIVE);
+
+ mcxtdumpHash = ShmemInitHash("mcxtdump hash",
+ SHMEM_MEMCONTEXT_SIZE,
+ SHMEM_MEMCONTEXT_SIZE,
+ &info,
+ HASH_ELEM | HASH_BLOBS);
+
+ LWLockRelease(McxtDumpHashLock);
+}
+
+/*
+ * RemoveMemcxtFile
+ * Remove dump files.
+ */
+void
+RemoveMemcxtFile(int pid)
+{
+ DIR *dir;
+ struct dirent *dumpfile;
+
+ if (pid == 0)
+ {
+ dir = AllocateDir(PG_MEMUSAGE_DIR);
+ while ((dumpfile = ReadDir(dir, PG_MEMUSAGE_DIR)) != NULL)
+ {
+ char dumpfilepath[32];
+
+ if (strcmp(dumpfile->d_name, ".") == 0 || strcmp(dumpfile->d_name, "..") == 0)
+ continue;
+
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, dumpfile->d_name);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ FreeDir(dir);
+ }
+ else
+ {
+ char str_pid[12];
+ char dumpfilepath[32];
+ struct stat stat_tmp;
+
+ pg_ltoa(pid, str_pid);
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, str_pid);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (stat(dumpfilepath, &stat_tmp) == 0)
+ {
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ }
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 6ab8216839..463337f661 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -33,6 +33,7 @@ volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t ProcSignalDumpMemoryPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index ee3bfa82f4..52cdb26272 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -221,7 +221,8 @@ static const char *const subdirs[] = {
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
- "pg_logical/mappings"
+ "pg_logical/mappings",
+ "pg_memusage"
};
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index f674a7c94e..340a80fc11 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -6,7 +6,7 @@ use File::Basename qw(basename dirname);
use File::Path qw(rmtree);
use PostgresNode;
use TestLib;
-use Test::More tests => 109;
+use Test::More tests => 110;
program_help_ok('pg_basebackup');
program_version_ok('pg_basebackup');
@@ -124,7 +124,7 @@ is_deeply(
# Contents of these directories should not be copied.
foreach my $dirname (
- qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans)
+ qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans pg_memusage)
)
{
is_deeply(
diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c
index 314b064b22..3b6ace6108 100644
--- a/src/bin/pg_rewind/filemap.c
+++ b/src/bin/pg_rewind/filemap.c
@@ -119,6 +119,9 @@ static const char *excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index c01da4bf01..374e21cfa7 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7847,12 +7847,11 @@
# memory context of local backend
{ oid => '2282',
descr => 'information about all memory contexts of local backend',
- proname => 'pg_get_backend_memory_contexts', prorows => '100',
- proretset => 't', provolatile => 'v', proparallel => 'r',
- prorettype => 'record', proargtypes => '',
- proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ proname => 'pg_get_backend_memory_contexts', prorows => '100', proisstrict => 'f',
+ proretset => 't', provolatile => 'v', proparallel => 'r', prorettype => 'record',
+ proargtypes => 'int4', proallargtypes => '{int4,text,text,text,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
# non-persistent series generator
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..812032bb15 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -83,6 +83,7 @@ extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t ProcSignalDumpMemoryPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 257e515bfe..27212830c9 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -958,7 +958,8 @@ typedef enum
WAIT_EVENT_REPLICATION_SLOT_DROP,
WAIT_EVENT_SAFE_SNAPSHOT,
WAIT_EVENT_SYNC_REP,
- WAIT_EVENT_XACT_GROUP_UPDATE
+ WAIT_EVENT_XACT_GROUP_UPDATE,
+ WAIT_EVENT_MEMORY_CONTEXT_DUMP
} WaitEventIPC;
/* ----------
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 5cb39697f3..d4a7ae0761 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_DUMP_MEMCXT, /* request dumping memory context interrupt */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/mcxtfuncs.h b/src/include/utils/mcxtfuncs.h
new file mode 100644
index 0000000000..93062ee179
--- /dev/null
+++ b/src/include/utils/mcxtfuncs.h
@@ -0,0 +1,52 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcxtfuncs.h
+ * Declarations for showing backend memory context.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mcxtfuncs.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MCXT_H
+#define MCXT_H
+
+/* Directory to store dumped memory files */
+#define PG_MEMUSAGE_DIR "pg_memusage"
+
+#define PG_MEMCONTEXT_FILE_FORMAT_ID 0x01B5BC9E
+
+/*
+ * Size of the shmem hash table size(not a hard limit).
+ *
+ * Although it may be better to increase this number in the future (e.g.,
+ * adding views for all the backend process of memory contexts), currently
+ * small number would be enough.
+ */
+#define SHMEM_MEMCONTEXT_SIZE 64
+
+typedef enum McxtDumpStatus
+{
+ MCXTDUMPSTATUS_REQUESTING,
+ MCXTDUMPSTATUS_DUMPING,
+ MCXTDUMPSTATUS_DONE,
+ MCXTDUMPSTATUS_CANCELING
+} McxtDumpStatus;
+
+typedef struct mcxtdumpEntry
+{
+ pid_t dst_pid; /* pid of the signal receiver */
+ pid_t src_pid; /* pid of the signal sender */
+ McxtDumpStatus dump_status; /* dump status */
+} mcxtdumpEntry;
+
+extern void ProcessDumpMemoryInterrupt(void);
+extern void HandleProcSignalDumpMemory(void);
+extern void McxtDumpShmemInit(void);
+extern void RemoveMcxtDumpFile(int);
+extern void RemoveMemcxtFile(int);
+#endif /* MCXT_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 097ff5d111..d3320e5b34 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1333,7 +1333,7 @@ pg_backend_memory_contexts| SELECT pg_get_backend_memory_contexts.name,
pg_get_backend_memory_contexts.free_bytes,
pg_get_backend_memory_contexts.free_chunks,
pg_get_backend_memory_contexts.used_bytes
- FROM pg_get_backend_memory_contexts() pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
+ FROM pg_get_backend_memory_contexts(NULL::integer) pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
pg_config| SELECT pg_config.name,
pg_config.setting
FROM pg_config() pg_config(name, setting);
--
2.18.1
On 2020/11/16 19:58, torikoshia wrote:
On 2020-10-28 15:32, torikoshia wrote:
On 2020-10-23 13:46, Kyotaro Horiguchi wrote:
I think we might need to step-back to basic design of this feature
since this patch seems to have unhandled corner cases that are
difficult to find.I've written out the basic design below and attached
corresponding patch.
I'm starting to study how this feature behaves. At first, when I executed
the following query, the function never returned. ISTM that since
the autovacuum launcher cannot respond to the request of memory
contexts dump, the function keeps waiting infinity. Is this a bug?
Probably we should exclude non-backend proceses from the target
processes to dump? Sorry if this was already discussed.
SELECT pg_get_backend_memory_contexts(pid) FROM pg_stat_activity;
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
Fujii Masao <masao.fujii@oss.nttdata.com> writes:
I'm starting to study how this feature behaves. At first, when I executed
the following query, the function never returned. ISTM that since
the autovacuum launcher cannot respond to the request of memory
contexts dump, the function keeps waiting infinity. Is this a bug?
Probably we should exclude non-backend proceses from the target
processes to dump? Sorry if this was already discussed.
SELECT pg_get_backend_memory_contexts(pid) FROM pg_stat_activity;
FWIW, I think this patch is fundamentally unsafe. It's got a
lot of the same problems that I complained about w.r.t. the
nearby proposal to allow cross-backend stack trace dumping.
It does avoid the trap of thinking that it can do work in
a signal handler, but instead it supposes that it can do
work involving very high-level objects such as shared hash tables
in anyplace that might execute CHECK_FOR_INTERRUPTS. That's
never going to be safe: the only real expectation the system
has is that CHECK_FOR_INTERRUPTS is called at places where our
state is sane enough that a transaction abort can clean up.
Trying to do things like taking LWLocks is going to lead to
deadlocks or worse. We need not even get into the hard questions,
such as what happens when one process or the other exits
unexpectedly.
I also find the idea that this should be the same SQL function
as pg_get_backend_memory_contexts to be a seriously bad decision.
That means that it's not possible to GRANT the right to examine
only your own process's memory --- with this proposal, that means
granting the right to inspect every other process as well.
Beyond that, the fact that there's no way to restrict the capability
to just, say, other processes owned by the same user means that
it's not really safe to GRANT to non-superusers anyway. Even with
such a restriction added, things are problematic, since for example
it would be possible to inquire into the workings of a
security-definer function executing in another process that
nominally is owned by your user.
Between the security and implementation issues here, I really
think we'd be best advised to just reject the concept, period.
regards, tom lane
On 2020-12-03 10:36, Tom Lane wrote:
Fujii Masao <masao.fujii@oss.nttdata.com> writes:
I'm starting to study how this feature behaves. At first, when I
executed
the following query, the function never returned. ISTM that since
the autovacuum launcher cannot respond to the request of memory
contexts dump, the function keeps waiting infinity. Is this a bug?
Probably we should exclude non-backend proceses from the target
processes to dump? Sorry if this was already discussed.SELECT pg_get_backend_memory_contexts(pid) FROM pg_stat_activity;
Thanks for trying it!
It was not discussed explicitly, and I was going to do it later
as commented.
+ /* TODO: Check also whether backend or not. */
FWIW, I think this patch is fundamentally unsafe. It's got a
lot of the same problems that I complained about w.r.t. the
nearby proposal to allow cross-backend stack trace dumping.
It does avoid the trap of thinking that it can do work in
a signal handler, but instead it supposes that it can do
work involving very high-level objects such as shared hash tables
in anyplace that might execute CHECK_FOR_INTERRUPTS. That's
never going to be safe: the only real expectation the system
has is that CHECK_FOR_INTERRUPTS is called at places where our
state is sane enough that a transaction abort can clean up.
Trying to do things like taking LWLocks is going to lead to
deadlocks or worse. We need not even get into the hard questions,
such as what happens when one process or the other exits
unexpectedly.
Thanks for reviewing!
I may misunderstand something, but the dumper works not at
CHECK_FOR_INTERRUPTS but during the client read, i.e.,
ProcessClientReadInterrupt().
Is it also unsafe?
BTW, since there was a comment that the shared hash table
used too much memory, I'm now rewriting this patch not to use
the shared hash table but a simpler static shared memory struct.
I also find the idea that this should be the same SQL function
as pg_get_backend_memory_contexts to be a seriously bad decision.
That means that it's not possible to GRANT the right to examine
only your own process's memory --- with this proposal, that means
granting the right to inspect every other process as well.Beyond that, the fact that there's no way to restrict the capability
to just, say, other processes owned by the same user means that
it's not really safe to GRANT to non-superusers anyway. Even with
such a restriction added, things are problematic, since for example
it would be possible to inquire into the workings of a
security-definer function executing in another process that
nominally is owned by your user.
I'm going to change the function name and restrict the executor to
superusers. Is it enough?
Regards,
On 2020-12-04 19:16, torikoshia wrote:
On 2020-12-03 10:36, Tom Lane wrote:
Fujii Masao <masao.fujii@oss.nttdata.com> writes:
I'm starting to study how this feature behaves. At first, when I
executed
the following query, the function never returned. ISTM that since
the autovacuum launcher cannot respond to the request of memory
contexts dump, the function keeps waiting infinity. Is this a bug?
Probably we should exclude non-backend proceses from the target
processes to dump? Sorry if this was already discussed.SELECT pg_get_backend_memory_contexts(pid) FROM
pg_stat_activity;Thanks for trying it!
It was not discussed explicitly, and I was going to do it later
as commented.+ /* TODO: Check also whether backend or not. */
FWIW, I think this patch is fundamentally unsafe. It's got a
lot of the same problems that I complained about w.r.t. the
nearby proposal to allow cross-backend stack trace dumping.
It does avoid the trap of thinking that it can do work in
a signal handler, but instead it supposes that it can do
work involving very high-level objects such as shared hash tables
in anyplace that might execute CHECK_FOR_INTERRUPTS. That's
never going to be safe: the only real expectation the system
has is that CHECK_FOR_INTERRUPTS is called at places where our
state is sane enough that a transaction abort can clean up.
Trying to do things like taking LWLocks is going to lead to
deadlocks or worse. We need not even get into the hard questions,
such as what happens when one process or the other exits
unexpectedly.Thanks for reviewing!
I may misunderstand something, but the dumper works not at
CHECK_FOR_INTERRUPTS but during the client read, i.e.,
ProcessClientReadInterrupt().Is it also unsafe?
BTW, since there was a comment that the shared hash table
used too much memory, I'm now rewriting this patch not to use
the shared hash table but a simpler static shared memory struct.
Attached a rewritten patch.
Accordingly, I also slightly modified the basic design as below.
---
# Communication flow between the dumper and the requester
- (1) When requesting memory context dumping, the dumper changes
the struct on the shared memory state from 'ACCEPTABLE' to
'REQUESTING'.
- (2) The dumper sends the signal to the dumper process and wait on
the latch.
- (3) When the dumper noticed the signal, it changes the state to
'DUMPING'.
- (4) When the dumper completes dumping, it changes the state to
'DONE' and set the latch.
- (5) The requestor reads the dump file and shows it to the user.
Finally, the requestor removes the dump file and reset the shared
memory state to 'ACCEPTABLE'.
# Query cancellation
- When the requestor cancels dumping, e.g. signaling using ctrl-C,
the requestor changes the state of the shared memory to 'CANCELING'.
- The dumper checks the state when it tries to change the state to
'DONE' at (4), and if the state is 'CANCELING', it initializes the
dump file and reset the shared memory state to 'ACCEPTABLE'.
# Cleanup dump file and the shared memory
- In the normal case, the dumper removes the dump file and resets
the shared memory entry as described in (5).
- When something like query cancellation or process termination
happens on the dumper after (1) and before (3), in other words,
the state is 'REQUESTING', the requestor does the cleanup.
- When something happens on the dumper or the requestor after (3)
and before (4), in other words, the state is 'DUMPING', the dumper
does the cleanup. Specifically, if the requestor cancels the query,
it just changes the state to 'CANCELING' and the dumper notices it
and cleans up things later.
OTOH, when the dumper fails to dump, it cleans up the dump file and
reset the shared memory state.
- When something happens on the requestor after (4), i.e., the state
is 'DONE', the requestor does the cleanup.
- In the case of receiving SIGKILL or power failure, all dump files
are removed in the crash recovery process.
---
I also find the idea that this should be the same SQL function
as pg_get_backend_memory_contexts to be a seriously bad decision.
That means that it's not possible to GRANT the right to examine
only your own process's memory --- with this proposal, that means
granting the right to inspect every other process as well.Beyond that, the fact that there's no way to restrict the capability
to just, say, other processes owned by the same user means that
it's not really safe to GRANT to non-superusers anyway. Even with
such a restriction added, things are problematic, since for example
it would be possible to inquire into the workings of a
security-definer function executing in another process that
nominally is owned by your user.I'm going to change the function name and restrict the executor to
superusers. Is it enough?
In the attached patch, I changed the function name to
pg_get_target_backend_memory_contexts() for now.
Regards,
Attachments:
v5-0001-Add-pg_get_target_backend_memory_contexts-to-coll.patchtext/x-diff; name=v5-0001-Add-pg_get_target_backend_memory_contexts-to-coll.patchDownload
From 120b2c33e2b295305dedc63921183c103e65ddc6 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Thu, 9 Dec 2020 09:52:15 +0900
Subject: [PATCH v5] Add pg_get_target_backend_memory_contexts(() to collect
arbitrary backend process's memory contexts.
After commit 3e98c0bafb28de, we can display the usage of the
memory contexts using pg_backend_memory_contexts system
view. However, its target is limited to the process attached
to the current session.
This patch introduces pg_get_target_backend_memory_contexts()
and makes it possible to collect memory contexts of the
specified process.
---
src/backend/access/transam/xlog.c | 7 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/postmaster/pgstat.c | 3 +
src/backend/replication/basebackup.c | 3 +
src/backend/storage/ipc/ipci.c | 2 +
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/tcop/postgres.c | 5 +
src/backend/utils/adt/mcxtfuncs.c | 691 ++++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/bin/initdb/initdb.c | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 4 +-
src/bin/pg_rewind/filemap.c | 3 +
src/include/catalog/pg_proc.dat | 12 +-
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 3 +-
src/include/storage/procsignal.h | 1 +
src/include/utils/mcxtfuncs.h | 44 ++
18 files changed, 770 insertions(+), 21 deletions(-)
create mode 100644 src/include/utils/mcxtfuncs.h
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 7e81ce4f17..2474a36ac1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -73,6 +73,7 @@
#include "storage/sync.h"
#include "utils/builtins.h"
#include "utils/guc.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/relmapper.h"
@@ -6982,6 +6983,12 @@ StartupXLOG(void)
*/
pgstat_reset_all();
+ /*
+ * Reset dump files in pg_memusage, because target processes do
+ * not exist any more.
+ */
+ RemoveMemcxtFile(0);
+
/*
* If there was a backup label file, it's done its job and the info
* has now been propagated into pg_control. We must get rid of the
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b140c210bc..77e97f4805 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -558,7 +558,8 @@ CREATE VIEW pg_backend_memory_contexts AS
SELECT * FROM pg_get_backend_memory_contexts();
REVOKE ALL ON pg_backend_memory_contexts FROM PUBLIC;
-REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_target_backend_memory_contexts FROM PUBLIC;
-- Statistics views
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 7c75a25d21..6bd471af2f 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4042,6 +4042,9 @@ pgstat_get_wait_ipc(WaitEventIPC w)
case WAIT_EVENT_XACT_GROUP_UPDATE:
event_name = "XactGroupUpdate";
break;
+ case WAIT_EVENT_DUMP_MEMORY_CONTEXT:
+ event_name = "DumpMemoryContext";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index 1d8d1742a7..c65a1b4f66 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -184,6 +184,9 @@ static const char *const excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dump files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 96c2aaabbd..92f21ad2bf 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -45,6 +45,7 @@
#include "storage/procsignal.h"
#include "storage/sinvaladt.h"
#include "storage/spin.h"
+#include "utils/mcxtfuncs.h"
#include "utils/snapmgr.h"
/* GUCs */
@@ -267,6 +268,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ McxtDumpShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index ffe67acea1..2b1ddd3a0d 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -28,6 +28,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/mcxtfuncs.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -567,6 +568,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_DUMP_MEMCXT))
+ HandleProcSignalDumpMemoryContext();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 774292fd94..6b4ff6f08b 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+McxtDumpLock 48
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3679799e50..9d7f2458c0 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -75,6 +75,7 @@
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
#include "utils/lsyscache.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -539,6 +540,10 @@ ProcessClientReadInterrupt(bool blocked)
/* Process notify interrupts, if any */
if (notifyInterruptPending)
ProcessNotifyInterrupt();
+
+ /* Process memory contexts dump interrupts, if any */
+ if (ProcSignalDumpMemoryContextPending)
+ ProcessDumpMemoryContextInterrupt();
}
else if (ProcDiePending)
{
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 50e1b07ff0..d2ffc88c37 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,30 +15,101 @@
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "common/logging.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "pgstat.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "storage/shmem.h"
#include "utils/builtins.h"
+#include "utils/mcxtfuncs.h"
+
+/* The max bytes for showing names and identifiers of MemoryContext. */
+#define MEMORY_CONTEXT_DISPLAY_SIZE 1024
+
+/* Number of columns in pg_backend_memory_contexts view */
+#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
-/* ----------
- * The max bytes for showing identifiers of MemoryContext.
- * ----------
+/* Shared memory struct for managing the status of memory context dump. */
+static mcxtdumpShmemStruct *mcxtdumpShmem = NULL;
+
+
+/*
+ * McxtReqKill
+ * Error cleanup callback for memory context requestor.
*/
-#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+static void
+McxtReqKill(int code, Datum arg)
+{
+ int dst_pid = DatumGetInt32(arg);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->src_pid != MyProcPid)
+ {
+ /*
+ * If the requestor is not me, simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+ return;
+ }
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING)
+ {
+ /*
+ * Since the dumper has not received the dump order yet, clean up
+ * things by myself.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DUMPING)
+ {
+ /*
+ * Since the dumper has received the request already,
+ * requestor just change the status and the dumper cleans up
+ * things later.
+ */
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_CANCELING;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DONE)
+ {
+ /*
+ * Since the dumper has already finished dumping, clean up things
+ * by myself.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ RemoveMemcxtFile(dst_pid);
+ }
+ LWLockRelease(McxtDumpLock);
+}
/*
* PutMemoryContextsStatsTupleStore
* One recursion level for pg_get_backend_memory_contexts.
+ *
+ * Note: When fpout is not NULL, ferror() check must be done by the caller.
*/
static void
PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
TupleDesc tupdesc, MemoryContext context,
- const char *parent, int level)
+ const char *parent, int level, FILE *fpout)
{
-#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
-
Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
MemoryContextCounters stat;
MemoryContext child;
const char *name;
@@ -74,14 +145,12 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
if (ident)
{
int idlen = strlen(ident);
- char clipped_ident[MEMORY_CONTEXT_IDENT_DISPLAY_SIZE];
-
/*
* Some identifiers such as SQL query string can be very long,
* truncate oversize identifiers.
*/
- if (idlen >= MEMORY_CONTEXT_IDENT_DISPLAY_SIZE)
- idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_IDENT_DISPLAY_SIZE - 1);
+ if (idlen >= MEMORY_CONTEXT_DISPLAY_SIZE)
+ idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_DISPLAY_SIZE - 1);
memcpy(clipped_ident, ident, idlen);
clipped_ident[idlen] = '\0';
@@ -101,18 +170,198 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
values[6] = Int64GetDatum(stat.freespace);
values[7] = Int64GetDatum(stat.freechunks);
values[8] = Int64GetDatum(stat.totalspace - stat.freespace);
- tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Since pg_get_backend_memory_contexts() is called from local process,
+ * simply put tuples.
+ */
+ if(fpout == NULL)
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Write out the current memory context information in the form of
+ * "key: value" pairs to the file specified by the requestor.
+ */
+ else
+ {
+ /*
+ * Make each memory context information starts with 'D'.
+ * This is checked by the requestor when reading the file.
+ */
+ fputc('D', fpout);
+
+ fprintf(fpout,
+ "name: %s, ident: %s, parent: %s, level: %d, total_bytes: %lu, \
+ total_nblocks: %lu, free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name,
+ ident ? clipped_ident : "none",
+ parent ? parent : "none", level,
+ stat.totalspace,
+ stat.nblocks,
+ stat.freespace,
+ stat.freechunks,
+ stat.totalspace - stat.freespace);
+ }
for (child = context->firstchild; child != NULL; child = child->nextchild)
{
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- child, name, level + 1);
+ child, name, level + 1, fpout);
}
}
+/*
+ * SetupMcxtdumpShem
+ * Setup shared memory struct for dumping specified PID.
+ */
+static void
+SetupMcxtdumpShmem(int pid)
+{
+ /*
+ * We only allow one session per target process to request memory
+ * contexts dump at a time.
+ * If someone uses mcxtdumpShmem, wait until it was changed to acceptable.
+ */
+ while (true)
+ {
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_REQUESTING;
+ mcxtdumpShmem->src_pid = MyProcPid;
+ mcxtdumpShmem->dst_pid = pid;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+ else
+ {
+ ereport(INFO,
+ (errmsg("PID %d is looked up by another process", pid)));
+
+ LWLockRelease(McxtDumpLock);
+
+ pg_usleep(5000000L);
+ }
+ }
+}
+
+/*
+ * PutDumpedValuesOnTuplestore
+ * Read specified memory context dump file and put its values
+ * on the tuplestore.
+ */
+static void
+PutDumpedValuesOnTuplestore(char *dumpfile, Tuplestorestate *tupstore,
+ TupleDesc tupdesc, int pid)
+{
+ FILE *fpin;
+ int format_id;
+
+ if ((fpin = AllocateFile(dumpfile, "r")) == NULL)
+ {
+ if (errno != ENOENT)
+ ereport(LOG, (errcode_for_file_access(),
+ errmsg("could not open memory context dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ /* Verify it's of the expected format. */
+ if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ format_id != PG_MCXT_FILE_FORMAT_ID)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"", dumpfile)));
+ goto done;
+ }
+
+ /* Read dump file and put values on tuple store. */
+ while (true)
+ {
+ Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char name[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char parent[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
+ int level;
+ Size total_bytes;
+ Size total_nblocks;
+ Size free_bytes;
+ Size free_chunks;
+ Size used_bytes;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ switch (fgetc(fpin))
+ {
+ /* 'D' A memory context information follows */
+ case 'D':
+ if (fscanf(fpin, "name: %1023[^,], ident: %1023[^,], parent: %1023[^,], \
+ level: %d, total_bytes: %lu, total_nblocks: %lu, \
+ free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name, clipped_ident, parent, &level, &total_bytes, &total_nblocks,
+ &free_bytes, &free_chunks, &used_bytes)
+ != PG_GET_BACKEND_MEMORY_CONTEXTS_COLS)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+
+ values[0] = CStringGetTextDatum(name);
+
+ if (strcmp(clipped_ident, "none"))
+ values[1] = CStringGetTextDatum(clipped_ident);
+ else
+ nulls[1] = true;
+
+ if (strcmp(parent, "none"))
+ values[2] = CStringGetTextDatum(parent);
+ else
+ nulls[2] = true;
+
+ values[3] = Int32GetDatum(level);
+ values[4] = Int64GetDatum(total_bytes);
+ values[5] = Int64GetDatum(total_nblocks);
+ values[6] = Int64GetDatum(free_bytes);
+ values[7] = Int64GetDatum(free_chunks);
+ values[8] = Int64GetDatum(used_bytes);
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ break;
+
+ case 'E':
+ goto done;
+
+ default:
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ }
+done:
+ FreeFile(fpin);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ Assert(mcxtdumpShmem->src_pid == MyProcPid);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+}
+
/*
* pg_get_backend_memory_contexts
- * SQL SRF showing backend memory context.
+ * SQL SRF showing local backend memory context.
*/
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
@@ -148,10 +397,422 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldcontext);
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- TopMemoryContext, NULL, 0);
+ TopMemoryContext, "", 0, NULL);
+
+ /* clean up and return the tuplestore */
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
+
+/*
+ * pg_get_target_backend_memory_contexts
+ * SQL SRF showing specified process's backend memory context.
+ */
+Datum
+pg_get_target_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int dst_pid = PG_GETARG_INT32(0);
+
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ char dumpfile[MAXPGPATH];
+ PGPROC *proc;
+ PgBackendStatus *beentry;
+
+ if (dst_pid == MyProcPid)
+ {
+ pg_get_backend_memory_contexts(fcinfo);
+ return (Datum) 0;
+ }
+
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, dst_pid);
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ /*
+ * Check whether the target process is PostgreSQL backend process.
+ * By the time we reach SendProcSignal(), a process for which we get
+ * a valid proc here might have terminated or got stuck.
+ * In these cases, users are expected to cancel the dump request.
+ */
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", dst_pid)));
+
+ return (Datum) 1;
+ }
+
+ beentry = pgstat_fetch_stat_beentry(proc->backendId);
+
+ if (beentry->st_backendType != B_BACKEND)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL backend process", dst_pid)));
+
+ return (Datum) 1;
+ }
+
+ /*
+ * The ENSURE stuff ensures we clean up the shared memory struct and files
+ * on failure.
+ */
+ PG_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid));
+ {
+ SetupMcxtdumpShmem(dst_pid);
+
+ SendProcSignal(dst_pid, PROCSIG_DUMP_MEMCXT, InvalidBackendId);
+
+ /* Wait until target process finishes dumping file. */
+ for (;;)
+ {
+ /* Check for dump cancel request. */
+ CHECK_FOR_INTERRUPTS();
+
+ /* Must reset the latch before testing state. */
+ ResetLatch(MyLatch);
+
+ LWLockAcquire(McxtDumpLock, LW_SHARED);
+
+ if (mcxtdumpShmem->src_pid != MyProcPid)
+ {
+ /*
+ * It seems the dumper fails and subsequently another
+ * process is requesting dumping.
+ */
+ ereport(INFO,
+ (errmsg("The request has failed and now PID %d is requsting dumping.",
+ mcxtdumpShmem->src_pid)));
+
+ LWLockRelease(McxtDumpLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ /*
+ * Dumper seems to clean up the enry because of failures or
+ * cancellation.
+ * Since the dumper has already removed the dump file,
+ * simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /*
+ * Request has been canceled. Exit without dumping.
+ */
+ LWLockRelease(McxtDumpLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DONE)
+ {
+ /* Dumping has completed. */
+ LWLockRelease(McxtDumpLock);
+ break;
+ }
+ /*
+ * The dumper must be in the middle of a dumping or the request
+ * hasn't been reached yet.
+ */
+ Assert(mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING ||
+ mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DUMPING);
+
+ LWLockRelease(McxtDumpLock);
+
+ /*
+ * Wait. We expect to get a latch signal back from the dumper.
+ * Use a timeout to enable cancellation.
+ */
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ 1000L, WAIT_EVENT_DUMP_MEMORY_CONTEXT);
+ }
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(McxtReqKill, (Datum) Int32GetDatum(dst_pid));
+
+ /* Read values from the dump file and put them on tuplestore. */
+ PutDumpedValuesOnTuplestore(dumpfile, tupstore, tupdesc, dst_pid);
/* clean up and return the tuplestore */
tuplestore_donestoring(tupstore);
return (Datum) 0;
}
+
+/*
+ * dump_memory_contexts
+ * Dump local memory contexts to a file.
+ */
+static void
+dump_memory_contexts(void)
+{
+ FILE *fpout;
+ char dumpfile[MAXPGPATH];
+ int format_id;
+ pid_t src_pid;
+ PGPROC *src_proc;
+
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, MyProcPid);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ /*
+ * The requestor canceled the request and initialized
+ * the shared memory. Simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /*
+ * The requestor canceled the request.
+ * Initialize the shared memory and exit.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ Assert(mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_REQUESTING);
+
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_DUMPING;
+ src_pid = mcxtdumpShmem->src_pid;
+
+ LWLockRelease(McxtDumpLock);
+
+ fpout = AllocateFile(dumpfile, "w");
+
+ if (fpout == NULL)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write memory context file \"%s\": %m",
+ dumpfile)));
+ FreeFile(fpout);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+ format_id = PG_MCXT_FILE_FORMAT_ID;
+ fwrite(&format_id, sizeof(format_id), 1, fpout);
+
+ /* Look into each memory context from TopMemoryContext recursively. */
+ PutMemoryContextsStatsTupleStore(NULL, NULL,
+ TopMemoryContext, NULL, 0, fpout);
+
+ /*
+ * Make dump file ends with 'E'.
+ * This is checked by the requestor later.
+ */
+ fputc('E', fpout);
+
+ if (ferror(fpout))
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write dump file \"%s\": %m",
+ dumpfile)));
+ FreeFile(fpout);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ /* No more output to be done. Close file. */
+ else if (FreeFile(fpout) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not close dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /* During dumping, the requestor canceled the request. */
+ unlink(dumpfile);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ /* Dumping has succeeded, notify it to the requestor. */
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_DONE;
+ LWLockRelease(McxtDumpLock);
+ src_proc = BackendPidGetProc(src_pid);
+ SetLatch(&(src_proc->procLatch));
+
+ return;
+}
+
+/*
+ * ProcessDumpMemoryContextInterrupt
+ * The portion of memory context dump interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessDumpMemoryContextInterrupt(void)
+{
+ ProcSignalDumpMemoryContextPending = false;
+ dump_memory_contexts();
+}
+
+/*
+ * HandleProcSignalDumpMemoryContext
+ * Handle receipt of an interrupt indicating a memory context dump.
+ * Signal handler portion of interrupt handling.
+ */
+void
+HandleProcSignalDumpMemoryContext(void)
+{
+ ProcSignalDumpMemoryContextPending = true;
+}
+
+/*
+ * McxtDumpShmemInit
+ * Initialize mcxtdump shared memory struct.
+ */
+void
+McxtDumpShmemInit(void)
+{
+ bool found;
+
+ mcxtdumpShmem = (mcxtdumpShmemStruct *)
+ ShmemInitStruct("Memory Context Dump Data",
+ sizeof(mcxtdumpShmemStruct),
+ &found);
+ if (!found)
+ {
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+ }
+}
+
+/*
+ * RemoveMemcxtFile
+ * Remove dump files.
+ */
+void
+RemoveMemcxtFile(int pid)
+{
+ DIR *dir;
+ struct dirent *dumpfile;
+
+ if (pid == 0)
+ {
+ /* delete all dump files */
+ dir = AllocateDir(PG_MEMUSAGE_DIR);
+ while ((dumpfile = ReadDir(dir, PG_MEMUSAGE_DIR)) != NULL)
+ {
+ char dumpfilepath[32];
+
+ if (strcmp(dumpfile->d_name, ".") == 0 || strcmp(dumpfile->d_name, "..") == 0)
+ continue;
+
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, dumpfile->d_name);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ FreeDir(dir);
+ }
+ else
+ {
+ /* delete specified dump file */
+ char str_pid[12];
+ char dumpfilepath[32];
+ struct stat stat_tmp;
+
+ pg_ltoa(pid, str_pid);
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, str_pid);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (stat(dumpfilepath, &stat_tmp) == 0)
+ {
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ }
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 6ab8216839..288eb8bc0c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -33,6 +33,7 @@ volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t ProcSignalDumpMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index ee3bfa82f4..52cdb26272 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -221,7 +221,8 @@ static const char *const subdirs[] = {
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
- "pg_logical/mappings"
+ "pg_logical/mappings",
+ "pg_memusage"
};
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index f674a7c94e..340a80fc11 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -6,7 +6,7 @@ use File::Basename qw(basename dirname);
use File::Path qw(rmtree);
use PostgresNode;
use TestLib;
-use Test::More tests => 109;
+use Test::More tests => 110;
program_help_ok('pg_basebackup');
program_version_ok('pg_basebackup');
@@ -124,7 +124,7 @@ is_deeply(
# Contents of these directories should not be copied.
foreach my $dirname (
- qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans)
+ qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans pg_memusage)
)
{
is_deeply(
diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c
index ba34dbac14..c4a79528ac 100644
--- a/src/bin/pg_rewind/filemap.c
+++ b/src/bin/pg_rewind/filemap.c
@@ -119,6 +119,9 @@ static const char *excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fc2202b843..70604faefc 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7844,15 +7844,25 @@
# memory context of local backend
{ oid => '2282',
- descr => 'information about all memory contexts of local backend',
proname => 'pg_get_backend_memory_contexts', prorows => '100',
proretset => 't', provolatile => 'v', proparallel => 'r',
prorettype => 'record', proargtypes => '',
proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',
proargmodes => '{o,o,o,o,o,o,o,o,o}',
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ descr => 'information about all memory contexts of local backend',
prosrc => 'pg_get_backend_memory_contexts' },
+# memory context of specified backend
+{ oid => '2860',
+ descr => 'information about all memory contexts of specified backend',
+ proname => 'pg_get_target_backend_memory_contexts', prorows => '100', proisstrict => 'f',
+ proretset => 't', provolatile => 'v', proparallel => 'r', prorettype => 'record',
+ proargtypes => 'int4', proallargtypes => '{int4,text,text,text,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ prosrc => 'pg_get_target_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 72e3352398..1ff03b86a0 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -83,6 +83,7 @@ extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t ProcSignalDumpMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 5954068dec..262adbb782 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -964,7 +964,8 @@ typedef enum
WAIT_EVENT_REPLICATION_SLOT_DROP,
WAIT_EVENT_SAFE_SNAPSHOT,
WAIT_EVENT_SYNC_REP,
- WAIT_EVENT_XACT_GROUP_UPDATE
+ WAIT_EVENT_XACT_GROUP_UPDATE,
+ WAIT_EVENT_DUMP_MEMORY_CONTEXT
} WaitEventIPC;
/* ----------
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 5cb39697f3..d4a7ae0761 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_DUMP_MEMCXT, /* request dumping memory context interrupt */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/mcxtfuncs.h b/src/include/utils/mcxtfuncs.h
new file mode 100644
index 0000000000..eff2c62cbb
--- /dev/null
+++ b/src/include/utils/mcxtfuncs.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcxtfuncs.h
+ * Declarations for showing backend memory context.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mcxtfuncs.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MCXT_H
+#define MCXT_H
+
+/* Directory to store dumped memory files */
+#define PG_MEMUSAGE_DIR "pg_memusage"
+
+#define PG_MCXT_FILE_FORMAT_ID 0x01B5BC9E
+
+typedef enum McxtDumpStatus
+{
+ MCXTDUMPSTATUS_ACCEPTABLE, /* no one is requesting dumping */
+ MCXTDUMPSTATUS_REQUESTING, /* request has been issued, but dumper has not received it yet */
+ MCXTDUMPSTATUS_DUMPING, /* dumper is dumping files */
+ MCXTDUMPSTATUS_DONE, /* dumper has finished dumping and the requestor is working */
+ MCXTDUMPSTATUS_CANCELING /* requestor canceled the dumping */
+} McxtDumpStatus;
+
+typedef struct mcxtdumpShmemStruct
+{
+ pid_t dst_pid; /* pid of the signal receiver */
+ pid_t src_pid; /* pid of the signal sender */
+ McxtDumpStatus dump_status; /* dump status */
+} mcxtdumpShmemStruct;
+
+extern void ProcessDumpMemoryContextInterrupt(void);
+extern void HandleProcSignalDumpMemoryContext(void);
+extern void McxtDumpShmemInit(void);
+extern void RemoveMcxtDumpFile(int);
+extern void RemoveMemcxtFile(int);
+#endif /* MCXT_H */
--
2.18.1
Hi,
On Thu, Dec 10, 2020 at 10:48 AM torikoshia <torikoshia@oss.nttdata.com> wrote:
On 2020-12-04 19:16, torikoshia wrote:
On 2020-12-03 10:36, Tom Lane wrote:
Fujii Masao <masao.fujii@oss.nttdata.com> writes:
I'm starting to study how this feature behaves. At first, when I
executed
the following query, the function never returned. ISTM that since
the autovacuum launcher cannot respond to the request of memory
contexts dump, the function keeps waiting infinity. Is this a bug?
Probably we should exclude non-backend proceses from the target
processes to dump? Sorry if this was already discussed.SELECT pg_get_backend_memory_contexts(pid) FROM
pg_stat_activity;Thanks for trying it!
It was not discussed explicitly, and I was going to do it later
as commented.+ /* TODO: Check also whether backend or not. */
FWIW, I think this patch is fundamentally unsafe. It's got a
lot of the same problems that I complained about w.r.t. the
nearby proposal to allow cross-backend stack trace dumping.
It does avoid the trap of thinking that it can do work in
a signal handler, but instead it supposes that it can do
work involving very high-level objects such as shared hash tables
in anyplace that might execute CHECK_FOR_INTERRUPTS. That's
never going to be safe: the only real expectation the system
has is that CHECK_FOR_INTERRUPTS is called at places where our
state is sane enough that a transaction abort can clean up.
Trying to do things like taking LWLocks is going to lead to
deadlocks or worse. We need not even get into the hard questions,
such as what happens when one process or the other exits
unexpectedly.Thanks for reviewing!
I may misunderstand something, but the dumper works not at
CHECK_FOR_INTERRUPTS but during the client read, i.e.,
ProcessClientReadInterrupt().Is it also unsafe?
BTW, since there was a comment that the shared hash table
used too much memory, I'm now rewriting this patch not to use
the shared hash table but a simpler static shared memory struct.Attached a rewritten patch.
Thanks for updating patch.
But when I had applyed the patch to the current HEAD and did make, I
got an error due to duplicate OIDs.
You need to rebase the patch.
Accordingly, I also slightly modified the basic design as below.
---
# Communication flow between the dumper and the requester
- (1) When requesting memory context dumping, the dumper changes
the struct on the shared memory state from 'ACCEPTABLE' to
'REQUESTING'.
- (2) The dumper sends the signal to the dumper process and wait on
the latch.
- (3) When the dumper noticed the signal, it changes the state to
'DUMPING'.
- (4) When the dumper completes dumping, it changes the state to
'DONE' and set the latch.
- (5) The requestor reads the dump file and shows it to the user.
Finally, the requestor removes the dump file and reset the shared
memory state to 'ACCEPTABLE'.# Query cancellation
- When the requestor cancels dumping, e.g. signaling using ctrl-C,
the requestor changes the state of the shared memory to 'CANCELING'.
- The dumper checks the state when it tries to change the state to
'DONE' at (4), and if the state is 'CANCELING', it initializes the
dump file and reset the shared memory state to 'ACCEPTABLE'.# Cleanup dump file and the shared memory
- In the normal case, the dumper removes the dump file and resets
the shared memory entry as described in (5).
- When something like query cancellation or process termination
happens on the dumper after (1) and before (3), in other words,
the state is 'REQUESTING', the requestor does the cleanup.
- When something happens on the dumper or the requestor after (3)
and before (4), in other words, the state is 'DUMPING', the dumper
does the cleanup. Specifically, if the requestor cancels the query,
it just changes the state to 'CANCELING' and the dumper notices it
and cleans up things later.
OTOH, when the dumper fails to dump, it cleans up the dump file and
reset the shared memory state.
- When something happens on the requestor after (4), i.e., the state
is 'DONE', the requestor does the cleanup.
- In the case of receiving SIGKILL or power failure, all dump files
are removed in the crash recovery process.
---
If the dumper is terminated before it dumps, the requestor will appear
to enter an
infinite loop because the status of mcxtdumpShmem will not change.
The following are the steps to reproduce.
- session1
BEGIN; LOCK TABLE t;
- session2
SELECT * FROM t; -- wait
- session3
select pg_get_target_backend_memory_contexts(<pid of session2>); -- wait
- session1
select pg_terminate_backend(<pid of session2>); -- kill session2
- session3 waits forever.
Therefore, you may need to set mcxtdumpShmem->dump_status to
MCXTDUMPSTATUS_CANCELING
or other status before the dumper terminates.
Also, although I have not been able to reproduce it, I believe that
with the current design,
if the requestor disappears right after the dumper dumps the memory information,
the dump file will remain.
Since the current design appears to allow only one requestor per
instance, when the requestor
requests a dump, it might be a good idea to delete any remaining dump
files, if any.
The following are comments on the code.
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", dst_pid)));
+
+ return (Datum) 1;
+ }
For now, background writer, checkpointer and WAL writer are belong to
the auxiliary process.
Therefore, if we specify the PIDs of these processes for
pg_get_target_backend_memory_contexts(),
"PID xxxx is not a PostgreSQL server process" whould be outoput.
This confuses the user.
How about use AuxiliaryPidGetProc() to determine these processes?
+ ereport(INFO,
+ (errmsg("The request has failed and now PID %d is
requsting dumping.",
+ mcxtdumpShmem->src_pid)));
+
+ LWLockRelease(McxtDumpLock);
You can release LWLock before ereport.
+ Assert(mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_REQUESTING);
typo?
It might be "mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING".
+ ereport(INFO,
+ (errmsg("PID %d is looked up by another process", pid)));
This message is not accurate.
Because, in the current design, only one session can request a dump,
so if there are multiple requests, this message will be output even if the
request is for a PID that is not looked up.
+ if (fpout == NULL)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write memory context file \"%s\": %m",
+ dumpfile)));
It would be better to use "dump file" instead of "memory context file".
+static void
+McxtReqKill(int code, Datum arg)
The function McxtReqKill looks like it should have a different name.
(Since it doesn't do any kill).
How about McxtReqcCleanup or something similar?
Best regards,
I also find the idea that this should be the same SQL function
as pg_get_backend_memory_contexts to be a seriously bad decision.
That means that it's not possible to GRANT the right to examine
only your own process's memory --- with this proposal, that means
granting the right to inspect every other process as well.Beyond that, the fact that there's no way to restrict the capability
to just, say, other processes owned by the same user means that
it's not really safe to GRANT to non-superusers anyway. Even with
such a restriction added, things are problematic, since for example
it would be possible to inquire into the workings of a
security-definer function executing in another process that
nominally is owned by your user.I'm going to change the function name and restrict the executor to
superusers. Is it enough?In the attached patch, I changed the function name to
pg_get_target_backend_memory_contexts() for now.Regards,
--
Tatsuhito Kasahara
kasahara.tatsuhito _at_ gmail.com
On Fri, Dec 25, 2020 at 6:08 PM Kasahara Tatsuhito
<kasahara.tatsuhito@gmail.com> wrote:
Thanks for reviewing and kind suggestion!
Attached a rewritten patch.
Thanks for updating patch.
But when I had applyed the patch to the current HEAD and did make, I
got an error due to duplicate OIDs.
You need to rebase the patch.
Assigned another OID.
Accordingly, I also slightly modified the basic design as below.
---
# Communication flow between the dumper and the requester
- (1) When requesting memory context dumping, the dumper changes
the struct on the shared memory state from 'ACCEPTABLE' to
'REQUESTING'.
- (2) The dumper sends the signal to the dumper process and wait on
the latch.
- (3) When the dumper noticed the signal, it changes the state to
'DUMPING'.
- (4) When the dumper completes dumping, it changes the state to
'DONE' and set the latch.
- (5) The requestor reads the dump file and shows it to the user.
Finally, the requestor removes the dump file and reset the shared
memory state to 'ACCEPTABLE'.# Query cancellation
- When the requestor cancels dumping, e.g. signaling using ctrl-C,
the requestor changes the state of the shared memory to 'CANCELING'.
- The dumper checks the state when it tries to change the state to
'DONE' at (4), and if the state is 'CANCELING', it initializes the
dump file and reset the shared memory state to 'ACCEPTABLE'.# Cleanup dump file and the shared memory
- In the normal case, the dumper removes the dump file and resets
the shared memory entry as described in (5).
- When something like query cancellation or process termination
happens on the dumper after (1) and before (3), in other words,
the state is 'REQUESTING', the requestor does the cleanup.
- When something happens on the dumper or the requestor after (3)
and before (4), in other words, the state is 'DUMPING', the dumper
does the cleanup. Specifically, if the requestor cancels the query,
it just changes the state to 'CANCELING' and the dumper notices it
and cleans up things later.
OTOH, when the dumper fails to dump, it cleans up the dump file and
reset the shared memory state.
- When something happens on the requestor after (4), i.e., the state
is 'DONE', the requestor does the cleanup.
- In the case of receiving SIGKILL or power failure, all dump files
are removed in the crash recovery process.
---If the dumper is terminated before it dumps, the requestor will appear
to enter an
infinite loop because the status of mcxtdumpShmem will not change.
The following are the steps to reproduce.- session1
BEGIN; LOCK TABLE t;
- session2
SELECT * FROM t; -- wait
- session3
select pg_get_target_backend_memory_contexts(<pid of session2>);
-- wait
- session1
select pg_terminate_backend(<pid of session2>); -- kill session2
- session3 waits forever.Therefore, you may need to set mcxtdumpShmem->dump_status to
MCXTDUMPSTATUS_CANCELING
or other status before the dumper terminates.
In this case, it may be difficult for the dumper to change dump_status
because
it's waiting for latch and dump_memory_contexts() is not called yet.
Instead, it's possible for the requestor to check the existence of the
dumper
process periodically during waiting.
I added this logic to the attached patch.
Also, although I have not been able to reproduce it, I believe that
with the current design,
if the requestor disappears right after the dumper dumps the memory
information,
the dump file will remain.
Since the current design appears to allow only one requestor per
instance, when the requestor
requests a dump, it might be a good idea to delete any remaining dump
files, if any.
Although I'm not sure when the dump file remains, deleting any remaining
dump
files seems good for safety.
I also added this idea to the attached patch.
The following are comments on the code.
+ proc = BackendPidGetProc(dst_pid); + + if (proc == NULL) + { + ereport(WARNING, + (errmsg("PID %d is not a PostgreSQL server process", dst_pid))); + + return (Datum) 1; + } For now, background writer, checkpointer and WAL writer are belong to the auxiliary process. Therefore, if we specify the PIDs of these processes for pg_get_target_backend_memory_contexts(), "PID xxxx is not a PostgreSQL server process" whould be outoput. This confuses the user. How about use AuxiliaryPidGetProc() to determine these processes?
Thanks and I modified the patch to output the below message when it's an
auxiliary process.
| PID %d is not a PostgreSQL backend process but an auxiliary process.
+ ereport(INFO, + (errmsg("The request has failed and now PID %d is requsting dumping.", + mcxtdumpShmem->src_pid))); + + LWLockRelease(McxtDumpLock); You can release LWLock before ereport.
Modified to release the lock before ereport.
+ Assert(mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_REQUESTING);
typo?
It might be "mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING".
Ops, it's a serious typo. Fixed it.
+ ereport(INFO, + (errmsg("PID %d is looked up by another process", pid))); This message is not accurate. Because, in the current design, only one session can request a dump, so if there are multiple requests, this message will be output even if the request is for a PID that is not looked up.
Exactly. Modified the message as below.
| Only one session can dump at a time and another session is dumping
currently.
+ if (fpout == NULL) + { + ereport(LOG, + (errcode_for_file_access(), + errmsg("could not write memory context file \"%s\": %m", + dumpfile))); It would be better to use "dump file" instead of "memory context file".+static void
+McxtReqKill(int code, Datum arg)
The function McxtReqKill looks like it should have a different name.
(Since it doesn't do any kill).
How about McxtReqcCleanup or something similar?
Thanks for your idea and I changed the name to McxtReqCleanup.
Regards,
Attachments:
v6-0001-Add-pg_get_target_backend_memory_contexts-to-coll.patchtext/x-diff; name=v6-0001-Add-pg_get_target_backend_memory_contexts-to-coll.patchDownload
From d7d99a5278c533ceae465b673b1d69f6ce51ee89 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 4 Jan 2021 12:43:18 +0900
Subject: [PATCH v6] After commit 3e98c0bafb28de, we can display the usage of
the memory contexts using pg_backend_memory_contexts system view. However,
its target is limited to the process attached to the current session. This
patch introduces pg_get_target_backend_memory_contexts() and makes it
possible to collect memory contexts of the specified process.
---
src/backend/access/transam/xlog.c | 7 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/postmaster/pgstat.c | 3 +
src/backend/replication/basebackup.c | 3 +
src/backend/storage/ipc/ipci.c | 2 +
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/tcop/postgres.c | 5 +
src/backend/utils/adt/mcxtfuncs.c | 730 ++++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/bin/initdb/initdb.c | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 4 +-
src/bin/pg_rewind/filemap.c | 3 +
src/include/catalog/pg_proc.dat | 12 +-
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 3 +-
src/include/storage/procsignal.h | 1 +
src/include/utils/mcxtfuncs.h | 44 ++
18 files changed, 809 insertions(+), 21 deletions(-)
create mode 100644 src/include/utils/mcxtfuncs.h
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index ede93ad7fd..4cab47a61d 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -73,6 +73,7 @@
#include "storage/sync.h"
#include "utils/builtins.h"
#include "utils/guc.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/relmapper.h"
@@ -6993,6 +6994,12 @@ StartupXLOG(void)
*/
pgstat_reset_all();
+ /*
+ * Reset dump files in pg_memusage, because target processes do
+ * not exist any more.
+ */
+ RemoveMemcxtFile(0);
+
/*
* If there was a backup label file, it's done its job and the info
* has now been propagated into pg_control. We must get rid of the
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index ab4603c69b..817669cd4f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -558,7 +558,8 @@ CREATE VIEW pg_backend_memory_contexts AS
SELECT * FROM pg_get_backend_memory_contexts();
REVOKE ALL ON pg_backend_memory_contexts FROM PUBLIC;
-REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_target_backend_memory_contexts FROM PUBLIC;
-- Statistics views
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 3f24a33ef1..8eb2d062b0 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4045,6 +4045,9 @@ pgstat_get_wait_ipc(WaitEventIPC w)
case WAIT_EVENT_XACT_GROUP_UPDATE:
event_name = "XactGroupUpdate";
break;
+ case WAIT_EVENT_DUMP_MEMORY_CONTEXT:
+ event_name = "DumpMemoryContext";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index 0f54635550..c67e71d79b 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -184,6 +184,9 @@ static const char *const excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dump files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..18a1dd5a74 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -45,6 +45,7 @@
#include "storage/procsignal.h"
#include "storage/sinvaladt.h"
#include "storage/spin.h"
+#include "utils/mcxtfuncs.h"
#include "utils/snapmgr.h"
/* GUCs */
@@ -267,6 +268,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ McxtDumpShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 583efaecff..106e125cc2 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -28,6 +28,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/mcxtfuncs.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -567,6 +568,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_DUMP_MEMCXT))
+ HandleProcSignalDumpMemoryContext();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 774292fd94..6b4ff6f08b 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+McxtDumpLock 48
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index dfa0d685a8..e07ecc43f7 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
#include "utils/lsyscache.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -540,6 +541,10 @@ ProcessClientReadInterrupt(bool blocked)
/* Process notify interrupts, if any */
if (notifyInterruptPending)
ProcessNotifyInterrupt();
+
+ /* Process memory contexts dump interrupts, if any */
+ if (ProcSignalDumpMemoryContextPending)
+ ProcessDumpMemoryContextInterrupt();
}
else if (ProcDiePending)
{
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..c52ae2aea2 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,30 +15,100 @@
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "common/logging.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "pgstat.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "storage/shmem.h"
#include "utils/builtins.h"
+#include "utils/mcxtfuncs.h"
+
+/* The max bytes for showing names and identifiers of MemoryContext. */
+#define MEMORY_CONTEXT_DISPLAY_SIZE 1024
+
+/* Number of columns in pg_backend_memory_contexts view */
+#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
+
+/* Shared memory struct for managing the status of memory context dump. */
+static mcxtdumpShmemStruct *mcxtdumpShmem = NULL;
-/* ----------
- * The max bytes for showing identifiers of MemoryContext.
- * ----------
+/*
+ * McxtReqCleanup
+ * Error cleanup callback for memory context requestor.
*/
-#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+static void
+McxtReqCleanup(int code, Datum arg)
+{
+ int dst_pid = DatumGetInt32(arg);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->src_pid != MyProcPid)
+ {
+ /*
+ * If the requestor is not me, simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+ return;
+ }
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING)
+ {
+ /*
+ * Since the dumper has not received the dump order yet, clean up
+ * things by myself.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DUMPING)
+ {
+ /*
+ * Since the dumper has received the request already,
+ * requestor just change the status and the dumper cleans up
+ * things later.
+ */
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_CANCELING;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DONE)
+ {
+ /*
+ * Since the dumper has already finished dumping, clean up things
+ * by myself.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ RemoveMemcxtFile(dst_pid);
+ }
+ LWLockRelease(McxtDumpLock);
+}
/*
* PutMemoryContextsStatsTupleStore
* One recursion level for pg_get_backend_memory_contexts.
+ *
+ * Note: When fpout is not NULL, ferror() check must be done by the caller.
*/
static void
PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
TupleDesc tupdesc, MemoryContext context,
- const char *parent, int level)
+ const char *parent, int level, FILE *fpout)
{
-#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
-
Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
MemoryContextCounters stat;
MemoryContext child;
const char *name;
@@ -74,14 +144,12 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
if (ident)
{
int idlen = strlen(ident);
- char clipped_ident[MEMORY_CONTEXT_IDENT_DISPLAY_SIZE];
-
/*
* Some identifiers such as SQL query string can be very long,
* truncate oversize identifiers.
*/
- if (idlen >= MEMORY_CONTEXT_IDENT_DISPLAY_SIZE)
- idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_IDENT_DISPLAY_SIZE - 1);
+ if (idlen >= MEMORY_CONTEXT_DISPLAY_SIZE)
+ idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_DISPLAY_SIZE - 1);
memcpy(clipped_ident, ident, idlen);
clipped_ident[idlen] = '\0';
@@ -101,18 +169,207 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
values[6] = Int64GetDatum(stat.freespace);
values[7] = Int64GetDatum(stat.freechunks);
values[8] = Int64GetDatum(stat.totalspace - stat.freespace);
- tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Since pg_get_backend_memory_contexts() is called from local process,
+ * simply put tuples.
+ */
+ if(fpout == NULL)
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Write out the current memory context information in the form of
+ * "key: value" pairs to the file specified by the requestor.
+ */
+ else
+ {
+ /*
+ * Make each memory context information starts with 'D'.
+ * This is checked by the requestor when reading the file.
+ */
+ fputc('D', fpout);
+
+ fprintf(fpout,
+ "name: %s, ident: %s, parent: %s, level: %d, total_bytes: %lu, \
+ total_nblocks: %lu, free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name,
+ ident ? clipped_ident : "none",
+ parent ? parent : "none", level,
+ stat.totalspace,
+ stat.nblocks,
+ stat.freespace,
+ stat.freechunks,
+ stat.totalspace - stat.freespace);
+ }
for (child = context->firstchild; child != NULL; child = child->nextchild)
{
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- child, name, level + 1);
+ child, name, level + 1, fpout);
+ }
+}
+
+/*
+ * SetupMcxtdumpShem
+ * Setup shared memory struct for dumping specified PID.
+ */
+static void
+SetupMcxtdumpShmem(int pid)
+{
+ /*
+ * We only allow one session per target process to request memory
+ * contexts dump at a time.
+ * If someone uses mcxtdumpShmem, wait until it was changed to acceptable.
+ */
+ while (true)
+ {
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_REQUESTING;
+ mcxtdumpShmem->src_pid = MyProcPid;
+ mcxtdumpShmem->dst_pid = pid;
+
+ /*
+ * Dump files should not exist now, but delete any of
+ * them just in case.
+ *
+ * Note: This is possible because only one session can
+ * request memory contexts per instance.
+ */
+ RemoveMemcxtFile(0);
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+ else
+ {
+ ereport(INFO,
+ (errmsg("Only one session can dump at a time and another session is dumping currently.")));
+
+ LWLockRelease(McxtDumpLock);
+
+ pg_usleep(5000000L);
+ }
+ }
+}
+
+/*
+ * PutDumpedValuesOnTuplestore
+ * Read specified memory context dump file and put its values
+ * on the tuplestore.
+ */
+static void
+PutDumpedValuesOnTuplestore(char *dumpfile, Tuplestorestate *tupstore,
+ TupleDesc tupdesc, int pid)
+{
+ FILE *fpin;
+ int format_id;
+
+ if ((fpin = AllocateFile(dumpfile, "r")) == NULL)
+ {
+ if (errno != ENOENT)
+ ereport(LOG, (errcode_for_file_access(),
+ errmsg("could not open memory context dump file \"%s\": %m",
+ dumpfile)));
}
+
+ /* Verify it's of the expected format. */
+ if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ format_id != PG_MCXT_FILE_FORMAT_ID)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"", dumpfile)));
+ goto done;
+ }
+
+ /* Read dump file and put values on tuple store. */
+ while (true)
+ {
+ Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char name[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char parent[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
+ int level;
+ Size total_bytes;
+ Size total_nblocks;
+ Size free_bytes;
+ Size free_chunks;
+ Size used_bytes;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ switch (fgetc(fpin))
+ {
+ /* 'D' A memory context information follows */
+ case 'D':
+ if (fscanf(fpin, "name: %1023[^,], ident: %1023[^,], parent: %1023[^,], \
+ level: %d, total_bytes: %lu, total_nblocks: %lu, \
+ free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name, clipped_ident, parent, &level, &total_bytes, &total_nblocks,
+ &free_bytes, &free_chunks, &used_bytes)
+ != PG_GET_BACKEND_MEMORY_CONTEXTS_COLS)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+
+ values[0] = CStringGetTextDatum(name);
+
+ if (strcmp(clipped_ident, "none"))
+ values[1] = CStringGetTextDatum(clipped_ident);
+ else
+ nulls[1] = true;
+
+ if (strcmp(parent, "none"))
+ values[2] = CStringGetTextDatum(parent);
+ else
+ nulls[2] = true;
+
+ values[3] = Int32GetDatum(level);
+ values[4] = Int64GetDatum(total_bytes);
+ values[5] = Int64GetDatum(total_nblocks);
+ values[6] = Int64GetDatum(free_bytes);
+ values[7] = Int64GetDatum(free_chunks);
+ values[8] = Int64GetDatum(used_bytes);
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ break;
+
+ case 'E':
+ goto done;
+
+ default:
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ }
+done:
+ FreeFile(fpin);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ Assert(mcxtdumpShmem->src_pid == MyProcPid);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
}
/*
* pg_get_backend_memory_contexts
- * SQL SRF showing backend memory context.
+ * SQL SRF showing local backend memory context.
*/
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
@@ -148,10 +405,453 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldcontext);
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- TopMemoryContext, NULL, 0);
+ TopMemoryContext, "", 0, NULL);
+
+ /* clean up and return the tuplestore */
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
+
+/*
+ * pg_get_target_backend_memory_contexts
+ * SQL SRF showing specified process's backend memory context.
+ */
+Datum
+pg_get_target_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int dst_pid = PG_GETARG_INT32(0);
+
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ char dumpfile[MAXPGPATH];
+ PGPROC *proc;
+ PgBackendStatus *beentry;
+
+ if (dst_pid == MyProcPid)
+ {
+ pg_get_backend_memory_contexts(fcinfo);
+ return (Datum) 0;
+ }
+
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, dst_pid);
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ /*
+ * Check whether the target process is PostgreSQL backend process.
+ */
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process.", dst_pid)));
+ else
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL backend process but an auxiliary process.", dst_pid)));
+
+ tuplestore_donestoring(tupstore);
+ return (Datum) 1;
+ }
+
+ beentry = pgstat_fetch_stat_beentry(proc->backendId);
+
+ if (beentry->st_backendType != B_BACKEND)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL backend process", dst_pid)));
+
+ tuplestore_donestoring(tupstore);
+ return (Datum) 1;
+ }
+
+ /*
+ * The ENSURE stuff ensures we clean up the shared memory struct and files
+ * on failure.
+ */
+ PG_ENSURE_ERROR_CLEANUP(McxtReqCleanup, (Datum) Int32GetDatum(dst_pid));
+ {
+ SetupMcxtdumpShmem(dst_pid);
+
+ SendProcSignal(dst_pid, PROCSIG_DUMP_MEMCXT, InvalidBackendId);
+
+ /* Wait until target process finishes dumping file. */
+ for (;;)
+ {
+ /* Check for dump cancel request. */
+ CHECK_FOR_INTERRUPTS();
+
+ /* Must reset the latch before testing state. */
+ ResetLatch(MyLatch);
+
+ LWLockAcquire(McxtDumpLock, LW_SHARED);
+
+ if (mcxtdumpShmem->src_pid != MyProcPid)
+ {
+ /*
+ * It seems the dumper exited and subsequently another
+ * process is requesting dumping.
+ */
+ LWLockRelease(McxtDumpLock);
+
+ ereport(INFO,
+ (errmsg("The request has failed and now PID %d is requsting dumping.",
+ mcxtdumpShmem->src_pid)));
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ /*
+ * Dumper seems to have cleaned up things already because
+ * of failures or cancellation.
+ * Since the dumper has already removed the dump file,
+ * simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /*
+ * Request has been canceled. Exit without dumping.
+ */
+ LWLockRelease(McxtDumpLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DONE)
+ {
+ /* Dumping has completed. */
+ LWLockRelease(McxtDumpLock);
+ break;
+ }
+ /*
+ * The dumper must be in the middle of a dumping or the request
+ * hasn't been reached yet.
+ */
+ Assert(mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING ||
+ mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DUMPING);
+
+ /*
+ * Although we have checked the target process,
+ * it might have been terminated after the check.
+ * Ensure it again.
+ */
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d seems to exit before dumping.", dst_pid)));
+
+ /* Initialize the shared memory and exit. */
+ LWLockRelease(McxtDumpLock);
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ tuplestore_donestoring(tupstore);
+ return (Datum) 1;
+ }
+ LWLockRelease(McxtDumpLock);
+
+ /*
+ * Wait. We expect to get a latch signal back from the dumper.
+ * Use a timeout to enable cancellation.
+ */
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ 1000L, WAIT_EVENT_DUMP_MEMORY_CONTEXT);
+ }
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(McxtReqCleanup, (Datum) Int32GetDatum(dst_pid));
+
+ /* Read values from the dump file and put them on tuplestore. */
+ PutDumpedValuesOnTuplestore(dumpfile, tupstore, tupdesc, dst_pid);
/* clean up and return the tuplestore */
tuplestore_donestoring(tupstore);
return (Datum) 0;
}
+
+/*
+ * dump_memory_contexts
+ * Dump local memory contexts to a file.
+ */
+static void
+dump_memory_contexts(void)
+{
+ FILE *fpout;
+ char dumpfile[MAXPGPATH];
+ int format_id;
+ pid_t src_pid;
+ PGPROC *src_proc;
+
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, MyProcPid);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ /*
+ * The requestor canceled the request and initialized
+ * the shared memory. Simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /*
+ * The requestor canceled the request.
+ * Initialize the shared memory and exit.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ Assert(mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING);
+
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_DUMPING;
+ src_pid = mcxtdumpShmem->src_pid;
+
+ LWLockRelease(McxtDumpLock);
+
+ fpout = AllocateFile(dumpfile, "w");
+
+ if (fpout == NULL)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write dump file \"%s\": %m",
+ dumpfile)));
+ FreeFile(fpout);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+ format_id = PG_MCXT_FILE_FORMAT_ID;
+ fwrite(&format_id, sizeof(format_id), 1, fpout);
+
+ /* Look into each memory context from TopMemoryContext recursively. */
+ PutMemoryContextsStatsTupleStore(NULL, NULL,
+ TopMemoryContext, NULL, 0, fpout);
+
+ /*
+ * Make dump file ends with 'E'.
+ * This is checked by the requestor later.
+ */
+ fputc('E', fpout);
+
+ if (ferror(fpout))
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write dump file \"%s\": %m",
+ dumpfile)));
+ FreeFile(fpout);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ /* No more output to be done. Close file. */
+ else if (FreeFile(fpout) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not close dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /* During dumping, the requestor canceled the request. */
+ unlink(dumpfile);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ /* Dumping has succeeded, notify it to the requestor. */
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_DONE;
+ LWLockRelease(McxtDumpLock);
+ src_proc = BackendPidGetProc(src_pid);
+ SetLatch(&(src_proc->procLatch));
+
+ return;
+}
+
+/*
+ * ProcessDumpMemoryContextInterrupt
+ * The portion of memory context dump interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessDumpMemoryContextInterrupt(void)
+{
+ ProcSignalDumpMemoryContextPending = false;
+ dump_memory_contexts();
+}
+
+/*
+ * HandleProcSignalDumpMemoryContext
+ * Handle receipt of an interrupt indicating a memory context dump.
+ * Signal handler portion of interrupt handling.
+ */
+void
+HandleProcSignalDumpMemoryContext(void)
+{
+ ProcSignalDumpMemoryContextPending = true;
+}
+
+/*
+ * McxtDumpShmemInit
+ * Initialize mcxtdump shared memory struct.
+ */
+void
+McxtDumpShmemInit(void)
+{
+ bool found;
+
+ mcxtdumpShmem = (mcxtdumpShmemStruct *)
+ ShmemInitStruct("Memory Context Dump Data",
+ sizeof(mcxtdumpShmemStruct),
+ &found);
+ if (!found)
+ {
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+ }
+}
+
+/*
+ * RemoveMemcxtFile
+ * Remove dump files.
+ */
+void
+RemoveMemcxtFile(int pid)
+{
+ DIR *dir;
+ struct dirent *dumpfile;
+
+ if (pid == 0)
+ {
+ /* delete all dump files */
+ dir = AllocateDir(PG_MEMUSAGE_DIR);
+ while ((dumpfile = ReadDir(dir, PG_MEMUSAGE_DIR)) != NULL)
+ {
+ char dumpfilepath[32];
+
+ if (strcmp(dumpfile->d_name, ".") == 0 || strcmp(dumpfile->d_name, "..") == 0)
+ continue;
+
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, dumpfile->d_name);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ FreeDir(dir);
+ }
+ else
+ {
+ /* delete specified dump file */
+ char str_pid[12];
+ char dumpfilepath[32];
+ struct stat stat_tmp;
+
+ pg_ltoa(pid, str_pid);
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, str_pid);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (stat(dumpfilepath, &stat_tmp) == 0)
+ {
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ }
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 3d5d6cc033..f85326fd60 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -33,6 +33,7 @@ volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t ProcSignalDumpMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 4453aeb0f0..ed90a4a339 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -221,7 +221,8 @@ static const char *const subdirs[] = {
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
- "pg_logical/mappings"
+ "pg_logical/mappings",
+ "pg_memusage"
};
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index f674a7c94e..340a80fc11 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -6,7 +6,7 @@ use File::Basename qw(basename dirname);
use File::Path qw(rmtree);
use PostgresNode;
use TestLib;
-use Test::More tests => 109;
+use Test::More tests => 110;
program_help_ok('pg_basebackup');
program_version_ok('pg_basebackup');
@@ -124,7 +124,7 @@ is_deeply(
# Contents of these directories should not be copied.
foreach my $dirname (
- qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans)
+ qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans pg_memusage)
)
{
is_deeply(
diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c
index 2618b4c957..33da570598 100644
--- a/src/bin/pg_rewind/filemap.c
+++ b/src/bin/pg_rewind/filemap.c
@@ -119,6 +119,9 @@ static const char *excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index d7b55f57ea..4e8aa46e48 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7852,15 +7852,25 @@
# memory context of local backend
{ oid => '2282',
- descr => 'information about all memory contexts of local backend',
proname => 'pg_get_backend_memory_contexts', prorows => '100',
proretset => 't', provolatile => 'v', proparallel => 'r',
prorettype => 'record', proargtypes => '',
proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',
proargmodes => '{o,o,o,o,o,o,o,o,o}',
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ descr => 'information about all memory contexts of local backend',
prosrc => 'pg_get_backend_memory_contexts' },
+# memory context of specified backend
+{ oid => '4543',
+ descr => 'information about all memory contexts of specified backend',
+ proname => 'pg_get_target_backend_memory_contexts', prorows => '100', proisstrict => 'f',
+ proretset => 't', provolatile => 'v', proparallel => 'r', prorettype => 'record',
+ proargtypes => 'int4', proallargtypes => '{int4,text,text,text,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ prosrc => 'pg_get_target_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 2c71db79c0..7812fa2499 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -83,6 +83,7 @@ extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t ProcSignalDumpMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 3a7e199750..b365dd7be4 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -964,7 +964,8 @@ typedef enum
WAIT_EVENT_REPLICATION_SLOT_DROP,
WAIT_EVENT_SAFE_SNAPSHOT,
WAIT_EVENT_SYNC_REP,
- WAIT_EVENT_XACT_GROUP_UPDATE
+ WAIT_EVENT_XACT_GROUP_UPDATE,
+ WAIT_EVENT_DUMP_MEMORY_CONTEXT
} WaitEventIPC;
/* ----------
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..d025381ae8 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_DUMP_MEMCXT, /* request dumping memory context interrupt */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/mcxtfuncs.h b/src/include/utils/mcxtfuncs.h
new file mode 100644
index 0000000000..eff2c62cbb
--- /dev/null
+++ b/src/include/utils/mcxtfuncs.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcxtfuncs.h
+ * Declarations for showing backend memory context.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mcxtfuncs.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MCXT_H
+#define MCXT_H
+
+/* Directory to store dumped memory files */
+#define PG_MEMUSAGE_DIR "pg_memusage"
+
+#define PG_MCXT_FILE_FORMAT_ID 0x01B5BC9E
+
+typedef enum McxtDumpStatus
+{
+ MCXTDUMPSTATUS_ACCEPTABLE, /* no one is requesting dumping */
+ MCXTDUMPSTATUS_REQUESTING, /* request has been issued, but dumper has not received it yet */
+ MCXTDUMPSTATUS_DUMPING, /* dumper is dumping files */
+ MCXTDUMPSTATUS_DONE, /* dumper has finished dumping and the requestor is working */
+ MCXTDUMPSTATUS_CANCELING /* requestor canceled the dumping */
+} McxtDumpStatus;
+
+typedef struct mcxtdumpShmemStruct
+{
+ pid_t dst_pid; /* pid of the signal receiver */
+ pid_t src_pid; /* pid of the signal sender */
+ McxtDumpStatus dump_status; /* dump status */
+} mcxtdumpShmemStruct;
+
+extern void ProcessDumpMemoryContextInterrupt(void);
+extern void HandleProcSignalDumpMemoryContext(void);
+extern void McxtDumpShmemInit(void);
+extern void RemoveMcxtDumpFile(int);
+extern void RemoveMemcxtFile(int);
+#endif /* MCXT_H */
--
2.18.1
v7 that fixes recent conflicts.
It also changed the behavior of requestor when another requestor is
already working for simplicity.
In this case, v6 patch makes the requestor wait. v7 patch makes the
requestor quit.
Regards,
--
Atsushi Torikoshi
Attachments:
v7-0001-Add-pg_get_target_backend_memory_contexts-to-coll.patchtext/x-diff; name=v7-0001-Add-pg_get_target_backend_memory_contexts-to-coll.patchDownload
From f20e48d99f2770bfec275805185aa5ce08661fce Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Tue, 12 Jan 2021 20:55:43 +0900
Subject: [PATCH v7] After commit 3e98c0bafb28de, we can display the usage of
the memory contexts using pg_backend_memory_contexts system view. However,
its target is limited to the process attached to the current session. This
patch introduces pg_get_target_backend_memory_contexts() and makes it
possible to collect memory contexts of the specified process.
---
src/backend/access/transam/xlog.c | 7 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/postmaster/pgstat.c | 3 +
src/backend/replication/basebackup.c | 3 +
src/backend/storage/ipc/ipci.c | 2 +
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/tcop/postgres.c | 5 +
src/backend/utils/adt/mcxtfuncs.c | 731 ++++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/bin/initdb/initdb.c | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 4 +-
src/bin/pg_rewind/filemap.c | 3 +
src/include/catalog/pg_proc.dat | 12 +-
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 3 +-
src/include/storage/procsignal.h | 1 +
src/include/utils/mcxtfuncs.h | 44 ++
18 files changed, 810 insertions(+), 21 deletions(-)
create mode 100644 src/include/utils/mcxtfuncs.h
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index ede93ad7fd..4cab47a61d 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -73,6 +73,7 @@
#include "storage/sync.h"
#include "utils/builtins.h"
#include "utils/guc.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/relmapper.h"
@@ -6993,6 +6994,12 @@ StartupXLOG(void)
*/
pgstat_reset_all();
+ /*
+ * Reset dump files in pg_memusage, because target processes do
+ * not exist any more.
+ */
+ RemoveMemcxtFile(0);
+
/*
* If there was a backup label file, it's done its job and the info
* has now been propagated into pg_control. We must get rid of the
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5d89e77dbe..7419c496b2 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -558,7 +558,8 @@ CREATE VIEW pg_backend_memory_contexts AS
SELECT * FROM pg_get_backend_memory_contexts();
REVOKE ALL ON pg_backend_memory_contexts FROM PUBLIC;
-REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_target_backend_memory_contexts FROM PUBLIC;
-- Statistics views
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 3f24a33ef1..8eb2d062b0 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4045,6 +4045,9 @@ pgstat_get_wait_ipc(WaitEventIPC w)
case WAIT_EVENT_XACT_GROUP_UPDATE:
event_name = "XactGroupUpdate";
break;
+ case WAIT_EVENT_DUMP_MEMORY_CONTEXT:
+ event_name = "DumpMemoryContext";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index 0f54635550..c67e71d79b 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -184,6 +184,9 @@ static const char *const excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dump files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..18a1dd5a74 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -45,6 +45,7 @@
#include "storage/procsignal.h"
#include "storage/sinvaladt.h"
#include "storage/spin.h"
+#include "utils/mcxtfuncs.h"
#include "utils/snapmgr.h"
/* GUCs */
@@ -267,6 +268,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ McxtDumpShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 583efaecff..106e125cc2 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -28,6 +28,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/mcxtfuncs.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -567,6 +568,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_DUMP_MEMCXT))
+ HandleProcSignalDumpMemoryContext();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 774292fd94..6b4ff6f08b 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+McxtDumpLock 48
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 28055680aa..986225e802 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
#include "utils/lsyscache.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -540,6 +541,10 @@ ProcessClientReadInterrupt(bool blocked)
/* Process notify interrupts, if any */
if (notifyInterruptPending)
ProcessNotifyInterrupt();
+
+ /* Process memory contexts dump interrupts, if any */
+ if (ProcSignalDumpMemoryContextPending)
+ ProcessDumpMemoryContextInterrupt();
}
else if (ProcDiePending)
{
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..71afef0904 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,30 +15,100 @@
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "common/logging.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "pgstat.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "storage/shmem.h"
#include "utils/builtins.h"
+#include "utils/mcxtfuncs.h"
+
+/* The max bytes for showing names and identifiers of MemoryContext. */
+#define MEMORY_CONTEXT_DISPLAY_SIZE 1024
+
+/* Number of columns in pg_backend_memory_contexts view */
+#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
+
+/* Shared memory struct for managing the status of memory context dump. */
+static mcxtdumpShmemStruct *mcxtdumpShmem = NULL;
-/* ----------
- * The max bytes for showing identifiers of MemoryContext.
- * ----------
+/*
+ * McxtReqCleanup
+ * Error cleanup callback for memory context requestor.
*/
-#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+static void
+McxtReqCleanup(int code, Datum arg)
+{
+ int dst_pid = DatumGetInt32(arg);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->src_pid != MyProcPid)
+ {
+ /*
+ * If the requestor is not me, simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+ return;
+ }
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING)
+ {
+ /*
+ * Since the dumper has not received the dump order yet, clean up
+ * things by myself.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DUMPING)
+ {
+ /*
+ * Since the dumper has received the request already,
+ * requestor just change the status and the dumper cleans up
+ * things later.
+ */
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_CANCELING;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DONE)
+ {
+ /*
+ * Since the dumper has already finished dumping, clean up things
+ * by myself.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ RemoveMemcxtFile(dst_pid);
+ }
+ LWLockRelease(McxtDumpLock);
+}
/*
* PutMemoryContextsStatsTupleStore
* One recursion level for pg_get_backend_memory_contexts.
+ *
+ * Note: When fpout is not NULL, ferror() check must be done by the caller.
*/
static void
PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
TupleDesc tupdesc, MemoryContext context,
- const char *parent, int level)
+ const char *parent, int level, FILE *fpout)
{
-#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
-
Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
MemoryContextCounters stat;
MemoryContext child;
const char *name;
@@ -74,14 +144,12 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
if (ident)
{
int idlen = strlen(ident);
- char clipped_ident[MEMORY_CONTEXT_IDENT_DISPLAY_SIZE];
-
/*
* Some identifiers such as SQL query string can be very long,
* truncate oversize identifiers.
*/
- if (idlen >= MEMORY_CONTEXT_IDENT_DISPLAY_SIZE)
- idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_IDENT_DISPLAY_SIZE - 1);
+ if (idlen >= MEMORY_CONTEXT_DISPLAY_SIZE)
+ idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_DISPLAY_SIZE - 1);
memcpy(clipped_ident, ident, idlen);
clipped_ident[idlen] = '\0';
@@ -101,18 +169,203 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
values[6] = Int64GetDatum(stat.freespace);
values[7] = Int64GetDatum(stat.freechunks);
values[8] = Int64GetDatum(stat.totalspace - stat.freespace);
- tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Since pg_get_backend_memory_contexts() is called from local process,
+ * simply put tuples.
+ */
+ if(fpout == NULL)
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Write out the current memory context information in the form of
+ * "key: value" pairs to the file specified by the requestor.
+ */
+ else
+ {
+ /*
+ * Make each memory context information starts with 'D'.
+ * This is checked by the requestor when reading the file.
+ */
+ fputc('D', fpout);
+
+ fprintf(fpout,
+ "name: %s, ident: %s, parent: %s, level: %d, total_bytes: %lu, \
+ total_nblocks: %lu, free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name,
+ ident ? clipped_ident : "none",
+ parent ? parent : "none", level,
+ stat.totalspace,
+ stat.nblocks,
+ stat.freespace,
+ stat.freechunks,
+ stat.totalspace - stat.freespace);
+ }
for (child = context->firstchild; child != NULL; child = child->nextchild)
{
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- child, name, level + 1);
+ child, name, level + 1, fpout);
}
}
+/*
+ * SetupMcxtdumpShem
+ * Setup shared memory struct for dumping specified PID.
+ */
+static bool
+SetupMcxtdumpShmem(int pid)
+{
+ /*
+ * We only allow one session per target process to request memory
+ * contexts dump at a time.
+ */
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_REQUESTING;
+ mcxtdumpShmem->src_pid = MyProcPid;
+ mcxtdumpShmem->dst_pid = pid;
+
+ /*
+ * Dump files should not exist now, but delete any of
+ * them just in case.
+ *
+ * Note: This is possible because only one session can
+ * request memory contexts per instance.
+ */
+ RemoveMemcxtFile(0);
+
+ LWLockRelease(McxtDumpLock);
+
+ return true;
+ }
+ else
+ {
+ LWLockRelease(McxtDumpLock);
+
+ ereport(WARNING,
+ (errmsg("Only one session can dump at a time and another session is dumping currently.")));
+
+ return false;
+ }
+}
+
+/*
+ * PutDumpedValuesOnTuplestore
+ * Read specified memory context dump file and put its values
+ * on the tuplestore.
+ */
+static void
+PutDumpedValuesOnTuplestore(char *dumpfile, Tuplestorestate *tupstore,
+ TupleDesc tupdesc, int pid)
+{
+ FILE *fpin;
+ int format_id;
+
+ if ((fpin = AllocateFile(dumpfile, "r")) == NULL)
+ {
+ if (errno != ENOENT)
+ ereport(LOG, (errcode_for_file_access(),
+ errmsg("could not open memory context dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ /* Verify it's of the expected format. */
+ if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ format_id != PG_MCXT_FILE_FORMAT_ID)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"", dumpfile)));
+ goto done;
+ }
+
+ /* Read dump file and put values on tuple store. */
+ while (true)
+ {
+ Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char name[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char parent[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
+ int level;
+ Size total_bytes;
+ Size total_nblocks;
+ Size free_bytes;
+ Size free_chunks;
+ Size used_bytes;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ switch (fgetc(fpin))
+ {
+ /* 'D' A memory context information follows */
+ case 'D':
+ if (fscanf(fpin, "name: %1023[^,], ident: %1023[^,], parent: %1023[^,], \
+ level: %d, total_bytes: %lu, total_nblocks: %lu, \
+ free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name, clipped_ident, parent, &level, &total_bytes, &total_nblocks,
+ &free_bytes, &free_chunks, &used_bytes)
+ != PG_GET_BACKEND_MEMORY_CONTEXTS_COLS)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+
+ values[0] = CStringGetTextDatum(name);
+
+ if (strcmp(clipped_ident, "none"))
+ values[1] = CStringGetTextDatum(clipped_ident);
+ else
+ nulls[1] = true;
+
+ if (strcmp(parent, "none"))
+ values[2] = CStringGetTextDatum(parent);
+ else
+ nulls[2] = true;
+
+ values[3] = Int32GetDatum(level);
+ values[4] = Int64GetDatum(total_bytes);
+ values[5] = Int64GetDatum(total_nblocks);
+ values[6] = Int64GetDatum(free_bytes);
+ values[7] = Int64GetDatum(free_chunks);
+ values[8] = Int64GetDatum(used_bytes);
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ break;
+
+ case 'E':
+ goto done;
+
+ default:
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ }
+done:
+ FreeFile(fpin);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ Assert(mcxtdumpShmem->src_pid == MyProcPid);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+}
+
/*
* pg_get_backend_memory_contexts
- * SQL SRF showing backend memory context.
+ * SQL SRF showing local backend memory context.
*/
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
@@ -148,10 +401,458 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldcontext);
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- TopMemoryContext, NULL, 0);
+ TopMemoryContext, "", 0, NULL);
+
+ /* clean up and return the tuplestore */
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
+
+/*
+ * pg_get_target_backend_memory_contexts
+ * SQL SRF showing specified process's backend memory context.
+ */
+Datum
+pg_get_target_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int dst_pid = PG_GETARG_INT32(0);
+
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ char dumpfile[MAXPGPATH];
+ PGPROC *proc;
+ PgBackendStatus *beentry;
+
+ if (dst_pid == MyProcPid)
+ {
+ pg_get_backend_memory_contexts(fcinfo);
+ return (Datum) 0;
+ }
+
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, dst_pid);
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ /*
+ * Check whether the target process is PostgreSQL backend process.
+ */
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process.", dst_pid)));
+ else
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL backend process but an auxiliary process.", dst_pid)));
+
+ tuplestore_donestoring(tupstore);
+ return (Datum) 1;
+ }
+
+ beentry = pgstat_fetch_stat_beentry(proc->backendId);
+
+ if (beentry->st_backendType != B_BACKEND)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL backend process", dst_pid)));
+
+ tuplestore_donestoring(tupstore);
+ return (Datum) 1;
+ }
+
+ /*
+ * The ENSURE stuff ensures we clean up the shared memory struct and files
+ * on failure.
+ */
+ PG_ENSURE_ERROR_CLEANUP(McxtReqCleanup, (Datum) Int32GetDatum(dst_pid));
+ {
+ if(!SetupMcxtdumpShmem(dst_pid))
+ {
+ /* Someone uses mcxtdumpShmem, simply exit. */
+ tuplestore_donestoring(tupstore);
+ return (Datum) 1;
+ }
+
+ SendProcSignal(dst_pid, PROCSIG_DUMP_MEMCXT, InvalidBackendId);
+
+ /* Wait until target process finishes dumping file. */
+ for (;;)
+ {
+ /* Check for dump cancel request. */
+ CHECK_FOR_INTERRUPTS();
+
+ /* Must reset the latch before testing state. */
+ ResetLatch(MyLatch);
+
+ LWLockAcquire(McxtDumpLock, LW_SHARED);
+
+ if (mcxtdumpShmem->src_pid != MyProcPid)
+ {
+ /*
+ * It seems the dumper exited and subsequently another
+ * process is requesting dumping.
+ */
+ LWLockRelease(McxtDumpLock);
+
+ ereport(INFO,
+ (errmsg("The request has failed and now PID %d is requsting dumping.",
+ mcxtdumpShmem->src_pid)));
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ /*
+ * Dumper seems to have cleaned up things already because
+ * of failures or cancellation.
+ * Since the dumper has already removed the dump file,
+ * simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /*
+ * Request has been canceled. Exit without dumping.
+ */
+ LWLockRelease(McxtDumpLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DONE)
+ {
+ /* Dumping has completed. */
+ LWLockRelease(McxtDumpLock);
+ break;
+ }
+ /*
+ * The dumper must be in the middle of a dumping or the request
+ * hasn't been reached yet.
+ */
+ Assert(mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING ||
+ mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DUMPING);
+
+ /*
+ * Although we have checked the target process,
+ * it might have been terminated after the check.
+ * Ensure it again.
+ */
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d seems to exit before dumping.", dst_pid)));
+
+ /* Initialize the shared memory and exit. */
+ LWLockRelease(McxtDumpLock);
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ tuplestore_donestoring(tupstore);
+ return (Datum) 1;
+ }
+ LWLockRelease(McxtDumpLock);
+
+ /*
+ * Wait. We expect to get a latch signal back from the dumper.
+ * Use a timeout to enable cancellation.
+ */
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ 1000L, WAIT_EVENT_DUMP_MEMORY_CONTEXT);
+ }
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(McxtReqCleanup, (Datum) Int32GetDatum(dst_pid));
+
+ /* Read values from the dump file and put them on tuplestore. */
+ PutDumpedValuesOnTuplestore(dumpfile, tupstore, tupdesc, dst_pid);
/* clean up and return the tuplestore */
tuplestore_donestoring(tupstore);
return (Datum) 0;
}
+
+/*
+ * dump_memory_contexts
+ * Dump local memory contexts to a file.
+ */
+static void
+dump_memory_contexts(void)
+{
+ FILE *fpout;
+ char dumpfile[MAXPGPATH];
+ int format_id;
+ pid_t src_pid;
+ PGPROC *src_proc;
+
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, MyProcPid);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ /*
+ * The requestor canceled the request and initialized
+ * the shared memory. Simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /*
+ * The requestor canceled the request.
+ * Initialize the shared memory and exit.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ Assert(mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING);
+
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_DUMPING;
+ src_pid = mcxtdumpShmem->src_pid;
+
+ LWLockRelease(McxtDumpLock);
+
+ fpout = AllocateFile(dumpfile, "w");
+
+ if (fpout == NULL)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write dump file \"%s\": %m",
+ dumpfile)));
+ FreeFile(fpout);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+ format_id = PG_MCXT_FILE_FORMAT_ID;
+ fwrite(&format_id, sizeof(format_id), 1, fpout);
+
+ /* Look into each memory context from TopMemoryContext recursively. */
+ PutMemoryContextsStatsTupleStore(NULL, NULL,
+ TopMemoryContext, NULL, 0, fpout);
+
+ /*
+ * Make dump file ends with 'E'.
+ * This is checked by the requestor later.
+ */
+ fputc('E', fpout);
+
+ if (ferror(fpout))
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write dump file \"%s\": %m",
+ dumpfile)));
+ FreeFile(fpout);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ /* No more output to be done. Close file. */
+ else if (FreeFile(fpout) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not close dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /* During dumping, the requestor canceled the request. */
+ unlink(dumpfile);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ /* Dumping has succeeded, notify it to the requestor. */
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_DONE;
+ LWLockRelease(McxtDumpLock);
+ src_proc = BackendPidGetProc(src_pid);
+ SetLatch(&(src_proc->procLatch));
+
+ return;
+}
+
+/*
+ * ProcessDumpMemoryContextInterrupt
+ * The portion of memory context dump interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessDumpMemoryContextInterrupt(void)
+{
+ ProcSignalDumpMemoryContextPending = false;
+ dump_memory_contexts();
+}
+
+/*
+ * HandleProcSignalDumpMemoryContext
+ * Handle receipt of an interrupt indicating a memory context dump.
+ * Signal handler portion of interrupt handling.
+ */
+void
+HandleProcSignalDumpMemoryContext(void)
+{
+ ProcSignalDumpMemoryContextPending = true;
+}
+
+/*
+ * McxtDumpShmemInit
+ * Initialize mcxtdump shared memory struct.
+ */
+void
+McxtDumpShmemInit(void)
+{
+ bool found;
+
+ mcxtdumpShmem = (mcxtdumpShmemStruct *)
+ ShmemInitStruct("Memory Context Dump Data",
+ sizeof(mcxtdumpShmemStruct),
+ &found);
+ if (!found)
+ {
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+ }
+}
+
+/*
+ * RemoveMemcxtFile
+ * Remove dump files.
+ */
+void
+RemoveMemcxtFile(int pid)
+{
+ DIR *dir;
+ struct dirent *dumpfile;
+
+ if (pid == 0)
+ {
+ /* delete all dump files */
+ dir = AllocateDir(PG_MEMUSAGE_DIR);
+ while ((dumpfile = ReadDir(dir, PG_MEMUSAGE_DIR)) != NULL)
+ {
+ char dumpfilepath[32];
+
+ if (strcmp(dumpfile->d_name, ".") == 0 || strcmp(dumpfile->d_name, "..") == 0)
+ continue;
+
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, dumpfile->d_name);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ FreeDir(dir);
+ }
+ else
+ {
+ /* delete specified dump file */
+ char str_pid[12];
+ char dumpfilepath[32];
+ struct stat stat_tmp;
+
+ pg_ltoa(pid, str_pid);
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, str_pid);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (stat(dumpfilepath, &stat_tmp) == 0)
+ {
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ }
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index ea28769d6a..437ae2213a 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -32,6 +32,7 @@ volatile sig_atomic_t QueryCancelPending = false;
volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
+volatile sig_atomic_t ProcSignalDumpMemoryContextPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile uint32 InterruptHoldoffCount = 0;
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index c854221a30..8ceba2fe42 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -221,7 +221,8 @@ static const char *const subdirs[] = {
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
- "pg_logical/mappings"
+ "pg_logical/mappings",
+ "pg_memusage"
};
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index f674a7c94e..340a80fc11 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -6,7 +6,7 @@ use File::Basename qw(basename dirname);
use File::Path qw(rmtree);
use PostgresNode;
use TestLib;
-use Test::More tests => 109;
+use Test::More tests => 110;
program_help_ok('pg_basebackup');
program_version_ok('pg_basebackup');
@@ -124,7 +124,7 @@ is_deeply(
# Contents of these directories should not be copied.
foreach my $dirname (
- qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans)
+ qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans pg_memusage)
)
{
is_deeply(
diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c
index 2618b4c957..33da570598 100644
--- a/src/bin/pg_rewind/filemap.c
+++ b/src/bin/pg_rewind/filemap.c
@@ -119,6 +119,9 @@ static const char *excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index d7b55f57ea..4e8aa46e48 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7852,15 +7852,25 @@
# memory context of local backend
{ oid => '2282',
- descr => 'information about all memory contexts of local backend',
proname => 'pg_get_backend_memory_contexts', prorows => '100',
proretset => 't', provolatile => 'v', proparallel => 'r',
prorettype => 'record', proargtypes => '',
proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',
proargmodes => '{o,o,o,o,o,o,o,o,o}',
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ descr => 'information about all memory contexts of local backend',
prosrc => 'pg_get_backend_memory_contexts' },
+# memory context of specified backend
+{ oid => '4543',
+ descr => 'information about all memory contexts of specified backend',
+ proname => 'pg_get_target_backend_memory_contexts', prorows => '100', proisstrict => 'f',
+ proretset => 't', provolatile => 'v', proparallel => 'r', prorettype => 'record',
+ proargtypes => 'int4', proallargtypes => '{int4,text,text,text,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ prosrc => 'pg_get_target_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..d058422bed 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -82,6 +82,7 @@ extern PGDLLIMPORT volatile sig_atomic_t InterruptPending;
extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t ProcSignalDumpMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index c38b689710..fd384183f6 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -964,7 +964,8 @@ typedef enum
WAIT_EVENT_REPLICATION_SLOT_DROP,
WAIT_EVENT_SAFE_SNAPSHOT,
WAIT_EVENT_SYNC_REP,
- WAIT_EVENT_XACT_GROUP_UPDATE
+ WAIT_EVENT_XACT_GROUP_UPDATE,
+ WAIT_EVENT_DUMP_MEMORY_CONTEXT
} WaitEventIPC;
/* ----------
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..d025381ae8 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_DUMP_MEMCXT, /* request dumping memory context interrupt */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/mcxtfuncs.h b/src/include/utils/mcxtfuncs.h
new file mode 100644
index 0000000000..eff2c62cbb
--- /dev/null
+++ b/src/include/utils/mcxtfuncs.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcxtfuncs.h
+ * Declarations for showing backend memory context.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mcxtfuncs.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MCXT_H
+#define MCXT_H
+
+/* Directory to store dumped memory files */
+#define PG_MEMUSAGE_DIR "pg_memusage"
+
+#define PG_MCXT_FILE_FORMAT_ID 0x01B5BC9E
+
+typedef enum McxtDumpStatus
+{
+ MCXTDUMPSTATUS_ACCEPTABLE, /* no one is requesting dumping */
+ MCXTDUMPSTATUS_REQUESTING, /* request has been issued, but dumper has not received it yet */
+ MCXTDUMPSTATUS_DUMPING, /* dumper is dumping files */
+ MCXTDUMPSTATUS_DONE, /* dumper has finished dumping and the requestor is working */
+ MCXTDUMPSTATUS_CANCELING /* requestor canceled the dumping */
+} McxtDumpStatus;
+
+typedef struct mcxtdumpShmemStruct
+{
+ pid_t dst_pid; /* pid of the signal receiver */
+ pid_t src_pid; /* pid of the signal sender */
+ McxtDumpStatus dump_status; /* dump status */
+} mcxtdumpShmemStruct;
+
+extern void ProcessDumpMemoryContextInterrupt(void);
+extern void HandleProcSignalDumpMemoryContext(void);
+extern void McxtDumpShmemInit(void);
+extern void RemoveMcxtDumpFile(int);
+extern void RemoveMemcxtFile(int);
+#endif /* MCXT_H */
--
2.18.1
Since pg_get_target_backend_memory_contexts() waits to dump memory and
it could lead dead lock as below.
- session1
BEGIN; TRUNCATE t;
- session2
BEGIN; TRUNCATE t; -- wait
- session1
SELECT * FROM pg_get_target_backend_memory_contexts(<pid of session
2>); --wait
Thanks for notifying me, Fujii-san.
Attached v8 patch that prohibited calling the function inside
transactions.
Regards,
--
Atsushi Torikoshi
Attachments:
v8-0001-Add-pg_get_target_backend_memory_contexts-to-coll.patchtext/x-diff; name=v8-0001-Add-pg_get_target_backend_memory_contexts-to-coll.patchDownload
From 840185c1ad40cb7bc40333ab38927667c4d48c1d Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Thu, 14 Jan 2021 18:20:43 +0900
Subject: [PATCH v8] After commit 3e98c0bafb28de, we can display the usage of
the memory contexts using pg_backend_memory_contexts system view. However,
its target is limited to the process attached to the current session. This
patch introduces pg_get_target_backend_memory_contexts() and makes it
possible to collect memory contexts of the specified process.
---
src/backend/access/transam/xlog.c | 7 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/postmaster/pgstat.c | 3 +
src/backend/replication/basebackup.c | 3 +
src/backend/storage/ipc/ipci.c | 2 +
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/tcop/postgres.c | 5 +
src/backend/utils/adt/mcxtfuncs.c | 742 ++++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/bin/initdb/initdb.c | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 4 +-
src/bin/pg_rewind/filemap.c | 3 +
src/include/catalog/pg_proc.dat | 12 +-
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 3 +-
src/include/storage/procsignal.h | 1 +
src/include/utils/mcxtfuncs.h | 44 ++
18 files changed, 821 insertions(+), 21 deletions(-)
create mode 100644 src/include/utils/mcxtfuncs.h
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index b18257c198..45381c343a 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -74,6 +74,7 @@
#include "storage/sync.h"
#include "utils/builtins.h"
#include "utils/guc.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/relmapper.h"
@@ -7009,6 +7010,12 @@ StartupXLOG(void)
*/
pgstat_reset_all();
+ /*
+ * Reset dump files in pg_memusage, because target processes do
+ * not exist any more.
+ */
+ RemoveMemcxtFile(0);
+
/*
* If there was a backup label file, it's done its job and the info
* has now been propagated into pg_control. We must get rid of the
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5d89e77dbe..7419c496b2 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -558,7 +558,8 @@ CREATE VIEW pg_backend_memory_contexts AS
SELECT * FROM pg_get_backend_memory_contexts();
REVOKE ALL ON pg_backend_memory_contexts FROM PUBLIC;
-REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_target_backend_memory_contexts FROM PUBLIC;
-- Statistics views
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 3f24a33ef1..8eb2d062b0 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4045,6 +4045,9 @@ pgstat_get_wait_ipc(WaitEventIPC w)
case WAIT_EVENT_XACT_GROUP_UPDATE:
event_name = "XactGroupUpdate";
break;
+ case WAIT_EVENT_DUMP_MEMORY_CONTEXT:
+ event_name = "DumpMemoryContext";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index 0f54635550..c67e71d79b 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -184,6 +184,9 @@ static const char *const excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dump files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..18a1dd5a74 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -45,6 +45,7 @@
#include "storage/procsignal.h"
#include "storage/sinvaladt.h"
#include "storage/spin.h"
+#include "utils/mcxtfuncs.h"
#include "utils/snapmgr.h"
/* GUCs */
@@ -267,6 +268,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ McxtDumpShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 583efaecff..106e125cc2 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -28,6 +28,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/mcxtfuncs.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -567,6 +568,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_DUMP_MEMCXT))
+ HandleProcSignalDumpMemoryContext();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 774292fd94..6b4ff6f08b 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+McxtDumpLock 48
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 28055680aa..986225e802 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
#include "utils/lsyscache.h"
+#include "utils/mcxtfuncs.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/snapmgr.h"
@@ -540,6 +541,10 @@ ProcessClientReadInterrupt(bool blocked)
/* Process notify interrupts, if any */
if (notifyInterruptPending)
ProcessNotifyInterrupt();
+
+ /* Process memory contexts dump interrupts, if any */
+ if (ProcSignalDumpMemoryContextPending)
+ ProcessDumpMemoryContextInterrupt();
}
else if (ProcDiePending)
{
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..26637079d8 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,30 +15,101 @@
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "access/xact.h"
+#include "common/logging.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "pgstat.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "storage/shmem.h"
#include "utils/builtins.h"
+#include "utils/mcxtfuncs.h"
+
+/* The max bytes for showing names and identifiers of MemoryContext. */
+#define MEMORY_CONTEXT_DISPLAY_SIZE 1024
+
+/* Number of columns in pg_backend_memory_contexts view */
+#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
+
+/* Shared memory struct for managing the status of memory context dump. */
+static mcxtdumpShmemStruct *mcxtdumpShmem = NULL;
-/* ----------
- * The max bytes for showing identifiers of MemoryContext.
- * ----------
+/*
+ * McxtReqCleanup
+ * Error cleanup callback for memory context requestor.
*/
-#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+static void
+McxtReqCleanup(int code, Datum arg)
+{
+ int dst_pid = DatumGetInt32(arg);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->src_pid != MyProcPid)
+ {
+ /*
+ * If the requestor is not me, simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+ return;
+ }
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING)
+ {
+ /*
+ * Since the dumper has not received the dump order yet, clean up
+ * things by myself.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DUMPING)
+ {
+ /*
+ * Since the dumper has received the request already,
+ * requestor just change the status and the dumper cleans up
+ * things later.
+ */
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_CANCELING;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DONE)
+ {
+ /*
+ * Since the dumper has already finished dumping, clean up things
+ * by myself.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ RemoveMemcxtFile(dst_pid);
+ }
+ LWLockRelease(McxtDumpLock);
+}
/*
* PutMemoryContextsStatsTupleStore
* One recursion level for pg_get_backend_memory_contexts.
+ *
+ * Note: When fpout is not NULL, ferror() check must be done by the caller.
*/
static void
PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
TupleDesc tupdesc, MemoryContext context,
- const char *parent, int level)
+ const char *parent, int level, FILE *fpout)
{
-#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9
-
Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
MemoryContextCounters stat;
MemoryContext child;
const char *name;
@@ -74,14 +145,12 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
if (ident)
{
int idlen = strlen(ident);
- char clipped_ident[MEMORY_CONTEXT_IDENT_DISPLAY_SIZE];
-
/*
* Some identifiers such as SQL query string can be very long,
* truncate oversize identifiers.
*/
- if (idlen >= MEMORY_CONTEXT_IDENT_DISPLAY_SIZE)
- idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_IDENT_DISPLAY_SIZE - 1);
+ if (idlen >= MEMORY_CONTEXT_DISPLAY_SIZE)
+ idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_DISPLAY_SIZE - 1);
memcpy(clipped_ident, ident, idlen);
clipped_ident[idlen] = '\0';
@@ -101,18 +170,203 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
values[6] = Int64GetDatum(stat.freespace);
values[7] = Int64GetDatum(stat.freechunks);
values[8] = Int64GetDatum(stat.totalspace - stat.freespace);
- tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Since pg_get_backend_memory_contexts() is called from local process,
+ * simply put tuples.
+ */
+ if(fpout == NULL)
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /*
+ * Write out the current memory context information in the form of
+ * "key: value" pairs to the file specified by the requestor.
+ */
+ else
+ {
+ /*
+ * Make each memory context information starts with 'D'.
+ * This is checked by the requestor when reading the file.
+ */
+ fputc('D', fpout);
+
+ fprintf(fpout,
+ "name: %s, ident: %s, parent: %s, level: %d, total_bytes: %lu, \
+ total_nblocks: %lu, free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name,
+ ident ? clipped_ident : "none",
+ parent ? parent : "none", level,
+ stat.totalspace,
+ stat.nblocks,
+ stat.freespace,
+ stat.freechunks,
+ stat.totalspace - stat.freespace);
+ }
for (child = context->firstchild; child != NULL; child = child->nextchild)
{
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- child, name, level + 1);
+ child, name, level + 1, fpout);
+ }
+}
+
+/*
+ * SetupMcxtdumpShem
+ * Setup shared memory struct for dumping specified PID.
+ */
+static bool
+SetupMcxtdumpShmem(int pid)
+{
+ /*
+ * We only allow one session per target process to request memory
+ * contexts dump at a time.
+ */
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_REQUESTING;
+ mcxtdumpShmem->src_pid = MyProcPid;
+ mcxtdumpShmem->dst_pid = pid;
+
+ /*
+ * Dump files should not exist now, but delete any of
+ * them just in case.
+ *
+ * Note: This is possible because only one session can
+ * request memory contexts per instance.
+ */
+ RemoveMemcxtFile(0);
+
+ LWLockRelease(McxtDumpLock);
+
+ return true;
+ }
+ else
+ {
+ LWLockRelease(McxtDumpLock);
+
+ ereport(WARNING,
+ (errmsg("Only one session can dump at a time and another session is dumping currently.")));
+
+ return false;
}
}
+/*
+ * PutDumpedValuesOnTuplestore
+ * Read specified memory context dump file and put its values
+ * on the tuplestore.
+ */
+static void
+PutDumpedValuesOnTuplestore(char *dumpfile, Tuplestorestate *tupstore,
+ TupleDesc tupdesc, int pid)
+{
+ FILE *fpin;
+ int format_id;
+
+ if ((fpin = AllocateFile(dumpfile, "r")) == NULL)
+ {
+ if (errno != ENOENT)
+ ereport(LOG, (errcode_for_file_access(),
+ errmsg("could not open memory context dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ /* Verify it's of the expected format. */
+ if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
+ format_id != PG_MCXT_FILE_FORMAT_ID)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"", dumpfile)));
+ goto done;
+ }
+
+ /* Read dump file and put values on tuple store. */
+ while (true)
+ {
+ Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];
+ char name[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char parent[MEMORY_CONTEXT_DISPLAY_SIZE];
+ char clipped_ident[MEMORY_CONTEXT_DISPLAY_SIZE];
+ int level;
+ Size total_bytes;
+ Size total_nblocks;
+ Size free_bytes;
+ Size free_chunks;
+ Size used_bytes;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ switch (fgetc(fpin))
+ {
+ /* 'D' A memory context information follows */
+ case 'D':
+ if (fscanf(fpin, "name: %1023[^,], ident: %1023[^,], parent: %1023[^,], \
+ level: %d, total_bytes: %lu, total_nblocks: %lu, \
+ free_bytes: %lu, free_chunks: %lu, used_bytes: %lu,\n",
+ name, clipped_ident, parent, &level, &total_bytes, &total_nblocks,
+ &free_bytes, &free_chunks, &used_bytes)
+ != PG_GET_BACKEND_MEMORY_CONTEXTS_COLS)
+ {
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+
+ values[0] = CStringGetTextDatum(name);
+
+ if (strcmp(clipped_ident, "none"))
+ values[1] = CStringGetTextDatum(clipped_ident);
+ else
+ nulls[1] = true;
+
+ if (strcmp(parent, "none"))
+ values[2] = CStringGetTextDatum(parent);
+ else
+ nulls[2] = true;
+
+ values[3] = Int32GetDatum(level);
+ values[4] = Int64GetDatum(total_bytes);
+ values[5] = Int64GetDatum(total_nblocks);
+ values[6] = Int64GetDatum(free_bytes);
+ values[7] = Int64GetDatum(free_chunks);
+ values[8] = Int64GetDatum(used_bytes);
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ break;
+
+ case 'E':
+ goto done;
+
+ default:
+ ereport(WARNING,
+ (errmsg("corrupted memory context dump file \"%s\"",
+ dumpfile)));
+ goto done;
+ }
+ }
+done:
+ FreeFile(fpin);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ Assert(mcxtdumpShmem->src_pid == MyProcPid);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+}
+
/*
* pg_get_backend_memory_contexts
- * SQL SRF showing backend memory context.
+ * SQL SRF showing local backend memory context.
*/
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
@@ -148,10 +402,468 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldcontext);
PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- TopMemoryContext, NULL, 0);
+ TopMemoryContext, "", 0, NULL);
+
+ /* clean up and return the tuplestore */
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
+
+/*
+ * pg_get_target_backend_memory_contexts
+ * SQL SRF showing specified process's backend memory context.
+ */
+Datum
+pg_get_target_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int dst_pid = PG_GETARG_INT32(0);
+
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ char dumpfile[MAXPGPATH];
+ PGPROC *proc;
+ PgBackendStatus *beentry;
+
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, dst_pid);
+
+ /*
+ * Prohibit calling this function inside a transaction since
+ * it can cause dead lock.
+ */
+ if (IsTransactionBlock())
+ ereport(ERROR,
+ (errcode(ERRCODE_ACTIVE_SQL_TRANSACTION),
+ errmsg("pg_get_target_backend_memory_contexts cannot run inside a transaction block")));
+
+ /* If the target process is itself, call a dedicated function. */
+ if (dst_pid == MyProcPid)
+ {
+ pg_get_backend_memory_contexts(fcinfo);
+ return (Datum) 0;
+ }
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ /*
+ * Check whether the target process is PostgreSQL backend process.
+ */
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process.", dst_pid)));
+ else
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL backend process but an auxiliary process.", dst_pid)));
+
+ tuplestore_donestoring(tupstore);
+ return (Datum) 1;
+ }
+
+ beentry = pgstat_fetch_stat_beentry(proc->backendId);
+
+ if (beentry->st_backendType != B_BACKEND)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL backend process", dst_pid)));
+
+ tuplestore_donestoring(tupstore);
+ return (Datum) 1;
+ }
+
+ /*
+ * The ENSURE stuff ensures we clean up the shared memory struct and files
+ * on failure.
+ */
+ PG_ENSURE_ERROR_CLEANUP(McxtReqCleanup, (Datum) Int32GetDatum(dst_pid));
+ {
+ if(!SetupMcxtdumpShmem(dst_pid))
+ {
+ /* Someone uses mcxtdumpShmem, simply exit. */
+ tuplestore_donestoring(tupstore);
+ return (Datum) 1;
+ }
+
+ SendProcSignal(dst_pid, PROCSIG_DUMP_MEMCXT, InvalidBackendId);
+
+ /* Wait until target process finishes dumping file. */
+ for (;;)
+ {
+ /* Check for dump cancel request. */
+ CHECK_FOR_INTERRUPTS();
+
+ /* Must reset the latch before testing state. */
+ ResetLatch(MyLatch);
+
+ LWLockAcquire(McxtDumpLock, LW_SHARED);
+
+ if (mcxtdumpShmem->src_pid != MyProcPid)
+ {
+ /*
+ * It seems the dumper exited and subsequently another
+ * process is requesting dumping.
+ */
+ LWLockRelease(McxtDumpLock);
+
+ ereport(INFO,
+ (errmsg("The request has failed and now PID %d is requsting dumping.",
+ mcxtdumpShmem->src_pid)));
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ /*
+ * Dumper seems to have cleaned up things already because
+ * of failures or cancellation.
+ * Since the dumper has already removed the dump file,
+ * simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /*
+ * Request has been canceled. Exit without dumping.
+ */
+ LWLockRelease(McxtDumpLock);
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+ }
+
+ else if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DONE)
+ {
+ /* Dumping has completed. */
+ LWLockRelease(McxtDumpLock);
+ break;
+ }
+ /*
+ * The dumper must be in the middle of a dumping or the request
+ * hasn't been reached yet.
+ */
+ Assert(mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING ||
+ mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_DUMPING);
+
+ /*
+ * Although we have checked the target process,
+ * it might have been terminated after the check.
+ * Ensure it again.
+ */
+ proc = BackendPidGetProc(dst_pid);
+
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ (errmsg("PID %d seems to exit before dumping.", dst_pid)));
+
+ /* Initialize the shared memory and exit. */
+ LWLockRelease(McxtDumpLock);
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ tuplestore_donestoring(tupstore);
+ return (Datum) 1;
+ }
+ LWLockRelease(McxtDumpLock);
+
+ /*
+ * Wait. We expect to get a latch signal back from the dumper.
+ * Use a timeout to enable cancellation.
+ */
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ 1000L, WAIT_EVENT_DUMP_MEMORY_CONTEXT);
+ }
+ }
+ PG_END_ENSURE_ERROR_CLEANUP(McxtReqCleanup, (Datum) Int32GetDatum(dst_pid));
+
+ /* Read values from the dump file and put them on tuplestore. */
+ PutDumpedValuesOnTuplestore(dumpfile, tupstore, tupdesc, dst_pid);
/* clean up and return the tuplestore */
tuplestore_donestoring(tupstore);
return (Datum) 0;
}
+
+/*
+ * dump_memory_contexts
+ * Dump local memory contexts to a file.
+ */
+static void
+dump_memory_contexts(void)
+{
+ FILE *fpout;
+ char dumpfile[MAXPGPATH];
+ int format_id;
+ pid_t src_pid;
+ PGPROC *src_proc;
+
+ snprintf(dumpfile, sizeof(dumpfile), "%s/%d", PG_MEMUSAGE_DIR, MyProcPid);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_ACCEPTABLE)
+ {
+ /*
+ * The requestor canceled the request and initialized
+ * the shared memory. Simply exit.
+ */
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /*
+ * The requestor canceled the request.
+ * Initialize the shared memory and exit.
+ */
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ Assert(mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_REQUESTING);
+
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_DUMPING;
+ src_pid = mcxtdumpShmem->src_pid;
+
+ LWLockRelease(McxtDumpLock);
+
+ fpout = AllocateFile(dumpfile, "w");
+
+ if (fpout == NULL)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write dump file \"%s\": %m",
+ dumpfile)));
+ FreeFile(fpout);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+ format_id = PG_MCXT_FILE_FORMAT_ID;
+ fwrite(&format_id, sizeof(format_id), 1, fpout);
+
+ /* Look into each memory context from TopMemoryContext recursively. */
+ PutMemoryContextsStatsTupleStore(NULL, NULL,
+ TopMemoryContext, NULL, 0, fpout);
+
+ /*
+ * Make dump file ends with 'E'.
+ * This is checked by the requestor later.
+ */
+ fputc('E', fpout);
+
+ if (ferror(fpout))
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write dump file \"%s\": %m",
+ dumpfile)));
+ FreeFile(fpout);
+ unlink(dumpfile);
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ /* No more output to be done. Close file. */
+ else if (FreeFile(fpout) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not close dump file \"%s\": %m",
+ dumpfile)));
+ }
+
+ LWLockAcquire(McxtDumpLock, LW_EXCLUSIVE);
+
+ if (mcxtdumpShmem->dump_status == MCXTDUMPSTATUS_CANCELING)
+ {
+ /* During dumping, the requestor canceled the request. */
+ unlink(dumpfile);
+
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+
+ LWLockRelease(McxtDumpLock);
+
+ return;
+ }
+
+ /* Dumping has succeeded, notify it to the requestor. */
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_DONE;
+ LWLockRelease(McxtDumpLock);
+ src_proc = BackendPidGetProc(src_pid);
+ SetLatch(&(src_proc->procLatch));
+
+ return;
+}
+
+/*
+ * ProcessDumpMemoryContextInterrupt
+ * The portion of memory context dump interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessDumpMemoryContextInterrupt(void)
+{
+ ProcSignalDumpMemoryContextPending = false;
+ dump_memory_contexts();
+}
+
+/*
+ * HandleProcSignalDumpMemoryContext
+ * Handle receipt of an interrupt indicating a memory context dump.
+ * Signal handler portion of interrupt handling.
+ */
+void
+HandleProcSignalDumpMemoryContext(void)
+{
+ ProcSignalDumpMemoryContextPending = true;
+}
+
+/*
+ * McxtDumpShmemInit
+ * Initialize mcxtdump shared memory struct.
+ */
+void
+McxtDumpShmemInit(void)
+{
+ bool found;
+
+ mcxtdumpShmem = (mcxtdumpShmemStruct *)
+ ShmemInitStruct("Memory Context Dump Data",
+ sizeof(mcxtdumpShmemStruct),
+ &found);
+ if (!found)
+ {
+ mcxtdumpShmem->dst_pid = 0;
+ mcxtdumpShmem->src_pid = 0;
+ mcxtdumpShmem->dump_status = MCXTDUMPSTATUS_ACCEPTABLE;
+ }
+}
+
+/*
+ * RemoveMemcxtFile
+ * Remove dump files.
+ */
+void
+RemoveMemcxtFile(int pid)
+{
+ DIR *dir;
+ struct dirent *dumpfile;
+
+ if (pid == 0)
+ {
+ /* delete all dump files */
+ dir = AllocateDir(PG_MEMUSAGE_DIR);
+ while ((dumpfile = ReadDir(dir, PG_MEMUSAGE_DIR)) != NULL)
+ {
+ char dumpfilepath[32];
+
+ if (strcmp(dumpfile->d_name, ".") == 0 || strcmp(dumpfile->d_name, "..") == 0)
+ continue;
+
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, dumpfile->d_name);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ FreeDir(dir);
+ }
+ else
+ {
+ /* delete specified dump file */
+ char str_pid[12];
+ char dumpfilepath[32];
+ struct stat stat_tmp;
+
+ pg_ltoa(pid, str_pid);
+ sprintf(dumpfilepath, "%s/%s", PG_MEMUSAGE_DIR, str_pid);
+
+ ereport(DEBUG2,
+ (errmsg("removing file \"%s\"", dumpfilepath)));
+
+ if (stat(dumpfilepath, &stat_tmp) == 0)
+ {
+ if (unlink(dumpfilepath) < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", dumpfilepath)));
+ }
+ }
+ }
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index ea28769d6a..437ae2213a 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -32,6 +32,7 @@ volatile sig_atomic_t QueryCancelPending = false;
volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
+volatile sig_atomic_t ProcSignalDumpMemoryContextPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile uint32 InterruptHoldoffCount = 0;
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index c854221a30..8ceba2fe42 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -221,7 +221,8 @@ static const char *const subdirs[] = {
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
- "pg_logical/mappings"
+ "pg_logical/mappings",
+ "pg_memusage"
};
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index f674a7c94e..340a80fc11 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -6,7 +6,7 @@ use File::Basename qw(basename dirname);
use File::Path qw(rmtree);
use PostgresNode;
use TestLib;
-use Test::More tests => 109;
+use Test::More tests => 110;
program_help_ok('pg_basebackup');
program_version_ok('pg_basebackup');
@@ -124,7 +124,7 @@ is_deeply(
# Contents of these directories should not be copied.
foreach my $dirname (
- qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans)
+ qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans pg_memusage)
)
{
is_deeply(
diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c
index 2618b4c957..33da570598 100644
--- a/src/bin/pg_rewind/filemap.c
+++ b/src/bin/pg_rewind/filemap.c
@@ -119,6 +119,9 @@ static const char *excludeDirContents[] =
/* Contents zeroed on startup, see StartupSUBTRANS(). */
"pg_subtrans",
+ /* Skip memory context dumped files. */
+ "pg_memusage",
+
/* end of list */
NULL
};
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index d27336adcd..9f7811987e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7854,15 +7854,25 @@
# memory context of local backend
{ oid => '2282',
- descr => 'information about all memory contexts of local backend',
proname => 'pg_get_backend_memory_contexts', prorows => '100',
proretset => 't', provolatile => 'v', proparallel => 'r',
prorettype => 'record', proargtypes => '',
proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',
proargmodes => '{o,o,o,o,o,o,o,o,o}',
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ descr => 'information about all memory contexts of local backend',
prosrc => 'pg_get_backend_memory_contexts' },
+# memory context of specified backend
+{ oid => '4543',
+ descr => 'information about all memory contexts of specified backend',
+ proname => 'pg_get_target_backend_memory_contexts', prorows => '100', proisstrict => 'f',
+ proretset => 't', provolatile => 'v', proparallel => 'r', prorettype => 'record',
+ proargtypes => 'int4', proallargtypes => '{int4,text,text,text,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ prosrc => 'pg_get_target_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bdc97e308..d058422bed 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -82,6 +82,7 @@ extern PGDLLIMPORT volatile sig_atomic_t InterruptPending;
extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t ProcSignalDumpMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index c38b689710..fd384183f6 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -964,7 +964,8 @@ typedef enum
WAIT_EVENT_REPLICATION_SLOT_DROP,
WAIT_EVENT_SAFE_SNAPSHOT,
WAIT_EVENT_SYNC_REP,
- WAIT_EVENT_XACT_GROUP_UPDATE
+ WAIT_EVENT_XACT_GROUP_UPDATE,
+ WAIT_EVENT_DUMP_MEMORY_CONTEXT
} WaitEventIPC;
/* ----------
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..d025381ae8 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_DUMP_MEMCXT, /* request dumping memory context interrupt */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/mcxtfuncs.h b/src/include/utils/mcxtfuncs.h
new file mode 100644
index 0000000000..eff2c62cbb
--- /dev/null
+++ b/src/include/utils/mcxtfuncs.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * mcxtfuncs.h
+ * Declarations for showing backend memory context.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mcxtfuncs.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef MCXT_H
+#define MCXT_H
+
+/* Directory to store dumped memory files */
+#define PG_MEMUSAGE_DIR "pg_memusage"
+
+#define PG_MCXT_FILE_FORMAT_ID 0x01B5BC9E
+
+typedef enum McxtDumpStatus
+{
+ MCXTDUMPSTATUS_ACCEPTABLE, /* no one is requesting dumping */
+ MCXTDUMPSTATUS_REQUESTING, /* request has been issued, but dumper has not received it yet */
+ MCXTDUMPSTATUS_DUMPING, /* dumper is dumping files */
+ MCXTDUMPSTATUS_DONE, /* dumper has finished dumping and the requestor is working */
+ MCXTDUMPSTATUS_CANCELING /* requestor canceled the dumping */
+} McxtDumpStatus;
+
+typedef struct mcxtdumpShmemStruct
+{
+ pid_t dst_pid; /* pid of the signal receiver */
+ pid_t src_pid; /* pid of the signal sender */
+ McxtDumpStatus dump_status; /* dump status */
+} mcxtdumpShmemStruct;
+
+extern void ProcessDumpMemoryContextInterrupt(void);
+extern void HandleProcSignalDumpMemoryContext(void);
+extern void McxtDumpShmemInit(void);
+extern void RemoveMcxtDumpFile(int);
+extern void RemoveMemcxtFile(int);
+#endif /* MCXT_H */
--
2.18.1
On 2021-01-14 19:11, torikoshia wrote:
Since pg_get_target_backend_memory_contexts() waits to dump memory and
it could lead dead lock as below.- session1
BEGIN; TRUNCATE t;- session2
BEGIN; TRUNCATE t; -- wait- session1
SELECT * FROM pg_get_target_backend_memory_contexts(<pid of session
2>); --waitThanks for notifying me, Fujii-san.
Attached v8 patch that prohibited calling the function inside
transactions.
Regrettably, this modification could not cope with the advisory lock and
I haven't come up with a good way to deal with it.
It seems to me that the architecture of the requestor waiting for the
dumper leads to this problem and complicates things.
Considering the discussion printing backtrace discussion[1]/messages/by-id/CALDaNm3ZzmFS-=r7oDUzj7y7BgQv+N06Kqyft6C3xZDoKnk_6w@mail.gmail.com, it seems
reasonable that the requestor just sends a signal and dumper dumps to
the log file.
Since I found a past discussion that was doing exactly what I thought
reasonable[2]/messages/by-id/20171212044330.3nclev2sfrab36tf@alap3.anarazel.de, I'm going to continue that discussion if there are no
objections.
Any thought?
[1]: /messages/by-id/CALDaNm3ZzmFS-=r7oDUzj7y7BgQv+N06Kqyft6C3xZDoKnk_6w@mail.gmail.com
/messages/by-id/CALDaNm3ZzmFS-=r7oDUzj7y7BgQv+N06Kqyft6C3xZDoKnk_6w@mail.gmail.com
[2]: /messages/by-id/20171212044330.3nclev2sfrab36tf@alap3.anarazel.de
/messages/by-id/20171212044330.3nclev2sfrab36tf@alap3.anarazel.de
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
On 2021/03/04 18:32, torikoshia wrote:
On 2021-01-14 19:11, torikoshia wrote:
Since pg_get_target_backend_memory_contexts() waits to dump memory and
it could lead dead lock as below.- session1
BEGIN; TRUNCATE t;- session2
BEGIN; TRUNCATE t; -- wait- session1
SELECT * FROM pg_get_target_backend_memory_contexts(<pid of session
2>); --waitThanks for notifying me, Fujii-san.
Attached v8 patch that prohibited calling the function inside transactions.
Regrettably, this modification could not cope with the advisory lock and
I haven't come up with a good way to deal with it.It seems to me that the architecture of the requestor waiting for the
dumper leads to this problem and complicates things.Considering the discussion printing backtrace discussion[1], it seems
reasonable that the requestor just sends a signal and dumper dumps to
the log file.
+1
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On 2021-03-05 14:22, Fujii Masao wrote:
On 2021/03/04 18:32, torikoshia wrote:
On 2021-01-14 19:11, torikoshia wrote:
Since pg_get_target_backend_memory_contexts() waits to dump memory
and
it could lead dead lock as below.- session1
BEGIN; TRUNCATE t;- session2
BEGIN; TRUNCATE t; -- wait- session1
SELECT * FROM pg_get_target_backend_memory_contexts(<pid of session
2>); --waitThanks for notifying me, Fujii-san.
Attached v8 patch that prohibited calling the function inside
transactions.Regrettably, this modification could not cope with the advisory lock
and
I haven't come up with a good way to deal with it.It seems to me that the architecture of the requestor waiting for the
dumper leads to this problem and complicates things.Considering the discussion printing backtrace discussion[1], it seems
reasonable that the requestor just sends a signal and dumper dumps to
the log file.+1
Thanks!
I remade the patch and introduced a function
pg_print_backend_memory_contexts(PID) which prints the memory contexts
of
the specified PID to elog.
=# SELECT pg_print_backend_memory_contexts(450855);
** log output **
2021-03-17 15:21:01.942 JST [450855] LOG: Printing memory contexts of
PID 450855
2021-03-17 15:21:01.942 JST [450855] LOG: level: 0 TopMemoryContext:
68720 total in 5 blocks; 16312 free (15 chunks); 52408 used
2021-03-17 15:21:01.942 JST [450855] LOG: level: 1 Prepared Queries:
65536 total in 4 blocks; 35088 free (14 chunks); 30448 used
2021-03-17 15:21:01.942 JST [450855] LOG: level: 1 pgstat
TabStatusArray lookup hash table: 8192 total in 1 blocks; 1408 free (0
chunks); 6784 used
..(snip)..
2021-03-17 15:21:01.942 JST [450855] LOG: level: 2 CachedPlanSource:
4096 total in 3 blocks; 680 free (0 chunks); 3416 used: PREPARE hoge_200
AS SELECT * FROM pgbench_accounts WHERE aid =
1111111111111111111111111111111111111...
2021-03-17 15:21:01.942 JST [450855] LOG: level: 3 CachedPlanQuery:
4096 total in 3 blocks; 464 free (0 chunks); 3632 used
..(snip)..
2021-03-17 15:21:01.945 JST [450855] LOG: level: 1 Timezones: 104128
total in 2 blocks; 2584 free (0 chunks); 101544 used
2021-03-17 15:21:01.945 JST [450855] LOG: level: 1 ErrorContext: 8192
total in 1 blocks; 7928 free (5 chunks); 264 used
2021-03-17 15:21:01.945 JST [450855] LOG: Grand total: 2802080 bytes
in 1399 blocks; 480568 free (178 chunks); 2321512 used
As above, the output is almost the same as MemoryContextStatsPrint()
except for the way of expression of the level.
MemoryContextStatsPrint() uses indents, but
pg_print_backend_memory_contexts() writes it as "level: %d".
Since there was discussion about enlarging StringInfo may cause
errors on OOM[1]/messages/by-id/CAMsr+YGh+sso5N6Q+FmYHLWC=BPCzA+5GbhYZSGruj2d0c7Vvg@mail.gmail.com "r_d/strengthen_perf/print_memcon.md" 110L, 5642C written, this patch calls elog for each context.
As with MemoryContextStatsPrint(), each context shows 100
children at most.
I once thought it should be configurable, but something like
pg_print_backend_memory_contexts(PID, num_children) needs to send
the 'num_children' from requestor to dumper and it seems to require
another infrastructure.
Creating a new GUC for this seems overkill.
If MemoryContextStatsPrint(), i.e. showing 100 children at most is
enough, this hard limit may be acceptable.
Only superusers can call pg_print_backend_memory_contexts().
I'm going to add documentation and regression tests.
Any thoughts?
[1]: /messages/by-id/CAMsr+YGh+sso5N6Q+FmYHLWC=BPCzA+5GbhYZSGruj2d0c7Vvg@mail.gmail.com "r_d/strengthen_perf/print_memcon.md" 110L, 5642C written
/messages/by-id/CAMsr+YGh+sso5N6Q+FmYHLWC=BPCzA+5GbhYZSGruj2d0c7Vvg@mail.gmail.com
"r_d/strengthen_perf/print_memcon.md" 110L, 5642C written
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
v1-0001-Add-memorycontext-elog-print.patchtext/x-diff; name=v1-0001-Add-memorycontext-elog-print.patchDownload
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index c6a8d4611e..e116f4a1be 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -440,6 +441,20 @@ HandleProcSignalBarrierInterrupt(void)
/* latch will be set by procsignal_sigusr1_handler */
}
+/*
+ * HandleProcSignalPrintMemoryContext
+ *
+ * Handle receipt of an interrupt indicating print memory context.
+ * Signal handler portion of interrupt handling.
+ */
+static void
+HandleProcSignalPrintMemoryContext(void)
+{
+ InterruptPending = true;
+ PrintMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
/*
* Perform global barrier related interrupt checking.
*
@@ -580,6 +595,25 @@ ProcessProcSignalBarrier(void)
ConditionVariableBroadcast(&MyProcSignalSlot->pss_barrierCV);
}
+/*
+ * ProcessPrintMemoryContextInterrupt
+ * The portion of print memory context interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessPrintMemoryContextInterrupt(void)
+{
+ PrintMemoryContextPending = false;
+
+ ereport(LOG,
+ (errmsg("Printing memory contexts of PID %d", MyProcPid)));
+
+ /* A hard-wired limit on the number of children is usually good enough */
+ MemoryContextStatsDetail(TopMemoryContext, 100, false);
+
+}
+
+
/*
* If it turns out that we couldn't absorb one or more barrier types, either
* because the barrier-processing functions returned false or due to an error,
@@ -675,6 +709,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_BUFFERPIN))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_BUFFERPIN);
+ if (CheckProcSignal(PROCSIG_PRINT_MEMORY_CONTEXT))
+ HandleProcSignalPrintMemoryContext();
+
SetLatch(MyLatch);
errno = save_errno;
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2b1b68109f..6fe47d1229 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3295,6 +3295,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (PrintMemoryContextPending)
+ ProcessPrintMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..07a95dd115 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -61,7 +61,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 73e0a672ae..32fe971386 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -34,6 +34,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t PrintMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index ec6c130d0f..8e7fe09a0f 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool is_dst_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 2b90034764..b286308aa4 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool is_dst_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 84472b9158..9ec0c0509d 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -55,9 +59,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool is_dst_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -487,6 +493,55 @@ MemoryContextMemAllocated(MemoryContext context, bool recurse)
return total;
}
+/*
+ * pg_print_backend_memory_contexts
+ * Print memory context of the specified backend process.
+ */
+Datum
+pg_print_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ PGPROC *proc = BackendPidGetProc(pid);
+
+ /*
+ * Check whether the target process is PostgreSQL backend process.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * According to the thread below, it might be better to
+ * change this message.
+ *
+ * https://www.postgresql.org/message-id/CALj2ACW7Rr-R7mBcBQiXWPp%3DJV5chajjTdudLiF5YcpW-BmHhg%40mail.gmail.com
+ *
+ * Until the conclusion of the discussion, output the same
+ * message as pg_cancel_backend().
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+
+ PG_RETURN_BOOL(false);
+ }
+
+ /* Only allow superusers to signal superuser-owned backends. */
+ if (superuser_arg(proc->roleId) && !superuser())
+ {
+ ereport(WARNING,
+ (errmsg("must be a superuser to print memory contexts")));
+ PG_RETURN_BOOL(false);
+ }
+
+ if(!SendProcSignal(pid, PROCSIG_PRINT_MEMORY_CONTEXT, InvalidBackendId))
+ PG_RETURN_BOOL(true);
+ else
+ {
+ ereport(WARNING,
+ (errmsg("failed to send signal: %m")));
+
+ PG_RETURN_BOOL(false);
+ }
+}
+
/*
* MemoryContextStats
* Print statistics about the named context and all its descendants.
@@ -499,7 +554,7 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);
+ MemoryContextStatsDetail(context, 100, true);
}
/*
@@ -508,19 +563,28 @@ MemoryContextStats(MemoryContext context)
* Entry point for use if you want to vary the number of child contexts shown.
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)
+MemoryContextStatsDetail(MemoryContext context, int max_children, bool is_dst_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);
+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, is_dst_stderr);
- fprintf(stderr,
- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
- grand_totals.totalspace, grand_totals.nblocks,
- grand_totals.freespace, grand_totals.freechunks,
- grand_totals.totalspace - grand_totals.freespace);
+ if (is_dst_stderr)
+ fprintf(stderr,
+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace);
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +597,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool is_dst_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,7 +610,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);
+ totals, is_dst_stderr);
/*
* Examine children. If there are more than max_children of them, we do
@@ -560,11 +625,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
if (ichild < max_children)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);
+ totals,
+ is_dst_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);
+ &local_totals,
+ is_dst_stderr);
}
/* Deal with excess children */
@@ -574,16 +641,31 @@ MemoryContextStatsInternal(MemoryContext context, int level,
{
int i;
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
+ if (is_dst_stderr)
+ {
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,12 +687,14 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)
+ const char *stats_string,
+ bool is_dst_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
int i;
+ char truncated_ident[110];
/*
* It seems preferable to label dynahash contexts with just the hash table
@@ -623,11 +707,11 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)
- fprintf(stderr, " ");
- fprintf(stderr, "%s: %s", name, stats_string);
+ truncated_ident[0] = '\0';
+
if (ident)
{
+ char delimiter[] = ": ";
/*
* Some contexts may have very long identifiers (e.g., SQL queries).
* Arbitrarily truncate at 100 bytes, but be careful not to break
@@ -637,24 +721,42 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcat(truncated_ident, delimiter);
+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");
+
+ i = strlen(delimiter);
+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);
+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';
+
if (truncated)
- fprintf(stderr, "...");
+ strcat(truncated_ident, "...");
}
- fputc('\n', stderr);
+
+ if (is_dst_stderr)
+ {
+ for (i = 0; i < level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d %s: %s%s",
+ level, name, stats_string, truncated_ident)));
}
/*
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 9213be7c95..479b08bedd 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool is_dst_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..0091806809 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7917,6 +7917,12 @@
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# print memory context of specified backend
+{ oid => '4543', descr => 'print memory contexts of specified backend',
+ proname => 'pg_print_backend_memory_contexts',
+ provolatile => 'v', prorettype => 'bool',
+ proargtypes => 'int4', prosrc => 'pg_print_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 013850ac28..711d6c2089 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t PrintMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index 9331ef80fd..87a8501c37 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool is_dst_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..0bfe290477 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,8 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_PRINT_MEMORY_CONTEXT, /* ask specified backend to print the
+ memory context */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
@@ -69,6 +71,7 @@ extern int SendProcSignal(pid_t pid, ProcSignalReason reason,
extern uint64 EmitProcSignalBarrier(ProcSignalBarrierType type);
extern void WaitForProcSignalBarrier(uint64 generation);
extern void ProcessProcSignalBarrier(void);
+extern void ProcessPrintMemoryContextInterrupt(void);
extern void procsignal_sigusr1_handler(SIGNAL_ARGS);
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 36aae4e51c..fd10f80eda 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -84,7 +84,8 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);
+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool is_dst_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
On 2021/03/17 22:24, torikoshia wrote:
I remade the patch and introduced a function
pg_print_backend_memory_contexts(PID) which prints the memory contexts of
the specified PID to elog.
Thanks for the patch!
=# SELECT pg_print_backend_memory_contexts(450855);
** log output **
2021-03-17 15:21:01.942 JST [450855] LOG: Printing memory contexts of PID 450855
2021-03-17 15:21:01.942 JST [450855] LOG: level: 0 TopMemoryContext: 68720 total in 5 blocks; 16312 free (15 chunks); 52408 used
2021-03-17 15:21:01.942 JST [450855] LOG: level: 1 Prepared Queries: 65536 total in 4 blocks; 35088 free (14 chunks); 30448 used
2021-03-17 15:21:01.942 JST [450855] LOG: level: 1 pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1408 free (0 chunks); 6784 used
..(snip)..
2021-03-17 15:21:01.942 JST [450855] LOG: level: 2 CachedPlanSource: 4096 total in 3 blocks; 680 free (0 chunks); 3416 used: PREPARE hoge_200 AS SELECT * FROM pgbench_accounts WHERE aid = 1111111111111111111111111111111111111...
2021-03-17 15:21:01.942 JST [450855] LOG: level: 3 CachedPlanQuery: 4096 total in 3 blocks; 464 free (0 chunks); 3632 used
..(snip)..
2021-03-17 15:21:01.945 JST [450855] LOG: level: 1 Timezones: 104128 total in 2 blocks; 2584 free (0 chunks); 101544 used
2021-03-17 15:21:01.945 JST [450855] LOG: level: 1 ErrorContext: 8192 total in 1 blocks; 7928 free (5 chunks); 264 used
2021-03-17 15:21:01.945 JST [450855] LOG: Grand total: 2802080 bytes in 1399 blocks; 480568 free (178 chunks); 2321512 usedAs above, the output is almost the same as MemoryContextStatsPrint()
except for the way of expression of the level.
MemoryContextStatsPrint() uses indents, but
pg_print_backend_memory_contexts() writes it as "level: %d".
This format looks better to me.
Since there was discussion about enlarging StringInfo may cause
errors on OOM[1], this patch calls elog for each context.As with MemoryContextStatsPrint(), each context shows 100
children at most.
I once thought it should be configurable, but something like
pg_print_backend_memory_contexts(PID, num_children) needs to send
the 'num_children' from requestor to dumper and it seems to require
another infrastructure.
Creating a new GUC for this seems overkill.
If MemoryContextStatsPrint(), i.e. showing 100 children at most is
enough, this hard limit may be acceptable.
Can't this number be passed via shared memory?
Only superusers can call pg_print_backend_memory_contexts().
+ /* Only allow superusers to signal superuser-owned backends. */
+ if (superuser_arg(proc->roleId) && !superuser())
The patch seems to allow even non-superuser to request to print the memory
contexts if the target backend is owned by non-superuser. Is this intentional?
I think that only superuser should be allowed to execute
pg_print_backend_memory_contexts() whoever owns the target backend.
Because that function can cause lots of log messages.
I'm going to add documentation and regression tests.
Thanks!
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On 2021-03-18 15:09, Fujii Masao wrote:
Thanks for your comments!
On 2021/03/17 22:24, torikoshia wrote:
I remade the patch and introduced a function
pg_print_backend_memory_contexts(PID) which prints the memory contexts
of
the specified PID to elog.Thanks for the patch!
=# SELECT pg_print_backend_memory_contexts(450855);
** log output **
2021-03-17 15:21:01.942 JST [450855] LOG: Printing memory contexts
of PID 450855
2021-03-17 15:21:01.942 JST [450855] LOG: level: 0
TopMemoryContext: 68720 total in 5 blocks; 16312 free (15 chunks);
52408 used
2021-03-17 15:21:01.942 JST [450855] LOG: level: 1 Prepared
Queries: 65536 total in 4 blocks; 35088 free (14 chunks); 30448 used
2021-03-17 15:21:01.942 JST [450855] LOG: level: 1 pgstat
TabStatusArray lookup hash table: 8192 total in 1 blocks; 1408 free (0
chunks); 6784 used
..(snip)..
2021-03-17 15:21:01.942 JST [450855] LOG: level: 2
CachedPlanSource: 4096 total in 3 blocks; 680 free (0 chunks); 3416
used: PREPARE hoge_200 AS SELECT * FROM pgbench_accounts WHERE aid =
1111111111111111111111111111111111111...
2021-03-17 15:21:01.942 JST [450855] LOG: level: 3
CachedPlanQuery: 4096 total in 3 blocks; 464 free (0 chunks); 3632
used
..(snip)..
2021-03-17 15:21:01.945 JST [450855] LOG: level: 1 Timezones:
104128 total in 2 blocks; 2584 free (0 chunks); 101544 used
2021-03-17 15:21:01.945 JST [450855] LOG: level: 1 ErrorContext:
8192 total in 1 blocks; 7928 free (5 chunks); 264 used
2021-03-17 15:21:01.945 JST [450855] LOG: Grand total: 2802080
bytes in 1399 blocks; 480568 free (178 chunks); 2321512 usedAs above, the output is almost the same as MemoryContextStatsPrint()
except for the way of expression of the level.
MemoryContextStatsPrint() uses indents, but
pg_print_backend_memory_contexts() writes it as "level: %d".This format looks better to me.
Since there was discussion about enlarging StringInfo may cause
errors on OOM[1], this patch calls elog for each context.As with MemoryContextStatsPrint(), each context shows 100
children at most.
I once thought it should be configurable, but something like
pg_print_backend_memory_contexts(PID, num_children) needs to send
the 'num_children' from requestor to dumper and it seems to require
another infrastructure.
Creating a new GUC for this seems overkill.
If MemoryContextStatsPrint(), i.e. showing 100 children at most is
enough, this hard limit may be acceptable.Can't this number be passed via shared memory?
The attached patch uses static shared memory to pass the number.
As documented, the current implementation allows that when multiple
pg_print_backend_memory_contexts() called in succession or
simultaneously, max_children can be the one of another
pg_print_backend_memory_contexts().
I had tried to avoid this by adding some state information and using
before_shmem_exit() in case of process termination for cleaning up the
state information as in the patch I presented earlier, but since kill()
returns success before the dumper called signal handler, it seemed
there were times when we couldn't clean up the state.
Since this happens only when multiple pg_print_backend_memory_contexts()
are called and their specified number of children are different, and the
effect is just the not intended number of children to print, it might be
acceptable.
Or it might be better to wait for some seconds if num_chilren on shared
memory is not the initialized value(meaning some other process is
requesting to print memory contexts).
Only superusers can call pg_print_backend_memory_contexts().
+ /* Only allow superusers to signal superuser-owned backends. */ + if (superuser_arg(proc->roleId) && !superuser())The patch seems to allow even non-superuser to request to print the
memory
contexts if the target backend is owned by non-superuser. Is this
intentional?
I think that only superuser should be allowed to execute
pg_print_backend_memory_contexts() whoever owns the target backend.
Because that function can cause lots of log messages.
Thanks, it's not intentional, modified it.
I'm going to add documentation and regression tests.
Added them.
Regards,
Attachments:
v2-0001-add-memorycontext-elog-print.patchtext/x-diff; name=v2-0001-add-memorycontext-elog-print.patchDownload
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..e834b923e4 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24781,6 +24781,33 @@ SELECT collation for ('foo' COLLATE "de_DE");
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_print_backend_memory_contexts</primary>
+ </indexterm>
+ <function>pg_print_backend_memory_contexts</function> (
+ <parameter>pid</parameter> <type>integer</type>,
+ <parameter>max_children</parameter> <type>integer</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Prints the memory contexts whose backend process has the specified
+ process ID.
+ <parameter>max_children</parameter> limits the max number of children
+ to print per one parent context.
+ Note that when multiple
+ <function>pg_print_backend_memory_contexts</function> called in
+ succession or simultaneously, <parameter>max_children</parameter> can
+ be the one of another
+ <function>pg_print_backend_memory_contexts</function>.
+ Backtrace will be printed based on the log configuration set. See
+ <xref linkend="runtime-config-logging"/> for more information.
+ Only superusers can call this function even when the specified process
+ is non-superuser backend.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 3e4ec53a97..785374719f 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -46,6 +46,7 @@
#include "storage/sinvaladt.h"
#include "storage/spin.h"
#include "utils/snapmgr.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -269,6 +270,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ McxtPrintShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index c6a8d4611e..dabb2025b1 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -440,6 +441,20 @@ HandleProcSignalBarrierInterrupt(void)
/* latch will be set by procsignal_sigusr1_handler */
}
+/*
+ * HandleProcSignalPrintMemoryContext
+ *
+ * Handle receipt of an interrupt indicating print memory context.
+ * Signal handler portion of interrupt handling.
+ */
+static void
+HandleProcSignalPrintMemoryContext(void)
+{
+ InterruptPending = true;
+ PrintMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
/*
* Perform global barrier related interrupt checking.
*
@@ -580,6 +595,29 @@ ProcessProcSignalBarrier(void)
ConditionVariableBroadcast(&MyProcSignalSlot->pss_barrierCV);
}
+/*
+ * ProcessPrintMemoryContextInterrupt
+ * The portion of print memory context interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessPrintMemoryContextInterrupt(void)
+{
+ int max_children;
+
+ PrintMemoryContextPending = false;
+
+ LWLockAcquire(McxtPrintLock, LW_SHARED);
+ max_children = *MaxChildrenPerContext;
+ LWLockRelease(McxtPrintLock);
+
+ ereport(LOG,
+ (errmsg("Printing memory contexts of PID %d. Max number of children per context is %d", MyProcPid, max_children)));
+
+ MemoryContextStatsDetail(TopMemoryContext, max_children, false);
+}
+
+
/*
* If it turns out that we couldn't absorb one or more barrier types, either
* because the barrier-processing functions returned false or due to an error,
@@ -675,6 +713,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_BUFFERPIN))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_BUFFERPIN);
+ if (CheckProcSignal(PROCSIG_PRINT_MEMORY_CONTEXT))
+ HandleProcSignalPrintMemoryContext();
+
SetLatch(MyLatch);
errno = save_errno;
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 6c7cf6c295..f2aa1852b1 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+McxtPrintLock 48
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2b1b68109f..6fe47d1229 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3295,6 +3295,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (PrintMemoryContextPending)
+ ProcessPrintMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..07a95dd115 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -61,7 +61,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 73e0a672ae..32fe971386 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -34,6 +34,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t PrintMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index ec6c130d0f..8e7fe09a0f 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool is_dst_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 2b90034764..b286308aa4 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool is_dst_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 84472b9158..5389cf4507 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -37,6 +41,12 @@
*/
MemoryContext CurrentMemoryContext = NULL;
+/*
+ * MaxChildrenPerContext
+ * Max number of children to print per one parent context.
+ */
+int *MaxChildrenPerContext = NULL;
+
/*
* Standard top-level contexts. For a description of the purpose of each
* of these contexts, refer to src/backend/utils/mmgr/README
@@ -55,9 +65,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool is_dst_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -487,6 +499,89 @@ MemoryContextMemAllocated(MemoryContext context, bool recurse)
return total;
}
+/*
+ * McxtPrintShmemInit
+ * Initialize shared memory area
+ */
+void
+McxtPrintShmemInit(void)
+{
+ bool found;
+
+ MaxChildrenPerContext = (int *)
+ ShmemInitStruct("Max number of children per context",
+ sizeof(int),
+ &found);
+ if (!found)
+ {
+ *MaxChildrenPerContext = 0;
+ }
+}
+
+/*
+ * pg_print_backend_memory_contexts
+ * Print memory contexts of the specified backend process.
+ */
+Datum
+pg_print_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ int max_children = PG_GETARG_INT32(1);
+
+ PGPROC *proc = BackendPidGetProc(pid);
+
+ if(max_children <= 0)
+ {
+ ereport(WARNING,
+ (errmsg("%d is invalid value", max_children),
+ errhint("second parameter is the number of context and it must be set to a value greater than or equal to 1")));
+
+ PG_RETURN_BOOL(false);
+ }
+
+ /*
+ * Check whether the target process is PostgreSQL backend process.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * According to the thread below, it might be better to
+ * change this message.
+ *
+ * https://www.postgresql.org/message-id/CALj2ACW7Rr-R7mBcBQiXWPp%3DJV5chajjTdudLiF5YcpW-BmHhg%40mail.gmail.com
+ *
+ * Until the conclusion of the discussion, output the same
+ * message as pg_cancel_backend().
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+
+ PG_RETURN_BOOL(false);
+ }
+
+ /* Only allow superusers to print memory contexts. */
+ if (!superuser())
+ {
+ ereport(WARNING,
+ (errmsg("must be a superuser to print memory contexts")));
+ PG_RETURN_BOOL(false);
+ }
+
+ LWLockAcquire(McxtPrintLock, LW_EXCLUSIVE);
+ *MaxChildrenPerContext = max_children;
+ LWLockRelease(McxtPrintLock);
+
+ if(!SendProcSignal(pid, PROCSIG_PRINT_MEMORY_CONTEXT, InvalidBackendId))
+ PG_RETURN_BOOL(true);
+ else
+ {
+ ereport(WARNING,
+ (errmsg("failed to send signal: %m")));
+
+ PG_RETURN_BOOL(false);
+ }
+}
+
/*
* MemoryContextStats
* Print statistics about the named context and all its descendants.
@@ -499,7 +594,7 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);
+ MemoryContextStatsDetail(context, 100, true);
}
/*
@@ -508,19 +603,28 @@ MemoryContextStats(MemoryContext context)
* Entry point for use if you want to vary the number of child contexts shown.
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)
+MemoryContextStatsDetail(MemoryContext context, int max_children, bool is_dst_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);
+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, is_dst_stderr);
- fprintf(stderr,
- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
- grand_totals.totalspace, grand_totals.nblocks,
- grand_totals.freespace, grand_totals.freechunks,
- grand_totals.totalspace - grand_totals.freespace);
+ if (is_dst_stderr)
+ fprintf(stderr,
+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace);
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +637,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool is_dst_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,7 +650,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);
+ totals, is_dst_stderr);
/*
* Examine children. If there are more than max_children of them, we do
@@ -560,11 +665,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
if (ichild < max_children)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);
+ totals,
+ is_dst_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);
+ &local_totals,
+ is_dst_stderr);
}
/* Deal with excess children */
@@ -574,16 +681,31 @@ MemoryContextStatsInternal(MemoryContext context, int level,
{
int i;
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
+ if (is_dst_stderr)
+ {
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,12 +727,14 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)
+ const char *stats_string,
+ bool is_dst_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
int i;
+ char truncated_ident[110];
/*
* It seems preferable to label dynahash contexts with just the hash table
@@ -623,11 +747,11 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)
- fprintf(stderr, " ");
- fprintf(stderr, "%s: %s", name, stats_string);
+ truncated_ident[0] = '\0';
+
if (ident)
{
+ char delimiter[] = ": ";
/*
* Some contexts may have very long identifiers (e.g., SQL queries).
* Arbitrarily truncate at 100 bytes, but be careful not to break
@@ -637,24 +761,42 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcat(truncated_ident, delimiter);
+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");
+
+ i = strlen(delimiter);
+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);
+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';
+
if (truncated)
- fprintf(stderr, "...");
+ strcat(truncated_ident, "...");
+ }
+
+ if (is_dst_stderr)
+ {
+ for (i = 0; i < level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
}
- fputc('\n', stderr);
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d %s: %s%s",
+ level, name, stats_string, truncated_ident)));
}
/*
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 9213be7c95..479b08bedd 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool is_dst_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..ff6a9d2186 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7917,6 +7917,12 @@
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# print memory context of specified backend
+{ oid => '4543', descr => 'print memory contexts of specified backend',
+ proname => 'pg_print_backend_memory_contexts',
+ provolatile => 'v', prorettype => 'bool',
+ proargtypes => 'int4 int4', prosrc => 'pg_print_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 013850ac28..711d6c2089 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t PrintMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index 9331ef80fd..87a8501c37 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool is_dst_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..0bfe290477 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,8 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_PRINT_MEMORY_CONTEXT, /* ask specified backend to print the
+ memory context */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
@@ -69,6 +71,7 @@ extern int SendProcSignal(pid_t pid, ProcSignalReason reason,
extern uint64 EmitProcSignalBarrier(ProcSignalBarrierType type);
extern void WaitForProcSignalBarrier(uint64 generation);
extern void ProcessProcSignalBarrier(void);
+extern void ProcessPrintMemoryContextInterrupt(void);
extern void procsignal_sigusr1_handler(SIGNAL_ARGS);
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 36aae4e51c..db638c2f3f 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -63,6 +63,9 @@ extern PGDLLIMPORT MemoryContext CurTransactionContext;
/* This is a transient link to the active portal's memory context: */
extern PGDLLIMPORT MemoryContext PortalContext;
+
+extern int *MaxChildrenPerContext;
+
/* Backwards compatibility macro */
#define MemoryContextResetAndDeleteChildren(ctx) MemoryContextReset(ctx)
@@ -84,9 +87,11 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);
+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool is_dst_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
+extern void McxtPrintShmemInit(void);
#ifdef MEMORY_CONTEXT_CHECKING
extern void MemoryContextCheck(MemoryContext context);
diff --git a/src/test/modules/test_misc/t/002_print_memory_context_validation.pl b/src/test/modules/test_misc/t/002_print_memory_context_validation.pl
new file mode 100644
index 0000000000..61e318806f
--- /dev/null
+++ b/src/test/modules/test_misc/t/002_print_memory_context_validation.pl
@@ -0,0 +1,89 @@
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# Set up node with logging collector
+my $node = get_new_node('primary');
+$node->init();
+$node->append_conf(
+ 'postgresql.conf', qq{
+ logging_collector = on
+ lc_messages = 'C'
+ });
+
+$node->start();
+
+# Verify that log output gets to the file
+$node->psql('postgres', 'select pg_print_backend_memory_contexts(pg_backend_pid(), 100)');
+
+# might need to retry if logging collector process is slow...
+my $max_attempts = 180 * 10;
+
+my $current_logfiles;
+for (my $attempts = 0; $attempts < $max_attempts; $attempts++)
+{
+ eval {
+ $current_logfiles = slurp_file($node->data_dir . '/current_logfiles');
+ };
+ last unless $@;
+ usleep(100_000);
+}
+die $@ if $@;
+
+note "current_logfiles = $current_logfiles";
+
+like(
+ $current_logfiles,
+ qr|^stderr log/postgresql-.*log$|,
+ 'current_logfiles is sane');
+
+my $lfname = $current_logfiles;
+$lfname =~ s/^stderr //;
+chomp $lfname;
+
+my $logfile;
+my $print_count;
+
+# Verify that the backtraces of the processes are logged into logfile.
+for (my $attempts = 0; $attempts < $max_attempts; $attempts++)
+{
+ $logfile = $node->data_dir . '/' . $lfname;
+ chomp $logfile;
+ print "file is $logfile";
+ open my $fh, '<', $logfile
+ or die "Could not open '$logfile' $!";
+ while (my $line = <$fh>)
+ {
+ chomp $line;
+ if ($line =~ m/Grand total: \d+ bytes in \d+ blocks/)
+ {
+ $print_count++;
+ }
+ }
+ last if $print_count == 1;
+ usleep(100_000);
+}
+
+is($print_count, 1, 'found expected memory context print in the log file');
+
+my $output;
+
+$node->psql('postgres', 'select pg_print_backend_memory_contexts(pg_backend_pid(), 0)',
+ stderr => \$output);
+
+like(
+ $output,
+ qr/0 is invalid value/,
+ 'max_children must be set to a value greater than or equal to 1');
+
+$node->psql('postgres', 'select pg_print_backend_memory_contexts(1, 10)',
+ stderr => \$output);
+
+like(
+ $output,
+ qr/PID 1 is not a PostgreSQL server process/,
+ 'target PID is not PostgreSQL server process');
+
+$node->stop('fast');
At Mon, 22 Mar 2021 15:09:58 +0900, torikoshia <torikoshia@oss.nttdata.com> wrote in
If MemoryContextStatsPrint(), i.e. showing 100 children at most is
enough, this hard limit may be acceptable.Can't this number be passed via shared memory?
The attached patch uses static shared memory to pass the number.
"pg_print_backend_memory_contexts"
That name looks like as if it returns the result as text when used on
command-line. We could have pg_get_backend_memory_context(bool
dump_to_log (or where to dump), int limit). Or couldn't we name it
differently even in the ase we add a separate function?
+/*
+ * MaxChildrenPerContext
+ * Max number of children to print per one parent context.
+ */
+int *MaxChildrenPerContext = NULL;
Perhaps it'd be better to have a struct even if it consists only of
one member. (Aligned) C-int values are atomic so we can omit the
McxtPrintLock. (I don't think it's a problem even if it is modifed
while reading^^:)
+ if(max_children <= 0)
+ {
+ ereport(WARNING,
+ (errmsg("%d is invalid value", max_children),
+ errhint("second parameter is the number of context and it must be set to a value greater than or equal to 1")));
It's annoying to choose a number large enough when I want to dump
children unlimitedly. Couldn't we use 0 to specify "unlimited"?
+ (errmsg("%d is invalid value", max_children),
+ errhint("second parameter is the number of context and it must be set to a value greater than or equal to 1")));
For the main message, (I think) we usually spell the "%d is invalid
value" as "maximum number of children must be positive" or such. For
the hint, we don't need a copy of the primary section of the
documentation here.
I think we should ERROR out for invalid parameters, at least for
max_children. I'm not sure about pid since we might call it based on
pg_stat_activity..
+ if(!SendProcSignal(pid, PROCSIG_PRINT_MEMORY_CONTEXT, InvalidBackendId))
We know the backendid of the process here.
+ if (is_dst_stderr)
+ {
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
The fprintf path is used nowhere in the patch at all. It can be used
while attaching debugger but I'm not sure we need that code. The
footprint of this patch is largely shrinked by removing it.
+ strcat(truncated_ident, delimiter);
strcpy is sufficient here. And we don't need the delimiter to be a
variable. (we can copy a string literal into truncate_ident, then
count the length of truncate_ident, instead of the delimiter
variable.)
+ $current_logfiles = slurp_file($node->data_dir . '/current_logfiles');
...
+my $lfname = $current_logfiles;
+$lfname =~ s/^stderr //;
+chomp $lfname;
$node->logfile is the current log file name.
+ 'target PID is not PostgreSQL server process');
Maybe "check if PID check is working" or such? And, we can do
something like the following to exercise in a more practical way.
select pg_print_backend...(pid,) from pg_stat_activity where backend_type = 'checkpointer';
As documented, the current implementation allows that when multiple
pg_print_backend_memory_contexts() called in succession or
simultaneously, max_children can be the one of another
pg_print_backend_memory_contexts().
I had tried to avoid this by adding some state information and using
before_shmem_exit() in case of process termination for cleaning up the
state information as in the patch I presented earlier, but since
kill()
returns success before the dumper called signal handler, it seemed
there were times when we couldn't clean up the state.
Since this happens only when multiple
pg_print_backend_memory_contexts()
are called and their specified number of children are different, and
the
effect is just the not intended number of children to print, it might
be
acceptable.
I see it as a non-issue. Even though the behavior looks somewhat
strange, that usage is stranger than the behavior.
Or it might be better to wait for some seconds if num_chilren on
shared
memory is not the initialized value(meaning some other process is
requesting to print memory contexts).Only superusers can call pg_print_backend_memory_contexts().
+ /* Only allow superusers to signal superuser-owned backends. */ + if (superuser_arg(proc->roleId) && !superuser()) The patch seems to allow even non-superuser to request to print the memory contexts if the target backend is owned by non-superuser. Is this intentional? I think that only superuser should be allowed to execute pg_print_backend_memory_contexts() whoever owns the target backend. Because that function can cause lots of log messages.Thanks, it's not intentional, modified it.
By the way we can send a procsig to other than a client backend. And
most of the postgres processes are using the standard signal handler
and intializes the procsig facility. So some of such kind of processes
can dump the memory context information as-is. Otherwise we can add
CHECK_FOR_INTERRUPT to appropriate place to allow that. I'm not sure
how it is useful for other kind of processes, but it might be useful
for autovacuum workers.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On 2021-03-23 17:24, Kyotaro Horiguchi wrote:
Thanks for reviewing and suggestions!
At Mon, 22 Mar 2021 15:09:58 +0900, torikoshia
<torikoshia@oss.nttdata.com> wrote inIf MemoryContextStatsPrint(), i.e. showing 100 children at most is
enough, this hard limit may be acceptable.Can't this number be passed via shared memory?
The attached patch uses static shared memory to pass the number.
"pg_print_backend_memory_contexts"
That name looks like as if it returns the result as text when used on
command-line. We could have pg_get_backend_memory_context(bool
dump_to_log (or where to dump), int limit). Or couldn't we name it
differently even in the ase we add a separate function?
Redefined pg_get_backend_memory_contexts() as
pg_get_backend_memory_contexts(pid, int max_children).
When pid equals 0, pg_get_backend_memory_contexts() prints local memory
contexts as original pg_get_backend_memory_contexts() does.
In this case, 'max_children' is ignored.
When 'pid' does not equal 0 and it is the PID of the client backend,
memory contexts are logged through elog().
+/* + * MaxChildrenPerContext + * Max number of children to print per one parent context. + */ +int *MaxChildrenPerContext = NULL;Perhaps it'd be better to have a struct even if it consists only of
one member. (Aligned) C-int values are atomic so we can omit the
McxtPrintLock. (I don't think it's a problem even if it is modifed
while reading^^:)
Fixed them.
+ if(max_children <= 0) + { + ereport(WARNING, + (errmsg("%d is invalid value", max_children), + errhint("second parameter is the number of context and it must be set to a value greater than or equal to 1")));It's annoying to choose a number large enough when I want to dump
children unlimitedly. Couldn't we use 0 to specify "unlimited"?
Modified as you suggested.
+ (errmsg("%d is invalid value", max_children), + errhint("second parameter is the number of context and it must be set to a value greater than or equal to 1")));For the main message, (I think) we usually spell the "%d is invalid
value" as "maximum number of children must be positive" or such. For
the hint, we don't need a copy of the primary section of the
documentation here.
Modified it to "The maximum number of children must be greater than 0".
I think we should ERROR out for invalid parameters, at least for
max_children. I'm not sure about pid since we might call it based on
pg_stat_activity..
Changed to ERROR out when the 'max_children' is less than 0.
Regarding pid, I left it untouched considering the consistency with
other
signal sending functions such as pg_cancel_backend().
+ if(!SendProcSignal(pid, PROCSIG_PRINT_MEMORY_CONTEXT,
InvalidBackendId))We know the backendid of the process here.
Added it.
+ if (is_dst_stderr) + { + for (i = 0; i <= level; i++) + fprintf(stderr, " ");The fprintf path is used nowhere in the patch at all. It can be used
while attaching debugger but I'm not sure we need that code. The
footprint of this patch is largely shrinked by removing it.
According to the past discussion[1]/messages/by-id/906.1513707472@sss.pgh.pa.us, people wanted MemoryContextStats
as it was, so I think it's better that MemoryContextStats can be used
as before.
+ strcat(truncated_ident, delimiter);
strcpy is sufficient here. And we don't need the delimiter to be a
variable. (we can copy a string literal into truncate_ident, then
count the length of truncate_ident, instead of the delimiter
variable.)
True.
+ $current_logfiles = slurp_file($node->data_dir . '/current_logfiles'); ... +my $lfname = $current_logfiles; +$lfname =~ s/^stderr //; +chomp $lfname;$node->logfile is the current log file name.
+ 'target PID is not PostgreSQL server process');
Maybe "check if PID check is working" or such? And, we can do
something like the following to exercise in a more practical way.select pg_print_backend...(pid,) from pg_stat_activity where
backend_type = 'checkpointer';
It seems better.
As documented, the current implementation allows that when multiple
pg_print_backend_memory_contexts() called in succession or
simultaneously, max_children can be the one of another
pg_print_backend_memory_contexts().
I had tried to avoid this by adding some state information and using
before_shmem_exit() in case of process termination for cleaning up the
state information as in the patch I presented earlier, but since
kill()
returns success before the dumper called signal handler, it seemed
there were times when we couldn't clean up the state.
Since this happens only when multiple
pg_print_backend_memory_contexts()
are called and their specified number of children are different, and
the
effect is just the not intended number of children to print, it might
be
acceptable.I see it as a non-issue. Even though the behavior looks somewhat
strange, that usage is stranger than the behavior.
Thanks for your comments!
Or it might be better to wait for some seconds if num_chilren on
shared
memory is not the initialized value(meaning some other process is
requesting to print memory contexts).Only superusers can call pg_print_backend_memory_contexts().
+ /* Only allow superusers to signal superuser-owned backends. */ + if (superuser_arg(proc->roleId) && !superuser()) The patch seems to allow even non-superuser to request to print the memory contexts if the target backend is owned by non-superuser. Is this intentional? I think that only superuser should be allowed to execute pg_print_backend_memory_contexts() whoever owns the target backend. Because that function can cause lots of log messages.Thanks, it's not intentional, modified it.
By the way we can send a procsig to other than a client backend. And
most of the postgres processes are using the standard signal handler
and intializes the procsig facility. So some of such kind of processes
can dump the memory context information as-is. Otherwise we can add
CHECK_FOR_INTERRUPT to appropriate place to allow that. I'm not sure
how it is useful for other kind of processes, but it might be useful
for autovacuum workers.
Yeah, I also think it's possible to get memory contexts other than the
client backend.
But I'm not sure whether people want to do it...
[1]: /messages/by-id/906.1513707472@sss.pgh.pa.us
regards.
Attachments:
v3-0001-add-memorycontext-elog-print.patchtext/x-diff; name=v3-0001-add-memorycontext-elog-print.patchDownload
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 1d3429fbd9..a4017a0760 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24821,6 +24821,37 @@ SELECT collation for ('foo' COLLATE "de_DE");
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_backend_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_backend_memory_contexts</function> (
+ <parameter>pid</parameter> <type>integer</type>,
+ <parameter>max_children</parameter> <type>integer</type> )
+ <returnvalue>setof record</returnvalue>
+ </para>
+ <para>
+ Get memory contexts whose backend process has the specified process ID.
+ <parameter>max_children</parameter> limits the max number of children
+ to print per one parent context. 0 means unlimited.
+ When <parameter>pid</parameter> equals 0,
+ <function>pg_get_backend_memory_contexts</function> displays all
+ the memory contexts of the local process regardless of
+ <parameter>max_children</parameter>.
+ When <parameter>pid</parameter> does not equal 0,
+ memory contexts will be printed based on the log configuration set.
+ See <xref linkend="runtime-config-logging"/> for more information.
+ Only superusers can call this function even when the specified process
+ is non-superuser backend.
+ Note that when multiple
+ <function>pg_get_backend_memory_contexts</function> called in
+ succession or simultaneously, <parameter>max_children</parameter> can
+ be the one of another
+ <function>pg_get_backend_memory_contexts</function>.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 0dca65dc7b..48a1a0e958 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -555,10 +555,10 @@ REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
CREATE VIEW pg_backend_memory_contexts AS
- SELECT * FROM pg_get_backend_memory_contexts();
+ SELECT * FROM pg_get_backend_memory_contexts(0, 0);
REVOKE ALL ON pg_backend_memory_contexts FROM PUBLIC;
-REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts FROM PUBLIC;
-- Statistics views
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 3e4ec53a97..ed5393324a 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -46,6 +46,7 @@
#include "storage/sinvaladt.h"
#include "storage/spin.h"
#include "utils/snapmgr.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -269,6 +270,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ McxtLogShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index c6a8d4611e..c61d5079e2 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -440,6 +441,20 @@ HandleProcSignalBarrierInterrupt(void)
/* latch will be set by procsignal_sigusr1_handler */
}
+/*
+ * HandleProcSignalLogMemoryContext
+ *
+ * Handle receipt of an interrupt indicating logging memory context.
+ * Signal handler portion of interrupt handling.
+ */
+static void
+HandleProcSignalLogMemoryContext(void)
+{
+ InterruptPending = true;
+ LogMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
/*
* Perform global barrier related interrupt checking.
*
@@ -580,6 +595,27 @@ ProcessProcSignalBarrier(void)
ConditionVariableBroadcast(&MyProcSignalSlot->pss_barrierCV);
}
+/*
+ * ProcessLogMemoryContextInterrupt
+ * The portion of logging memory context interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessLogMemoryContextInterrupt(void)
+{
+ int max_children;
+
+ LogMemoryContextPending = false;
+
+ max_children = mcxtLogData->maxChildrenPerContext;
+
+ ereport(LOG,
+ (errmsg("Logging memory contexts of PID %d. Max number of children per context is %d", MyProcPid, max_children)));
+
+ MemoryContextStatsDetail(TopMemoryContext, max_children, false);
+}
+
+
/*
* If it turns out that we couldn't absorb one or more barrier types, either
* because the barrier-processing functions returned false or due to an error,
@@ -675,6 +711,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_BUFFERPIN))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_BUFFERPIN);
+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
+ HandleProcSignalLogMemoryContext();
+
SetLatch(MyLatch);
errno = save_errno;
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2b1b68109f..afaf3b1cce 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3295,6 +3295,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (LogMemoryContextPending)
+ ProcessLogMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..de2040640c 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -19,6 +19,7 @@
#include "miscadmin.h"
#include "mb/pg_wchar.h"
#include "utils/builtins.h"
+#include "utils/mcxtfuncs.h"
/* ----------
* The max bytes for showing identifiers of MemoryContext.
@@ -61,7 +62,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
@@ -117,6 +118,9 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
{
+ int pid = PG_GETARG_INT32(0);
+ int max_children = PG_GETARG_INT32(1);
+
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -147,8 +151,11 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldcontext);
- PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- TopMemoryContext, NULL, 0);
+ if( pid == 0)
+ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
+ TopMemoryContext, NULL, 0);
+ else
+ pg_log_backend_memory_contexts(pid, max_children);
/* clean up and return the tuplestore */
tuplestore_donestoring(tupstore);
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 73e0a672ae..6c27065f96 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -34,6 +34,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t LogMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index ec6c130d0f..8e7fe09a0f 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool is_dst_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 2b90034764..b286308aa4 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool is_dst_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 84472b9158..f02331c36c 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -37,6 +41,13 @@
*/
MemoryContext CurrentMemoryContext = NULL;
+/*
+ * McxtLogData
+ * Data for communication between memory context logging
+ * dumper and requestor.
+ */
+McxtLogData *mcxtLogData = NULL;
+
/*
* Standard top-level contexts. For a description of the purpose of each
* of these contexts, refer to src/backend/utils/mmgr/README
@@ -55,9 +66,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool is_dst_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -487,6 +500,84 @@ MemoryContextMemAllocated(MemoryContext context, bool recurse)
return total;
}
+/*
+ * McxtLogShmemInit
+ * Initialize shared memory area
+ */
+void
+McxtLogShmemInit(void)
+{
+ bool found;
+
+ mcxtLogData = (McxtLogData *)
+ ShmemInitStruct("Memory Context Logging Data",
+ sizeof(int),
+ &found);
+ if (!found)
+ {
+ mcxtLogData->maxChildrenPerContext = 0;
+ }
+}
+
+/*
+ * pg_log_backend_memory_contexts
+ * Log memory contexts of the specified backend process.
+ */
+Datum
+pg_log_backend_memory_contexts(int pid, int max_children)
+{
+ PGPROC *proc = BackendPidGetProc(pid);
+
+ if(max_children < 0)
+ {
+ ereport(ERROR,
+ (errmsg("The maximum number of children must be greater than or equal to 0")));
+
+ PG_RETURN_BOOL(false);
+ }
+
+ /*
+ * Check whether the target process is PostgreSQL backend process.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * According to the thread below, it might be better to
+ * change this message.
+ *
+ * https://www.postgresql.org/message-id/CALj2ACW7Rr-R7mBcBQiXWPp%3DJV5chajjTdudLiF5YcpW-BmHhg%40mail.gmail.com
+ *
+ * Until the conclusion of the discussion, output the same
+ * message as pg_cancel_backend().
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+
+ PG_RETURN_BOOL(false);
+ }
+
+ /* Only allow superusers to print memory contexts. */
+ if (!superuser())
+ {
+ ereport(WARNING,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be a superuser to log memory contexts")));
+ PG_RETURN_BOOL(false);
+ }
+
+ mcxtLogData->maxChildrenPerContext = max_children;
+
+ if(!SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId))
+ PG_RETURN_BOOL(true);
+ else
+ {
+ ereport(WARNING,
+ (errmsg("failed to send signal: %m")));
+
+ PG_RETURN_BOOL(false);
+ }
+}
+
/*
* MemoryContextStats
* Print statistics about the named context and all its descendants.
@@ -499,7 +590,7 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);
+ MemoryContextStatsDetail(context, 100, true);
}
/*
@@ -508,19 +599,28 @@ MemoryContextStats(MemoryContext context)
* Entry point for use if you want to vary the number of child contexts shown.
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)
+MemoryContextStatsDetail(MemoryContext context, int max_children, bool is_dst_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);
+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, is_dst_stderr);
- fprintf(stderr,
- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
- grand_totals.totalspace, grand_totals.nblocks,
- grand_totals.freespace, grand_totals.freechunks,
- grand_totals.totalspace - grand_totals.freespace);
+ if (is_dst_stderr)
+ fprintf(stderr,
+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace);
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +633,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool is_dst_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,11 +646,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);
+ totals, is_dst_stderr);
/*
* Examine children. If there are more than max_children of them, we do
* not print the rest explicitly, but just summarize them.
+ *
+ * Note that specifying 0 to max_children means umlimited.
*/
memset(&local_totals, 0, sizeof(local_totals));
@@ -557,33 +660,50 @@ MemoryContextStatsInternal(MemoryContext context, int level,
child != NULL;
child = child->nextchild, ichild++)
{
- if (ichild < max_children)
+ if (ichild < max_children || max_children == 0)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);
+ totals,
+ is_dst_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);
+ &local_totals,
+ is_dst_stderr);
}
- /* Deal with excess children */
- if (ichild > max_children)
+ /*Deal with excess children */
+ if (ichild > max_children && max_children != 0)
{
if (print)
{
int i;
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
+ if (is_dst_stderr)
+ {
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,12 +725,14 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)
+ const char *stats_string,
+ bool is_dst_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
int i;
+ char truncated_ident[110];
/*
* It seems preferable to label dynahash contexts with just the hash table
@@ -623,9 +745,8 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)
- fprintf(stderr, " ");
- fprintf(stderr, "%s: %s", name, stats_string);
+ truncated_ident[0] = '\0';
+
if (ident)
{
/*
@@ -637,24 +758,42 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcpy(truncated_ident, ": ");
+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");
+
+ i = strlen(truncated_ident);
+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);
+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';
+
if (truncated)
- fprintf(stderr, "...");
+ strcat(truncated_ident, "...");
+ }
+
+ if (is_dst_stderr)
+ {
+ for (i = 0; i < level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
}
- fputc('\n', stderr);
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d %s: %s%s",
+ level, name, stats_string, truncated_ident)));
}
/*
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 9213be7c95..479b08bedd 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool is_dst_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 464fa8d614..fa20889105 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7921,12 +7921,14 @@
descr => 'information about all memory contexts of local backend',
proname => 'pg_get_backend_memory_contexts', prorows => '100',
proretset => 't', provolatile => 'v', proparallel => 'r',
- prorettype => 'record', proargtypes => '',
- proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ prorettype => 'record', proargtypes => 'int4 int4',
+ proallargtypes => '{int4,int4,text,text,text,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, max_children, name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# print memory context of specified backend
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 013850ac28..081822823c 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index 9331ef80fd..87a8501c37 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool is_dst_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..1144536031 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,8 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_LOG_MEMORY_CONTEXT, /* ask specified backend to log the
+ memory context */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
@@ -69,6 +71,7 @@ extern int SendProcSignal(pid_t pid, ProcSignalReason reason,
extern uint64 EmitProcSignalBarrier(ProcSignalBarrierType type);
extern void WaitForProcSignalBarrier(uint64 generation);
extern void ProcessProcSignalBarrier(void);
+extern void ProcessLogMemoryContextInterrupt(void);
extern void procsignal_sigusr1_handler(SIGNAL_ARGS);
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 36aae4e51c..9596aa7f14 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -46,6 +46,12 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
+typedef struct McxtLogData
+{
+ /* Max number of children to print per one parent context */
+ int maxChildrenPerContext;
+} McxtLogData;
+
/*
* Standard top-level memory contexts.
*
@@ -63,6 +69,8 @@ extern PGDLLIMPORT MemoryContext CurTransactionContext;
/* This is a transient link to the active portal's memory context: */
extern PGDLLIMPORT MemoryContext PortalContext;
+extern McxtLogData *mcxtLogData;
+
/* Backwards compatibility macro */
#define MemoryContextResetAndDeleteChildren(ctx) MemoryContextReset(ctx)
@@ -84,9 +92,12 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);
+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool is_dst_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
+extern void McxtLogShmemInit(void);
+extern Datum pg_log_backend_memory_contexts(int pid, int max_children);
#ifdef MEMORY_CONTEXT_CHECKING
extern void MemoryContextCheck(MemoryContext context);
diff --git a/src/test/modules/test_misc/t/002_print_memory_context_validation.pl b/src/test/modules/test_misc/t/002_print_memory_context_validation.pl
new file mode 100644
index 0000000000..056ced1da2
--- /dev/null
+++ b/src/test/modules/test_misc/t/002_print_memory_context_validation.pl
@@ -0,0 +1,90 @@
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# Set up node with logging collector
+my $node = get_new_node('primary');
+$node->init();
+$node->append_conf(
+ 'postgresql.conf', qq{
+ logging_collector = on
+ lc_messages = 'C'
+ });
+
+$node->start();
+
+# Verify that log output gets to the file
+$node->psql('postgres', 'select pg_get_backend_memory_contexts(pg_backend_pid(), 100)');
+
+# might need to retry if logging collector process is slow...
+my $max_attempts = 180 * 10;
+
+my $current_logfiles;
+for (my $attempts = 0; $attempts < $max_attempts; $attempts++)
+{
+ eval {
+ $current_logfiles = slurp_file($node->data_dir . '/current_logfiles');
+ };
+ last unless $@;
+ usleep(100_000);
+}
+die $@ if $@;
+
+note "current_logfiles = $current_logfiles";
+
+like(
+ $current_logfiles,
+ qr|^stderr log/postgresql-.*log$|,
+ 'current_logfiles is sane');
+
+my $lfname = $current_logfiles;
+$lfname =~ s/^stderr //;
+chomp $lfname;
+
+my $logfile;
+my $print_count;
+
+# Verify that the backtraces of the processes are logged into logfile.
+for (my $attempts = 0; $attempts < $max_attempts; $attempts++)
+{
+ $logfile = $node->data_dir . '/' . $lfname;
+ chomp $logfile;
+ print "file is $logfile";
+ open my $fh, '<', $logfile
+ or die "Could not open '$logfile' $!";
+ while (my $line = <$fh>)
+ {
+ chomp $line;
+ if ($line =~ m/Grand total: \d+ bytes in \d+ blocks/)
+ {
+ $print_count++;
+ }
+ }
+ last if $print_count == 1;
+ usleep(100_000);
+}
+
+is($print_count, 1, 'found expected memory context print in the log file');
+
+my $output;
+
+$node->psql('postgres', 'select pg_get_backend_memory_contexts(pg_backend_pid(), -1)',
+ stderr => \$output);
+
+like(
+ $output,
+ qr/The maximum number of children must be greater than or equal to 0/,
+ 'check if max_children check is working');
+
+$node->psql('postgres',
+ 'select pg_get_backend_memory_contexts(pid, 100) from pg_stat_activity where backend_type = \'checkpointer\'',
+ stderr => \$output);
+
+like(
+ $output,
+ qr/PID \d+ is not a PostgreSQL server process/,
+ 'check if PID check is working');
+
+$node->stop('fast');
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 9b12cc122a..5841e5cc12 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1333,7 +1333,7 @@ pg_backend_memory_contexts| SELECT pg_get_backend_memory_contexts.name,
pg_get_backend_memory_contexts.free_bytes,
pg_get_backend_memory_contexts.free_chunks,
pg_get_backend_memory_contexts.used_bytes
- FROM pg_get_backend_memory_contexts() pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
+ FROM pg_get_backend_memory_contexts(0, 0) pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
pg_config| SELECT pg_config.name,
pg_config.setting
FROM pg_config() pg_config(name, setting);
On 2021/03/25 0:17, torikoshia wrote:
On 2021-03-23 17:24, Kyotaro Horiguchi wrote:
Thanks for reviewing and suggestions!
The patched version failed to be compiled as follows. Could you fix this issue?
mcxtfuncs.c:22:10: fatal error: utils/mcxtfuncs.h: No such file or directory
#include "utils/mcxtfuncs.h"
^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[4]: *** [<builtin>: mcxtfuncs.o] Error 1
make[4]: *** Waiting for unfinished jobs....
make[3]: *** [../../../src/backend/common.mk:39: adt-recursive] Error 2
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [common.mk:39: utils-recursive] Error 2
make[1]: *** [Makefile:42: all-backend-recurse] Error 2
make: *** [GNUmakefile:11: all-src-recurse] Error 2
https://cirrus-ci.com/task/4621477321375744
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On 2021-03-25 22:02, Fujii Masao wrote:
On 2021/03/25 0:17, torikoshia wrote:
On 2021-03-23 17:24, Kyotaro Horiguchi wrote:
Thanks for reviewing and suggestions!
The patched version failed to be compiled as follows. Could you fix
this issue?
Sorry, it included a header file that's not contained in
the current version patch.
Attached new one.
Show quoted text
mcxtfuncs.c:22:10: fatal error: utils/mcxtfuncs.h: No such file or
directory
#include "utils/mcxtfuncs.h"
^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[4]: *** [<builtin>: mcxtfuncs.o] Error 1
make[4]: *** Waiting for unfinished jobs....
make[3]: *** [../../../src/backend/common.mk:39: adt-recursive] Error 2
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [common.mk:39: utils-recursive] Error 2
make[1]: *** [Makefile:42: all-backend-recurse] Error 2
make: *** [GNUmakefile:11: all-src-recurse] Error 2https://cirrus-ci.com/task/4621477321375744
Regards,
Attachments:
v4-0001-add-memorycontext-elog-print.patchtext/x-diff; name=v4-0001-add-memorycontext-elog-print.patchDownload
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 1d3429fbd9..a4017a0760 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24821,6 +24821,37 @@ SELECT collation for ('foo' COLLATE "de_DE");
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_backend_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_backend_memory_contexts</function> (
+ <parameter>pid</parameter> <type>integer</type>,
+ <parameter>max_children</parameter> <type>integer</type> )
+ <returnvalue>setof record</returnvalue>
+ </para>
+ <para>
+ Get memory contexts whose backend process has the specified process ID.
+ <parameter>max_children</parameter> limits the max number of children
+ to print per one parent context. 0 means unlimited.
+ When <parameter>pid</parameter> equals 0,
+ <function>pg_get_backend_memory_contexts</function> displays all
+ the memory contexts of the local process regardless of
+ <parameter>max_children</parameter>.
+ When <parameter>pid</parameter> does not equal 0,
+ memory contexts will be printed based on the log configuration set.
+ See <xref linkend="runtime-config-logging"/> for more information.
+ Only superusers can call this function even when the specified process
+ is non-superuser backend.
+ Note that when multiple
+ <function>pg_get_backend_memory_contexts</function> called in
+ succession or simultaneously, <parameter>max_children</parameter> can
+ be the one of another
+ <function>pg_get_backend_memory_contexts</function>.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 0dca65dc7b..48a1a0e958 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -555,10 +555,10 @@ REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
CREATE VIEW pg_backend_memory_contexts AS
- SELECT * FROM pg_get_backend_memory_contexts();
+ SELECT * FROM pg_get_backend_memory_contexts(0, 0);
REVOKE ALL ON pg_backend_memory_contexts FROM PUBLIC;
-REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts FROM PUBLIC;
-- Statistics views
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 3e4ec53a97..ed5393324a 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -46,6 +46,7 @@
#include "storage/sinvaladt.h"
#include "storage/spin.h"
#include "utils/snapmgr.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -269,6 +270,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ McxtLogShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index c6a8d4611e..c61d5079e2 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -440,6 +441,20 @@ HandleProcSignalBarrierInterrupt(void)
/* latch will be set by procsignal_sigusr1_handler */
}
+/*
+ * HandleProcSignalLogMemoryContext
+ *
+ * Handle receipt of an interrupt indicating logging memory context.
+ * Signal handler portion of interrupt handling.
+ */
+static void
+HandleProcSignalLogMemoryContext(void)
+{
+ InterruptPending = true;
+ LogMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
/*
* Perform global barrier related interrupt checking.
*
@@ -580,6 +595,27 @@ ProcessProcSignalBarrier(void)
ConditionVariableBroadcast(&MyProcSignalSlot->pss_barrierCV);
}
+/*
+ * ProcessLogMemoryContextInterrupt
+ * The portion of logging memory context interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessLogMemoryContextInterrupt(void)
+{
+ int max_children;
+
+ LogMemoryContextPending = false;
+
+ max_children = mcxtLogData->maxChildrenPerContext;
+
+ ereport(LOG,
+ (errmsg("Logging memory contexts of PID %d. Max number of children per context is %d", MyProcPid, max_children)));
+
+ MemoryContextStatsDetail(TopMemoryContext, max_children, false);
+}
+
+
/*
* If it turns out that we couldn't absorb one or more barrier types, either
* because the barrier-processing functions returned false or due to an error,
@@ -675,6 +711,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_BUFFERPIN))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_BUFFERPIN);
+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
+ HandleProcSignalLogMemoryContext();
+
SetLatch(MyLatch);
errno = save_errno;
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2b1b68109f..afaf3b1cce 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3295,6 +3295,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (LogMemoryContextPending)
+ ProcessLogMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..de2040640c 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -61,7 +62,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
@@ -117,6 +118,9 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
{
+ int pid = PG_GETARG_INT32(0);
+ int max_children = PG_GETARG_INT32(1);
+
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -147,8 +151,11 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldcontext);
- PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
- TopMemoryContext, NULL, 0);
+ if( pid == 0)
+ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,
+ TopMemoryContext, NULL, 0);
+ else
+ pg_log_backend_memory_contexts(pid, max_children);
/* clean up and return the tuplestore */
tuplestore_donestoring(tupstore);
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 73e0a672ae..6c27065f96 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -34,6 +34,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t LogMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index ec6c130d0f..8e7fe09a0f 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool is_dst_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 2b90034764..b286308aa4 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool is_dst_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 84472b9158..f02331c36c 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -37,6 +41,13 @@
*/
MemoryContext CurrentMemoryContext = NULL;
+/*
+ * McxtLogData
+ * Data for communication between memory context logging
+ * dumper and requestor.
+ */
+McxtLogData *mcxtLogData = NULL;
+
/*
* Standard top-level contexts. For a description of the purpose of each
* of these contexts, refer to src/backend/utils/mmgr/README
@@ -55,9 +66,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool is_dst_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -487,6 +500,84 @@ MemoryContextMemAllocated(MemoryContext context, bool recurse)
return total;
}
+/*
+ * McxtLogShmemInit
+ * Initialize shared memory area
+ */
+void
+McxtLogShmemInit(void)
+{
+ bool found;
+
+ mcxtLogData = (McxtLogData *)
+ ShmemInitStruct("Memory Context Logging Data",
+ sizeof(int),
+ &found);
+ if (!found)
+ {
+ mcxtLogData->maxChildrenPerContext = 0;
+ }
+}
+
+/*
+ * pg_log_backend_memory_contexts
+ * Log memory contexts of the specified backend process.
+ */
+Datum
+pg_log_backend_memory_contexts(int pid, int max_children)
+{
+ PGPROC *proc = BackendPidGetProc(pid);
+
+ if(max_children < 0)
+ {
+ ereport(ERROR,
+ (errmsg("The maximum number of children must be greater than or equal to 0")));
+
+ PG_RETURN_BOOL(false);
+ }
+
+ /*
+ * Check whether the target process is PostgreSQL backend process.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * According to the thread below, it might be better to
+ * change this message.
+ *
+ * https://www.postgresql.org/message-id/CALj2ACW7Rr-R7mBcBQiXWPp%3DJV5chajjTdudLiF5YcpW-BmHhg%40mail.gmail.com
+ *
+ * Until the conclusion of the discussion, output the same
+ * message as pg_cancel_backend().
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+
+ PG_RETURN_BOOL(false);
+ }
+
+ /* Only allow superusers to print memory contexts. */
+ if (!superuser())
+ {
+ ereport(WARNING,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be a superuser to log memory contexts")));
+ PG_RETURN_BOOL(false);
+ }
+
+ mcxtLogData->maxChildrenPerContext = max_children;
+
+ if(!SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId))
+ PG_RETURN_BOOL(true);
+ else
+ {
+ ereport(WARNING,
+ (errmsg("failed to send signal: %m")));
+
+ PG_RETURN_BOOL(false);
+ }
+}
+
/*
* MemoryContextStats
* Print statistics about the named context and all its descendants.
@@ -499,7 +590,7 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);
+ MemoryContextStatsDetail(context, 100, true);
}
/*
@@ -508,19 +599,28 @@ MemoryContextStats(MemoryContext context)
* Entry point for use if you want to vary the number of child contexts shown.
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)
+MemoryContextStatsDetail(MemoryContext context, int max_children, bool is_dst_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);
+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, is_dst_stderr);
- fprintf(stderr,
- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
- grand_totals.totalspace, grand_totals.nblocks,
- grand_totals.freespace, grand_totals.freechunks,
- grand_totals.totalspace - grand_totals.freespace);
+ if (is_dst_stderr)
+ fprintf(stderr,
+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace);
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +633,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool is_dst_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,11 +646,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);
+ totals, is_dst_stderr);
/*
* Examine children. If there are more than max_children of them, we do
* not print the rest explicitly, but just summarize them.
+ *
+ * Note that specifying 0 to max_children means umlimited.
*/
memset(&local_totals, 0, sizeof(local_totals));
@@ -557,33 +660,50 @@ MemoryContextStatsInternal(MemoryContext context, int level,
child != NULL;
child = child->nextchild, ichild++)
{
- if (ichild < max_children)
+ if (ichild < max_children || max_children == 0)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);
+ totals,
+ is_dst_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);
+ &local_totals,
+ is_dst_stderr);
}
- /* Deal with excess children */
- if (ichild > max_children)
+ /*Deal with excess children */
+ if (ichild > max_children && max_children != 0)
{
if (print)
{
int i;
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
+ if (is_dst_stderr)
+ {
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,12 +725,14 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)
+ const char *stats_string,
+ bool is_dst_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
int i;
+ char truncated_ident[110];
/*
* It seems preferable to label dynahash contexts with just the hash table
@@ -623,9 +745,8 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)
- fprintf(stderr, " ");
- fprintf(stderr, "%s: %s", name, stats_string);
+ truncated_ident[0] = '\0';
+
if (ident)
{
/*
@@ -637,24 +758,42 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcpy(truncated_ident, ": ");
+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");
+
+ i = strlen(truncated_ident);
+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);
+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';
+
if (truncated)
- fprintf(stderr, "...");
+ strcat(truncated_ident, "...");
+ }
+
+ if (is_dst_stderr)
+ {
+ for (i = 0; i < level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
}
- fputc('\n', stderr);
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d %s: %s%s",
+ level, name, stats_string, truncated_ident)));
}
/*
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 9213be7c95..479b08bedd 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool is_dst_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 464fa8d614..fa20889105 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7921,12 +7921,14 @@
descr => 'information about all memory contexts of local backend',
proname => 'pg_get_backend_memory_contexts', prorows => '100',
proretset => 't', provolatile => 'v', proparallel => 'r',
- prorettype => 'record', proargtypes => '',
- proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
+ prorettype => 'record', proargtypes => 'int4 int4',
+ proallargtypes => '{int4,int4,text,text,text,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, max_children, name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# print memory context of specified backend
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 013850ac28..081822823c 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index 9331ef80fd..87a8501c37 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool is_dst_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..1144536031 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,8 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_LOG_MEMORY_CONTEXT, /* ask specified backend to log the
+ memory context */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
@@ -69,6 +71,7 @@ extern int SendProcSignal(pid_t pid, ProcSignalReason reason,
extern uint64 EmitProcSignalBarrier(ProcSignalBarrierType type);
extern void WaitForProcSignalBarrier(uint64 generation);
extern void ProcessProcSignalBarrier(void);
+extern void ProcessLogMemoryContextInterrupt(void);
extern void procsignal_sigusr1_handler(SIGNAL_ARGS);
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 36aae4e51c..9596aa7f14 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -46,6 +46,12 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
+typedef struct McxtLogData
+{
+ /* Max number of children to print per one parent context */
+ int maxChildrenPerContext;
+} McxtLogData;
+
/*
* Standard top-level memory contexts.
*
@@ -63,6 +69,8 @@ extern PGDLLIMPORT MemoryContext CurTransactionContext;
/* This is a transient link to the active portal's memory context: */
extern PGDLLIMPORT MemoryContext PortalContext;
+extern McxtLogData *mcxtLogData;
+
/* Backwards compatibility macro */
#define MemoryContextResetAndDeleteChildren(ctx) MemoryContextReset(ctx)
@@ -84,9 +92,12 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);
+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool is_dst_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
+extern void McxtLogShmemInit(void);
+extern Datum pg_log_backend_memory_contexts(int pid, int max_children);
#ifdef MEMORY_CONTEXT_CHECKING
extern void MemoryContextCheck(MemoryContext context);
diff --git a/src/test/modules/test_misc/t/002_print_memory_context_validation.pl b/src/test/modules/test_misc/t/002_print_memory_context_validation.pl
new file mode 100644
index 0000000000..056ced1da2
--- /dev/null
+++ b/src/test/modules/test_misc/t/002_print_memory_context_validation.pl
@@ -0,0 +1,90 @@
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# Set up node with logging collector
+my $node = get_new_node('primary');
+$node->init();
+$node->append_conf(
+ 'postgresql.conf', qq{
+ logging_collector = on
+ lc_messages = 'C'
+ });
+
+$node->start();
+
+# Verify that log output gets to the file
+$node->psql('postgres', 'select pg_get_backend_memory_contexts(pg_backend_pid(), 100)');
+
+# might need to retry if logging collector process is slow...
+my $max_attempts = 180 * 10;
+
+my $current_logfiles;
+for (my $attempts = 0; $attempts < $max_attempts; $attempts++)
+{
+ eval {
+ $current_logfiles = slurp_file($node->data_dir . '/current_logfiles');
+ };
+ last unless $@;
+ usleep(100_000);
+}
+die $@ if $@;
+
+note "current_logfiles = $current_logfiles";
+
+like(
+ $current_logfiles,
+ qr|^stderr log/postgresql-.*log$|,
+ 'current_logfiles is sane');
+
+my $lfname = $current_logfiles;
+$lfname =~ s/^stderr //;
+chomp $lfname;
+
+my $logfile;
+my $print_count;
+
+# Verify that the backtraces of the processes are logged into logfile.
+for (my $attempts = 0; $attempts < $max_attempts; $attempts++)
+{
+ $logfile = $node->data_dir . '/' . $lfname;
+ chomp $logfile;
+ print "file is $logfile";
+ open my $fh, '<', $logfile
+ or die "Could not open '$logfile' $!";
+ while (my $line = <$fh>)
+ {
+ chomp $line;
+ if ($line =~ m/Grand total: \d+ bytes in \d+ blocks/)
+ {
+ $print_count++;
+ }
+ }
+ last if $print_count == 1;
+ usleep(100_000);
+}
+
+is($print_count, 1, 'found expected memory context print in the log file');
+
+my $output;
+
+$node->psql('postgres', 'select pg_get_backend_memory_contexts(pg_backend_pid(), -1)',
+ stderr => \$output);
+
+like(
+ $output,
+ qr/The maximum number of children must be greater than or equal to 0/,
+ 'check if max_children check is working');
+
+$node->psql('postgres',
+ 'select pg_get_backend_memory_contexts(pid, 100) from pg_stat_activity where backend_type = \'checkpointer\'',
+ stderr => \$output);
+
+like(
+ $output,
+ qr/PID \d+ is not a PostgreSQL server process/,
+ 'check if PID check is working');
+
+$node->stop('fast');
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 9b12cc122a..5841e5cc12 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1333,7 +1333,7 @@ pg_backend_memory_contexts| SELECT pg_get_backend_memory_contexts.name,
pg_get_backend_memory_contexts.free_bytes,
pg_get_backend_memory_contexts.free_chunks,
pg_get_backend_memory_contexts.used_bytes
- FROM pg_get_backend_memory_contexts() pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
+ FROM pg_get_backend_memory_contexts(0, 0) pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
pg_config| SELECT pg_config.name,
pg_config.setting
FROM pg_config() pg_config(name, setting);
At Thu, 25 Mar 2021 23:45:17 +0900, torikoshia <torikoshia@oss.nttdata.com> wrote in
Attached new one.
Thanks!
- SELECT * FROM pg_get_backend_memory_contexts();
+ SELECT * FROM pg_get_backend_memory_contexts(0, 0);
However we can freely change the signature since it's new in 14, but I
see the (void) signature as useful. I vaguely thought of defining
pg_get_backend_memory_contexts(void) in pg_proc.dat then defining
(int, int) as a different function in system_views.sql. That allows
the function of the second signature to output nothing. You can make a
distinction between the two signatures by using PG_NARGS() in the C
function.
That being said, it's somewhat uneasy that two functions with the same
name returns different values. I'd like to hear others what they feel
like about the behavior.
+# print memory context of specified backend
+
This looks like a garbage.
+ if( pid == 0)
Odd spacing:p
+note "current_logfiles = $current_logfiles";
+
+like(
+ $current_logfiles,
+ qr|^stderr log/postgresql-.*log$|,
+ 'current_logfiles is sane');
+
+my $lfname = $current_logfiles;
+$lfname =~ s/^stderr //;
+chomp $lfname;
+
+my $logfile;
+my $print_count;
+
+# Verify that the backtraces of the processes are logged into logfile.
+for (my $attempts = 0; $attempts < $max_attempts; $attempts++)
+{
+ $logfile = $node->data_dir . '/' . $lfname;
It seems that you forgot to include the change on this part in v4.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On 2021/03/26 12:01, Kyotaro Horiguchi wrote:
At Thu, 25 Mar 2021 23:45:17 +0900, torikoshia <torikoshia@oss.nttdata.com> wrote in
Attached new one.
Regarding the argument max_children, isn't it better to set its default value,
e.g., 100 like MemoryContextStats() uses?
+ ereport(WARNING,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be a superuser to log memory contexts")));
IMO this type of error, i.e., permission error, should be treated as ERROR
rather than WARNING, like pg_terminate_backend() does.
+ ereport(WARNING,
+ (errmsg("failed to send signal: %m")));
Per message style rule, "could not send signal to process %d: %m" is better?
Thanks!
- SELECT * FROM pg_get_backend_memory_contexts(); + SELECT * FROM pg_get_backend_memory_contexts(0, 0);However we can freely change the signature since it's new in 14, but I
see the (void) signature as useful. I vaguely thought of defining
pg_get_backend_memory_contexts(void) in pg_proc.dat then defining
(int, int) as a different function in system_views.sql. That allows
the function of the second signature to output nothing. You can make a
distinction between the two signatures by using PG_NARGS() in the C
function.That being said, it's somewhat uneasy that two functions with the same
name returns different values. I'd like to hear others what they feel
like about the behavior.
I think that it's confusing to merge two different features into one function.
Isn't it better to leave pg_get_backend_memory_contexts() as it is, and
make the log-memory-contexts feature as separate function, e.g.,
pg_log_backend_memory_contexts()?
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On 2021/03/26 12:26, Fujii Masao wrote:
On 2021/03/26 12:01, Kyotaro Horiguchi wrote:
At Thu, 25 Mar 2021 23:45:17 +0900, torikoshia <torikoshia@oss.nttdata.com> wrote in
Attached new one.
Regarding the argument max_children, isn't it better to set its default value,
e.g., 100 like MemoryContextStats() uses?
+ mcxtLogData->maxChildrenPerContext = max_children;
+
+ if(!SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId))
+ PG_RETURN_BOOL(true);
Do we need memory barrier here? Otherwise, there is a race condition
where maxChildrenPerContext has not been set yet when the target backend
that received signal read that shared variable. No?
+ Note that when multiple
+ <function>pg_get_backend_memory_contexts</function> called in
+ succession or simultaneously, <parameter>max_children</parameter> can
+ be the one of another
+ <function>pg_get_backend_memory_contexts</function>.
+ </para></entry>
This might not be an issue in practice as Horiguchi-san said upthread.
But this looks a bit ugly and might be footgun later. The current approach
using shared memory to pass this information to the target backends
might be overkill, and too complicated to polish the design at the stage
just before feature freeze. So I'd withdraw my previous comment and
am OK to use the hard-coded value as max_children at the first version
of this feature. Thought?
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
At Fri, 26 Mar 2021 12:26:31 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in
On 2021/03/26 12:01, Kyotaro Horiguchi wrote:
At Thu, 25 Mar 2021 23:45:17 +0900, torikoshia
<torikoshia@oss.nttdata.com> wrote inAttached new one.
Regarding the argument max_children, isn't it better to set its default value, e.g., 100 like MemoryContextStats() uses? + ereport(WARNING, + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("must be a superuser to log memory contexts")));IMO this type of error, i.e., permission error, should be treated as ERROR rather than WARNING, like pg_terminate_backend() does. + ereport(WARNING, + (errmsg("failed to send signal: %m")));Per message style rule, "could not send signal to process %d: %m" is
better?
+1 x 3 for the above.
Thanks! - SELECT * FROM pg_get_backend_memory_contexts(); + SELECT * FROM pg_get_backend_memory_contexts(0, 0); However we can freely change the signature since it's new in 14, but I see the (void) signature as useful. I vaguely thought of defining pg_get_backend_memory_contexts(void) in pg_proc.dat then defining (int, int) as a different function in system_views.sql. That allows the function of the second signature to output nothing. You can make a distinction between the two signatures by using PG_NARGS() in the C function. That being said, it's somewhat uneasy that two functions with the same name returns different values. I'd like to hear others what they feel like about the behavior.I think that it's confusing to merge two different features into one
function.
Isn't it better to leave pg_get_backend_memory_contexts() as it is,
and
make the log-memory-contexts feature as separate function, e.g.,
pg_log_backend_memory_contexts()?
Yeah, that name looks better than pg_print_ba.. and I agree to
separate the two features. Sorry for wagging the discussion
back-and-forth.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
At Fri, 26 Mar 2021 12:49:16 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in
On 2021/03/26 12:26, Fujii Masao wrote:
On 2021/03/26 12:01, Kyotaro Horiguchi wrote:
At Thu, 25 Mar 2021 23:45:17 +0900, torikoshia
<torikoshia@oss.nttdata.com> wrote inAttached new one.
Regarding the argument max_children, isn't it better to set its
default value,
e.g., 100 like MemoryContextStats() uses?+ mcxtLogData->maxChildrenPerContext = max_children; + + if(!SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId)) + PG_RETURN_BOOL(true);Do we need memory barrier here? Otherwise, there is a race condition
where maxChildrenPerContext has not been set yet when the target
backend
that received signal read that shared variable. No?+ Note that when multiple + <function>pg_get_backend_memory_contexts</function> called in + succession or simultaneously, <parameter>max_children</parameter> can + be the one of another + <function>pg_get_backend_memory_contexts</function>. + </para></entry>This might not be an issue in practice as Horiguchi-san said upthread.
But this looks a bit ugly and might be footgun later. The current
approach
using shared memory to pass this information to the target backends
might be overkill, and too complicated to polish the design at the
stage
just before feature freeze. So I'd withdraw my previous comment and
am OK to use the hard-coded value as max_children at the first version
of this feature. Thought?
The dumper function silently suppresses logging when there are too
many children. Suppressing a part of output when the user didn't told
anything looks like just a misbehavior (even if it is written in the
documentation..), especially when the suppressed contexts occupy the
majority of memory usage. I think the fixed-limit of lines works if
the lines are in descending order of memory usage.
At least we need an additional line to inform the suppression.
"some contexts are omitted"
"n child contexts: total_bytes = ..."
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
At Fri, 26 Mar 2021 13:17:21 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in
At Fri, 26 Mar 2021 12:49:16 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in
On 2021/03/26 12:26, Fujii Masao wrote:
On 2021/03/26 12:01, Kyotaro Horiguchi wrote:
At Thu, 25 Mar 2021 23:45:17 +0900, torikoshia
<torikoshia@oss.nttdata.com> wrote inAttached new one.
Regarding the argument max_children, isn't it better to set its
default value,
e.g., 100 like MemoryContextStats() uses?+ mcxtLogData->maxChildrenPerContext = max_children; + + if(!SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId)) + PG_RETURN_BOOL(true);Do we need memory barrier here? Otherwise, there is a race condition
where maxChildrenPerContext has not been set yet when the target
backend
that received signal read that shared variable. No?+ Note that when multiple + <function>pg_get_backend_memory_contexts</function> called in + succession or simultaneously, <parameter>max_children</parameter> can + be the one of another + <function>pg_get_backend_memory_contexts</function>. + </para></entry>This might not be an issue in practice as Horiguchi-san said upthread.
But this looks a bit ugly and might be footgun later. The current
approach
using shared memory to pass this information to the target backends
might be overkill, and too complicated to polish the design at the
stage
just before feature freeze. So I'd withdraw my previous comment and
am OK to use the hard-coded value as max_children at the first version
of this feature. Thought?The dumper function silently suppresses logging when there are too
many children. Suppressing a part of output when the user didn't told
anything looks like just a misbehavior (even if it is written in the
documentation..), especially when the suppressed contexts occupy the
majority of memory usage. I think the fixed-limit of lines works if
the lines are in descending order of memory usage.At least we need an additional line to inform the suppression.
"some contexts are omitted"
"n child contexts: total_bytes = ..."
Sorry I missed that is already implemented. So my opnion is I agree
with limiting with a fixed-number, and preferablly sorted in
descending order of... totalspace/nblocks?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On 2021/03/26 13:28, Kyotaro Horiguchi wrote:
"some contexts are omitted"
"n child contexts: total_bytes = ..."Sorry I missed that is already implemented. So my opnion is I agree
with limiting with a fixed-number, and preferablly sorted in
descending order of... totalspace/nblocks?
This may be an improvement, but makes us modify MemoryContextStatsInternal()
very much. I'm afraid that it's too late to do that at this stage...
What about leaving the output order as it is at the first version?
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
At Fri, 26 Mar 2021 14:02:49 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in
On 2021/03/26 13:28, Kyotaro Horiguchi wrote:
"some contexts are omitted"
"n child contexts: total_bytes = ..."Sorry I missed that is already implemented. So my opnion is I agree
with limiting with a fixed-number, and preferablly sorted in
descending order of... totalspace/nblocks?This may be an improvement, but makes us modify
MemoryContextStatsInternal()
very much. I'm afraid that it's too late to do that at this stage...
What about leaving the output order as it is at the first version?
So I said "preferably":p (with a misspelling...)
I'm fine with that.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On 2021-03-26 14:08, Kyotaro Horiguchi wrote:
At Fri, 26 Mar 2021 14:02:49 +0900, Fujii Masao
<masao.fujii@oss.nttdata.com> wrote inOn 2021/03/26 13:28, Kyotaro Horiguchi wrote:
"some contexts are omitted"
"n child contexts: total_bytes = ..."Sorry I missed that is already implemented. So my opnion is I agree
with limiting with a fixed-number, and preferablly sorted in
descending order of... totalspace/nblocks?This may be an improvement, but makes us modify
MemoryContextStatsInternal()
very much. I'm afraid that it's too late to do that at this stage...
What about leaving the output order as it is at the first version?So I said "preferably":p (with a misspelling...)
I'm fine with that.regards.
Thanks for the comments!
Attached a new patch.
It adds pg_log_backend_memory_contexts(pid) which logs memory contexts
of the specified backend process.
The number of child contexts to be logged per parent is limited to 100
as with MemoryContextStats().
As written in commit 7b5ef8f2d07, which limits the verbosity of
memory context statistics dumps, it supposes that practical cases
where the dump gets long will typically be huge numbers of
siblings under the same parent context; while the additional
debugging value from seeing details about individual siblings
beyond 100 will not be large.
Thoughts?
Regards.
Attachments:
v5-0001-add-memorycontext-elog-print.patchtext/x-diff; name=v5-0001-add-memorycontext-elog-print.patchDownload
From e5ab553c1e5b7fa53c51e0e4fa4472bdaeced4e1 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 29 Mar 2021 09:30:23 +0900
Subject: [PATCH] After commit 3e98c0bafb28de, we can display the usage of
memory contexts using pg_backend_memory_contexts system view. However, its
target process is limited to the backend which is showing the view. This
patch introduces pg_log_backend_memory_contexts(pid) which logs memory
contexts of the specified backend process.
Currently the number of child contexts to be logged per parent
is limited to 100.
As with MemoryContextStats(), it supposes that practical cases
where the dump gets long will typically be huge numbers of
siblings under the same parent context; while the additional
debugging value from seeing details about individual siblings
beyond 100 will not be large.
---
doc/src/sgml/func.sgml | 20 +++
src/backend/storage/ipc/procsignal.c | 37 ++++
src/backend/tcop/postgres.c | 3 +
src/backend/utils/adt/mcxtfuncs.c | 2 +-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/mmgr/aset.c | 8 +-
src/backend/utils/mmgr/generation.c | 8 +-
src/backend/utils/mmgr/mcxt.c | 164 ++++++++++++++----
src/backend/utils/mmgr/slab.c | 9 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/miscadmin.h | 1 +
src/include/nodes/memnodes.h | 6 +-
src/include/storage/procsignal.h | 3 +
src/include/utils/memutils.h | 3 +-
.../t/002_log_memory_context_validation.pl | 31 ++++
15 files changed, 257 insertions(+), 45 deletions(-)
create mode 100644 src/test/modules/test_misc/t/002_log_memory_context_validation.pl
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 19285ae136..7a80607366 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24871,6 +24871,26 @@ SELECT collation for ('foo' COLLATE "de_DE");
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_log_backend_memory_contexts</primary>
+ </indexterm>
+ <function>pg_log_backend_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Log the memory contexts whose backend process has the specified
+ process ID.
+ Memory contexts will be printed based on the log configuration set.
+ See <xref linkend="runtime-config-logging"/> for more information.
+ The number of child contexts per parent is limited to 100.
+ For contexts with more than 100 children, summary will be shown.
+ Only superusers can log the memory contexts even when the specified
+ process is non-superuser backend.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index c6a8d4611e..550aa2ffea 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -440,6 +441,20 @@ HandleProcSignalBarrierInterrupt(void)
/* latch will be set by procsignal_sigusr1_handler */
}
+/*
+ * HandleProcSignalLogMemoryContext
+ *
+ * Handle receipt of an interrupt indicating log memory context.
+ * Signal handler portion of interrupt handling.
+ */
+static void
+HandleProcSignalLogMemoryContext(void)
+{
+ InterruptPending = true;
+ LogMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
/*
* Perform global barrier related interrupt checking.
*
@@ -580,6 +595,25 @@ ProcessProcSignalBarrier(void)
ConditionVariableBroadcast(&MyProcSignalSlot->pss_barrierCV);
}
+/*
+ * ProcessLogMemoryContextInterrupt
+ *
+ * The portion of logging memory context interrupt handling that runs
+ * outside of the signal handler.
+ */
+void
+ProcessLogMemoryContextInterrupt(void)
+{
+ LogMemoryContextPending = false;
+
+ ereport(LOG,
+ (errmsg("Logging memory contexts of PID %d", MyProcPid)));
+
+ /* A hard-wired limit on the number of children is usually good enough */
+ MemoryContextStatsDetail(TopMemoryContext, 100, false);
+}
+
+
/*
* If it turns out that we couldn't absorb one or more barrier types, either
* because the barrier-processing functions returned false or due to an error,
@@ -654,6 +688,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_WALSND_INIT_STOPPING))
HandleWalSndInitStopping();
+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
+ HandleProcSignalLogMemoryContext();
+
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2b1b68109f..afaf3b1cce 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3295,6 +3295,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (LogMemoryContextPending)
+ ProcessLogMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..07a95dd115 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -61,7 +61,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 73e0a672ae..6c27065f96 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -34,6 +34,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t LogMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index ec6c130d0f..980c092558 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output is stderr or elog.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool is_dst_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 2b90034764..b286308aa4 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool is_dst_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 84472b9158..bb0cc0425e 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -55,9 +59,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool is_dst_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -487,6 +493,54 @@ MemoryContextMemAllocated(MemoryContext context, bool recurse)
return total;
}
+/*
+ * pg_log_backend_memory_contexts
+ * Print memory context of the specified backend process.
+ */
+Datum
+pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ PGPROC *proc = BackendPidGetProc(pid);
+
+ /* Check whether the target process is PostgreSQL backend process. */
+ if (proc == NULL)
+ {
+ /*
+ * According to the thread below, it might be better to
+ * change this message.
+ *
+ * https://www.postgresql.org/message-id/CALj2ACW7Rr-R7mBcBQiXWPp%3DJV5chajjTdudLiF5YcpW-BmHhg%40mail.gmail.com
+ *
+ * Until the conclusion of the discussion, output the same
+ * message as pg_cancel_backend().
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+
+ PG_RETURN_BOOL(false);
+ }
+
+ /* Only allow superusers to log memory contexts. */
+ if (!superuser())
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be a superuser to log memory contexts")));
+ PG_RETURN_BOOL(false);
+ }
+
+ if(!SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId))
+ PG_RETURN_BOOL(true);
+ else
+ {
+ ereport(WARNING,
+ (errmsg("could not send signal to process %d: %m", pid)));
+
+ PG_RETURN_BOOL(false);
+ }
+}
+
/*
* MemoryContextStats
* Print statistics about the named context and all its descendants.
@@ -499,7 +553,7 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);
+ MemoryContextStatsDetail(context, 100, true);
}
/*
@@ -508,19 +562,28 @@ MemoryContextStats(MemoryContext context)
* Entry point for use if you want to vary the number of child contexts shown.
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)
+MemoryContextStatsDetail(MemoryContext context, int max_children, bool is_dst_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);
+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, is_dst_stderr);
- fprintf(stderr,
- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
- grand_totals.totalspace, grand_totals.nblocks,
- grand_totals.freespace, grand_totals.freechunks,
- grand_totals.totalspace - grand_totals.freespace);
+ if (is_dst_stderr)
+ fprintf(stderr,
+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace);
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +596,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool is_dst_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,7 +609,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);
+ totals, is_dst_stderr);
/*
* Examine children. If there are more than max_children of them, we do
@@ -560,11 +624,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
if (ichild < max_children)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);
+ totals,
+ is_dst_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);
+ &local_totals,
+ is_dst_stderr);
}
/* Deal with excess children */
@@ -574,16 +640,31 @@ MemoryContextStatsInternal(MemoryContext context, int level,
{
int i;
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
+ if (is_dst_stderr)
+ {
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,11 +686,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)
+ const char *stats_string,
+ bool is_dst_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
+ char truncated_ident[110];
int i;
/*
@@ -623,9 +706,8 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)
- fprintf(stderr, " ");
- fprintf(stderr, "%s: %s", name, stats_string);
+ truncated_ident[0] = '\0';
+
if (ident)
{
/*
@@ -637,24 +719,42 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcpy(truncated_ident, ": ");
+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");
+
+ i = strlen(truncated_ident);
+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);
+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';
+
if (truncated)
- fprintf(stderr, "...");
+ strcat(truncated_ident, "...");
}
- fputc('\n', stderr);
+
+ if (is_dst_stderr)
+ {
+ for (i = 0; i < level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d %s: %s%s",
+ level, name, stats_string, truncated_ident)));
}
/*
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 9213be7c95..479b08bedd 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool is_dst_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index cc7d90d2b0..7e09bad8db 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7946,6 +7946,12 @@
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# log memory context of specified backend
+{ oid => '4543', descr => 'log memory contexts of specified backend',
+ proname => 'pg_log_backend_memory_contexts',
+ provolatile => 'v', prorettype => 'bool',
+ proargtypes => 'int4', prosrc => 'pg_log_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 013850ac28..081822823c 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index 9331ef80fd..87a8501c37 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool is_dst_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..1144536031 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,8 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_LOG_MEMORY_CONTEXT, /* ask specified backend to log the
+ memory context */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
@@ -69,6 +71,7 @@ extern int SendProcSignal(pid_t pid, ProcSignalReason reason,
extern uint64 EmitProcSignalBarrier(ProcSignalBarrierType type);
extern void WaitForProcSignalBarrier(uint64 generation);
extern void ProcessProcSignalBarrier(void);
+extern void ProcessLogMemoryContextInterrupt(void);
extern void procsignal_sigusr1_handler(SIGNAL_ARGS);
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 36aae4e51c..fd10f80eda 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -84,7 +84,8 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);
+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool is_dst_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
diff --git a/src/test/modules/test_misc/t/002_log_memory_context_validation.pl b/src/test/modules/test_misc/t/002_log_memory_context_validation.pl
new file mode 100644
index 0000000000..541616c5fa
--- /dev/null
+++ b/src/test/modules/test_misc/t/002_log_memory_context_validation.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+# Set up node with logging collector
+my $node = get_new_node('primary');
+$node->init();
+$node->start();
+
+# Verify that memory contexts are logged
+$node->psql('postgres', 'select pg_log_backend_memory_contexts(pg_backend_pid())');
+
+my $logfile = slurp_file($node->logfile);
+like($logfile, qr/Grand total: \d+ bytes in \d+ blocks/,
+ 'found expected memory context print in the log file');
+
+# Verify PID check is working
+my $output;
+
+$node->psql('postgres',
+ 'select pg_log_backend_memory_contexts(pid) from pg_stat_activity where backend_type = \'checkpointer\'',
+ stderr => \$output);
+
+like(
+ $output,
+ qr/PID \d+ is not a PostgreSQL server process/,
+ 'expected PID check working');
+
+$node->stop('fast');
--
2.18.1
On 2021/03/29 11:59, torikoshia wrote:
On 2021-03-26 14:08, Kyotaro Horiguchi wrote:
At Fri, 26 Mar 2021 14:02:49 +0900, Fujii Masao
<masao.fujii@oss.nttdata.com> wrote inOn 2021/03/26 13:28, Kyotaro Horiguchi wrote:
"some contexts are omitted"
"n child contexts: total_bytes = ..."Sorry I missed that is already implemented.� So my opnion is I agree
with limiting with a fixed-number, and preferablly sorted in
descending order of... totalspace/nblocks?This may be an improvement, but makes us modify
MemoryContextStatsInternal()
very much. I'm afraid that it's too late to do that at this stage...
What about leaving the output order as it is at the first version?So I said "preferably":p� (with a misspelling...)
I'm fine with that.regards.
Thanks for the comments!
Attached a new patch.
Thanks!
It adds pg_log_backend_memory_contexts(pid) which logs memory contexts
of the specified backend process.The number of child contexts to be logged per parent is limited to 100
as with MemoryContextStats().As written in commit 7b5ef8f2d07, which limits the verbosity of
memory context statistics dumps, it supposes that practical cases
where the dump gets long will typically be huge numbers of
siblings under the same parent context; while the additional
debugging value from seeing details about individual siblings
beyond 100 will not be large.Thoughts?
I'm OK with 100. We should comment why we chose 100 for that.
Here are some review comments.
Isn't it better to move HandleProcSignalLogMemoryContext() and
ProcessLogMemoryContextInterrupt() to mcxt.c from procsignal.c
(like the functions for notify interrupt are defined in async.c)
because they are the functions for memory contexts?
+ * HandleProcSignalLogMemoryContext
+ *
+ * Handle receipt of an interrupt indicating log memory context.
+ * Signal handler portion of interrupt handling.
IMO it's better to comment why we need to separate the function into two,
i.e., HandleProcSignalLogMemoryContext() and ProcessLogMemoryContextInterrupt(),
like the comment for other similar function explains. What about the followings?
-------------------------------
HandleLogMemoryContextInterrupt
Handle receipt of an interrupt indicating logging of memory contexts.
All the actual work is deferred to ProcessLogMemoryContextInterrupt(),
because we cannot safely emit a log message inside the signal handler.
-------------------------------
ProcessLogMemoryContextInterrupt
Perform logging of memory contexts of this backend process.
Any backend that participates in ProcSignal signaling must arrange to
call this function if we see LogMemoryContextPending set. It is called
from CHECK_FOR_INTERRUPTS(), which is enough because the target process
for logging of memory contexts is a backend.
-------------------------------
+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
+ HandleProcSignalLogMemoryContext();
+
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
The code for memory context logging interrupt came after barrier interrupt
in other places, e.g., procsignal.h. Why is this order of code different?
+/*
+ * pg_log_backend_memory_contexts
+ * Print memory context of the specified backend process.
Isn't it better to move pg_log_backend_memory_contexts() to mcxtfuncs.c
from mcxt.c because this is the SQL function memory contexts?
IMO we should comment why we allow only superuser to call this function.
What about the following?
-----------------
Signal a backend process to log its memory contexts.
Only superusers are allowed to signal to log the memory contexts
because allowing any users to issue this request at an unbounded rate
would cause lots of log messages and which can lead to denial of service.
-----------------
+ PGPROC *proc = BackendPidGetProc(pid);
+
+ /* Check whether the target process is PostgreSQL backend process. */
+ if (proc == NULL)
What about adding more comments as follows?
---------------------------------
+ /*
+ * BackendPidGetProc returns NULL if the pid isn't valid; but by the time
+ * we reach kill(), a process for which we get a valid proc here might
+ * have terminated on its own. There's no way to acquire a lock on an
+ * arbitrary process to prevent that. But since this mechanism is usually
+ * used to debug a backend running and consuming lots of memory,
+ * that it might end on its own first and its memory contexts are not
+ * logged is not a problem.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend logged its memory contexts during the run.
+ */
+ ereport(WARNING,
---------------------------------
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be a superuser to log memory contexts")));
+ PG_RETURN_BOOL(false);
This PG_RETURN_BOOL(false) is unnecessary because ereport() will emit an ERROR?
+$node->psql('postgres', 'select pg_log_backend_memory_contexts(pg_backend_pid())');
+
+my $logfile = slurp_file($node->logfile);
+like($logfile, qr/Grand total: \d+ bytes in \d+ blocks/,
+ 'found expected memory context print in the log file');
Isn't there the case where the memory contexts have not been logged yet
when slurp_file() is called? It's rare, though. If there is the risk for that,
we should wait for the memory contexts to be actually logged?
BTW, I agree that we should have the regression test that calls
pg_log_backend_memory_contexts() and checks, e.g., that it doesn't cause
segmentation fault. But I'm just wondering if it's a bit overkill to make perl
scriptand start new node to test only this function...
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On 2021-03-30 02:28, Fujii Masao wrote:
Thanks for reviewing and kind suggestions!
It adds pg_log_backend_memory_contexts(pid) which logs memory contexts
of the specified backend process.The number of child contexts to be logged per parent is limited to 100
as with MemoryContextStats().As written in commit 7b5ef8f2d07, which limits the verbosity of
memory context statistics dumps, it supposes that practical cases
where the dump gets long will typically be huge numbers of
siblings under the same parent context; while the additional
debugging value from seeing details about individual siblings
beyond 100 will not be large.Thoughts?
I'm OK with 100. We should comment why we chose 100 for that.
Added following comments.
+ /*
+ * When a backend process is consuming huge memory, logging all
its
+ * memory contexts might overrun available disk space. To
prevent
+ * this, we limit the number of child contexts per parent to
100.
+ *
+ * As with MemoryContextStats(), we suppose that practical cases
+ * where the dump gets long will typically be huge numbers of
+ * siblings under the same parent context; while the additional
+ * debugging value from seeing details about individual siblings
+ * beyond 100 will not be large.
+ */
+ MemoryContextStatsDetail(TopMemoryContext, 100, false);
Here are some review comments.
Isn't it better to move HandleProcSignalLogMemoryContext() and
ProcessLogMemoryContextInterrupt() to mcxt.c from procsignal.c
(like the functions for notify interrupt are defined in async.c)
because they are the functions for memory contexts?
Agreed.
Also renamed HandleProcSignalLogMemoryContext to
HandleLogMemoryContextInterrupt.
+ * HandleProcSignalLogMemoryContext + * + * Handle receipt of an interrupt indicating log memory context. + * Signal handler portion of interrupt handling.IMO it's better to comment why we need to separate the function into
two,
i.e., HandleProcSignalLogMemoryContext() and
ProcessLogMemoryContextInterrupt(),
like the comment for other similar function explains. What about the
followings?
Thanks! Changed them to the suggested one.
-------------------------------
HandleLogMemoryContextInterruptHandle receipt of an interrupt indicating logging of memory contexts.
All the actual work is deferred to ProcessLogMemoryContextInterrupt(),
because we cannot safely emit a log message inside the signal handler.
-------------------------------
ProcessLogMemoryContextInterruptPerform logging of memory contexts of this backend process.
Any backend that participates in ProcSignal signaling must arrange to
call this function if we see LogMemoryContextPending set. It is called
from CHECK_FOR_INTERRUPTS(), which is enough because the target process
for logging of memory contexts is a backend.
-------------------------------+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT)) + HandleProcSignalLogMemoryContext(); + if (CheckProcSignal(PROCSIG_BARRIER)) HandleProcSignalBarrierInterrupt();The code for memory context logging interrupt came after barrier
interrupt
in other places, e.g., procsignal.h. Why is this order of code
different?
Fixed.
+/* + * pg_log_backend_memory_contexts + * Print memory context of the specified backend process.Isn't it better to move pg_log_backend_memory_contexts() to mcxtfuncs.c
from mcxt.c because this is the SQL function memory contexts?
Agreed.
IMO we should comment why we allow only superuser to call this
function.
What about the following?
Thanks!
Modified the patch according to the suggestions.
-----------------
Signal a backend process to log its memory contexts.Only superusers are allowed to signal to log the memory contexts
because allowing any users to issue this request at an unbounded rate
would cause lots of log messages and which can lead to denial of
service.
-----------------+ PGPROC *proc = BackendPidGetProc(pid); + + /* Check whether the target process is PostgreSQL backend process. */ + if (proc == NULL)What about adding more comments as follows?
--------------------------------- + /* + * BackendPidGetProc returns NULL if the pid isn't valid; but by the time + * we reach kill(), a process for which we get a valid proc here might + * have terminated on its own. There's no way to acquire a lock on an + * arbitrary process to prevent that. But since this mechanism is usually + * used to debug a backend running and consuming lots of memory, + * that it might end on its own first and its memory contexts are not + * logged is not a problem. + */ + if (proc == NULL) + { + /* + * This is just a warning so a loop-through-resultset will not abort + * if one backend logged its memory contexts during the run. + */ + ereport(WARNING, ---------------------------------+ ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("must be a superuser to log memory contexts"))); + PG_RETURN_BOOL(false);This PG_RETURN_BOOL(false) is unnecessary because ereport() will emit
an ERROR?+$node->psql('postgres', 'select pg_log_backend_memory_contexts(pg_backend_pid())'); + +my $logfile = slurp_file($node->logfile); +like($logfile, qr/Grand total: \d+ bytes in \d+ blocks/, + 'found expected memory context print in the log file');Isn't there the case where the memory contexts have not been logged yet
when slurp_file() is called? It's rare, though. If there is the risk
for that,
we should wait for the memory contexts to be actually logged?BTW, I agree that we should have the regression test that calls
pg_log_backend_memory_contexts() and checks, e.g., that it doesn't
cause
segmentation fault. But I'm just wondering if it's a bit overkill to
make perl
scriptand start new node to test only this function...
OK, I removed the perl TAP test and added a regression test running
pg_log_backend_memory_contexts(pg_backend_pid()) and checking it returns
't'.
Regards.
Attachments:
v6-0001-add-memorycontext-elog-print.patchtext/x-diff; name=v6-0001-add-memorycontext-elog-print.patchDownload
From 76a0be1a697939e2568ec39eb6365cffd79569bf Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Tue, 30 Mar 2021 01:58:18 +0900
Subject: [PATCH] After commit 3e98c0bafb28de, we can display the usage of
memory contexts using pg_backend_memory_contexts system view. However, its
target process is limited to the backend which is showing the view. This
patch introduces pg_log_backend_memory_contexts(pid) which logs memory
contexts of the specified backend process.
Currently the number of child contexts to be logged per parent is limited
to 100.
As with MemoryContextStats(), it supposes that practical cases where the
dump gets long will typically be huge numbers of siblings under the same
parent context; while the additional debugging value from seeing details
about individual siblings beyond 100 will not be large.
---
doc/src/sgml/func.sgml | 20 +++
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/tcop/postgres.c | 3 +
src/backend/utils/adt/mcxtfuncs.c | 57 ++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/mmgr/aset.c | 8 +-
src/backend/utils/mmgr/generation.c | 8 +-
src/backend/utils/mmgr/mcxt.c | 163 +++++++++++++++----
src/backend/utils/mmgr/slab.c | 9 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/miscadmin.h | 1 +
src/include/nodes/memnodes.h | 6 +-
src/include/storage/procsignal.h | 3 +
src/include/utils/memutils.h | 4 +-
src/test/regress/expected/misc_functions.out | 13 ++
src/test/regress/sql/misc_functions.sql | 9 +
16 files changed, 270 insertions(+), 45 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index fbf6062d0a..489ef5cd95 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24832,6 +24832,26 @@ SELECT collation for ('foo' COLLATE "de_DE");
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_log_backend_memory_contexts</primary>
+ </indexterm>
+ <function>pg_log_backend_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Log the memory contexts whose backend process has the specified
+ process ID.
+ Memory contexts will be logged based on the log configuration set.
+ See <xref linkend="runtime-config-logging"/> for more information.
+ The number of child contexts per parent is limited to 100.
+ For more than 100 contexts, a summary will be shown.
+ Only superusers can log the memory contexts even when the specified
+ process is non-superuser backend.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index c6a8d4611e..eac6895141 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -657,6 +658,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
+ HandleLogMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2b1b68109f..afaf3b1cce 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3295,6 +3295,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (LogMemoryContextPending)
+ ProcessLogMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..2dbdab406a 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -18,6 +18,8 @@
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
#include "utils/builtins.h"
/* ----------
@@ -61,7 +63,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
@@ -155,3 +157,56 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
return (Datum) 0;
}
+
+/*
+ * pg_log_backend_memory_contexts
+ * Signal a backend process to log its memory contexts.
+ *
+ * Only superusers are allowed to signal to log the memory contexts
+ * because allowing any users to issue this request at an unbounded
+ * rate would cause lots of log messages and which can lead to
+ * denial of service.
+ */
+Datum
+pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ PGPROC *proc = BackendPidGetProc(pid);
+
+ /*
+ * BackendPidGetProc returns NULL if the pid isn't valid; but by the
+ * time we reach kill(), a process for which we get a valid proc here
+ * might have terminated on its own. There's no way to acquire a
+ * lock on an arbitrary process to prevent that. But since this
+ * mechanism is usually used to debug a backend running and consuming
+ * lots of memory, that it might end on its own first and its memory
+ * contexts are not logged is not a problem.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend logged its memory contexts during the run.
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+
+ PG_RETURN_BOOL(false);
+ }
+
+ /* Only allow superusers to log memory contexts. */
+ if (!superuser())
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be a superuser to log memory contexts")));
+
+ if(!SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId))
+ PG_RETURN_BOOL(true);
+ else
+ {
+ ereport(WARNING,
+ (errmsg("could not send signal to process %d: %m", pid)));
+
+ PG_RETURN_BOOL(false);
+ }
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 73e0a672ae..6c27065f96 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -34,6 +34,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t LogMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index ec6c130d0f..980c092558 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output is stderr or elog.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool is_dst_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 2b90034764..b286308aa4 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool is_dst_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 84472b9158..f1ed61e5ad 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -55,9 +59,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool is_dst_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -499,7 +505,7 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);
+ MemoryContextStatsDetail(context, 100, true);
}
/*
@@ -508,19 +514,28 @@ MemoryContextStats(MemoryContext context)
* Entry point for use if you want to vary the number of child contexts shown.
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)
+MemoryContextStatsDetail(MemoryContext context, int max_children, bool is_dst_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);
+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, is_dst_stderr);
- fprintf(stderr,
- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
- grand_totals.totalspace, grand_totals.nblocks,
- grand_totals.freespace, grand_totals.freechunks,
- grand_totals.totalspace - grand_totals.freespace);
+ if (is_dst_stderr)
+ fprintf(stderr,
+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace);
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +548,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool is_dst_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,7 +561,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);
+ totals, is_dst_stderr);
/*
* Examine children. If there are more than max_children of them, we do
@@ -560,11 +576,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
if (ichild < max_children)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);
+ totals,
+ is_dst_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);
+ &local_totals,
+ is_dst_stderr);
}
/* Deal with excess children */
@@ -574,16 +592,31 @@ MemoryContextStatsInternal(MemoryContext context, int level,
{
int i;
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
+ if (is_dst_stderr)
+ {
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,11 +638,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)
+ const char *stats_string,
+ bool is_dst_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
+ char truncated_ident[110];
int i;
/*
@@ -623,9 +658,8 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)
- fprintf(stderr, " ");
- fprintf(stderr, "%s: %s", name, stats_string);
+ truncated_ident[0] = '\0';
+
if (ident)
{
/*
@@ -637,24 +671,42 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcpy(truncated_ident, ": ");
+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");
+
+ i = strlen(truncated_ident);
+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);
+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';
+
if (truncated)
- fprintf(stderr, "...");
+ strcat(truncated_ident, "...");
}
- fputc('\n', stderr);
+
+ if (is_dst_stderr)
+ {
+ for (i = 0; i < level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d %s: %s%s",
+ level, name, stats_string, truncated_ident)));
}
/*
@@ -946,6 +998,53 @@ MemoryContextAllocExtended(MemoryContext context, Size size, int flags)
return ret;
}
+/*
+ * HandleLogMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating logging of memory
+ * contexts.
+ *
+ * All the actual work is deferred to ProcessLogMemoryContextInterrupt(),
+ * because we cannot safely emit a log message inside the signal handler.
+ */
+void
+HandleLogMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ LogMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessLogMemoryContextInterrupt
+ * Perform logging of memory contexts of this backend process.
+ *
+ * Any backend that participates in ProcSignal signaling must arrange
+ * to call this function if we see LogMemoryContextPending set.
+ * It is called from CHECK_FOR_INTERRUPTS(), which is enough because
+ * the target process for logging of memory contexts is a backend.
+ */
+void
+ProcessLogMemoryContextInterrupt(void)
+{
+ LogMemoryContextPending = false;
+
+ ereport(LOG,
+ (errmsg("Logging memory contexts of PID %d", MyProcPid)));
+
+ /*
+ * When a backend process is consuming huge memory, logging all its
+ * memory contexts might overrun available disk space. To prevent
+ * this, we limit the number of child contexts per parent to 100.
+ *
+ * As with MemoryContextStats(), we suppose that practical cases
+ * where the dump gets long will typically be huge numbers of
+ * siblings under the same parent context; while the additional
+ * debugging value from seeing details about individual siblings
+ * beyond 100 will not be large.
+ */
+ MemoryContextStatsDetail(TopMemoryContext, 100, false);
+}
+
void *
palloc(Size size)
{
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 9213be7c95..479b08bedd 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool is_dst_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index bfb89e0575..8049587707 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7946,6 +7946,12 @@
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# log memory contexts of the specified backend
+{ oid => '4543', descr => 'log memory contexts of the specified backend',
+ proname => 'pg_log_backend_memory_contexts',
+ provolatile => 'v', prorettype => 'bool',
+ proargtypes => 'int4', prosrc => 'pg_log_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 013850ac28..081822823c 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index 9331ef80fd..87a8501c37 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool is_dst_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..55ea7104b3 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,8 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_LOG_MEMORY_CONTEXT, /* ask specified backend to log the
+ memory contexts */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
@@ -69,6 +71,7 @@ extern int SendProcSignal(pid_t pid, ProcSignalReason reason,
extern uint64 EmitProcSignalBarrier(ProcSignalBarrierType type);
extern void WaitForProcSignalBarrier(uint64 generation);
extern void ProcessProcSignalBarrier(void);
+extern void ProcessLogMemoryContextInterrupt(void);
extern void procsignal_sigusr1_handler(SIGNAL_ARGS);
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 36aae4e51c..1ebf5797cb 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -84,7 +84,8 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);
+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool is_dst_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
@@ -144,6 +145,7 @@ extern void MemoryContextCreate(MemoryContext node,
MemoryContext parent,
const char *name);
+extern void HandleLogMemoryContextInterrupt(void);
/*
* Memory-context-type-specific functions
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index d3acb98d04..e845042d38 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -133,6 +133,19 @@ ERROR: function num_nulls() does not exist
LINE 1: SELECT num_nulls();
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
+--
+-- pg_log_backend_memory_contexts()
+--
+-- Memory contexts are logged and they are not returned to the function.
+-- Furthermore, their contents can vary depending on the timing. However,
+-- we can at least verify that the code doesn't fail.
+--
+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+
--
-- Test some built-in SRFs
--
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 094e8f8296..a398349afc 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -30,6 +30,15 @@ SELECT num_nulls(VARIADIC '{}'::int[]);
SELECT num_nonnulls();
SELECT num_nulls();
+--
+-- pg_log_backend_memory_contexts()
+--
+-- Memory contexts are logged and they are not returned to the function.
+-- Furthermore, their contents can vary depending on the timing. However,
+-- we can at least verify that the code doesn't fail.
+--
+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());
+
--
-- Test some built-in SRFs
--
--
2.18.1
On 2021/03/30 22:06, torikoshia wrote:
Modified the patch according to the suggestions.
Thanks for updating the patch!
I applied the cosmetic changes to the patch and added the example of
the function call into the document. Attached is the updated version
of the patch. Could you check this version?
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
Attachments:
v7-0001-add-memorycontext-elog-print.patchtext/plain; charset=UTF-8; name=v7-0001-add-memorycontext-elog-print.patch; x-mac-creator=0; x-mac-type=0Download
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index fbf6062d0a..aa9080bddc 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24917,6 +24917,23 @@ SELECT collation for ('foo' COLLATE "de_DE");
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_log_backend_memory_contexts</primary>
+ </indexterm>
+ <function>pg_log_backend_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Logs the memory contexts whose backend process has the specified
+ process ID.
+ Memory contexts will be logged based on the log configuration set.
+ See <xref linkend="runtime-config-logging"/> for more information.
+ Only superusers can log the memory contexts.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
@@ -24987,6 +25004,35 @@ SELECT collation for ('foo' COLLATE "de_DE");
<structname>pg_stat_activity</structname> view.
</para>
+ <para>
+ <function>pg_log_backend_memory_contexts</function> can be used
+ to log the memory contexts of the backend process. For example,
+<programlisting>
+postgres=# SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+
+The memory contexts will be logged in the log file. For example:
+LOG: logging memory contexts of PID 10377
+STATEMENT: SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+LOG: level: 0; TopMemoryContext: 80800 total in 6 blocks; 14432 free (5 chunks); 66368 used
+LOG: level: 1; pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1408 free (0 chunks); 6784 used
+LOG: level: 1; TopTransactionContext: 8192 total in 1 blocks; 7720 free (1 chunks); 472 used
+LOG: level: 1; RowDescriptionContext: 8192 total in 1 blocks; 6880 free (0 chunks); 1312 used
+LOG: level: 1; MessageContext: 16384 total in 2 blocks; 5152 free (0 chunks); 11232 used
+LOG: level: 1; Operator class cache: 8192 total in 1 blocks; 512 free (0 chunks); 7680 used
+LOG: level: 1; smgr relation table: 16384 total in 2 blocks; 4544 free (3 chunks); 11840 used
+LOG: level: 1; TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
+...
+LOG: level: 1; ErrorContext: 8192 total in 1 blocks; 7928 free (3 chunks); 264 used
+LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 used
+</programlisting>
+ For more than 100 child contexts under the same parent one,
+ 100 child contexts and a summary of the remaining ones will be logged.
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index c6a8d4611e..eac6895141 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -657,6 +658,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
+ HandleLogMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2b1b68109f..afaf3b1cce 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3295,6 +3295,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (LogMemoryContextPending)
+ ProcessLogMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..fe9b7979e2 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -18,6 +18,8 @@
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
#include "utils/builtins.h"
/* ----------
@@ -61,7 +63,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
@@ -155,3 +157,59 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
return (Datum) 0;
}
+
+/*
+ * pg_log_backend_memory_contexts
+ * Signal a backend process to log its memory contexts.
+ *
+ * Only superusers are allowed to signal to log the memory contexts
+ * because allowing any users to issue this request at an unbounded
+ * rate would cause lots of log messages and which can lead to
+ * denial of service.
+ *
+ * On receipt of this signal, a backend sets the flag in the signal
+ * handler, and then which causes the next CHECK_FOR_INTERRUPTS()
+ * to log the memory contexts.
+ */
+Datum
+pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ PGPROC *proc = BackendPidGetProc(pid);
+
+ /*
+ * BackendPidGetProc returns NULL if the pid isn't valid; but by the time
+ * we reach kill(), a process for which we get a valid proc here might
+ * have terminated on its own. There's no way to acquire a lock on an
+ * arbitrary process to prevent that. But since this mechanism is usually
+ * used to debug a backend running and consuming lots of memory, that it
+ * might end on its own first and its memory contexts are not logged is
+ * not a problem.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend logged its memory contexts during the run.
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ /* Only allow superusers to log memory contexts. */
+ if (!superuser())
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be a superuser to log memory contexts")));
+
+ if (SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId) < 0)
+ {
+ /* Again, just a warning to allow loops */
+ ereport(WARNING,
+ (errmsg("could not send signal to process %d: %m", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ PG_RETURN_BOOL(true);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 73e0a672ae..6c27065f96 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -34,6 +34,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t LogMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index ec6c130d0f..77872e77bc 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool print_to_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 2b90034764..584cd614da 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool print_to_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 84472b9158..bd41f8d31a 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -55,9 +59,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool print_to_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -499,7 +505,7 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);
+ MemoryContextStatsDetail(context, 100, true);
}
/*
@@ -508,19 +514,34 @@ MemoryContextStats(MemoryContext context)
* Entry point for use if you want to vary the number of child contexts shown.
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)
+MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool print_to_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);
+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, print_to_stderr);
- fprintf(stderr,
- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
- grand_totals.totalspace, grand_totals.nblocks,
- grand_totals.freespace, grand_totals.freechunks,
- grand_totals.totalspace - grand_totals.freespace);
+ if (print_to_stderr)
+ fprintf(stderr,
+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace);
+ else
+
+ /*
+ * Use LOG_SERVER_ONLY to prevent the memory contexts from being sent
+ * to the connected client.
+ */
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +554,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool print_to_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,7 +567,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);
+ totals, print_to_stderr);
/*
* Examine children. If there are more than max_children of them, we do
@@ -560,11 +582,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
if (ichild < max_children)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);
+ totals,
+ print_to_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);
+ &local_totals,
+ print_to_stderr);
}
/* Deal with excess children */
@@ -572,18 +596,33 @@ MemoryContextStatsInternal(MemoryContext context, int level,
{
if (print)
{
- int i;
+ if (print_to_stderr)
+ {
+ int i;
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,11 +644,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)
+ const char *stats_string,
+ bool print_to_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
+ char truncated_ident[110];
int i;
/*
@@ -623,9 +664,8 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)
- fprintf(stderr, " ");
- fprintf(stderr, "%s: %s", name, stats_string);
+ truncated_ident[0] = '\0';
+
if (ident)
{
/*
@@ -637,24 +677,41 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcpy(truncated_ident, ": ");
+ i = strlen(truncated_ident);
+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");
+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);
+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';
+
if (truncated)
- fprintf(stderr, "...");
+ strcat(truncated_ident, "...");
}
- fputc('\n', stderr);
+
+ if (print_to_stderr)
+ {
+ for (i = 0; i < level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %s: %s%s",
+ level, name, stats_string, truncated_ident)));
}
/*
@@ -946,6 +1003,52 @@ MemoryContextAllocExtended(MemoryContext context, Size size, int flags)
return ret;
}
+/*
+ * HandleLogMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating logging of memory
+ * contexts.
+ *
+ * All the actual work is deferred to ProcessLogMemoryContextInterrupt(),
+ * because we cannot safely emit a log message inside the signal handler.
+ */
+void
+HandleLogMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ LogMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessLogMemoryContextInterrupt
+ * Perform logging of memory contexts of this backend process.
+ *
+ * Any backend that participates in ProcSignal signaling must arrange
+ * to call this function if we see LogMemoryContextPending set.
+ * It is called from CHECK_FOR_INTERRUPTS(), which is enough because
+ * the target process for logging of memory contexts is a backend.
+ */
+void
+ProcessLogMemoryContextInterrupt(void)
+{
+ LogMemoryContextPending = false;
+
+ ereport(LOG,
+ (errmsg("logging memory contexts of PID %d", MyProcPid)));
+
+ /*
+ * When a backend process is consuming huge memory, logging all its memory
+ * contexts might overrun available disk space. To prevent this, we limit
+ * the number of child contexts to log per parent to 100.
+ *
+ * As with MemoryContextStats(), we suppose that practical cases where the
+ * dump gets long will typically be huge numbers of siblings under the
+ * same parent context; while the additional debugging value from seeing
+ * details about individual siblings beyond 100 will not be large.
+ */
+ MemoryContextStatsDetail(TopMemoryContext, 100, false);
+}
+
void *
palloc(Size size)
{
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 9213be7c95..553dd7f667 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool print_to_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index bfb89e0575..8049587707 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7946,6 +7946,12 @@
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# log memory contexts of the specified backend
+{ oid => '4543', descr => 'log memory contexts of the specified backend',
+ proname => 'pg_log_backend_memory_contexts',
+ provolatile => 'v', prorettype => 'bool',
+ proargtypes => 'int4', prosrc => 'pg_log_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 013850ac28..081822823c 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index 9331ef80fd..e6a757d6a0 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool print_to_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..eec186be2e 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 36aae4e51c..ff872274d4 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -84,7 +84,8 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);
+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool print_to_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
@@ -144,6 +145,8 @@ extern void MemoryContextCreate(MemoryContext node,
MemoryContext parent,
const char *name);
+extern void HandleLogMemoryContextInterrupt(void);
+extern void ProcessLogMemoryContextInterrupt(void);
/*
* Memory-context-type-specific functions
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index d3acb98d04..e845042d38 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -133,6 +133,19 @@ ERROR: function num_nulls() does not exist
LINE 1: SELECT num_nulls();
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
+--
+-- pg_log_backend_memory_contexts()
+--
+-- Memory contexts are logged and they are not returned to the function.
+-- Furthermore, their contents can vary depending on the timing. However,
+-- we can at least verify that the code doesn't fail.
+--
+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+
--
-- Test some built-in SRFs
--
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 094e8f8296..a398349afc 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -30,6 +30,15 @@ SELECT num_nulls(VARIADIC '{}'::int[]);
SELECT num_nonnulls();
SELECT num_nulls();
+--
+-- pg_log_backend_memory_contexts()
+--
+-- Memory contexts are logged and they are not returned to the function.
+-- Furthermore, their contents can vary depending on the timing. However,
+-- we can at least verify that the code doesn't fail.
+--
+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());
+
--
-- Test some built-in SRFs
--
On 2021-03-31 04:36, Fujii Masao wrote:
On 2021/03/30 22:06, torikoshia wrote:
Modified the patch according to the suggestions.
Thanks for updating the patch!
I applied the cosmetic changes to the patch and added the example of
the function call into the document. Attached is the updated version
of the patch. Could you check this version?
Thanks a lot!
+The memory contexts will be logged in the log file. For example:
When 'log_destination = stderr' and 'logging_collector = off', it does
not log in the file but in the stderr.
Description like below would be a bit more accurate but I'm wondering
it's repeating the same words.
+ The memory contexts will be logged based on the log configuration set.
For example:
How do you think?
+<programlisting>
+postgres=# SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+
+The memory contexts will be logged in the log file. For example:
+LOG: logging memory contexts of PID 10377
+STATEMENT: SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+LOG: level: 0; TopMemoryContext: 80800 total in 6 blocks; 14432 free
(5 chunks); 66368 used
+LOG: level: 1; pgstat TabStatusArray lookup hash table: 8192 total in
1 blocks; 1408 free (0 chunks); 6784 used
The line "The memory contexts will be logged in the log file. For
example:"
is neither nor SQL command and its outputs, it might be better to
differentiate it.
What about the following like attached patch?
+<programlisting>
+postgres=# SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+</programlisting>
+The memory contexts will be logged in the log file. For example:
+<screen>
+LOG: logging memory contexts of PID 10377
+STATEMENT: SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+LOG: level: 0; TopMemoryContext: 80800 total in 6 blocks; 14432 free
(5 chunks); 66368 used
+LOG: level: 1; pgstat TabStatusArray lookup hash table: 8192 total in
1 blocks; 1408 free (0 chunks); 6784 used
...(snip)...
+LOG: level: 1; ErrorContext: 8192 total in 1 blocks; 7928 free (3
chunks); 264 used
+LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88
chunks); 1029560 used
+</screen>
Regards.
Attachments:
v8-0001-add-memorycontext-elog-print.patchtext/x-diff; name=v8-0001-add-memorycontext-elog-print.patchDownload
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index fbf6062d0a..ce01d51b21 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24917,6 +24917,23 @@ SELECT collation for ('foo' COLLATE "de_DE");
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_log_backend_memory_contexts</primary>
+ </indexterm>
+ <function>pg_log_backend_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Logs the memory contexts whose backend process has the specified
+ process ID.
+ Memory contexts will be logged based on the log configuration set.
+ See <xref linkend="runtime-config-logging"/> for more information.
+ Only superusers can log the memory contexts.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
@@ -24987,6 +25004,36 @@ SELECT collation for ('foo' COLLATE "de_DE");
<structname>pg_stat_activity</structname> view.
</para>
+ <para>
+ <function>pg_log_backend_memory_contexts</function> can be used
+ to log the memory contexts of the backend process. For example,
+<programlisting>
+postgres=# SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+</programlisting>
+The memory contexts will be logged in the log file. For example:
+<screen>
+LOG: logging memory contexts of PID 10377
+STATEMENT: SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+LOG: level: 0; TopMemoryContext: 80800 total in 6 blocks; 14432 free (5 chunks); 66368 used
+LOG: level: 1; pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1408 free (0 chunks); 6784 used
+LOG: level: 1; TopTransactionContext: 8192 total in 1 blocks; 7720 free (1 chunks); 472 used
+LOG: level: 1; RowDescriptionContext: 8192 total in 1 blocks; 6880 free (0 chunks); 1312 used
+LOG: level: 1; MessageContext: 16384 total in 2 blocks; 5152 free (0 chunks); 11232 used
+LOG: level: 1; Operator class cache: 8192 total in 1 blocks; 512 free (0 chunks); 7680 used
+LOG: level: 1; smgr relation table: 16384 total in 2 blocks; 4544 free (3 chunks); 11840 used
+LOG: level: 1; TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
+...
+LOG: level: 1; ErrorContext: 8192 total in 1 blocks; 7928 free (3 chunks); 264 used
+LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 used
+</screen>
+ For more than 100 child contexts under the same parent one,
+ 100 child contexts and a summary of the remaining ones will be logged.
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index c6a8d4611e..eac6895141 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -657,6 +658,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
+ HandleLogMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2b1b68109f..afaf3b1cce 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3295,6 +3295,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (LogMemoryContextPending)
+ ProcessLogMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..fe9b7979e2 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -18,6 +18,8 @@
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
#include "utils/builtins.h"
/* ----------
@@ -61,7 +63,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
@@ -155,3 +157,59 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
return (Datum) 0;
}
+
+/*
+ * pg_log_backend_memory_contexts
+ * Signal a backend process to log its memory contexts.
+ *
+ * Only superusers are allowed to signal to log the memory contexts
+ * because allowing any users to issue this request at an unbounded
+ * rate would cause lots of log messages and which can lead to
+ * denial of service.
+ *
+ * On receipt of this signal, a backend sets the flag in the signal
+ * handler, and then which causes the next CHECK_FOR_INTERRUPTS()
+ * to log the memory contexts.
+ */
+Datum
+pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ PGPROC *proc = BackendPidGetProc(pid);
+
+ /*
+ * BackendPidGetProc returns NULL if the pid isn't valid; but by the time
+ * we reach kill(), a process for which we get a valid proc here might
+ * have terminated on its own. There's no way to acquire a lock on an
+ * arbitrary process to prevent that. But since this mechanism is usually
+ * used to debug a backend running and consuming lots of memory, that it
+ * might end on its own first and its memory contexts are not logged is
+ * not a problem.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend logged its memory contexts during the run.
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ /* Only allow superusers to log memory contexts. */
+ if (!superuser())
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be a superuser to log memory contexts")));
+
+ if (SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId) < 0)
+ {
+ /* Again, just a warning to allow loops */
+ ereport(WARNING,
+ (errmsg("could not send signal to process %d: %m", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ PG_RETURN_BOOL(true);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 73e0a672ae..6c27065f96 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -34,6 +34,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t LogMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index ec6c130d0f..77872e77bc 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool print_to_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 2b90034764..584cd614da 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool print_to_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 84472b9158..bd41f8d31a 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -55,9 +59,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool print_to_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -499,7 +505,7 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);
+ MemoryContextStatsDetail(context, 100, true);
}
/*
@@ -508,19 +514,34 @@ MemoryContextStats(MemoryContext context)
* Entry point for use if you want to vary the number of child contexts shown.
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)
+MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool print_to_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);
+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, print_to_stderr);
- fprintf(stderr,
- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
- grand_totals.totalspace, grand_totals.nblocks,
- grand_totals.freespace, grand_totals.freechunks,
- grand_totals.totalspace - grand_totals.freespace);
+ if (print_to_stderr)
+ fprintf(stderr,
+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace);
+ else
+
+ /*
+ * Use LOG_SERVER_ONLY to prevent the memory contexts from being sent
+ * to the connected client.
+ */
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +554,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool print_to_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,7 +567,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);
+ totals, print_to_stderr);
/*
* Examine children. If there are more than max_children of them, we do
@@ -560,11 +582,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
if (ichild < max_children)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);
+ totals,
+ print_to_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);
+ &local_totals,
+ print_to_stderr);
}
/* Deal with excess children */
@@ -572,18 +596,33 @@ MemoryContextStatsInternal(MemoryContext context, int level,
{
if (print)
{
- int i;
-
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
+ if (print_to_stderr)
+ {
+ int i;
+
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,11 +644,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)
+ const char *stats_string,
+ bool print_to_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
+ char truncated_ident[110];
int i;
/*
@@ -623,9 +664,8 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)
- fprintf(stderr, " ");
- fprintf(stderr, "%s: %s", name, stats_string);
+ truncated_ident[0] = '\0';
+
if (ident)
{
/*
@@ -637,24 +677,41 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcpy(truncated_ident, ": ");
+ i = strlen(truncated_ident);
+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");
+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);
+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';
+
if (truncated)
- fprintf(stderr, "...");
+ strcat(truncated_ident, "...");
}
- fputc('\n', stderr);
+
+ if (print_to_stderr)
+ {
+ for (i = 0; i < level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %s: %s%s",
+ level, name, stats_string, truncated_ident)));
}
/*
@@ -946,6 +1003,52 @@ MemoryContextAllocExtended(MemoryContext context, Size size, int flags)
return ret;
}
+/*
+ * HandleLogMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating logging of memory
+ * contexts.
+ *
+ * All the actual work is deferred to ProcessLogMemoryContextInterrupt(),
+ * because we cannot safely emit a log message inside the signal handler.
+ */
+void
+HandleLogMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ LogMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessLogMemoryContextInterrupt
+ * Perform logging of memory contexts of this backend process.
+ *
+ * Any backend that participates in ProcSignal signaling must arrange
+ * to call this function if we see LogMemoryContextPending set.
+ * It is called from CHECK_FOR_INTERRUPTS(), which is enough because
+ * the target process for logging of memory contexts is a backend.
+ */
+void
+ProcessLogMemoryContextInterrupt(void)
+{
+ LogMemoryContextPending = false;
+
+ ereport(LOG,
+ (errmsg("logging memory contexts of PID %d", MyProcPid)));
+
+ /*
+ * When a backend process is consuming huge memory, logging all its memory
+ * contexts might overrun available disk space. To prevent this, we limit
+ * the number of child contexts to log per parent to 100.
+ *
+ * As with MemoryContextStats(), we suppose that practical cases where the
+ * dump gets long will typically be huge numbers of siblings under the
+ * same parent context; while the additional debugging value from seeing
+ * details about individual siblings beyond 100 will not be large.
+ */
+ MemoryContextStatsDetail(TopMemoryContext, 100, false);
+}
+
void *
palloc(Size size)
{
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 9213be7c95..553dd7f667 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool print_to_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index bfb89e0575..8049587707 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7946,6 +7946,12 @@
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# log memory contexts of the specified backend
+{ oid => '4543', descr => 'log memory contexts of the specified backend',
+ proname => 'pg_log_backend_memory_contexts',
+ provolatile => 'v', prorettype => 'bool',
+ proargtypes => 'int4', prosrc => 'pg_log_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 013850ac28..081822823c 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index 9331ef80fd..e6a757d6a0 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool print_to_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..eec186be2e 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 36aae4e51c..ff872274d4 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -84,7 +84,8 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);
+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool print_to_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
@@ -144,6 +145,8 @@ extern void MemoryContextCreate(MemoryContext node,
MemoryContext parent,
const char *name);
+extern void HandleLogMemoryContextInterrupt(void);
+extern void ProcessLogMemoryContextInterrupt(void);
/*
* Memory-context-type-specific functions
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index d3acb98d04..e845042d38 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -133,6 +133,19 @@ ERROR: function num_nulls() does not exist
LINE 1: SELECT num_nulls();
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
+--
+-- pg_log_backend_memory_contexts()
+--
+-- Memory contexts are logged and they are not returned to the function.
+-- Furthermore, their contents can vary depending on the timing. However,
+-- we can at least verify that the code doesn't fail.
+--
+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+
--
-- Test some built-in SRFs
--
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 094e8f8296..a398349afc 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -30,6 +30,15 @@ SELECT num_nulls(VARIADIC '{}'::int[]);
SELECT num_nonnulls();
SELECT num_nulls();
+--
+-- pg_log_backend_memory_contexts()
+--
+-- Memory contexts are logged and they are not returned to the function.
+-- Furthermore, their contents can vary depending on the timing. However,
+-- we can at least verify that the code doesn't fail.
+--
+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());
+
--
-- Test some built-in SRFs
--
At Wed, 31 Mar 2021 15:02:02 +0900, torikoshia <torikoshia@oss.nttdata.com> wrote in
On 2021-03-31 04:36, Fujii Masao wrote:
On 2021/03/30 22:06, torikoshia wrote:
Modified the patch according to the suggestions.
Thanks for updating the patch!
I applied the cosmetic changes to the patch and added the example of
the function call into the document. Attached is the updated version
of the patch. Could you check this version?Thanks a lot!
+The memory contexts will be logged in the log file. For example:
When 'log_destination = stderr' and 'logging_collector = off', it does
not log in the file but in the stderr.Description like below would be a bit more accurate but I'm wondering
it's repeating the same words.+ The memory contexts will be logged based on the log configuration
set. For example:How do you think?
How about "The memory contexts will be logged in the server log" ?
I think "server log" doesn't suggest any concrete target.
+<programlisting> +postgres=# SELECT pg_log_backend_memory_contexts(pg_backend_pid()); + pg_log_backend_memory_contexts +-------------------------------- + t +(1 row) + +The memory contexts will be logged in the log file. For example: +LOG: logging memory contexts of PID 10377 +STATEMENT: SELECT pg_log_backend_memory_contexts(pg_backend_pid()); +LOG: level: 0; TopMemoryContext: 80800 total in 6 blocks; 14432 free (5 chunks); 66368 used +LOG: level: 1; pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1408 free (0 chunks); 6784 usedThe line "The memory contexts will be logged in the log file. For
example:"
is neither nor SQL command and its outputs, it might be better to
differentiate it.What about the following like attached patch?
+<programlisting> +postgres=# SELECT pg_log_backend_memory_contexts(pg_backend_pid()); + pg_log_backend_memory_contexts +-------------------------------- + t +(1 row) +</programlisting> +The memory contexts will be logged in the log file. For example: +<screen> +LOG: logging memory contexts of PID 10377 +STATEMENT: SELECT pg_log_backend_memory_contexts(pg_backend_pid()); +LOG: level: 0; TopMemoryContext: 80800 total in 6 blocks; 14432 free (5 chunks); 66368 used +LOG: level: 1; pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1408 free (0 chunks); 6784 used ...(snip)... +LOG: level: 1; ErrorContext: 8192 total in 1 blocks; 7928 free (3 chunks); 264 used +LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 used +</screen>
+1 from me. It looks like more compliant with the standard?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On 2021/03/31 15:16, Kyotaro Horiguchi wrote:
+ The memory contexts will be logged based on the log configuration
set. For example:How do you think?
How about "The memory contexts will be logged in the server log" ?
I think "server log" doesn't suggest any concrete target.
Or just using "logged" is enough?
Also I'd like to document that one message for each memory context is logged.
So what about the following?
One message for each memory context will be logged. For example,
+1 from me. It looks like more compliant with the standard?
+1, too.
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On 2021-04-01 19:13, Fujii Masao wrote:
On 2021/03/31 15:16, Kyotaro Horiguchi wrote:
+ The memory contexts will be logged based on the log configuration
set. For example:How do you think?
How about "The memory contexts will be logged in the server log" ?
I think "server log" doesn't suggest any concrete target.Or just using "logged" is enough?
Also I'd like to document that one message for each memory context is
logged.
So what about the following?One message for each memory context will be logged. For example,
Agreed.
BTW, there was a conflict since c30f54ad732(Detect POLLHUP/POLLRDHUP
while
running queries), attached v9.
Regards,
Attachments:
v9-0001-add-memorycontext-elog-print.patchtext/x-diff; name=v9-0001-add-memorycontext-elog-print.patchDownload
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 3cf243a16a..a20be435ca 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24913,6 +24913,23 @@ SELECT collation for ('foo' COLLATE "de_DE");
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_log_backend_memory_contexts</primary>
+ </indexterm>
+ <function>pg_log_backend_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Logs the memory contexts whose backend process has the specified
+ process ID.
+ Memory contexts will be logged based on the log configuration set.
+ See <xref linkend="runtime-config-logging"/> for more information.
+ Only superusers can log the memory contexts.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
@@ -24983,6 +25000,36 @@ SELECT collation for ('foo' COLLATE "de_DE");
<structname>pg_stat_activity</structname> view.
</para>
+ <para>
+ <function>pg_log_backend_memory_contexts</function> can be used
+ to log the memory contexts of the backend process. For example,
+<programlisting>
+postgres=# SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+</programlisting>
+One message for each memory context will be logged. For example:
+<screen>
+LOG: logging memory contexts of PID 10377
+STATEMENT: SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+LOG: level: 0; TopMemoryContext: 80800 total in 6 blocks; 14432 free (5 chunks); 66368 used
+LOG: level: 1; pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1408 free (0 chunks); 6784 used
+LOG: level: 1; TopTransactionContext: 8192 total in 1 blocks; 7720 free (1 chunks); 472 used
+LOG: level: 1; RowDescriptionContext: 8192 total in 1 blocks; 6880 free (0 chunks); 1312 used
+LOG: level: 1; MessageContext: 16384 total in 2 blocks; 5152 free (0 chunks); 11232 used
+LOG: level: 1; Operator class cache: 8192 total in 1 blocks; 512 free (0 chunks); 7680 used
+LOG: level: 1; smgr relation table: 16384 total in 2 blocks; 4544 free (3 chunks); 11840 used
+LOG: level: 1; TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
+...
+LOG: level: 1; ErrorContext: 8192 total in 1 blocks; 7928 free (3 chunks); 264 used
+LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 used
+</screen>
+ For more than 100 child contexts under the same parent one,
+ 100 child contexts and a summary of the remaining ones will be logged.
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index c6a8d4611e..eac6895141 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -657,6 +658,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
+ HandleLogMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index ad351e2fd1..330ec5b028 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3327,6 +3327,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (LogMemoryContextPending)
+ ProcessLogMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..fe9b7979e2 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -18,6 +18,8 @@
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
#include "utils/builtins.h"
/* ----------
@@ -61,7 +63,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
@@ -155,3 +157,59 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
return (Datum) 0;
}
+
+/*
+ * pg_log_backend_memory_contexts
+ * Signal a backend process to log its memory contexts.
+ *
+ * Only superusers are allowed to signal to log the memory contexts
+ * because allowing any users to issue this request at an unbounded
+ * rate would cause lots of log messages and which can lead to
+ * denial of service.
+ *
+ * On receipt of this signal, a backend sets the flag in the signal
+ * handler, and then which causes the next CHECK_FOR_INTERRUPTS()
+ * to log the memory contexts.
+ */
+Datum
+pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ PGPROC *proc = BackendPidGetProc(pid);
+
+ /*
+ * BackendPidGetProc returns NULL if the pid isn't valid; but by the time
+ * we reach kill(), a process for which we get a valid proc here might
+ * have terminated on its own. There's no way to acquire a lock on an
+ * arbitrary process to prevent that. But since this mechanism is usually
+ * used to debug a backend running and consuming lots of memory, that it
+ * might end on its own first and its memory contexts are not logged is
+ * not a problem.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend logged its memory contexts during the run.
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ /* Only allow superusers to log memory contexts. */
+ if (!superuser())
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be a superuser to log memory contexts")));
+
+ if (SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId) < 0)
+ {
+ /* Again, just a warning to allow loops */
+ ereport(WARNING,
+ (errmsg("could not send signal to process %d: %m", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ PG_RETURN_BOOL(true);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index a9f0fc3017..381d9e548d 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -35,6 +35,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t LogMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index ec6c130d0f..77872e77bc 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool print_to_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 2b90034764..584cd614da 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool print_to_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 84472b9158..bd41f8d31a 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -55,9 +59,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool print_to_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -499,7 +505,7 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);
+ MemoryContextStatsDetail(context, 100, true);
}
/*
@@ -508,19 +514,34 @@ MemoryContextStats(MemoryContext context)
* Entry point for use if you want to vary the number of child contexts shown.
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)
+MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool print_to_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);
+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, print_to_stderr);
- fprintf(stderr,
- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
- grand_totals.totalspace, grand_totals.nblocks,
- grand_totals.freespace, grand_totals.freechunks,
- grand_totals.totalspace - grand_totals.freespace);
+ if (print_to_stderr)
+ fprintf(stderr,
+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace);
+ else
+
+ /*
+ * Use LOG_SERVER_ONLY to prevent the memory contexts from being sent
+ * to the connected client.
+ */
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +554,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool print_to_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,7 +567,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);
+ totals, print_to_stderr);
/*
* Examine children. If there are more than max_children of them, we do
@@ -560,11 +582,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
if (ichild < max_children)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);
+ totals,
+ print_to_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);
+ &local_totals,
+ print_to_stderr);
}
/* Deal with excess children */
@@ -572,18 +596,33 @@ MemoryContextStatsInternal(MemoryContext context, int level,
{
if (print)
{
- int i;
-
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
+ if (print_to_stderr)
+ {
+ int i;
+
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,11 +644,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)
+ const char *stats_string,
+ bool print_to_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
+ char truncated_ident[110];
int i;
/*
@@ -623,9 +664,8 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)
- fprintf(stderr, " ");
- fprintf(stderr, "%s: %s", name, stats_string);
+ truncated_ident[0] = '\0';
+
if (ident)
{
/*
@@ -637,24 +677,41 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcpy(truncated_ident, ": ");
+ i = strlen(truncated_ident);
+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");
+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);
+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';
+
if (truncated)
- fprintf(stderr, "...");
+ strcat(truncated_ident, "...");
}
- fputc('\n', stderr);
+
+ if (print_to_stderr)
+ {
+ for (i = 0; i < level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %s: %s%s",
+ level, name, stats_string, truncated_ident)));
}
/*
@@ -946,6 +1003,52 @@ MemoryContextAllocExtended(MemoryContext context, Size size, int flags)
return ret;
}
+/*
+ * HandleLogMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating logging of memory
+ * contexts.
+ *
+ * All the actual work is deferred to ProcessLogMemoryContextInterrupt(),
+ * because we cannot safely emit a log message inside the signal handler.
+ */
+void
+HandleLogMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ LogMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessLogMemoryContextInterrupt
+ * Perform logging of memory contexts of this backend process.
+ *
+ * Any backend that participates in ProcSignal signaling must arrange
+ * to call this function if we see LogMemoryContextPending set.
+ * It is called from CHECK_FOR_INTERRUPTS(), which is enough because
+ * the target process for logging of memory contexts is a backend.
+ */
+void
+ProcessLogMemoryContextInterrupt(void)
+{
+ LogMemoryContextPending = false;
+
+ ereport(LOG,
+ (errmsg("logging memory contexts of PID %d", MyProcPid)));
+
+ /*
+ * When a backend process is consuming huge memory, logging all its memory
+ * contexts might overrun available disk space. To prevent this, we limit
+ * the number of child contexts to log per parent to 100.
+ *
+ * As with MemoryContextStats(), we suppose that practical cases where the
+ * dump gets long will typically be huge numbers of siblings under the
+ * same parent context; while the additional debugging value from seeing
+ * details about individual siblings beyond 100 will not be large.
+ */
+ MemoryContextStatsDetail(TopMemoryContext, 100, false);
+}
+
void *
palloc(Size size)
{
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 9213be7c95..553dd7f667 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool print_to_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 69ffd0c3f4..7208fbe70d 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7946,6 +7946,12 @@
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# log memory contexts of the specified backend
+{ oid => '4543', descr => 'log memory contexts of the specified backend',
+ proname => 'pg_log_backend_memory_contexts',
+ provolatile => 'v', prorettype => 'bool',
+ proargtypes => 'int4', prosrc => 'pg_log_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 6f8251e0b0..95202d37af 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index 9331ef80fd..e6a757d6a0 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool print_to_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..eec186be2e 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 36aae4e51c..ff872274d4 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -84,7 +84,8 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);
+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool print_to_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
@@ -144,6 +145,8 @@ extern void MemoryContextCreate(MemoryContext node,
MemoryContext parent,
const char *name);
+extern void HandleLogMemoryContextInterrupt(void);
+extern void ProcessLogMemoryContextInterrupt(void);
/*
* Memory-context-type-specific functions
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index d3acb98d04..e845042d38 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -133,6 +133,19 @@ ERROR: function num_nulls() does not exist
LINE 1: SELECT num_nulls();
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
+--
+-- pg_log_backend_memory_contexts()
+--
+-- Memory contexts are logged and they are not returned to the function.
+-- Furthermore, their contents can vary depending on the timing. However,
+-- we can at least verify that the code doesn't fail.
+--
+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+
--
-- Test some built-in SRFs
--
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 094e8f8296..a398349afc 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -30,6 +30,15 @@ SELECT num_nulls(VARIADIC '{}'::int[]);
SELECT num_nonnulls();
SELECT num_nulls();
+--
+-- pg_log_backend_memory_contexts()
+--
+-- Memory contexts are logged and they are not returned to the function.
+-- Furthermore, their contents can vary depending on the timing. However,
+-- we can at least verify that the code doesn't fail.
+--
+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());
+
--
-- Test some built-in SRFs
--
On Sun, Apr 4, 2021 at 7:56 PM torikoshia <torikoshia@oss.nttdata.com>
wrote:
On 2021-04-01 19:13, Fujii Masao wrote:
On 2021/03/31 15:16, Kyotaro Horiguchi wrote:
+ The memory contexts will be logged based on the log configuration
set. For example:How do you think?
How about "The memory contexts will be logged in the server log" ?
I think "server log" doesn't suggest any concrete target.Or just using "logged" is enough?
Also I'd like to document that one message for each memory context is
logged.
So what about the following?One message for each memory context will be logged. For example,
Agreed.
BTW, there was a conflict since c30f54ad732(Detect POLLHUP/POLLRDHUP
while
running queries), attached v9.Regards,
Hi,
+ * On receipt of this signal, a backend sets the flag in the signal
+ * handler, and then which causes the next CHECK_FOR_INTERRUPTS()
I think the 'and then' is not needed:
handler which causes the next ...
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend logged its memory contexts during the run.
The pid given by arg 0 is not a PostgreSQL server process. Which other
backend could it be ?
Thanks
On 2021/04/05 12:20, Zhihong Yu wrote:
+ * This is just a warning so a loop-through-resultset will not abort + * if one backend logged its memory contexts during the run.The pid given by arg 0 is not a PostgreSQL server process. Which other backend could it be ?
This is the comment that I added wrongly. So the comment should be
"This is just a warning so a loop-through-resultset will not abort
if one backend terminated on its own during the run.",
like pg_signal_backend(). Thought?
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On 2021-04-05 12:59, Fujii Masao wrote:
On 2021/04/05 12:20, Zhihong Yu wrote:
Thanks for reviewing!
+ * On receipt of this signal, a backend sets the flag in the signal + * handler, and then which causes the next CHECK_FOR_INTERRUPTS()
I think the 'and then' is not needed:
Although I wonder either would be fine, removed the words.
+ * This is just a warning so a loop-through-resultset will not abort + * if one backend logged its memory contexts during the run.The pid given by arg 0 is not a PostgreSQL server process. Which other
backend could it be ?This is the comment that I added wrongly. So the comment should be
"This is just a warning so a loop-through-resultset will not abort
if one backend terminated on its own during the run.",
like pg_signal_backend(). Thought?
+1.
Attached v10 patch.
Regards,
Attachments:
v10-0001-add-memorycontext-elog-print.patchtext/x-diff; name=v10-0001-add-memorycontext-elog-print.patchDownload
From 8931099cbf3d6e6ef24150496cb795413785f808 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 5 Apr 2021 20:40:12 +0900
Subject: [PATCH v10] After commit 3e98c0bafb28de, we can display the usage of
memory contexts using pg_backend_memory_contexts system view. However, its
target process is limited to the backend which is showing the view. This
patch introduces pg_log_backend_memory_contexts(pid) which logs memory
contexts of the specified backend process.
Currently the number of child contexts to be logged per parent is limited
to 100.
As with MemoryContextStats(), it supposes that practical cases where the
dump gets long will typically be huge numbers of siblings under the same
parent context; while the additional debugging value from seeing details
about individual siblings beyond 100 will not be large.
---
doc/src/sgml/func.sgml | 47 +++++
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/tcop/postgres.c | 3 +
src/backend/utils/adt/mcxtfuncs.c | 60 ++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/mmgr/aset.c | 8 +-
src/backend/utils/mmgr/generation.c | 8 +-
src/backend/utils/mmgr/mcxt.c | 171 +++++++++++++++----
src/backend/utils/mmgr/slab.c | 9 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/miscadmin.h | 1 +
src/include/nodes/memnodes.h | 6 +-
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 5 +-
src/test/regress/expected/misc_functions.out | 13 ++
src/test/regress/sql/misc_functions.sql | 9 +
16 files changed, 305 insertions(+), 47 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 3cf243a16a..a20be435ca 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24913,6 +24913,23 @@ SELECT collation for ('foo' COLLATE "de_DE");
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_log_backend_memory_contexts</primary>
+ </indexterm>
+ <function>pg_log_backend_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Logs the memory contexts whose backend process has the specified
+ process ID.
+ Memory contexts will be logged based on the log configuration set.
+ See <xref linkend="runtime-config-logging"/> for more information.
+ Only superusers can log the memory contexts.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
@@ -24983,6 +25000,36 @@ SELECT collation for ('foo' COLLATE "de_DE");
<structname>pg_stat_activity</structname> view.
</para>
+ <para>
+ <function>pg_log_backend_memory_contexts</function> can be used
+ to log the memory contexts of the backend process. For example,
+<programlisting>
+postgres=# SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+</programlisting>
+One message for each memory context will be logged. For example:
+<screen>
+LOG: logging memory contexts of PID 10377
+STATEMENT: SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+LOG: level: 0; TopMemoryContext: 80800 total in 6 blocks; 14432 free (5 chunks); 66368 used
+LOG: level: 1; pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1408 free (0 chunks); 6784 used
+LOG: level: 1; TopTransactionContext: 8192 total in 1 blocks; 7720 free (1 chunks); 472 used
+LOG: level: 1; RowDescriptionContext: 8192 total in 1 blocks; 6880 free (0 chunks); 1312 used
+LOG: level: 1; MessageContext: 16384 total in 2 blocks; 5152 free (0 chunks); 11232 used
+LOG: level: 1; Operator class cache: 8192 total in 1 blocks; 512 free (0 chunks); 7680 used
+LOG: level: 1; smgr relation table: 16384 total in 2 blocks; 4544 free (3 chunks); 11840 used
+LOG: level: 1; TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
+...
+LOG: level: 1; ErrorContext: 8192 total in 1 blocks; 7928 free (3 chunks); 264 used
+LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 used
+</screen>
+ For more than 100 child contexts under the same parent one,
+ 100 child contexts and a summary of the remaining ones will be logged.
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index c6a8d4611e..eac6895141 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -657,6 +658,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
+ HandleLogMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index ad351e2fd1..330ec5b028 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3327,6 +3327,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (LogMemoryContextPending)
+ ProcessLogMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..e2b87a7ed9 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -18,6 +18,8 @@
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
#include "utils/builtins.h"
/* ----------
@@ -61,7 +63,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
@@ -155,3 +157,59 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
return (Datum) 0;
}
+
+/*
+ * pg_log_backend_memory_contexts
+ * Signal a backend process to log its memory contexts.
+ *
+ * Only superusers are allowed to signal to log the memory contexts
+ * because allowing any users to issue this request at an unbounded
+ * rate would cause lots of log messages and which can lead to
+ * denial of service.
+ *
+ * On receipt of this signal, a backend sets the flag in the signal
+ * handler, which causes the next CHECK_FOR_INTERRUPTS() to log the
+ * memory contexts.
+ */
+Datum
+pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ PGPROC *proc = BackendPidGetProc(pid);
+
+ /*
+ * BackendPidGetProc returns NULL if the pid isn't valid; but by the time
+ * we reach kill(), a process for which we get a valid proc here might
+ * have terminated on its own. There's no way to acquire a lock on an
+ * arbitrary process to prevent that. But since this mechanism is usually
+ * used to debug a backend running and consuming lots of memory, that it
+ * might end on its own first and its memory contexts are not logged is
+ * not a problem.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ /* Only allow superusers to log memory contexts. */
+ if (!superuser())
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be a superuser to log memory contexts")));
+
+ if (SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId) < 0)
+ {
+ /* Again, just a warning to allow loops */
+ ereport(WARNING,
+ (errmsg("could not send signal to process %d: %m", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ PG_RETURN_BOOL(true);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index a9f0fc3017..381d9e548d 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -35,6 +35,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t LogMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index ec6c130d0f..77872e77bc 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool print_to_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 2b90034764..584cd614da 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool print_to_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 84472b9158..bd41f8d31a 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -55,9 +59,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool print_to_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -499,7 +505,7 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);
+ MemoryContextStatsDetail(context, 100, true);
}
/*
@@ -508,19 +514,34 @@ MemoryContextStats(MemoryContext context)
* Entry point for use if you want to vary the number of child contexts shown.
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)
+MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool print_to_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);
+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, print_to_stderr);
- fprintf(stderr,
- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
- grand_totals.totalspace, grand_totals.nblocks,
- grand_totals.freespace, grand_totals.freechunks,
- grand_totals.totalspace - grand_totals.freespace);
+ if (print_to_stderr)
+ fprintf(stderr,
+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace);
+ else
+
+ /*
+ * Use LOG_SERVER_ONLY to prevent the memory contexts from being sent
+ * to the connected client.
+ */
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +554,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool print_to_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,7 +567,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);
+ totals, print_to_stderr);
/*
* Examine children. If there are more than max_children of them, we do
@@ -560,11 +582,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
if (ichild < max_children)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);
+ totals,
+ print_to_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);
+ &local_totals,
+ print_to_stderr);
}
/* Deal with excess children */
@@ -572,18 +596,33 @@ MemoryContextStatsInternal(MemoryContext context, int level,
{
if (print)
{
- int i;
-
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
+ if (print_to_stderr)
+ {
+ int i;
+
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,11 +644,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)
+ const char *stats_string,
+ bool print_to_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
+ char truncated_ident[110];
int i;
/*
@@ -623,9 +664,8 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)
- fprintf(stderr, " ");
- fprintf(stderr, "%s: %s", name, stats_string);
+ truncated_ident[0] = '\0';
+
if (ident)
{
/*
@@ -637,24 +677,41 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcpy(truncated_ident, ": ");
+ i = strlen(truncated_ident);
+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");
+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);
+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';
+
if (truncated)
- fprintf(stderr, "...");
+ strcat(truncated_ident, "...");
}
- fputc('\n', stderr);
+
+ if (print_to_stderr)
+ {
+ for (i = 0; i < level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %s: %s%s",
+ level, name, stats_string, truncated_ident)));
}
/*
@@ -946,6 +1003,52 @@ MemoryContextAllocExtended(MemoryContext context, Size size, int flags)
return ret;
}
+/*
+ * HandleLogMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating logging of memory
+ * contexts.
+ *
+ * All the actual work is deferred to ProcessLogMemoryContextInterrupt(),
+ * because we cannot safely emit a log message inside the signal handler.
+ */
+void
+HandleLogMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ LogMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessLogMemoryContextInterrupt
+ * Perform logging of memory contexts of this backend process.
+ *
+ * Any backend that participates in ProcSignal signaling must arrange
+ * to call this function if we see LogMemoryContextPending set.
+ * It is called from CHECK_FOR_INTERRUPTS(), which is enough because
+ * the target process for logging of memory contexts is a backend.
+ */
+void
+ProcessLogMemoryContextInterrupt(void)
+{
+ LogMemoryContextPending = false;
+
+ ereport(LOG,
+ (errmsg("logging memory contexts of PID %d", MyProcPid)));
+
+ /*
+ * When a backend process is consuming huge memory, logging all its memory
+ * contexts might overrun available disk space. To prevent this, we limit
+ * the number of child contexts to log per parent to 100.
+ *
+ * As with MemoryContextStats(), we suppose that practical cases where the
+ * dump gets long will typically be huge numbers of siblings under the
+ * same parent context; while the additional debugging value from seeing
+ * details about individual siblings beyond 100 will not be large.
+ */
+ MemoryContextStatsDetail(TopMemoryContext, 100, false);
+}
+
void *
palloc(Size size)
{
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 9213be7c95..553dd7f667 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool print_to_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 69ffd0c3f4..73c22c8b4d 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7946,6 +7946,12 @@
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# logging memory contexts of the specified backend
+{ oid => '4543', descr => 'log memory contexts of the specified backend',
+ proname => 'pg_log_backend_memory_contexts',
+ provolatile => 'v', prorettype => 'bool',
+ proargtypes => 'int4', prosrc => 'pg_log_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 6f8251e0b0..95202d37af 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index 9331ef80fd..e6a757d6a0 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool print_to_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..eec186be2e 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 36aae4e51c..ff872274d4 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -84,7 +84,8 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);
+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool print_to_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
@@ -144,6 +145,8 @@ extern void MemoryContextCreate(MemoryContext node,
MemoryContext parent,
const char *name);
+extern void HandleLogMemoryContextInterrupt(void);
+extern void ProcessLogMemoryContextInterrupt(void);
/*
* Memory-context-type-specific functions
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index d3acb98d04..e845042d38 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -133,6 +133,19 @@ ERROR: function num_nulls() does not exist
LINE 1: SELECT num_nulls();
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
+--
+-- pg_log_backend_memory_contexts()
+--
+-- Memory contexts are logged and they are not returned to the function.
+-- Furthermore, their contents can vary depending on the timing. However,
+-- we can at least verify that the code doesn't fail.
+--
+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+
--
-- Test some built-in SRFs
--
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 094e8f8296..a398349afc 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -30,6 +30,15 @@ SELECT num_nulls(VARIADIC '{}'::int[]);
SELECT num_nonnulls();
SELECT num_nulls();
+--
+-- pg_log_backend_memory_contexts()
+--
+-- Memory contexts are logged and they are not returned to the function.
+-- Furthermore, their contents can vary depending on the timing. However,
+-- we can at least verify that the code doesn't fail.
+--
+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());
+
--
-- Test some built-in SRFs
--
--
2.18.1
On 2021/04/05 21:03, torikoshia wrote:
On 2021-04-05 12:59, Fujii Masao wrote:
On 2021/04/05 12:20, Zhihong Yu wrote:
Thanks for reviewing!
+ * On receipt of this signal, a backend sets the flag in the signal + * handler, and then which causes the next CHECK_FOR_INTERRUPTS()I think the 'and then' is not needed:
Although I wonder either would be fine, removed the words.
+ * This is just a warning so a loop-through-resultset will not abort + * if one backend logged its memory contexts during the run.The pid given by arg 0 is not a PostgreSQL server process. Which other backend could it be ?
This is the comment that I added wrongly. So the comment should be
"This is just a warning so a loop-through-resultset will not abort
if one backend terminated on its own during the run.",
like pg_signal_backend(). Thought?+1.
Attached v10 patch.
Thanks for updating the patch!
I updated the patch as follows. Could you check the attached patch?
+ Memory contexts will be logged based on the log configuration set.
+ See <xref linkend="runtime-config-logging"/> for more information.
Those memory contexts are logged at LOG level, but they are not sent to
a client whatever the setting of client_min_messages. I think
this information should be documented. So I updated the document as follows.
These memory contexts will be logged at <literal>LOG</literal>
message level. They will appear in the server log based on
the log configuration set (See <xref linkend="runtime-config-logging"/>
for more information), but will not be sent to the client whatever
the setting of <xref linkend="guc-client-min-messages"/>.
+ Only superusers can log the memory contexts.
We can read this description as "only superusers can request to log ...".
But ISTM that we can also read this as "only superusers can log (dump)
the memory contexts of its backend". Right? To avoid this confusion,
I updated this description as follows.
Only superusers can request to log the memory contexts.
I added the following note about the performance overhead by this function.
Note that frequent calls to this function could incur significant overhead,
because it may generate a large number of log messages.
I also added some comments.
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
Attachments:
v11-0001-add-memorycontext-elog-print.patchtext/plain; charset=UTF-8; name=v11-0001-add-memorycontext-elog-print.patch; x-mac-creator=0; x-mac-type=0Download
From 16700e49635737c44f2342aaa79618249077b021 Mon Sep 17 00:00:00 2001
From: Fujii Masao <fujii@postgresql.org>
Date: Mon, 5 Apr 2021 23:44:36 +0900
Subject: [PATCH v11] After commit 3e98c0bafb28de, we can display the usage of
memory contexts using pg_backend_memory_contexts system view. However, its
target process is limited to the backend which is showing the view. This
patch introduces pg_log_backend_memory_contexts(pid) which logs memory
contexts of the specified backend process.
Currently the number of child contexts to be logged per parent is limited
to 100.
As with MemoryContextStats(), it supposes that practical cases where the
dump gets long will typically be huge numbers of siblings under the same
parent context; while the additional debugging value from seeing details
about individual siblings beyond 100 will not be large.
---
doc/src/sgml/func.sgml | 52 ++++++
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/tcop/postgres.c | 3 +
src/backend/utils/adt/mcxtfuncs.c | 60 ++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/mmgr/aset.c | 8 +-
src/backend/utils/mmgr/generation.c | 8 +-
src/backend/utils/mmgr/mcxt.c | 180 +++++++++++++++----
src/backend/utils/mmgr/slab.c | 9 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/miscadmin.h | 1 +
src/include/nodes/memnodes.h | 6 +-
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 5 +-
src/test/regress/expected/misc_functions.out | 13 ++
src/test/regress/sql/misc_functions.sql | 9 +
16 files changed, 319 insertions(+), 47 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 3cf243a16a..9fcee74910 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -24913,6 +24913,26 @@ SELECT collation for ('foo' COLLATE "de_DE");
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_log_backend_memory_contexts</primary>
+ </indexterm>
+ <function>pg_log_backend_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Requests to log the memory contexts whose backend process has
+ the specified process ID. These memory contexts will be logged at
+ <literal>LOG</literal> message level. They will appear in
+ the server log based on the log configuration set
+ (See <xref linkend="runtime-config-logging"/> for more information),
+ but will not be sent to the client whatever the setting of
+ <xref linkend="guc-client-min-messages"/>.
+ Only superusers can request to log the memory contexts.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
@@ -24983,6 +25003,38 @@ SELECT collation for ('foo' COLLATE "de_DE");
<structname>pg_stat_activity</structname> view.
</para>
+ <para>
+ <function>pg_log_backend_memory_contexts</function> can be used
+ to log the memory contexts of the backend process. For example,
+<programlisting>
+postgres=# SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+</programlisting>
+One message for each memory context will be logged. For example:
+<screen>
+LOG: logging memory contexts of PID 10377
+STATEMENT: SELECT pg_log_backend_memory_contexts(pg_backend_pid());
+LOG: level: 0; TopMemoryContext: 80800 total in 6 blocks; 14432 free (5 chunks); 66368 used
+LOG: level: 1; pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1408 free (0 chunks); 6784 used
+LOG: level: 1; TopTransactionContext: 8192 total in 1 blocks; 7720 free (1 chunks); 472 used
+LOG: level: 1; RowDescriptionContext: 8192 total in 1 blocks; 6880 free (0 chunks); 1312 used
+LOG: level: 1; MessageContext: 16384 total in 2 blocks; 5152 free (0 chunks); 11232 used
+LOG: level: 1; Operator class cache: 8192 total in 1 blocks; 512 free (0 chunks); 7680 used
+LOG: level: 1; smgr relation table: 16384 total in 2 blocks; 4544 free (3 chunks); 11840 used
+LOG: level: 1; TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
+...
+LOG: level: 1; ErrorContext: 8192 total in 1 blocks; 7928 free (3 chunks); 264 used
+LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 used
+</screen>
+ For more than 100 child contexts under the same parent one,
+ 100 child contexts and a summary of the remaining ones will be logged.
+ Note that frequent calls to this function could incur significant overhead,
+ because it may generate a large number of log messages.
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index c6a8d4611e..eac6895141 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -657,6 +658,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
+ HandleLogMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index ad351e2fd1..330ec5b028 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3327,6 +3327,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (LogMemoryContextPending)
+ ProcessLogMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c02fa47550..e2b87a7ed9 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -18,6 +18,8 @@
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
#include "utils/builtins.h"
/* ----------
@@ -61,7 +63,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);
+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
@@ -155,3 +157,59 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
return (Datum) 0;
}
+
+/*
+ * pg_log_backend_memory_contexts
+ * Signal a backend process to log its memory contexts.
+ *
+ * Only superusers are allowed to signal to log the memory contexts
+ * because allowing any users to issue this request at an unbounded
+ * rate would cause lots of log messages and which can lead to
+ * denial of service.
+ *
+ * On receipt of this signal, a backend sets the flag in the signal
+ * handler, which causes the next CHECK_FOR_INTERRUPTS() to log the
+ * memory contexts.
+ */
+Datum
+pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ PGPROC *proc = BackendPidGetProc(pid);
+
+ /*
+ * BackendPidGetProc returns NULL if the pid isn't valid; but by the time
+ * we reach kill(), a process for which we get a valid proc here might
+ * have terminated on its own. There's no way to acquire a lock on an
+ * arbitrary process to prevent that. But since this mechanism is usually
+ * used to debug a backend running and consuming lots of memory, that it
+ * might end on its own first and its memory contexts are not logged is
+ * not a problem.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ (errmsg("PID %d is not a PostgreSQL server process", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ /* Only allow superusers to log memory contexts. */
+ if (!superuser())
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be a superuser to log memory contexts")));
+
+ if (SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId) < 0)
+ {
+ /* Again, just a warning to allow loops */
+ ereport(WARNING,
+ (errmsg("could not send signal to process %d: %m", pid)));
+ PG_RETURN_BOOL(false);
+ }
+
+ PG_RETURN_BOOL(true);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index a9f0fc3017..381d9e548d 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -35,6 +35,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t LogMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index ec6c130d0f..77872e77bc 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool print_to_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 2b90034764..584cd614da 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals, bool print_to_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 84472b9158..6919a73280 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"
+#include "storage/procarray.h"
+#include "storage/procsignal.h"
+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -55,9 +59,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool print_to_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -499,28 +505,52 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);
+ MemoryContextStatsDetail(context, 100, true);
}
/*
* MemoryContextStatsDetail
*
* Entry point for use if you want to vary the number of child contexts shown.
+ *
+ * If print_to_stderr is true, print statistics about the memory contexts
+ * with fprintf(stderr), otherwise use ereport().
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)
+MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool print_to_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);
+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, print_to_stderr);
+
+ if (print_to_stderr)
+ fprintf(stderr,
+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace);
+ else
- fprintf(stderr,
- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",
- grand_totals.totalspace, grand_totals.nblocks,
- grand_totals.freespace, grand_totals.freechunks,
- grand_totals.totalspace - grand_totals.freespace);
+ /*
+ * Use LOG_SERVER_ONLY to prevent the memory contexts from being sent
+ * to the connected client.
+ *
+ * We don't buffer the information about all memory contexts in a
+ * backend into StringInfo and log it as one message. Otherwise which
+ * may require the buffer to be enlarged very much and lead to OOM
+ * error since there can be a large number of memory contexts in a
+ * backend. Instead, we log one message per memory context.
+ */
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",
+ grand_totals.totalspace, grand_totals.nblocks,
+ grand_totals.freespace, grand_totals.freechunks,
+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +563,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool print_to_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,7 +576,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);
+ totals, print_to_stderr);
/*
* Examine children. If there are more than max_children of them, we do
@@ -560,11 +591,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
if (ichild < max_children)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);
+ totals,
+ print_to_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);
+ &local_totals,
+ print_to_stderr);
}
/* Deal with excess children */
@@ -572,18 +605,33 @@ MemoryContextStatsInternal(MemoryContext context, int level,
{
if (print)
{
- int i;
-
- for (i = 0; i <= level; i++)
- fprintf(stderr, " ");
- fprintf(stderr,
- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
- ichild - max_children,
- local_totals.totalspace,
- local_totals.nblocks,
- local_totals.freespace,
- local_totals.freechunks,
- local_totals.totalspace - local_totals.freespace);
+ if (print_to_stderr)
+ {
+ int i;
+
+ for (i = 0; i <= level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr,
+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace);
+ }
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",
+ level,
+ ichild - max_children,
+ local_totals.totalspace,
+ local_totals.nblocks,
+ local_totals.freespace,
+ local_totals.freechunks,
+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,11 +653,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)
+ const char *stats_string,
+ bool print_to_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
+ char truncated_ident[110];
int i;
/*
@@ -623,9 +673,8 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)
- fprintf(stderr, " ");
- fprintf(stderr, "%s: %s", name, stats_string);
+ truncated_ident[0] = '\0';
+
if (ident)
{
/*
@@ -637,24 +686,41 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcpy(truncated_ident, ": ");
+ i = strlen(truncated_ident);
+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");
+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);
+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';
+
if (truncated)
- fprintf(stderr, "...");
+ strcat(truncated_ident, "...");
+ }
+
+ if (print_to_stderr)
+ {
+ for (i = 0; i < level; i++)
+ fprintf(stderr, " ");
+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
}
- fputc('\n', stderr);
+ else
+ ereport(LOG_SERVER_ONLY,
+ (errhidestmt(true),
+ errhidecontext(true),
+ errmsg_internal("level: %d; %s: %s%s",
+ level, name, stats_string, truncated_ident)));
}
/*
@@ -946,6 +1012,52 @@ MemoryContextAllocExtended(MemoryContext context, Size size, int flags)
return ret;
}
+/*
+ * HandleLogMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating logging of memory
+ * contexts.
+ *
+ * All the actual work is deferred to ProcessLogMemoryContextInterrupt(),
+ * because we cannot safely emit a log message inside the signal handler.
+ */
+void
+HandleLogMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ LogMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessLogMemoryContextInterrupt
+ * Perform logging of memory contexts of this backend process.
+ *
+ * Any backend that participates in ProcSignal signaling must arrange
+ * to call this function if we see LogMemoryContextPending set.
+ * It is called from CHECK_FOR_INTERRUPTS(), which is enough because
+ * the target process for logging of memory contexts is a backend.
+ */
+void
+ProcessLogMemoryContextInterrupt(void)
+{
+ LogMemoryContextPending = false;
+
+ ereport(LOG,
+ (errmsg("logging memory contexts of PID %d", MyProcPid)));
+
+ /*
+ * When a backend process is consuming huge memory, logging all its memory
+ * contexts might overrun available disk space. To prevent this, we limit
+ * the number of child contexts to log per parent to 100.
+ *
+ * As with MemoryContextStats(), we suppose that practical cases where the
+ * dump gets long will typically be huge numbers of siblings under the
+ * same parent context; while the additional debugging value from seeing
+ * details about individual siblings beyond 100 will not be large.
+ */
+ MemoryContextStatsDetail(TopMemoryContext, 100, false);
+}
+
void *
palloc(Size size)
{
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 9213be7c95..553dd7f667 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * print_to_stderr: print stats to stderr if true, elog otherwise.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)
+ MemoryContextCounters *totals,
+ bool print_to_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);
+ printfunc(context, passthru, stats_string, print_to_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 69ffd0c3f4..73c22c8b4d 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7946,6 +7946,12 @@
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# logging memory contexts of the specified backend
+{ oid => '4543', descr => 'log memory contexts of the specified backend',
+ proname => 'pg_log_backend_memory_contexts',
+ provolatile => 'v', prorettype => 'bool',
+ proargtypes => 'int4', prosrc => 'pg_log_backend_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 6f8251e0b0..95202d37af 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index 9331ef80fd..e6a757d6a0 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);
+ const char *stats_string,
+ bool print_to_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 4ae7dc33b8..eec186be2e 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -34,6 +34,7 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 36aae4e51c..ff872274d4 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -84,7 +84,8 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);
+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,
+ bool print_to_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
@@ -144,6 +145,8 @@ extern void MemoryContextCreate(MemoryContext node,
MemoryContext parent,
const char *name);
+extern void HandleLogMemoryContextInterrupt(void);
+extern void ProcessLogMemoryContextInterrupt(void);
/*
* Memory-context-type-specific functions
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index d3acb98d04..e845042d38 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -133,6 +133,19 @@ ERROR: function num_nulls() does not exist
LINE 1: SELECT num_nulls();
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
+--
+-- pg_log_backend_memory_contexts()
+--
+-- Memory contexts are logged and they are not returned to the function.
+-- Furthermore, their contents can vary depending on the timing. However,
+-- we can at least verify that the code doesn't fail.
+--
+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());
+ pg_log_backend_memory_contexts
+--------------------------------
+ t
+(1 row)
+
--
-- Test some built-in SRFs
--
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 094e8f8296..a398349afc 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -30,6 +30,15 @@ SELECT num_nulls(VARIADIC '{}'::int[]);
SELECT num_nonnulls();
SELECT num_nulls();
+--
+-- pg_log_backend_memory_contexts()
+--
+-- Memory contexts are logged and they are not returned to the function.
+-- Furthermore, their contents can vary depending on the timing. However,
+-- we can at least verify that the code doesn't fail.
+--
+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());
+
--
-- Test some built-in SRFs
--
--
2.27.0
On 2021-04-06 00:08, Fujii Masao wrote:
On 2021/04/05 21:03, torikoshia wrote:
On 2021-04-05 12:59, Fujii Masao wrote:
On 2021/04/05 12:20, Zhihong Yu wrote:
Thanks for reviewing!
+ * On receipt of this signal, a backend sets the flag in the signal + * handler, and then which causes the next CHECK_FOR_INTERRUPTS()I think the 'and then' is not needed:
Although I wonder either would be fine, removed the words.
+ * This is just a warning so a loop-through-resultset will not abort + * if one backend logged its memory contexts during the run.The pid given by arg 0 is not a PostgreSQL server process. Which
other backend could it be ?This is the comment that I added wrongly. So the comment should be
"This is just a warning so a loop-through-resultset will not abort
if one backend terminated on its own during the run.",
like pg_signal_backend(). Thought?+1.
Attached v10 patch.
Thanks for updating the patch!
I updated the patch as follows. Could you check the attached patch?
Thanks a lot!
I don't have any objections to your improvements.
Regards,
On 2021/04/06 10:57, torikoshia wrote:
I don't have any objections to your improvements.
Thanks for the check! I pushed the patch. Thanks a lot!
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION