pg_shmem_allocations view
Hi,
I've more than once wanted to know what allocated shared memory in
postgres installation is used for. Especially with more an more
extensions around that's quite useful.
Thus I've written a patch to add a new SRF/VIEW
pg_get_shmem_allocations/pg_shmem_allocations that shows the contents of
the shared memory index:
postgres=# SELECT * FROM pg_shmem_allocations ORDER BY size DESC;
key | off | size | allocated
-------------------------------------+-------------+-------------+-----------
Buffer Blocks | 286242528 | 17179869184 | t
Buffer Descriptors | 152024800 | 134217728 | t
Checkpointer Data | 17584720352 | 41943080 | t
XLOG Ctl | 134234112 | 16804496 | t
CLOG Ctl | 151038624 | 525312 | t
| 17627719648 | 366624 | f
Backend Activity Buffer | 17584379168 | 272000 | t
SUBTRANS Ctl | 151563936 | 263168 | t
OldSerXid SLRU Ctl | 17584225696 | 131648 | t
MultiXactMember Ctl | 151892960 | 131648 | t
Shared Buffer Lookup Table | 17466111712 | 131184 | t
shmInvalBuffer | 17584653184 | 66104 | t
Async Ctl | 17626666048 | 65856 | t
MultiXactOffset Ctl | 151827104 | 65856 | t
Fast Path Strong Relation Lock Data | 17583882752 | 4100 | t
Backend Status Array | 17584373304 | 3672 | t
PROCLOCK hash | 17583785856 | 2160 | t
PREDICATELOCKTARGET hash | 17583886856 | 2160 | t
PREDICATELOCK hash | 17583957632 | 2160 | t
pg_stat_statements hash | 17626731952 | 2160 | t
SERIALIZABLEXID hash | 17584175184 | 2160 | t
LOCK hash | 17583684096 | 2160 | t
Background Worker Data | 17584651184 | 1992 | t
Wal Receiver Ctl | 17626663712 | 1192 | t
Backend Client Host Name Buffer | 17584378064 | 1088 | t
Backend Application Name Buffer | 17584376976 | 1088 | t
ProcSignalSlots | 17584719472 | 864 | t
Sync Scan Locations List | 17626665120 | 656 | t
Async Queue Control | 17626665776 | 244 | t
Control File | 134233856 | 240 | t
AutoVacuum Data | 17626663432 | 224 | t
BTree Vacuum State | 17626664904 | 216 | t
PMSignalState | 17584719288 | 180 | t
Shared MultiXact State | 152024608 | 176 | t
Proc Array | 17584373192 | 108 | t
PredXactList | 17584149248 | 88 | t
Proc Header | 17584357360 | 88 | t
Wal Sender Ctl | 17626663656 | 56 | t
pg_stat_statements | 17626731904 | 48 | t
Buffer Strategy Status | 17583684064 | 32 | t
RWConflictPool | 17584184832 | 24 | t
Prepared Transaction Table | 17584651168 | 16 | t
FinishedSerializableTransactions | 17584225664 | 16 | t
OldSerXidControlData | 17584357344 | 16 | t
(44 rows)
I think this is quite worthwile information. It'd possibly be better of
in an extension, but the relevant dastructures aren't public.
The attached patch doesn't contain documentation because I wasn't sure
that others would be interested in the feature.
Greetings,
Andres Freund
PS: Yes, the checkpointer's allocation is crazy. The fsync queue is
sized by NBuffers which is absurd.
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
0001-Add-support-for-wrapping-to-psql-s-extended-mode.-Th.patchtext/x-patch; charset=us-asciiDownload
>From 6513633b94173fc1d9e2b213c43f9422ddbf5faa Mon Sep 17 00:00:00 2001
From: Greg Stark <stark@mit.edu>
Date: Mon, 28 Apr 2014 18:41:36 +0100
Subject: [PATCH] Add support for wrapping to psql's "extended" mode. This
makes it very feasible to display tables that have both many columns and some
large data in some columns (such as pg_stats).
Emre Hasegeli with review and rewriting from Sergey Muraviov and
reviewed by Greg Stark
---
src/bin/psql/print.c | 181 ++++++-
src/test/regress/expected/psql.out | 944 +++++++++++++++++++++++++++++++++++++
src/test/regress/sql/psql.sql | 120 +++++
3 files changed, 1222 insertions(+), 23 deletions(-)
diff --git a/src/bin/psql/print.c b/src/bin/psql/print.c
index 79fc43e..0ebffff 100644
--- a/src/bin/psql/print.c
+++ b/src/bin/psql/print.c
@@ -1161,6 +1161,7 @@ print_aligned_vertical(const printTableContent *cont, FILE *fout)
struct lineptr *hlineptr,
*dlineptr;
bool is_pager = false;
+ int output_columns = 0; /* Width of interactive console */
if (cancel_pressed)
return;
@@ -1234,24 +1235,86 @@ print_aligned_vertical(const printTableContent *cont, FILE *fout)
fprintf(fout, "%s\n", cont->title);
}
+ /*
+ * Choose target output width: \pset columns, or $COLUMNS, or ioctl
+ */
+ if (cont->opt->columns > 0)
+ output_columns = cont->opt->columns;
+ else if ((fout == stdout && isatty(fileno(stdout))) || is_pager)
+ {
+ if (cont->opt->env_columns > 0)
+ output_columns = cont->opt->env_columns;
+#ifdef TIOCGWINSZ
+ else
+ {
+ struct winsize screen_size;
+
+ if (ioctl(fileno(stdout), TIOCGWINSZ, &screen_size) != -1)
+ output_columns = screen_size.ws_col;
+ }
+#endif
+ }
+
+ if (cont->opt->format == PRINT_WRAPPED)
+ {
+ /* Calculate the available width to wrap the columns to after
+ * subtracting the maximum header width and separators. At a minimum
+ * enough to print "[ RECORD N ]" */
+ unsigned int width, swidth;
+
+ if (opt_border == 0)
+ swidth = 1; /* "header data" */
+ else if (opt_border == 1)
+ swidth = 3; /* "header | data" */
+ else if (opt_border > 1)
+ swidth = 7; /* "| header | data |" */
+
+ /* Wrap to maximum width */
+ width = dwidth + swidth + hwidth;
+ if ((output_columns > 0) && (width > output_columns))
+ {
+ dwidth = output_columns - hwidth - swidth;
+ width = output_columns;
+ }
+
+ /* Wrap to minimum width */
+ if (!opt_tuples_only)
+ {
+ int delta = 1 + log10(cont->nrows) - width;
+ if (opt_border == 0)
+ delta += 6; /* "* RECORD " */
+ else if (opt_border == 1)
+ delta += 10; /* "-[ RECORD ]" */
+ else if (opt_border == 2)
+ delta += 15; /* "+-[ RECORD ]-+" */
+
+ if (delta > 0)
+ dwidth += delta;
+ }
+ else if (dwidth < 3)
+ dwidth = 3;
+ }
+
/* print records */
for (i = 0, ptr = cont->cells; *ptr; i++, ptr++)
{
printTextRule pos;
- int line_count,
+ int dline,
+ hline,
dcomplete,
- hcomplete;
+ hcomplete,
+ offset,
+ chars_to_output;
if (cancel_pressed)
break;
if (i == 0)
pos = PRINT_RULE_TOP;
- else if (!(*(ptr + 1)))
- pos = PRINT_RULE_BOTTOM;
else
pos = PRINT_RULE_MIDDLE;
+ /* Print record header (e.g. "[ RECORD N ]") above each record */
if (i % cont->ncolumns == 0)
{
if (!opt_tuples_only)
@@ -1270,48 +1333,120 @@ print_aligned_vertical(const printTableContent *cont, FILE *fout)
pg_wcsformat((const unsigned char *) *ptr, strlen(*ptr), encoding,
dlineptr, dheight);
- line_count = 0;
+ /* Loop through header and data in parallel dealing with newlines and
+ * wrapped lines until they're both exhausted */
+ dline = hline = 0;
dcomplete = hcomplete = 0;
+ offset = 0;
+ chars_to_output = dlineptr[dline].width;
while (!dcomplete || !hcomplete)
{
+ /* Left border */
if (opt_border == 2)
- fprintf(fout, "%s ", dformat->leftvrule);
+ fprintf(fout, "%s", dformat->leftvrule);
+
+ /* Header (never wrapped so just need to deal with newlines) */
if (!hcomplete)
{
- fprintf(fout, "%-s%*s", hlineptr[line_count].ptr,
- hwidth - hlineptr[line_count].width, "");
+ int swidth, twidth = hwidth + 1;
+ fputs(hline? format->header_nl_left: " ", fout);
+ strlen_max_width((char *) hlineptr[hline].ptr, &twidth,
+ encoding);
+ fprintf(fout, "%-s", hlineptr[hline].ptr);
+
+ swidth = hwidth - twidth;
+ if (swidth > 0) /* spacer */
+ fprintf(fout, "%*s", swidth, " ");
- if (!hlineptr[line_count + 1].ptr)
+ if (hlineptr[hline + 1].ptr)
+ {
+ /* More lines after this one due to a newline */
+ fputs(format->header_nl_right, fout);
+ hline++;
+ }
+ else
+ {
+ /* This was the last line of the header */
+ fputs(" ", fout);
hcomplete = 1;
+ }
}
else
- fprintf(fout, "%*s", hwidth, "");
+ {
+ /* Header exhausted but more data for column */
+ fprintf(fout, "%*s", hwidth + 2, "");
+ }
+ /* Separator */
if (opt_border > 0)
- fprintf(fout, " %s ", dformat->midvrule);
- else
- fputc(' ', fout);
+ {
+ if (offset)
+ fputs(format->midvrule_wrap, fout);
+ else if (!dline)
+ fputs(dformat->midvrule, fout);
+ else if (dline)
+ fputs(format->midvrule_nl, fout);
+ else
+ fputs(format->midvrule_blank, fout);
+ }
+ /* Data */
if (!dcomplete)
{
- if (opt_border < 2)
- fprintf(fout, "%s\n", dlineptr[line_count].ptr);
- else
- fprintf(fout, "%-s%*s %s\n", dlineptr[line_count].ptr,
- dwidth - dlineptr[line_count].width, "",
- dformat->rightvrule);
+ int target_width,
+ bytes_to_output,
+ swidth;
+
+ fputs(!dcomplete && !offset? " ": format->wrap_left, fout);
- if (!dlineptr[line_count + 1].ptr)
+ target_width = dwidth;
+ bytes_to_output = strlen_max_width(dlineptr[dline].ptr + offset,
+ &target_width, encoding);
+ fputnbytes(fout, (char *)(dlineptr[dline].ptr + offset),
+ bytes_to_output);
+
+ chars_to_output -= target_width;
+ offset += bytes_to_output;
+
+ /* spacer */
+ swidth = dwidth - target_width;
+ if (swidth > 0)
+ fprintf(fout, "%*s", swidth, "");
+
+ if (chars_to_output)
+ {
+ /* continuing a wrapped column */
+ fputs(format->wrap_right, fout);
+ }
+ else if (dlineptr[dline + 1].ptr)
+ {
+ /* reached a newline in the column */
+ fputs(format->nl_right, fout);
+ dline++;
+ offset = 0;
+ chars_to_output = dlineptr[dline].width;
+ }
+ else
+ {
+ /* reached the end of the cell */
+ fputs(" ", fout);
dcomplete = 1;
+ }
+
+ if (opt_border == 2)
+ fputs(dformat->rightvrule, fout);
+
+ fputs("\n", fout);
}
else
{
+ /* data exhausted (this can occur if header is longer than the
+ * data due to newlines in the header) */
if (opt_border < 2)
- fputc('\n', fout);
+ fputs("\n", fout);
else
- fprintf(fout, "%*s %s\n", dwidth, "", dformat->rightvrule);
+ fprintf(fout, "%*s %s\n", dwidth, "", dformat->rightvrule);
}
- line_count++;
}
}
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 2bbee7d..c7dbd54 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -68,3 +68,947 @@ Record separator (recordsep) is <newline>.
Table attributes (tableattr) unset.
Title (title) unset.
Tuples only (tuples_only) is off.
+-- test multi-line headers, wrapping, and newline indicators
+prepare q as select array_to_string(array_agg(repeat('x',2*n)),E'\n') as "a
+
+b", array_to_string(array_agg(repeat('y',20-2*n)),E'\n') as "a
+b" from generate_series(1,10) as n(n) group by n>1 ;
+\pset linestyle ascii
+\pset expanded off
+\pset columns 40
+\pset border 0
+\pset format unaligned
+execute q;
+a
+
+b|a
+b
+xx|yyyyyyyyyyyyyyyyyy
+xxxx
+xxxxxx
+xxxxxxxx
+xxxxxxxxxx
+xxxxxxxxxxxx
+xxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxxxx|yyyyyyyyyyyyyyyy
+yyyyyyyyyyyyyy
+yyyyyyyyyyyy
+yyyyyyyyyy
+yyyyyyyy
+yyyyyy
+yyyy
+yy
+
+(2 rows)
+\pset format aligned
+execute q;
+ a + a +
+ + b
+ b
+-------------------- ------------------
+xx yyyyyyyyyyyyyyyyyy
+xxxx +yyyyyyyyyyyyyyyy +
+xxxxxx +yyyyyyyyyyyyyy +
+xxxxxxxx +yyyyyyyyyyyy +
+xxxxxxxxxx +yyyyyyyyyy +
+xxxxxxxxxxxx +yyyyyyyy +
+xxxxxxxxxxxxxx +yyyyyy +
+xxxxxxxxxxxxxxxx +yyyy +
+xxxxxxxxxxxxxxxxxx +yy +
+xxxxxxxxxxxxxxxxxxxx
+(2 rows)
+
+\pset format wrapped
+execute q;
+ a + a +
+ + b
+ b
+-------------------- ------------------
+xx yyyyyyyyyyyyyyyyyy
+xxxx +yyyyyyyyyyyyyyyy +
+xxxxxx +yyyyyyyyyyyyyy +
+xxxxxxxx +yyyyyyyyyyyy +
+xxxxxxxxxx +yyyyyyyyyy +
+xxxxxxxxxxxx +yyyyyyyy +
+xxxxxxxxxxxxxx +yyyyyy +
+xxxxxxxxxxxxxxxx +yyyy +
+xxxxxxxxxxxxxxxxxx +yy +
+xxxxxxxxxxxxxxxxxxxx
+(2 rows)
+
+\pset border 1
+\pset format unaligned
+execute q;
+a
+
+b|a
+b
+xx|yyyyyyyyyyyyyyyyyy
+xxxx
+xxxxxx
+xxxxxxxx
+xxxxxxxxxx
+xxxxxxxxxxxx
+xxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxxxx|yyyyyyyyyyyyyyyy
+yyyyyyyyyyyyyy
+yyyyyyyyyyyy
+yyyyyyyyyy
+yyyyyyyy
+yyyyyy
+yyyy
+yy
+
+(2 rows)
+\pset format aligned
+execute q;
+ a +| a +
+ +| b
+ b |
+----------------------+--------------------
+ xx | yyyyyyyyyyyyyyyyyy
+ xxxx +| yyyyyyyyyyyyyyyy +
+ xxxxxx +| yyyyyyyyyyyyyy +
+ xxxxxxxx +| yyyyyyyyyyyy +
+ xxxxxxxxxx +| yyyyyyyyyy +
+ xxxxxxxxxxxx +| yyyyyyyy +
+ xxxxxxxxxxxxxx +| yyyyyy +
+ xxxxxxxxxxxxxxxx +| yyyy +
+ xxxxxxxxxxxxxxxxxx +| yy +
+ xxxxxxxxxxxxxxxxxxxx |
+(2 rows)
+
+\pset format wrapped
+execute q;
+ a +| a +
+ +| b
+ b |
+-------------------+--------------------
+ xx | yyyyyyyyyyyyyyyyyy
+ xxxx +| yyyyyyyyyyyyyyyy +
+ xxxxxx +| yyyyyyyyyyyyyy +
+ xxxxxxxx +| yyyyyyyyyyyy +
+ xxxxxxxxxx +| yyyyyyyyyy +
+ xxxxxxxxxxxx +| yyyyyyyy +
+ xxxxxxxxxxxxxx +| yyyyyy +
+ xxxxxxxxxxxxxxxx +| yyyy +
+ xxxxxxxxxxxxxxxxx.| yy +
+.x +|
+ xxxxxxxxxxxxxxxxx.|
+.xxx |
+(2 rows)
+
+\pset border 2
+\pset format unaligned
+execute q;
+a
+
+b|a
+b
+xx|yyyyyyyyyyyyyyyyyy
+xxxx
+xxxxxx
+xxxxxxxx
+xxxxxxxxxx
+xxxxxxxxxxxx
+xxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxxxx|yyyyyyyyyyyyyyyy
+yyyyyyyyyyyyyy
+yyyyyyyyyyyy
+yyyyyyyyyy
+yyyyyyyy
+yyyyyy
+yyyy
+yy
+
+(2 rows)
+\pset format aligned
+execute q;
++----------------------+--------------------+
+| a +| a +|
+| +| b |
+| b | |
++----------------------+--------------------+
+| xx | yyyyyyyyyyyyyyyyyy |
+| xxxx +| yyyyyyyyyyyyyyyy +|
+| xxxxxx +| yyyyyyyyyyyyyy +|
+| xxxxxxxx +| yyyyyyyyyyyy +|
+| xxxxxxxxxx +| yyyyyyyyyy +|
+| xxxxxxxxxxxx +| yyyyyyyy +|
+| xxxxxxxxxxxxxx +| yyyyyy +|
+| xxxxxxxxxxxxxxxx +| yyyy +|
+| xxxxxxxxxxxxxxxxxx +| yy +|
+| xxxxxxxxxxxxxxxxxxxx | |
++----------------------+--------------------+
+(2 rows)
+
+\pset format wrapped
+execute q;
++-----------------+--------------------+
+| a +| a +|
+| +| b |
+| b | |
++-----------------+--------------------+
+| xx | yyyyyyyyyyyyyyyyyy |
+| xxxx +| yyyyyyyyyyyyyyyy +|
+| xxxxxx +| yyyyyyyyyyyyyy +|
+| xxxxxxxx +| yyyyyyyyyyyy +|
+| xxxxxxxxxx +| yyyyyyyyyy +|
+| xxxxxxxxxxxx +| yyyyyyyy +|
+| xxxxxxxxxxxxxx +| yyyyyy +|
+| xxxxxxxxxxxxxxx.| yyyy +|
+|.x +| yy +|
+| xxxxxxxxxxxxxxx.| |
+|.xxx +| |
+| xxxxxxxxxxxxxxx.| |
+|.xxxxx | |
++-----------------+--------------------+
+(2 rows)
+
+\pset expanded on
+\pset columns 20
+\pset border 0
+\pset format unaligned
+execute q;
+a
+
+b|xx
+a
+b|yyyyyyyyyyyyyyyyyy
+
+a
+
+b|xxxx
+xxxxxx
+xxxxxxxx
+xxxxxxxxxx
+xxxxxxxxxxxx
+xxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxxxx
+a
+b|yyyyyyyyyyyyyyyy
+yyyyyyyyyyyyyy
+yyyyyyyyyyyy
+yyyyyyyyyy
+yyyyyyyy
+yyyyyy
+yyyy
+yy
+
+\pset format aligned
+execute q;
+* Record 1
+ a+ xx
+ +
+ b
+ a+ yyyyyyyyyyyyyyyyyy
+ b
+* Record 2
+ a+ xxxx +
+ + xxxxxx +
+ b xxxxxxxx +
+ xxxxxxxxxx +
+ xxxxxxxxxxxx +
+ xxxxxxxxxxxxxx +
+ xxxxxxxxxxxxxxxx +
+ xxxxxxxxxxxxxxxxxx +
+ xxxxxxxxxxxxxxxxxxxx
+ a+ yyyyyyyyyyyyyyyy +
+ b yyyyyyyyyyyyyy +
+ yyyyyyyyyyyy +
+ yyyyyyyyyy +
+ yyyyyyyy +
+ yyyyyy +
+ yyyy +
+ yy +
+
+
+\pset format wrapped
+execute q;
+* Record 1
+ a+ xx
+ +
+ b
+ a+ yyyyyyyyyyyyyyyyyy
+ b
+* Record 2
+ a+ xxxx +
+ + xxxxxx +
+ b xxxxxxxx +
+ xxxxxxxxxx +
+ xxxxxxxxxxxx +
+ xxxxxxxxxxxxxx +
+ xxxxxxxxxxxxxxxx +
+ xxxxxxxxxxxxxxxxxx+
+ xxxxxxxxxxxxxxxxxx.
+ .xx
+ a+ yyyyyyyyyyyyyyyy +
+ b yyyyyyyyyyyyyy +
+ yyyyyyyyyyyy +
+ yyyyyyyyyy +
+ yyyyyyyy +
+ yyyyyy +
+ yyyy +
+ yy +
+
+
+\pset border 1
+\pset format unaligned
+execute q;
+a
+
+b|xx
+a
+b|yyyyyyyyyyyyyyyyyy
+
+a
+
+b|xxxx
+xxxxxx
+xxxxxxxx
+xxxxxxxxxx
+xxxxxxxxxxxx
+xxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxxxx
+a
+b|yyyyyyyyyyyyyyyy
+yyyyyyyyyyyyyy
+yyyyyyyyyyyy
+yyyyyyyyyy
+yyyyyyyy
+yyyyyy
+yyyy
+yy
+
+\pset format aligned
+execute q;
+-[ RECORD 1 ]-----------
+ a+| xx
+ +|
+ b |
+ a+| yyyyyyyyyyyyyyyyyy
+ b |
+-[ RECORD 2 ]-----------
+ a+| xxxx +
+ +| xxxxxx +
+ b | xxxxxxxx +
+ | xxxxxxxxxx +
+ | xxxxxxxxxxxx +
+ | xxxxxxxxxxxxxx +
+ | xxxxxxxxxxxxxxxx +
+ | xxxxxxxxxxxxxxxxxx +
+ | xxxxxxxxxxxxxxxxxxxx
+ a+| yyyyyyyyyyyyyyyy +
+ b | yyyyyyyyyyyyyy +
+ | yyyyyyyyyyyy +
+ | yyyyyyyyyy +
+ | yyyyyyyy +
+ | yyyyyy +
+ | yyyy +
+ | yy +
+ |
+
+\pset format wrapped
+execute q;
+-[ RECORD 1 ]-------
+ a+| xx
+ +|
+ b |
+ a+| yyyyyyyyyyyyyyyy.
+ b |.yy
+-[ RECORD 2 ]-------
+ a+| xxxx +
+ +| xxxxxx +
+ b | xxxxxxxx +
+ | xxxxxxxxxx +
+ | xxxxxxxxxxxx +
+ | xxxxxxxxxxxxxx +
+ | xxxxxxxxxxxxxxxx+
+ | xxxxxxxxxxxxxxxx.
+ |.xx +
+ | xxxxxxxxxxxxxxxx.
+ |.xxxx
+ a+| yyyyyyyyyyyyyyyy+
+ b | yyyyyyyyyyyyyy +
+ | yyyyyyyyyyyy +
+ | yyyyyyyyyy +
+ | yyyyyyyy +
+ | yyyyyy +
+ | yyyy +
+ | yy +
+ |
+
+\pset border 2
+\pset format unaligned
+execute q;
+a
+
+b|xx
+a
+b|yyyyyyyyyyyyyyyyyy
+
+a
+
+b|xxxx
+xxxxxx
+xxxxxxxx
+xxxxxxxxxx
+xxxxxxxxxxxx
+xxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxxxx
+a
+b|yyyyyyyyyyyyyyyy
+yyyyyyyyyyyyyy
+yyyyyyyyyyyy
+yyyyyyyyyy
+yyyyyyyy
+yyyyyy
+yyyy
+yy
+
+\pset format aligned
+execute q;
++-[ RECORD 1 ]-------------+
+| a+| xx |
+| +| |
+| b | |
+| a+| yyyyyyyyyyyyyyyyyy |
+| b | |
++-[ RECORD 2 ]-------------+
+| a+| xxxx +|
+| +| xxxxxx +|
+| b | xxxxxxxx +|
+| | xxxxxxxxxx +|
+| | xxxxxxxxxxxx +|
+| | xxxxxxxxxxxxxx +|
+| | xxxxxxxxxxxxxxxx +|
+| | xxxxxxxxxxxxxxxxxx +|
+| | xxxxxxxxxxxxxxxxxxxx |
+| a+| yyyyyyyyyyyyyyyy +|
+| b | yyyyyyyyyyyyyy +|
+| | yyyyyyyyyyyy +|
+| | yyyyyyyyyy +|
+| | yyyyyyyy +|
+| | yyyyyy +|
+| | yyyy +|
+| | yy +|
+| | |
++---+----------------------+
+
+\pset format wrapped
+execute q;
++-[ RECORD 1 ]-----+
+| a+| xx |
+| +| |
+| b | |
+| a+| yyyyyyyyyyyy.|
+| b |.yyyyyy |
++-[ RECORD 2 ]-----+
+| a+| xxxx +|
+| +| xxxxxx +|
+| b | xxxxxxxx +|
+| | xxxxxxxxxx +|
+| | xxxxxxxxxxxx+|
+| | xxxxxxxxxxxx.|
+| |.xx +|
+| | xxxxxxxxxxxx.|
+| |.xxxx +|
+| | xxxxxxxxxxxx.|
+| |.xxxxxx +|
+| | xxxxxxxxxxxx.|
+| |.xxxxxxxx |
+| a+| yyyyyyyyyyyy.|
+| b |.yyyy +|
+| | yyyyyyyyyyyy.|
+| |.yy +|
+| | yyyyyyyyyyyy+|
+| | yyyyyyyyyy +|
+| | yyyyyyyy +|
+| | yyyyyy +|
+| | yyyy +|
+| | yy +|
+| | |
++---+--------------+
+
+\pset linestyle old-ascii
+\pset expanded off
+\pset columns 40
+\pset border 0
+\pset format unaligned
+execute q;
+a
+
+b|a
+b
+xx|yyyyyyyyyyyyyyyyyy
+xxxx
+xxxxxx
+xxxxxxxx
+xxxxxxxxxx
+xxxxxxxxxxxx
+xxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxxxx|yyyyyyyyyyyyyyyy
+yyyyyyyyyyyyyy
+yyyyyyyyyyyy
+yyyyyyyyyy
+yyyyyyyy
+yyyyyy
+yyyy
+yy
+
+(2 rows)
+\pset format aligned
+execute q;
+ a a
+ + b
+ b +
+-------------------- ------------------
+xx yyyyyyyyyyyyyyyyyy
+xxxx yyyyyyyyyyyyyyyy
+xxxxxx yyyyyyyyyyyyyy
+xxxxxxxx yyyyyyyyyyyy
+xxxxxxxxxx yyyyyyyyyy
+xxxxxxxxxxxx yyyyyyyy
+xxxxxxxxxxxxxx yyyyyy
+xxxxxxxxxxxxxxxx yyyy
+xxxxxxxxxxxxxxxxxx yy
+xxxxxxxxxxxxxxxxxxxx
+(2 rows)
+
+\pset format wrapped
+execute q;
+ a a
+ + b
+ b +
+-------------------- ------------------
+xx yyyyyyyyyyyyyyyyyy
+xxxx yyyyyyyyyyyyyyyy
+xxxxxx yyyyyyyyyyyyyy
+xxxxxxxx yyyyyyyyyyyy
+xxxxxxxxxx yyyyyyyyyy
+xxxxxxxxxxxx yyyyyyyy
+xxxxxxxxxxxxxx yyyyyy
+xxxxxxxxxxxxxxxx yyyy
+xxxxxxxxxxxxxxxxxx yy
+xxxxxxxxxxxxxxxxxxxx
+(2 rows)
+
+\pset border 1
+\pset format unaligned
+execute q;
+a
+
+b|a
+b
+xx|yyyyyyyyyyyyyyyyyy
+xxxx
+xxxxxx
+xxxxxxxx
+xxxxxxxxxx
+xxxxxxxxxxxx
+xxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxxxx|yyyyyyyyyyyyyyyy
+yyyyyyyyyyyyyy
+yyyyyyyyyyyy
+yyyyyyyyyy
+yyyyyyyy
+yyyyyy
+yyyy
+yy
+
+(2 rows)
+\pset format aligned
+execute q;
+ a | a
++ |+ b
++ b |+
+----------------------+--------------------
+ xx | yyyyyyyyyyyyyyyyyy
+ xxxx | yyyyyyyyyyyyyyyy
+ xxxxxx : yyyyyyyyyyyyyy
+ xxxxxxxx : yyyyyyyyyyyy
+ xxxxxxxxxx : yyyyyyyyyy
+ xxxxxxxxxxxx : yyyyyyyy
+ xxxxxxxxxxxxxx : yyyyyy
+ xxxxxxxxxxxxxxxx : yyyy
+ xxxxxxxxxxxxxxxxxx : yy
+ xxxxxxxxxxxxxxxxxxxx :
+(2 rows)
+
+\pset format wrapped
+execute q;
+ a | a
++ |+ b
++ b |+
+-------------------+--------------------
+ xx | yyyyyyyyyyyyyyyyyy
+ xxxx | yyyyyyyyyyyyyyyy
+ xxxxxx : yyyyyyyyyyyyyy
+ xxxxxxxx : yyyyyyyyyyyy
+ xxxxxxxxxx : yyyyyyyyyy
+ xxxxxxxxxxxx : yyyyyyyy
+ xxxxxxxxxxxxxx : yyyyyy
+ xxxxxxxxxxxxxxxx : yyyy
+ xxxxxxxxxxxxxxxxx : yy
+ x :
+ xxxxxxxxxxxxxxxxx
+ xxx
+(2 rows)
+
+\pset border 2
+\pset format unaligned
+execute q;
+a
+
+b|a
+b
+xx|yyyyyyyyyyyyyyyyyy
+xxxx
+xxxxxx
+xxxxxxxx
+xxxxxxxxxx
+xxxxxxxxxxxx
+xxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxxxx|yyyyyyyyyyyyyyyy
+yyyyyyyyyyyyyy
+yyyyyyyyyyyy
+yyyyyyyyyy
+yyyyyyyy
+yyyyyy
+yyyy
+yy
+
+(2 rows)
+\pset format aligned
+execute q;
++----------------------+--------------------+
+| a | a |
+|+ |+ b |
+|+ b |+ |
++----------------------+--------------------+
+| xx | yyyyyyyyyyyyyyyyyy |
+| xxxx | yyyyyyyyyyyyyyyy |
+| xxxxxx : yyyyyyyyyyyyyy |
+| xxxxxxxx : yyyyyyyyyyyy |
+| xxxxxxxxxx : yyyyyyyyyy |
+| xxxxxxxxxxxx : yyyyyyyy |
+| xxxxxxxxxxxxxx : yyyyyy |
+| xxxxxxxxxxxxxxxx : yyyy |
+| xxxxxxxxxxxxxxxxxx : yy |
+| xxxxxxxxxxxxxxxxxxxx : |
++----------------------+--------------------+
+(2 rows)
+
+\pset format wrapped
+execute q;
++-----------------+--------------------+
+| a | a |
+|+ |+ b |
+|+ b |+ |
++-----------------+--------------------+
+| xx | yyyyyyyyyyyyyyyyyy |
+| xxxx | yyyyyyyyyyyyyyyy |
+| xxxxxx : yyyyyyyyyyyyyy |
+| xxxxxxxx : yyyyyyyyyyyy |
+| xxxxxxxxxx : yyyyyyyyyy |
+| xxxxxxxxxxxx : yyyyyyyy |
+| xxxxxxxxxxxxxx : yyyyyy |
+| xxxxxxxxxxxxxxx : yyyy |
+| x : yy |
+| xxxxxxxxxxxxxxx : |
+| xxx |
+| xxxxxxxxxxxxxxx |
+| xxxxx |
++-----------------+--------------------+
+(2 rows)
+
+\pset expanded on
+\pset columns 20
+\pset border 0
+\pset format unaligned
+execute q;
+a
+
+b|xx
+a
+b|yyyyyyyyyyyyyyyyyy
+
+a
+
+b|xxxx
+xxxxxx
+xxxxxxxx
+xxxxxxxxxx
+xxxxxxxxxxxx
+xxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxxxx
+a
+b|yyyyyyyyyyyyyyyy
+yyyyyyyyyyyyyy
+yyyyyyyyyyyy
+yyyyyyyyyy
+yyyyyyyy
+yyyyyy
+yyyy
+yy
+
+\pset format aligned
+execute q;
+* Record 1
+ a xx
++
++b
+ a yyyyyyyyyyyyyyyyyy
++b
+* Record 2
+ a xxxx
++ xxxxxx
++b xxxxxxxx
+ xxxxxxxxxx
+ xxxxxxxxxxxx
+ xxxxxxxxxxxxxx
+ xxxxxxxxxxxxxxxx
+ xxxxxxxxxxxxxxxxxx
+ xxxxxxxxxxxxxxxxxxxx
+ a yyyyyyyyyyyyyyyy
++b yyyyyyyyyyyyyy
+ yyyyyyyyyyyy
+ yyyyyyyyyy
+ yyyyyyyy
+ yyyyyy
+ yyyy
+ yy
+
+
+\pset format wrapped
+execute q;
+* Record 1
+ a xx
++
++b
+ a yyyyyyyyyyyyyyyyyy
++b
+* Record 2
+ a xxxx
++ xxxxxx
++b xxxxxxxx
+ xxxxxxxxxx
+ xxxxxxxxxxxx
+ xxxxxxxxxxxxxx
+ xxxxxxxxxxxxxxxx
+ xxxxxxxxxxxxxxxxxx
+ xxxxxxxxxxxxxxxxxx
+ xx
+ a yyyyyyyyyyyyyyyy
++b yyyyyyyyyyyyyy
+ yyyyyyyyyyyy
+ yyyyyyyyyy
+ yyyyyyyy
+ yyyyyy
+ yyyy
+ yy
+
+
+\pset border 1
+\pset format unaligned
+execute q;
+a
+
+b|xx
+a
+b|yyyyyyyyyyyyyyyyyy
+
+a
+
+b|xxxx
+xxxxxx
+xxxxxxxx
+xxxxxxxxxx
+xxxxxxxxxxxx
+xxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxxxx
+a
+b|yyyyyyyyyyyyyyyy
+yyyyyyyyyyyyyy
+yyyyyyyyyyyy
+yyyyyyyyyy
+yyyyyyyy
+yyyyyy
+yyyy
+yy
+
+\pset format aligned
+execute q;
+-[ RECORD 1 ]-----------
+ a | xx
++ ;
++b ;
+ a | yyyyyyyyyyyyyyyyyy
++b ;
+-[ RECORD 2 ]-----------
+ a | xxxx
++ : xxxxxx
++b : xxxxxxxx
+ : xxxxxxxxxx
+ : xxxxxxxxxxxx
+ : xxxxxxxxxxxxxx
+ : xxxxxxxxxxxxxxxx
+ : xxxxxxxxxxxxxxxxxx
+ : xxxxxxxxxxxxxxxxxxxx
+ a | yyyyyyyyyyyyyyyy
++b : yyyyyyyyyyyyyy
+ : yyyyyyyyyyyy
+ : yyyyyyyyyy
+ : yyyyyyyy
+ : yyyyyy
+ : yyyy
+ : yy
+ :
+
+\pset format wrapped
+execute q;
+-[ RECORD 1 ]-------
+ a | xx
++ ;
++b ;
+ a | yyyyyyyyyyyyyyyy
++b ; yy
+-[ RECORD 2 ]-------
+ a | xxxx
++ : xxxxxx
++b : xxxxxxxx
+ : xxxxxxxxxx
+ : xxxxxxxxxxxx
+ : xxxxxxxxxxxxxx
+ : xxxxxxxxxxxxxxxx
+ : xxxxxxxxxxxxxxxx
+ ; xx
+ : xxxxxxxxxxxxxxxx
+ ; xxxx
+ a | yyyyyyyyyyyyyyyy
++b : yyyyyyyyyyyyyy
+ : yyyyyyyyyyyy
+ : yyyyyyyyyy
+ : yyyyyyyy
+ : yyyyyy
+ : yyyy
+ : yy
+ :
+
+\pset border 2
+\pset format unaligned
+execute q;
+a
+
+b|xx
+a
+b|yyyyyyyyyyyyyyyyyy
+
+a
+
+b|xxxx
+xxxxxx
+xxxxxxxx
+xxxxxxxxxx
+xxxxxxxxxxxx
+xxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxxxxxxx
+a
+b|yyyyyyyyyyyyyyyy
+yyyyyyyyyyyyyy
+yyyyyyyyyyyy
+yyyyyyyyyy
+yyyyyyyy
+yyyyyy
+yyyy
+yy
+
+\pset format aligned
+execute q;
++-[ RECORD 1 ]-------------+
+| a | xx |
+|+ ; |
+|+b ; |
+| a | yyyyyyyyyyyyyyyyyy |
+|+b ; |
++-[ RECORD 2 ]-------------+
+| a | xxxx |
+|+ : xxxxxx |
+|+b : xxxxxxxx |
+| : xxxxxxxxxx |
+| : xxxxxxxxxxxx |
+| : xxxxxxxxxxxxxx |
+| : xxxxxxxxxxxxxxxx |
+| : xxxxxxxxxxxxxxxxxx |
+| : xxxxxxxxxxxxxxxxxxxx |
+| a | yyyyyyyyyyyyyyyy |
+|+b : yyyyyyyyyyyyyy |
+| : yyyyyyyyyyyy |
+| : yyyyyyyyyy |
+| : yyyyyyyy |
+| : yyyyyy |
+| : yyyy |
+| : yy |
+| : |
++---+----------------------+
+
+\pset format wrapped
+execute q;
++-[ RECORD 1 ]-----+
+| a | xx |
+|+ ; |
+|+b ; |
+| a | yyyyyyyyyyyy |
+|+b ; yyyyyy |
++-[ RECORD 2 ]-----+
+| a | xxxx |
+|+ : xxxxxx |
+|+b : xxxxxxxx |
+| : xxxxxxxxxx |
+| : xxxxxxxxxxxx |
+| : xxxxxxxxxxxx |
+| ; xx |
+| : xxxxxxxxxxxx |
+| ; xxxx |
+| : xxxxxxxxxxxx |
+| ; xxxxxx |
+| : xxxxxxxxxxxx |
+| ; xxxxxxxx |
+| a | yyyyyyyyyyyy |
+|+b ; yyyy |
+| : yyyyyyyyyyyy |
+| ; yy |
+| : yyyyyyyyyyyy |
+| : yyyyyyyyyy |
+| : yyyyyyyy |
+| : yyyyyy |
+| : yyyy |
+| : yy |
+| : |
++---+--------------+
+
+deallocate q;
diff --git a/src/test/regress/sql/psql.sql b/src/test/regress/sql/psql.sql
index 99ad5b4..a7d5eeb 100644
--- a/src/test/regress/sql/psql.sql
+++ b/src/test/regress/sql/psql.sql
@@ -40,3 +40,123 @@ select 10 as test01, 20 as test02 from generate_series(1,0) \gset
-- show all pset options
\pset
+
+-- test multi-line headers, wrapping, and newline indicators
+prepare q as select array_to_string(array_agg(repeat('x',2*n)),E'\n') as "a
+
+b", array_to_string(array_agg(repeat('y',20-2*n)),E'\n') as "a
+b" from generate_series(1,10) as n(n) group by n>1 ;
+
+\pset linestyle ascii
+
+\pset expanded off
+\pset columns 40
+
+\pset border 0
+\pset format unaligned
+execute q;
+\pset format aligned
+execute q;
+\pset format wrapped
+execute q;
+
+\pset border 1
+\pset format unaligned
+execute q;
+\pset format aligned
+execute q;
+\pset format wrapped
+execute q;
+
+\pset border 2
+\pset format unaligned
+execute q;
+\pset format aligned
+execute q;
+\pset format wrapped
+execute q;
+
+\pset expanded on
+\pset columns 20
+
+\pset border 0
+\pset format unaligned
+execute q;
+\pset format aligned
+execute q;
+\pset format wrapped
+execute q;
+
+\pset border 1
+\pset format unaligned
+execute q;
+\pset format aligned
+execute q;
+\pset format wrapped
+execute q;
+
+\pset border 2
+\pset format unaligned
+execute q;
+\pset format aligned
+execute q;
+\pset format wrapped
+execute q;
+
+\pset linestyle old-ascii
+
+\pset expanded off
+\pset columns 40
+
+\pset border 0
+\pset format unaligned
+execute q;
+\pset format aligned
+execute q;
+\pset format wrapped
+execute q;
+
+\pset border 1
+\pset format unaligned
+execute q;
+\pset format aligned
+execute q;
+\pset format wrapped
+execute q;
+
+\pset border 2
+\pset format unaligned
+execute q;
+\pset format aligned
+execute q;
+\pset format wrapped
+execute q;
+
+\pset expanded on
+\pset columns 20
+
+\pset border 0
+\pset format unaligned
+execute q;
+\pset format aligned
+execute q;
+\pset format wrapped
+execute q;
+
+\pset border 1
+\pset format unaligned
+execute q;
+\pset format aligned
+execute q;
+\pset format wrapped
+execute q;
+
+\pset border 2
+\pset format unaligned
+execute q;
+\pset format aligned
+execute q;
+\pset format wrapped
+execute q;
+
+deallocate q;
--
1.8.5.rc2.dirty
Hi,
On 2014-05-04 13:44:17 +0200, Andres Freund wrote:
postgres=# SELECT * FROM pg_shmem_allocations ORDER BY size DESC;
key | off | size | allocated
-------------------------------------+-------------+-------------+-----------
Buffer Blocks | 286242528 | 17179869184 | t
Buffer Descriptors | 152024800 | 134217728 | t
...
OldSerXidControlData | 17584357344 | 16 | t
(44 rows)
Thinking about this, I think it was a mistake to not add a 'name' field
to dynamic shared memory's dsm_control_item. Right now it's very hard to
figure out which extension allocated a dsm segment. Imo we should change
that before 9.4 is out. I am not suggesting to use it to identify
segments, but just as an identifier, passed in into dsm_create().
Imo there should be a corresponding pg_dynshmem_allocations to
pg_shmem_allocations.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Hi,
On 2014-05-04 13:44:17 +0200, Andres Freund wrote:
postgres=# SELECT * FROM pg_shmem_allocations ORDER BY size DESC;
key | off | size | allocated
-------------------------------------+-------------+-------------+-----------
Buffer Blocks | 286242528 | 17179869184 | t
Buffer Descriptors | 152024800 | 134217728 | t
Abhijit notified me that I've attached the wrong patch. Corrected.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
0001-Add-pg_shmem_allocations-view.patchtext/x-patch; charset=us-asciiDownload
>From e8a7576f3a593f4f88bd619ae2504ee320e61db2 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 4 May 2014 13:37:20 +0200
Subject: [PATCH] Add pg_shmem_allocations view.
---
src/backend/catalog/system_views.sql | 3 ++
src/backend/storage/ipc/shmem.c | 97 ++++++++++++++++++++++++++++++++++++
src/include/catalog/pg_proc.h | 2 +
src/include/utils/builtins.h | 3 ++
src/test/regress/expected/rules.out | 5 ++
5 files changed, 110 insertions(+)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 42a4c00..104491a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -387,6 +387,9 @@ CREATE VIEW pg_timezone_abbrevs AS
CREATE VIEW pg_timezone_names AS
SELECT * FROM pg_timezone_names();
+CREATE VIEW pg_shmem_allocations AS
+ SELECT * FROM pg_get_shmem_allocations();
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 1d27a89..5722c78 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -66,11 +66,14 @@
#include "postgres.h"
#include "access/transam.h"
+#include "fmgr.h"
+#include "funcapi.h"
#include "miscadmin.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
#include "storage/shmem.h"
#include "storage/spin.h"
+#include "utils/builtins.h"
/* shared memory global variables */
@@ -459,3 +462,97 @@ mul_size(Size s1, Size s2)
errmsg("requested shared memory size overflows size_t")));
return result;
}
+
+/* SQL SRF showing allocated shared memory */
+Datum
+pg_get_shmem_allocations(PG_FUNCTION_ARGS)
+{
+#define PG_GET_SHMEM_SIZES_COLS 4
+
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ HASH_SEQ_STATUS hstat;
+ ShmemIndexEnt *ent;
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not " \
+ "allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ hash_seq_init(&hstat, ShmemIndex);
+
+ /* output all allocated entries */
+ while ((ent = (ShmemIndexEnt *) hash_seq_search(&hstat)) != NULL)
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key */
+ values[0] = CStringGetTextDatum(ent->key);
+ nulls[0] = false;
+
+ /* off */
+ values[1] = Int64GetDatum((char *) ent->location - (char *) ShmemSegHdr);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(ent->size);
+ nulls[2] = false;
+
+ /* allocated */
+ values[3] = BoolGetDatum(true);
+ nulls[3] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* output as-of-yet unallocated memory */
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key, show unallocated as NULL */
+ nulls[0] = true;
+
+ /* off */
+ values[1] = Int64GetDatum(ShmemSegHdr->freeoffset);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(ShmemSegHdr->totalsize - ShmemSegHdr->freeoffset);
+ nulls[2] = false;
+
+ /* allocated */
+ values[3] = BoolGetDatum(false);
+ nulls[3] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 98c183b..d018e6f 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -3899,6 +3899,8 @@ DATA(insert OID = 3035 ( pg_listening_channels PGNSP PGUID 12 1 10 0 0 f f f f
DESCR("get the channels that the current backend listens to");
DATA(insert OID = 3036 ( pg_notify PGNSP PGUID 12 1 0 0 0 f f f f f f v 2 0 2278 "25 25" _null_ _null_ _null_ _null_ pg_notify _null_ _null_ _null_ ));
DESCR("send a notification event");
+DATA(insert OID = 86 ( pg_get_shmem_allocations PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{25,20,20,16}" "{o,o,o,o}" "{key, off, size, allocated}" _null_ pg_get_shmem_allocations _null_ _null_ _null_ ));
+DESCR("show shared memory allocations");
/* non-persistent series generator */
DATA(insert OID = 1066 ( generate_series PGNSP PGUID 12 1 1000 0 0 f f f f t t i 3 0 23 "23 23 23" _null_ _null_ _null_ _null_ generate_series_step_int4 _null_ _null_ _null_ ));
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index 33b6dca..540015f 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -1209,4 +1209,7 @@ extern Datum pg_prepared_statement(PG_FUNCTION_ARGS);
/* utils/mmgr/portalmem.c */
extern Datum pg_cursor(PG_FUNCTION_ARGS);
+/* backend/storage/ipc/shmem.c */
+extern Datum pg_get_shmem_allocations(PG_FUNCTION_ARGS);
+
#endif /* BUILTINS_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 87870cf..58a125b 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1591,6 +1591,11 @@ pg_shadow| SELECT pg_authid.rolname AS usename,
FROM (pg_authid
LEFT JOIN pg_db_role_setting s ON (((pg_authid.oid = s.setrole) AND (s.setdatabase = (0)::oid))))
WHERE pg_authid.rolcanlogin;
+pg_shmem_allocations| SELECT pg_get_shmem_allocations.key,
+ pg_get_shmem_allocations.off,
+ pg_get_shmem_allocations.size,
+ pg_get_shmem_allocations.allocated
+ FROM pg_get_shmem_allocations() pg_get_shmem_allocations(key, off, size, allocated);
pg_stat_activity| SELECT s.datid,
d.datname,
s.pid,
--
1.8.5.rc2.dirty
On 04-05-2014 08:44, Andres Freund wrote:
I've more than once wanted to know what allocated shared memory in
postgres installation is used for. Especially with more an more
extensions around that's quite useful.
A few years ago I had to provide such information an did something
similar. Is it useful? Yes. However, it is a developer's feature.
On 2014-05-04 13:44:17 +0200, Andres Freund wrote:
Thinking about this, I think it was a mistake to not add a 'name' field
to dynamic shared memory's dsm_control_item. Right now it's very hard to
figure out which extension allocated a dsm segment. Imo we should change
that before 9.4 is out. I am not suggesting to use it to identify
segments, but just as an identifier, passed in into dsm_create().
+1.
Imo there should be a corresponding pg_dynshmem_allocations to
pg_shmem_allocations.
... or another boolean column (say 'dynamic') and just one view.
--
Euler Taveira Timbira - http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sun, May 4, 2014 at 7:50 AM, Andres Freund <andres@2ndquadrant.com> wrote:
On 2014-05-04 13:44:17 +0200, Andres Freund wrote:
postgres=# SELECT * FROM pg_shmem_allocations ORDER BY size DESC;
key | off | size | allocated
-------------------------------------+-------------+-------------+-----------
Buffer Blocks | 286242528 | 17179869184 | t
Buffer Descriptors | 152024800 | 134217728 | t
...
OldSerXidControlData | 17584357344 | 16 | t
(44 rows)Thinking about this, I think it was a mistake to not add a 'name' field
to dynamic shared memory's dsm_control_item. Right now it's very hard to
figure out which extension allocated a dsm segment. Imo we should change
that before 9.4 is out. I am not suggesting to use it to identify
segments, but just as an identifier, passed in into dsm_create().Imo there should be a corresponding pg_dynshmem_allocations to
pg_shmem_allocations.
Well, right now a dsm_control_item is 8 bytes. If we add a name field
of our usual 64 bytes, they'll each be 9 times bigger. We're not
talking about a lot of bytes in absolute terms, but I guess I'm not in
favor of an 800% size increase without somewhat more justification
than you've provided here. Who is using dynamic shared memory for
enough different things at this point to get confused?
I'm quite in favor of having something like this for the main shared
memory segment, but I think that's 9.5 material at this point.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
On Sun, May 4, 2014 at 7:50 AM, Andres Freund <andres@2ndquadrant.com> wrote:
Thinking about this, I think it was a mistake to not add a 'name' field
to dynamic shared memory's dsm_control_item.
Well, right now a dsm_control_item is 8 bytes. If we add a name field
of our usual 64 bytes, they'll each be 9 times bigger.
And the controlled shared segment is likely to be how big exactly? It's
probably not even possible for it to be smaller than a page size, 4K or
so depending on the OS. I agree with Andres that a name would be a good
idea; complaining about the space needed to hold it is penny-wise and
pound-foolish.
I'm quite in favor of having something like this for the main shared
memory segment, but I think that's 9.5 material at this point.
If you're prepared to break the current APIs later to add a name parameter
(which would have to be required, if it's to be useful at all), then sure,
put the question off till 9.5.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-05-05 15:04:07 -0400, Robert Haas wrote:
On Sun, May 4, 2014 at 7:50 AM, Andres Freund <andres@2ndquadrant.com> wrote:
On 2014-05-04 13:44:17 +0200, Andres Freund wrote:
postgres=# SELECT * FROM pg_shmem_allocations ORDER BY size DESC;
key | off | size | allocated
-------------------------------------+-------------+-------------+-----------
Buffer Blocks | 286242528 | 17179869184 | t
Buffer Descriptors | 152024800 | 134217728 | t
...
OldSerXidControlData | 17584357344 | 16 | t
(44 rows)Thinking about this, I think it was a mistake to not add a 'name' field
to dynamic shared memory's dsm_control_item. Right now it's very hard to
figure out which extension allocated a dsm segment. Imo we should change
that before 9.4 is out. I am not suggesting to use it to identify
segments, but just as an identifier, passed in into dsm_create().Imo there should be a corresponding pg_dynshmem_allocations to
pg_shmem_allocations.Well, right now a dsm_control_item is 8 bytes. If we add a name field
of our usual 64 bytes, they'll each be 9 times bigger. We're not
talking about a lot of bytes in absolute terms, but I guess I'm not in
favor of an 800% size increase without somewhat more justification
than you've provided here. Who is using dynamic shared memory for
enough different things at this point to get confused?
The kernel side overhead of creating a shared memory segment are so much
higher that this really isn't a meaningful saving. Also, are you really
considering a couple hundred bytes to be a problem?
I think it's quite a sensible thing for an administrator to ask where
all the memory has gone. The more users for dsm there the more important
that'll get. Right now pretty much the only thing a admin could do is to
poke around in /proc to see which backend has mapped the segment and try
to figure out via the logs what caused it to do so. Not nice.
I'm quite in favor of having something like this for the main shared
memory segment, but I think that's 9.5 material at this point.
Clearly. For one the version I posted here missed allocations which
aren't done via ShmemInitStruct (lwlock main array and hash table
allocations primarily). For another it's too late ;)
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-05-05 15:09:02 -0400, Tom Lane wrote:
I'm quite in favor of having something like this for the main shared
memory segment, but I think that's 9.5 material at this point.If you're prepared to break the current APIs later to add a name parameter
(which would have to be required, if it's to be useful at all), then sure,
put the question off till 9.5.
I understood Robert to mean that it's too late for my proposed view for
9.4 - and I agree - but I wholeheartedly agree with you that we should
add a name parameter to the dsm API *now*. We can just Assert() that it's
nonzero if we don't think it's useful for now.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, May 5, 2014 at 3:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
On Sun, May 4, 2014 at 7:50 AM, Andres Freund <andres@2ndquadrant.com> wrote:
Thinking about this, I think it was a mistake to not add a 'name' field
to dynamic shared memory's dsm_control_item.Well, right now a dsm_control_item is 8 bytes. If we add a name field
of our usual 64 bytes, they'll each be 9 times bigger.And the controlled shared segment is likely to be how big exactly? It's
probably not even possible for it to be smaller than a page size, 4K or
so depending on the OS. I agree with Andres that a name would be a good
idea; complaining about the space needed to hold it is penny-wise and
pound-foolish.
The control segment is sized to support a number of dynamic shared
memory segments not exceeding 64 + 2 *MaxBackends. With default
settings, that currently works out to 288 segments, or 2306 bytes.
So, adding a 64-byte name to each of those structures would increase
the size from 2k to about 20k.
So, sure, that's not a lot of memory. But I'm still not convinced
that's it's very useful. What I think is going to happen is that (1)
most people won't be used dynamic shared memory at all, so they won't
have any use for this; (2) those people who do run an extension that
uses dynamic shared memory will most likely only be running one such
extension, so they won't need a name to know what the segments are
being used for; and (3) if and when we eventually get parallel query,
dynamic shared memory segments will be widely used, but a bunch of
segments that are all named "parallel_query" or "parallel_query.$PID"
isn't going to be too informative.
Now, all that having been said, I recognize that human-readable names
are a generally useful thing, so I'm not going to hold my breath until
I turn blue if other people really want this, and it may turn out to
be useful someday. But if anyone is curious whether I'm *confident*
that it will be useful someday: at this point, no.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
On Mon, May 5, 2014 at 3:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
And the controlled shared segment is likely to be how big exactly? It's
probably not even possible for it to be smaller than a page size, 4K or
so depending on the OS. I agree with Andres that a name would be a good
idea; complaining about the space needed to hold it is penny-wise and
pound-foolish....
Now, all that having been said, I recognize that human-readable names
are a generally useful thing, so I'm not going to hold my breath until
I turn blue if other people really want this, and it may turn out to
be useful someday. But if anyone is curious whether I'm *confident*
that it will be useful someday: at this point, no.
I'm not confident that it'll be useful either. But I am confident that
if we don't put it in now, and decide we want it later, there will be
complaints when we change the API. Better to have an ignored parameter
than no parameter.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, May 5, 2014 at 6:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
On Mon, May 5, 2014 at 3:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
And the controlled shared segment is likely to be how big exactly? It's
probably not even possible for it to be smaller than a page size, 4K or
so depending on the OS. I agree with Andres that a name would be a good
idea; complaining about the space needed to hold it is penny-wise and
pound-foolish....
Now, all that having been said, I recognize that human-readable names
are a generally useful thing, so I'm not going to hold my breath until
I turn blue if other people really want this, and it may turn out to
be useful someday. But if anyone is curious whether I'm *confident*
that it will be useful someday: at this point, no.I'm not confident that it'll be useful either. But I am confident that
if we don't put it in now, and decide we want it later, there will be
complaints when we change the API. Better to have an ignored parameter
than no parameter.
I'm generally skeptical of that philosophy. If we put in an ignored
parameter, people may pass pointers to NULL or to garbage or to an
overly-long string, and they won't know it's broken until we make it
do something; at which point their code will begin to fail without
warning. Speaking as an employee of a company that maintains several
PostgreSQL extensions that sometimes need to be updated for newer
server versions, I'd rather have a clean API break that makes the
build fail than a "soft" break that supposedly lets things continue
working but maybe breaks them in subtler ways. Another problem with
this idea is that we might never get around to making it do anything,
and then the dead parameter is just a stupid and unnecessary wart.
If we're going to do anything at all here for 9.4, I recommend
ignoring the fact we're in feature freeze and going whole hog: add the
name, add the monitoring view, and add the monitoring view for the
main shared memory segment just for good measure. That way, if we get
the design wrong or something, we have a chance of getting some
feedback. If we're not going to do that, then I vote for doing
nothing and considering later whether to break it for 9.5, by which
time we may have some evidence as to whether and how this code is
really being used. Anyone who expects PostgreSQL's C API to be
completely stable is going to be regularly disappointed, as most
recently demonstrated by the Enormous Header Churn of the 9.3 era. I
don't particularly mind being the cause of further disappointment; as
long as the breakage is obvious rather than subtle, the fix usually
takes about 10 minutes.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 5 May 2014 21:54, Robert Haas <robertmhaas@gmail.com> wrote:
On Mon, May 5, 2014 at 3:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
On Sun, May 4, 2014 at 7:50 AM, Andres Freund <andres@2ndquadrant.com> wrote:
Thinking about this, I think it was a mistake to not add a 'name' field
to dynamic shared memory's dsm_control_item.Well, right now a dsm_control_item is 8 bytes. If we add a name field
of our usual 64 bytes, they'll each be 9 times bigger.And the controlled shared segment is likely to be how big exactly? It's
probably not even possible for it to be smaller than a page size, 4K or
so depending on the OS. I agree with Andres that a name would be a good
idea; complaining about the space needed to hold it is penny-wise and
pound-foolish.The control segment is sized to support a number of dynamic shared
memory segments not exceeding 64 + 2 *MaxBackends. With default
settings, that currently works out to 288 segments, or 2306 bytes.
So, adding a 64-byte name to each of those structures would increase
the size from 2k to about 20k.So, sure, that's not a lot of memory. But I'm still not convinced
that's it's very useful. What I think is going to happen is that (1)
most people won't be used dynamic shared memory at all, so they won't
have any use for this; (2) those people who do run an extension that
uses dynamic shared memory will most likely only be running one such
extension, so they won't need a name to know what the segments are
being used for; and (3) if and when we eventually get parallel query,
dynamic shared memory segments will be widely used, but a bunch of
segments that are all named "parallel_query" or "parallel_query.$PID"
isn't going to be too informative.
Not sure your arguments hold any water.
Most people don't use most features... and so we're not allowed
features that can be debugged?
How do you know people will only use one extension that uses dshmem?
Why would we call multiple segments the same thing??
If names are a problem, lets give them numbers. Seems a minor point.
Perhaps we can allocate space for names dynamically??
Not being able to tell segments apart from each other is just daft, if
we are trying to supply bug free software for the world to use.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-05-05 23:20:43 -0400, Robert Haas wrote:
On Mon, May 5, 2014 at 6:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I'm not confident that it'll be useful either. But I am confident that
if we don't put it in now, and decide we want it later, there will be
complaints when we change the API. Better to have an ignored parameter
than no parameter.I'm generally skeptical of that philosophy. If we put in an ignored
parameter, people may pass pointers to NULL or to garbage or to an
overly-long string, and they won't know it's broken until we make it
do something; at which point their code will begin to fail without
warning.
If it were a complex change, maybe. But I don't think that's likely
here.
Assert(name != NULL && strlen(name) > 0 && strlen(name) < NAMEDATALEN);
should perfectly do the trick.
If we're going to do anything at all here for 9.4, I recommend
ignoring the fact we're in feature freeze and going whole hog: add the
name, add the monitoring view, and add the monitoring view for the
main shared memory segment just for good measure.
We can do that as well. If there's agreement on that path I'll update
the patch to also show dynamic statements.
Anyone who expects PostgreSQL's C API to be
completely stable is going to be regularly disappointed, as most
recently demonstrated by the Enormous Header Churn of the 9.3 era. I
don't particularly mind being the cause of further disappointment; as
long as the breakage is obvious rather than subtle, the fix usually
takes about 10 minutes.
Didn't you complain rather loudly about that change?
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Tue, May 6, 2014 at 7:45 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
The control segment is sized to support a number of dynamic shared
memory segments not exceeding 64 + 2 *MaxBackends. With default
settings, that currently works out to 288 segments, or 2306 bytes.
So, adding a 64-byte name to each of those structures would increase
the size from 2k to about 20k.So, sure, that's not a lot of memory. But I'm still not convinced
that's it's very useful. What I think is going to happen is that (1)
most people won't be used dynamic shared memory at all, so they won't
have any use for this; (2) those people who do run an extension that
uses dynamic shared memory will most likely only be running one such
extension, so they won't need a name to know what the segments are
being used for; and (3) if and when we eventually get parallel query,
dynamic shared memory segments will be widely used, but a bunch of
segments that are all named "parallel_query" or "parallel_query.$PID"
isn't going to be too informative.Not sure your arguments hold any water.
I'm not, either.
Most people don't use most features... and so we're not allowed
features that can be debugged?
I certainly didn't say that.
How do you know people will only use one extension that uses dshmem?
I don't. If they do, that's a good argument for adding this.
Why would we call multiple segments the same thing??
It's not clear to me how someone is going to intelligently name
multiple segments used by the same extension. Maybe they'll give them
all the same name. Maybe they'll name them all extension_name.pid.
More than likely, different extensions will use different conventions.
:-(
It might be sensible to add a "creator PID" field to the DSM control
items. Of course, that PID might have exited, but it could still
possibly be useful for debugging purposes.
If names are a problem, lets give them numbers. Seems a minor point.
Perhaps we can allocate space for names dynamically??
A static buffer, as proposed by Andres, seems a lot simper.
Not being able to tell segments apart from each other is just daft, if
we are trying to supply bug free software for the world to use.
I can see I'm losing this argument.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 05/06/2014 02:59 PM, Robert Haas wrote:
Why would we call multiple segments the same thing??
It's not clear to me how someone is going to intelligently name
multiple segments used by the same extension. Maybe they'll give them
all the same name. Maybe they'll name them all extension_name.pid.
More than likely, different extensions will use different conventions.
:-(
That seems sensible to me. The best scheme will depend on how the
segments are used. Best to leave it to the extension author.
- Heikki
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 6 May 2014 13:06, Heikki Linnakangas <hlinnakangas@vmware.com> wrote:
The best scheme will depend on how the segments
are used. Best to leave it to the extension author.
+1
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Hi,
On 2014-05-06 13:56:41 +0200, Andres Freund wrote:
On 2014-05-05 23:20:43 -0400, Robert Haas wrote:
On Mon, May 5, 2014 at 6:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I'm not confident that it'll be useful either. But I am confident that
if we don't put it in now, and decide we want it later, there will be
complaints when we change the API. Better to have an ignored parameter
than no parameter.I'm generally skeptical of that philosophy. If we put in an ignored
parameter, people may pass pointers to NULL or to garbage or to an
overly-long string, and they won't know it's broken until we make it
do something; at which point their code will begin to fail without
warning.If it were a complex change, maybe. But I don't think that's likely
here.
Assert(name != NULL && strlen(name) > 0 && strlen(name) < NAMEDATALEN);
should perfectly do the trick.
Attached are two patches:
a) Patch addin a 'name' parameter to dsm_create(). I think we should
apply this to 9.4.
b) pg_dynamic_shmem_allocations and pg_static_shmem_allocations
views. The previous version didn't include dsm support and didn't
take the required lock.
I am not so sure whether b) should be applied together with a) in 9.4,
but I'd be happy enough to add docs if people agree with the naming.
FWIW, I like dsm_create()'s internals more after this patch...
postgres=# \d pg_dynamic_shmem_allocations
View "pg_catalog.pg_dynamic_shmem_allocations"
Column | Type | Modifiers
--------+--------+-----------
handle | bigint |
name | text |
size | bigint |
refcnt | bigint |
postgres=# \d pg_static_shmem_allocations
View "pg_catalog.pg_static_shmem_allocations"
Column | Type | Modifiers
-----------+---------+-----------
key | text |
off | bigint |
size | bigint |
allocated | boolean |
postgres=# SELECT * FROM pg_dynamic_shmem_allocations;
handle | name | size | refcnt
------------+-------------+-------+--------
1120921036 | test_shm_mq | 65656 | 1
(1 row)
postgres=# SELECT * FROM pg_static_shmem_allocations ORDER BY key NULLS FIRST;
key | off | size | allocated
-------------------------------------+------------+------------+-----------
| 2222605024 | 1727776 | f
| | 34844752 | t
Async Ctl | 2222539168 | 65856 | t
Async Queue Control | 2222537784 | 1384 | t
AutoVacuum Data | 2222533576 | 224 | t
Backend Activity Buffer | 2217099552 | 114688 | t
Backend Application Name Buffer | 2217085216 | 7168 | t
Backend Client Host Name Buffer | 2217092384 | 7168 | t
Backend Status Array | 2217061024 | 24192 | t
Background Worker Data | 2217214256 | 1992 | t
BTree Vacuum State | 2222535768 | 1356 | t
Buffer Blocks | 51365312 | 2147483648 | t
Buffer Descriptors | 34588096 | 16777216 | t
Buffer Strategy Status | 2213546176 | 32 | t
Checkpointer Data | 2217290656 | 5242920 | t
CLOG Ctl | 33601152 | 525312 | t
Control File | 16796384 | 240 | t
Fast Path Strong Relation Lock Data | 2214767072 | 4100 | t
FinishedSerializableTransactions | 2216841952 | 16 | t
LOCK hash | 2213546208 | 2160 | t
MultiXactMember Ctl | 34455488 | 131648 | t
MultiXactOffset Ctl | 34389632 | 65856 | t
OldSerXidControlData | 2216973632 | 16 | t
OldSerXid SLRU Ctl | 2216841984 | 131648 | t
PMSignalState | 2217285400 | 940 | t
PREDICATELOCK hash | 2215182944 | 2160 | t
PREDICATELOCKTARGET hash | 2214771176 | 2160 | t
PredXactList | 2216348384 | 88 | t
Prepared Transaction Table | 2217214240 | 16 | t
Proc Array | 2217060536 | 488 | t
Proc Header | 2216973648 | 88 | t
PROCLOCK hash | 2214183264 | 2160 | t
ProcSignalSlots | 2217286344 | 4284 | t
RWConflictPool | 2216573120 | 24 | t
SERIALIZABLEXID hash | 2216518720 | 2160 | t
Shared Buffer Lookup Table | 2198848960 | 16496 | t
Shared MultiXact State | 34587136 | 936 | t
shmInvalBuffer | 2217216256 | 69144 | t
SUBTRANS Ctl | 34126464 | 263168 | t
Sync Scan Locations List | 2222537128 | 656 | t
Wal Receiver Ctl | 2222534576 | 1192 | t
Wal Sender Ctl | 2222533800 | 776 | t
XLOG Ctl | 16796640 | 16804496 | t
(43 rows)
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
0001-Associate-names-to-created-dynamic-shared-memory-seg.patchtext/x-patch; charset=us-asciiDownload
>From 43ae2a5397fba3b83afced6ec813449a1c87f8c0 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 6 May 2014 19:42:36 +0200
Subject: [PATCH 1/2] Associate names to created dynamic shared memory
segments.
At some later point we want to add a view show all allocated dynamic
shared memory segments so admins can understand resource usage. To
avoid breaking the API in 9.5 add the necessary name now.
---
contrib/test_shm_mq/setup.c | 2 +-
src/backend/storage/ipc/dsm.c | 60 ++++++++++++++++++++++++++-----------------
src/include/storage/dsm.h | 2 +-
3 files changed, 39 insertions(+), 25 deletions(-)
diff --git a/contrib/test_shm_mq/setup.c b/contrib/test_shm_mq/setup.c
index 572cf88..897c47b 100644
--- a/contrib/test_shm_mq/setup.c
+++ b/contrib/test_shm_mq/setup.c
@@ -125,7 +125,7 @@ setup_dynamic_shared_memory(int64 queue_size, int nworkers,
segsize = shm_toc_estimate(&e);
/* Create the shared memory segment and establish a table of contents. */
- seg = dsm_create(shm_toc_estimate(&e));
+ seg = dsm_create("test_shm_mq", shm_toc_estimate(&e));
toc = shm_toc_create(PG_TEST_SHM_MQ_MAGIC, dsm_segment_address(seg),
segsize);
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index a5c0084..66e24f0 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -80,8 +80,10 @@ struct dsm_segment
/* Shared-memory state for a dynamic shared memory segment. */
typedef struct dsm_control_item
{
- dsm_handle handle;
+ dsm_handle handle; /* segment identifier */
uint32 refcnt; /* 2+ = active, 1 = moribund, 0 = gone */
+ Size size; /* current size */
+ char name[SHMEM_INDEX_KEYSIZE]; /* informational name */
} dsm_control_item;
/* Layout of the dynamic shared memory control segment. */
@@ -454,14 +456,16 @@ dsm_set_control_handle(dsm_handle h)
* Create a new dynamic shared memory segment.
*/
dsm_segment *
-dsm_create(Size size)
+dsm_create(const char *name, Size size)
{
dsm_segment *seg = dsm_create_descriptor();
- uint32 i;
- uint32 nitems;
+ dsm_control_item *item;
+ uint32 slot;
/* Unsafe in postmaster (and pointless in a stand-alone backend). */
Assert(IsUnderPostmaster);
+ Assert(name != NULL && strlen(name) > 0 &&
+ strlen(name) < SHMEM_INDEX_KEYSIZE);
if (!dsm_init_done)
dsm_backend_startup();
@@ -479,33 +483,39 @@ dsm_create(Size size)
/* Lock the control segment so we can register the new segment. */
LWLockAcquire(DynamicSharedMemoryControlLock, LW_EXCLUSIVE);
- /* Search the control segment for an unused slot. */
- nitems = dsm_control->nitems;
- for (i = 0; i < nitems; ++i)
+ /*
+ * Search the control segment for an unused slot that's previously been
+ * used. If we don't find one initialize a new one if there's still space.
+ */
+ for (slot = 0; slot < dsm_control->nitems; ++slot)
{
- if (dsm_control->item[i].refcnt == 0)
- {
- dsm_control->item[i].handle = seg->handle;
- /* refcnt of 1 triggers destruction, so start at 2 */
- dsm_control->item[i].refcnt = 2;
- seg->control_slot = i;
- LWLockRelease(DynamicSharedMemoryControlLock);
- return seg;
- }
+ if (dsm_control->item[slot].refcnt == 0)
+ break;
}
- /* Verify that we can support an additional mapping. */
- if (nitems >= dsm_control->maxitems)
+ /* Verify that we can support the mapping. */
+ if (slot >= dsm_control->maxitems)
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_RESOURCES),
errmsg("too many dynamic shared memory segments")));
- /* Enter the handle into a new array slot. */
- dsm_control->item[nitems].handle = seg->handle;
+ item = &dsm_control->item[slot];
+ item->handle = seg->handle;
/* refcnt of 1 triggers destruction, so start at 2 */
- dsm_control->item[nitems].refcnt = 2;
- seg->control_slot = nitems;
- dsm_control->nitems++;
+ item->refcnt = 2;
+ item->size = size;
+ strncpy(item->name, name, SHMEM_INDEX_SIZE - 1);
+ item->name[SHMEM_INDEX_SIZE] = 0;
+
+ seg->control_slot = slot;
+
+ /*
+ * Increase number of initilized slots if we didn't reuse a previously
+ * used one.
+ */
+ if (slot >= dsm_control->nitems)
+ dsm_control->nitems++;
+
LWLockRelease(DynamicSharedMemoryControlLock);
return seg;
@@ -658,6 +668,10 @@ dsm_resize(dsm_segment *seg, Size size)
Assert(seg->control_slot != INVALID_CONTROL_SLOT);
dsm_impl_op(DSM_OP_RESIZE, seg->handle, size, &seg->impl_private,
&seg->mapped_address, &seg->mapped_size, ERROR);
+
+ /* persist the changed size */
+ dsm_control->item[seg->control_slot].size = size;
+
return seg->mapped_address;
}
diff --git a/src/include/storage/dsm.h b/src/include/storage/dsm.h
index 1d0110d..3dbe53b 100644
--- a/src/include/storage/dsm.h
+++ b/src/include/storage/dsm.h
@@ -29,7 +29,7 @@ extern void dsm_set_control_handle(dsm_handle h);
#endif
/* Functions that create, update, or remove mappings. */
-extern dsm_segment *dsm_create(Size size);
+extern dsm_segment *dsm_create(const char *name, Size size);
extern dsm_segment *dsm_attach(dsm_handle h);
extern void *dsm_resize(dsm_segment *seg, Size size);
extern void *dsm_remap(dsm_segment *seg);
--
1.8.5.rc2.dirty
0002-Add-views-to-see-shared-memory-allocations.patchtext/x-patch; charset=us-asciiDownload
>From 24c42133a4017a9b6caf60c0f824b8f8eaed9d51 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 4 May 2014 13:37:20 +0200
Subject: [PATCH 2/2] Add views to see shared memory allocations.
---
src/backend/catalog/system_views.sql | 6 ++
src/backend/storage/ipc/dsm.c | 80 ++++++++++++++++++++++
src/backend/storage/ipc/shmem.c | 124 +++++++++++++++++++++++++++++++++++
src/include/catalog/pg_proc.h | 4 ++
src/include/utils/builtins.h | 5 ++
src/test/regress/expected/rules.out | 9 +++
6 files changed, 228 insertions(+)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 42a4c00..c414260 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -387,6 +387,12 @@ CREATE VIEW pg_timezone_abbrevs AS
CREATE VIEW pg_timezone_names AS
SELECT * FROM pg_timezone_names();
+CREATE VIEW pg_static_shmem_allocations AS
+ SELECT * FROM pg_get_static_shmem_allocations();
+
+CREATE VIEW pg_dynamic_shmem_allocations AS
+ SELECT * FROM pg_get_dynamic_shmem_allocations();
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index 66e24f0..a8c6c38 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -34,12 +34,15 @@
#endif
#include <sys/stat.h>
+#include "fmgr.h"
+#include "funcapi.h"
#include "lib/ilist.h"
#include "miscadmin.h"
#include "storage/dsm.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
+#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/memutils.h"
#include "utils/resowner_private.h"
@@ -1022,3 +1025,80 @@ dsm_control_bytes_needed(uint32 nitems)
return offsetof(dsm_control_header, item)
+sizeof(dsm_control_item) * (uint64) nitems;
}
+
+/* SQL SRF showing allocated shared memory */
+Datum
+pg_get_dynamic_shmem_allocations(PG_FUNCTION_ARGS)
+{
+#define PG_GET_SHMEM_SIZES_COLS 4
+
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ int i;
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not " \
+ "allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ LWLockAcquire(DynamicSharedMemoryControlLock, LW_SHARED);
+
+ for (i = 0; i < dsm_control->nitems; ++i)
+ {
+ dsm_control_item *item;
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* don't look at unused or about to be destroyed items */
+ if (dsm_control->item[i].refcnt < 2)
+ continue;
+
+ item = &dsm_control->item[i];
+
+ /* handle */
+ values[0] = Int64GetDatum(item->handle);
+ nulls[0] = false;
+
+ /* name */
+ values[1] = CStringGetTextDatum(item->name);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(item->size);
+ nulls[2] = false;
+
+ /* refcnt */
+ values[3] = Int64GetDatum(item->refcnt - 2);
+ nulls[3] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+ LWLockRelease(DynamicSharedMemoryControlLock);
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 2ea2216..9f08c4e 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -66,11 +66,14 @@
#include "postgres.h"
#include "access/transam.h"
+#include "fmgr.h"
+#include "funcapi.h"
#include "miscadmin.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
#include "storage/shmem.h"
#include "storage/spin.h"
+#include "utils/builtins.h"
/* shared memory global variables */
@@ -459,3 +462,124 @@ mul_size(Size s1, Size s2)
errmsg("requested shared memory size overflows size_t")));
return result;
}
+
+/* SQL SRF showing allocated shared memory */
+Datum
+pg_get_static_shmem_allocations(PG_FUNCTION_ARGS)
+{
+#define PG_GET_SHMEM_SIZES_COLS 4
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ HASH_SEQ_STATUS hstat;
+ ShmemIndexEnt *ent;
+ Size named_allocated = 0;
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not " \
+ "allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ hash_seq_init(&hstat, ShmemIndex);
+
+ LWLockAcquire(ShmemIndexLock, LW_SHARED);
+
+ /* output all allocated entries */
+ while ((ent = (ShmemIndexEnt *) hash_seq_search(&hstat)) != NULL)
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key */
+ values[0] = CStringGetTextDatum(ent->key);
+ nulls[0] = false;
+
+ /* off */
+ values[1] = Int64GetDatum((char *) ent->location - (char *) ShmemSegHdr);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(ent->size);
+ nulls[2] = false;
+ named_allocated += ent->size;
+
+ /* allocated */
+ values[3] = BoolGetDatum(true);
+ nulls[3] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* output shared memory allocated but not counted via the shmem index */
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key, show unnamed as NULL */
+ nulls[0] = true;
+
+ /* off */
+ nulls[1] = true;
+
+ /* size */
+ values[2] = Int64GetDatum(ShmemSegHdr->freeoffset - named_allocated);
+ nulls[2] = false;
+
+ /* allocated */
+ values[3] = BoolGetDatum(true);
+ nulls[3] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* output as-of-yet unused shared memory */
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key, show unallocated as NULL */
+ nulls[0] = true;
+
+ /* off */
+ values[1] = Int64GetDatum(ShmemSegHdr->freeoffset);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(ShmemSegHdr->totalsize - ShmemSegHdr->freeoffset);
+ nulls[2] = false;
+
+ /* allocated */
+ values[3] = BoolGetDatum(false);
+ nulls[3] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ LWLockRelease(ShmemIndexLock);
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index e601ccd..b59cfee 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -3899,6 +3899,10 @@ DATA(insert OID = 3035 ( pg_listening_channels PGNSP PGUID 12 1 10 0 0 f f f f
DESCR("get the channels that the current backend listens to");
DATA(insert OID = 3036 ( pg_notify PGNSP PGUID 12 1 0 0 0 f f f f f f v 2 0 2278 "25 25" _null_ _null_ _null_ _null_ pg_notify _null_ _null_ _null_ ));
DESCR("send a notification event");
+DATA(insert OID = 86 ( pg_get_static_shmem_allocations PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{25,20,20,16}" "{o,o,o,o}" "{key, off, size, allocated}" _null_ pg_get_static_shmem_allocations _null_ _null_ _null_ ));
+DESCR("show static shared memory allocations");
+DATA(insert OID = 87 ( pg_get_dynamic_shmem_allocations PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{20,25,20,20}" "{o,o,o,o}" "{handle, name, size, refcnt}" _null_ pg_get_dynamic_shmem_allocations _null_ _null_ _null_ ));
+DESCR("show dynamic shared memory allocations");
/* non-persistent series generator */
DATA(insert OID = 1066 ( generate_series PGNSP PGUID 12 1 1000 0 0 f f f f t t i 3 0 23 "23 23 23" _null_ _null_ _null_ _null_ generate_series_step_int4 _null_ _null_ _null_ ));
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index bbb5d39..ea91014 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -1209,4 +1209,9 @@ extern Datum pg_prepared_statement(PG_FUNCTION_ARGS);
/* utils/mmgr/portalmem.c */
extern Datum pg_cursor(PG_FUNCTION_ARGS);
+/* backend/storage/ipc/shmem.c */
+extern Datum pg_get_static_shmem_allocations(PG_FUNCTION_ARGS);
+/* backend/storage/ipc/dsm.c */
+extern Datum pg_get_dynamic_shmem_allocations(PG_FUNCTION_ARGS);
+
#endif /* BUILTINS_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 87870cf..66937d5 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1308,6 +1308,10 @@ pg_cursors| SELECT c.name,
c.is_scrollable,
c.creation_time
FROM pg_cursor() c(name, statement, is_holdable, is_binary, is_scrollable, creation_time);
+pg_dynamic_shmem_allocations| SELECT pg_get_dynamic_shmem_allocations.key,
+ pg_get_dynamic_shmem_allocations.size,
+ pg_get_dynamic_shmem_allocations.refcnt
+ FROM pg_get_dynamic_shmem_allocations() pg_get_dynamic_shmem_allocations(key, size, refcnt);
pg_group| SELECT pg_authid.rolname AS groname,
pg_authid.oid AS grosysid,
ARRAY( SELECT pg_auth_members.member
@@ -1848,6 +1852,11 @@ pg_stat_xact_user_tables| SELECT pg_stat_xact_all_tables.relid,
pg_stat_xact_all_tables.n_tup_hot_upd
FROM pg_stat_xact_all_tables
WHERE ((pg_stat_xact_all_tables.schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (pg_stat_xact_all_tables.schemaname !~ '^pg_toast'::text));
+pg_static_shmem_allocations| SELECT pg_get_static_shmem_allocations.key,
+ pg_get_static_shmem_allocations.off,
+ pg_get_static_shmem_allocations.size,
+ pg_get_static_shmem_allocations.allocated
+ FROM pg_get_static_shmem_allocations() pg_get_static_shmem_allocations(key, off, size, allocated);
pg_statio_all_indexes| SELECT c.oid AS relid,
i.oid AS indexrelid,
n.nspname AS schemaname,
--
1.8.5.rc2.dirty
Andres Freund <andres@2ndquadrant.com> writes:
Attached are two patches:
a) Patch addin a 'name' parameter to dsm_create(). I think we should
apply this to 9.4.
b) pg_dynamic_shmem_allocations and pg_static_shmem_allocations
views. The previous version didn't include dsm support and didn't
take the required lock.
I am not so sure whether b) should be applied together with a) in 9.4,
but I'd be happy enough to add docs if people agree with the naming.
FWIW, I vote for fixing (a) now but holding (b) for 9.5.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Tue, May 6, 2014 at 2:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Andres Freund <andres@2ndquadrant.com> writes:
Attached are two patches:
a) Patch addin a 'name' parameter to dsm_create(). I think we should
apply this to 9.4.
b) pg_dynamic_shmem_allocations and pg_static_shmem_allocations
views. The previous version didn't include dsm support and didn't
take the required lock.I am not so sure whether b) should be applied together with a) in 9.4,
but I'd be happy enough to add docs if people agree with the naming.FWIW, I vote for fixing (a) now but holding (b) for 9.5.
I guess I'll vote for applying both. I don't see a lot of risk, and I
think doing one with out the other is somewhat pointless.
Regarding patch 0002, I don't think we're using the term "static
shmem" anywhere else, so I vote for dropping the word static, so that
we have pg_get_shmem_allocations() and
pg_get_dynamic_shmem_allocations(). Also, I think using the
"allocated" column is pretty weird. There are always exactly two
entries with allocated = false, one of which is for actual free memory
and the other of which is for memory that actually IS allocated but
without using ShmemIndex (maybe the latter was supposed to have
allocated = true but still key = null?). I guess I'd vote for
ditching the allocated column completely and outputting the memory
allocated without ShmemIndex using some fixed tag (like "ShmemIndex"
or "Bootstrap" or "Overhead" or something).
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
On Tue, May 6, 2014 at 2:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
FWIW, I vote for fixing (a) now but holding (b) for 9.5.
I guess I'll vote for applying both. I don't see a lot of risk, and I
think doing one with out the other is somewhat pointless.
The difference is that there's not consensus about the details of the
views ... as borne out by your next paragraph.
Now admittedly, we could always redefine the views in 9.5, but
I'd rather not be doing this sort of thing in haste. Something
as user-visible as a system view really ought to have baked awhile
before we ship it. Patch (a) is merely institutionalizing the
expectation that DSM segments should have names, which is a much
lower-risk bet.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 6 May 2014 20:44, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
On Tue, May 6, 2014 at 2:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
FWIW, I vote for fixing (a) now but holding (b) for 9.5.
I guess I'll vote for applying both. I don't see a lot of risk, and I
think doing one with out the other is somewhat pointless.The difference is that there's not consensus about the details of the
views ... as borne out by your next paragraph.Now admittedly, we could always redefine the views in 9.5, but
I'd rather not be doing this sort of thing in haste. Something
as user-visible as a system view really ought to have baked awhile
before we ship it. Patch (a) is merely institutionalizing the
expectation that DSM segments should have names, which is a much
lower-risk bet.
As long as all the functions are exposed to allow b) to run as an
extension, I don't see we lose anything by waiting a while.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-05-06 15:37:04 -0400, Robert Haas wrote:
On Tue, May 6, 2014 at 2:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Andres Freund <andres@2ndquadrant.com> writes:
Attached are two patches:
a) Patch addin a 'name' parameter to dsm_create(). I think we should
apply this to 9.4.
b) pg_dynamic_shmem_allocations and pg_static_shmem_allocations
views. The previous version didn't include dsm support and didn't
take the required lock.I am not so sure whether b) should be applied together with a) in 9.4,
but I'd be happy enough to add docs if people agree with the naming.FWIW, I vote for fixing (a) now but holding (b) for 9.5.
I guess I'll vote for applying both. I don't see a lot of risk, and I
think doing one with out the other is somewhat pointless.
Fine with me. I guess if we don't do b) for now we can just do the
additional parameter and the Assert() from the patch. Without actually
storing the name to shared memory.
Regarding patch 0002, I don't think we're using the term "static
shmem" anywhere else, so I vote for dropping the word static, so that
we have pg_get_shmem_allocations() and
pg_get_dynamic_shmem_allocations().
Fine #2.
Also, I think using the
"allocated" column is pretty weird. There are always exactly two
entries with allocated = false
Hm. There shouldn't be. And at least in my installation there isn't and
I don't see a anything in the code that'd allow that? The example from
my last email has:
postgres=# SELECT * FROM pg_static_shmem_allocations ORDER BY key NULLS FIRST;
key | off | size | allocated
-------------------------------------+------------+------------+-----------
| 2222605024 | 1727776 | f
| | 34844752 | t
Async Ctl | 2222539168 | 65856 | t
, one of which is for actual free memory
and the other of which is for memory that actually IS allocated but
without using ShmemIndex (maybe the latter was supposed to have
allocated = true but still key = null?).
Yes, that's how I'd imagined it.
I guess I'd vote for
ditching the allocated column completely and outputting the memory
allocated without ShmemIndex using some fixed tag (like "ShmemIndex"
or "Bootstrap" or "Overhead" or something).
My way feels slightly cleaner, but I'd be ok with that as well. There's
no possible conflicts with an actual segment... In your variant the
unallocated/slop memory would continue to have a NULL key?
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-05-06 22:04:04 +0100, Simon Riggs wrote:
On 6 May 2014 20:44, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
On Tue, May 6, 2014 at 2:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
FWIW, I vote for fixing (a) now but holding (b) for 9.5.
I guess I'll vote for applying both. I don't see a lot of risk, and I
think doing one with out the other is somewhat pointless.The difference is that there's not consensus about the details of the
views ... as borne out by your next paragraph.Now admittedly, we could always redefine the views in 9.5, but
I'd rather not be doing this sort of thing in haste. Something
as user-visible as a system view really ought to have baked awhile
before we ship it. Patch (a) is merely institutionalizing the
expectation that DSM segments should have names, which is a much
lower-risk bet.As long as all the functions are exposed to allow b) to run as an
extension, I don't see we lose anything by waiting a while.
They aren't exposed. It's touching implementation details in both
shmem.c and dsm.c. I think that's actually fine.
Imo it's not too bad if we don't get either in 9.4. It's not a critical
feature.What I *would* like to avoid is a pointless API break between
9.4 and 9.5. Because I will push for the patch in 9.5 CF1...
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Tue, May 6, 2014 at 6:09 PM, Andres Freund <andres@2ndquadrant.com> wrote:
I guess I'd vote for
ditching the allocated column completely and outputting the memory
allocated without ShmemIndex using some fixed tag (like "ShmemIndex"
or "Bootstrap" or "Overhead" or something).My way feels slightly cleaner, but I'd be ok with that as well. There's
no possible conflicts with an actual segment... In your variant the
unallocated/slop memory would continue to have a NULL key?
Yeah, that seems all right.
One way to avoid conflict with an actual segment would be to add an
after-the-fact entry into ShmemIndex representing the amount of memory
that was used to bootstrap it.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-05-07 17:48:15 -0400, Robert Haas wrote:
On Tue, May 6, 2014 at 6:09 PM, Andres Freund <andres@2ndquadrant.com> wrote:
I guess I'd vote for
ditching the allocated column completely and outputting the memory
allocated without ShmemIndex using some fixed tag (like "ShmemIndex"
or "Bootstrap" or "Overhead" or something).My way feels slightly cleaner, but I'd be ok with that as well. There's
no possible conflicts with an actual segment... In your variant the
unallocated/slop memory would continue to have a NULL key?Yeah, that seems all right.
Hm. Not sure what you're ACKing here ;).
One way to avoid conflict with an actual segment would be to add an
after-the-fact entry into ShmemIndex representing the amount of memory
that was used to bootstrap it.
There's lots of allocations from shmem that cannot be associated with
any index entry though. Not just ShmemIndex's own entry. Most
prominently most of the memory used for SharedBufHash isn't actually
associated with the "Shared Buffer Lookup Table" entry - imo a
dynahash.c defficiency.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, May 7, 2014 at 5:54 PM, Andres Freund <andres@2ndquadrant.com> wrote:
On 2014-05-07 17:48:15 -0400, Robert Haas wrote:
On Tue, May 6, 2014 at 6:09 PM, Andres Freund <andres@2ndquadrant.com> wrote:
I guess I'd vote for
ditching the allocated column completely and outputting the memory
allocated without ShmemIndex using some fixed tag (like "ShmemIndex"
or "Bootstrap" or "Overhead" or something).My way feels slightly cleaner, but I'd be ok with that as well. There's
no possible conflicts with an actual segment... In your variant the
unallocated/slop memory would continue to have a NULL key?Yeah, that seems all right.
Hm. Not sure what you're ACKing here ;).
The idea of giving the unallocated memory a NULL key.
One way to avoid conflict with an actual segment would be to add an
after-the-fact entry into ShmemIndex representing the amount of memory
that was used to bootstrap it.There's lots of allocations from shmem that cannot be associated with
any index entry though. Not just ShmemIndex's own entry. Most
prominently most of the memory used for SharedBufHash isn't actually
associated with the "Shared Buffer Lookup Table" entry - imo a
dynahash.c defficiency.
Hmm, I don't know what to do about that.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-05-08 07:58:34 -0400, Robert Haas wrote:
On Wed, May 7, 2014 at 5:54 PM, Andres Freund <andres@2ndquadrant.com> wrote:
Hm. Not sure what you're ACKing here ;).
The idea of giving the unallocated memory a NULL key.
Ok. A new version of the patches implementing that are
attached. Including a couple of small fixups and docs. The latter aren't
extensive, but that doesn't seem to be warranted anyway.
There's lots of allocations from shmem that cannot be associated with
any index entry though. Not just ShmemIndex's own entry. Most
prominently most of the memory used for SharedBufHash isn't actually
associated with the "Shared Buffer Lookup Table" entry - imo a
dynahash.c defficiency.
Hmm, I don't know what to do about that.
Well, we have to live with it for now :)
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
0001-Associate-names-to-created-dynamic-shared-memory-seg.patchtext/x-patch; charset=us-asciiDownload
>From c219c03a173fef962c1caba9f016d5d87448fd8f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 6 May 2014 19:42:36 +0200
Subject: [PATCH 1/4] Associate names to created dynamic shared memory
segments.
At some later point we want to add a view show all allocated dynamic
shared memory segments so admins can understand resource usage. To
avoid breaking the API in 9.5 add the necessary name now.
---
contrib/test_shm_mq/setup.c | 2 +-
src/backend/storage/ipc/dsm.c | 60 ++++++++++++++++++++++++++-----------------
src/include/storage/dsm.h | 2 +-
3 files changed, 39 insertions(+), 25 deletions(-)
diff --git a/contrib/test_shm_mq/setup.c b/contrib/test_shm_mq/setup.c
index 572cf88..897c47b 100644
--- a/contrib/test_shm_mq/setup.c
+++ b/contrib/test_shm_mq/setup.c
@@ -125,7 +125,7 @@ setup_dynamic_shared_memory(int64 queue_size, int nworkers,
segsize = shm_toc_estimate(&e);
/* Create the shared memory segment and establish a table of contents. */
- seg = dsm_create(shm_toc_estimate(&e));
+ seg = dsm_create("test_shm_mq", shm_toc_estimate(&e));
toc = shm_toc_create(PG_TEST_SHM_MQ_MAGIC, dsm_segment_address(seg),
segsize);
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index a5c0084..c8fdf6e 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -80,8 +80,10 @@ struct dsm_segment
/* Shared-memory state for a dynamic shared memory segment. */
typedef struct dsm_control_item
{
- dsm_handle handle;
+ dsm_handle handle; /* segment identifier */
uint32 refcnt; /* 2+ = active, 1 = moribund, 0 = gone */
+ Size size; /* current size */
+ char name[SHMEM_INDEX_KEYSIZE]; /* informational name */
} dsm_control_item;
/* Layout of the dynamic shared memory control segment. */
@@ -454,14 +456,16 @@ dsm_set_control_handle(dsm_handle h)
* Create a new dynamic shared memory segment.
*/
dsm_segment *
-dsm_create(Size size)
+dsm_create(const char *name, Size size)
{
dsm_segment *seg = dsm_create_descriptor();
- uint32 i;
- uint32 nitems;
+ dsm_control_item *item;
+ uint32 slot;
/* Unsafe in postmaster (and pointless in a stand-alone backend). */
Assert(IsUnderPostmaster);
+ Assert(name != NULL && strlen(name) > 0 &&
+ strlen(name) < SHMEM_INDEX_KEYSIZE - 1);
if (!dsm_init_done)
dsm_backend_startup();
@@ -479,33 +483,39 @@ dsm_create(Size size)
/* Lock the control segment so we can register the new segment. */
LWLockAcquire(DynamicSharedMemoryControlLock, LW_EXCLUSIVE);
- /* Search the control segment for an unused slot. */
- nitems = dsm_control->nitems;
- for (i = 0; i < nitems; ++i)
+ /*
+ * Search the control segment for an unused slot that's previously been
+ * used. If we don't find one initialize a new one if there's still space.
+ */
+ for (slot = 0; slot < dsm_control->nitems; ++slot)
{
- if (dsm_control->item[i].refcnt == 0)
- {
- dsm_control->item[i].handle = seg->handle;
- /* refcnt of 1 triggers destruction, so start at 2 */
- dsm_control->item[i].refcnt = 2;
- seg->control_slot = i;
- LWLockRelease(DynamicSharedMemoryControlLock);
- return seg;
- }
+ if (dsm_control->item[slot].refcnt == 0)
+ break;
}
- /* Verify that we can support an additional mapping. */
- if (nitems >= dsm_control->maxitems)
+ /* Verify that we can support the mapping. */
+ if (slot >= dsm_control->maxitems)
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_RESOURCES),
errmsg("too many dynamic shared memory segments")));
- /* Enter the handle into a new array slot. */
- dsm_control->item[nitems].handle = seg->handle;
+ item = &dsm_control->item[slot];
+ item->handle = seg->handle;
/* refcnt of 1 triggers destruction, so start at 2 */
- dsm_control->item[nitems].refcnt = 2;
- seg->control_slot = nitems;
- dsm_control->nitems++;
+ item->refcnt = 2;
+ item->size = size;
+ strncpy(item->name, name, SHMEM_INDEX_KEYSIZE);
+ item->name[SHMEM_INDEX_KEYSIZE - 1] = 0;
+
+ seg->control_slot = slot;
+
+ /*
+ * Increase number of initilized slots if we didn't reuse a previously
+ * used one.
+ */
+ if (slot >= dsm_control->nitems)
+ dsm_control->nitems++;
+
LWLockRelease(DynamicSharedMemoryControlLock);
return seg;
@@ -658,6 +668,10 @@ dsm_resize(dsm_segment *seg, Size size)
Assert(seg->control_slot != INVALID_CONTROL_SLOT);
dsm_impl_op(DSM_OP_RESIZE, seg->handle, size, &seg->impl_private,
&seg->mapped_address, &seg->mapped_size, ERROR);
+
+ /* persist the changed size */
+ dsm_control->item[seg->control_slot].size = size;
+
return seg->mapped_address;
}
diff --git a/src/include/storage/dsm.h b/src/include/storage/dsm.h
index 1d0110d..3dbe53b 100644
--- a/src/include/storage/dsm.h
+++ b/src/include/storage/dsm.h
@@ -29,7 +29,7 @@ extern void dsm_set_control_handle(dsm_handle h);
#endif
/* Functions that create, update, or remove mappings. */
-extern dsm_segment *dsm_create(Size size);
+extern dsm_segment *dsm_create(const char *name, Size size);
extern dsm_segment *dsm_attach(dsm_handle h);
extern void *dsm_resize(dsm_segment *seg, Size size);
extern void *dsm_remap(dsm_segment *seg);
--
1.8.3.251.g1462b67
0002-Add-views-to-see-shared-memory-allocations.patchtext/x-patch; charset=us-asciiDownload
>From b5d301c847062cbede97a85d8f569ade9a896396 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 4 May 2014 13:37:20 +0200
Subject: [PATCH 2/4] Add views to see shared memory allocations.
Add the pg_shmem_allocations and pg_dynamic_shmem_allocations
views. These are useful to see what memory is being used for.
---
doc/src/sgml/catalogs.sgml | 132 +++++++++++++++++++++++++++++++++++
doc/src/sgml/xfunc.sgml | 2 +-
src/backend/catalog/system_views.sql | 6 ++
src/backend/storage/ipc/dsm.c | 80 +++++++++++++++++++++
src/backend/storage/ipc/shmem.c | 113 ++++++++++++++++++++++++++++++
src/include/catalog/pg_proc.h | 4 ++
src/include/utils/builtins.h | 5 ++
src/test/regress/expected/rules.out | 9 +++
8 files changed, 350 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index b4a06e4..f106dd6 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7549,6 +7549,70 @@
</sect1>
+ <sect1 id="view-pg-dynamic-shmem-allocations">
+ <title><structname>pg_dynamic_shmem_allocations</structname></title>
+
+ <indexterm zone="view-pg-dynamic-shmem-allocations">
+ <primary>pg_dynamic_shmem_allocations</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_dynamic_shmem_allocations</structname> view shows
+ information about the currently existing dynamic shared memory segments..
+ </para>
+
+ <para>
+ Note that this view does not include shared memory allocated at server
+ startup. That is shown in
+ the <link linkend="view-pg-shmem-allocations"><structname>pg_shmem_allocations</structname></link>
+ view.
+ </para>
+
+ <table>
+ <title><structname>pg_dynamic_shmem_allocations</structname> Columns</title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry><structfield>handle</structfield></entry>
+ <entry><type>int64</type></entry>
+ <entry>The identifier used to refer to this specific segment.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>name</structfield></entry>
+ <entry><type>text</type></entry>
+ <entry>Informational name for this segment.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>size</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Size of the segment.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>refcnt</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Number of backends attached to this segment.</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ The <structname>pg_shmem_allocations</structname> view is read only.
+ </para>
+ </sect1>
+
<sect1 id="view-pg-group">
<title><structname>pg_group</structname></title>
@@ -8873,6 +8937,74 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</sect1>
+ <sect1 id="view-pg-shmem-allocations">
+ <title><structname>pg_shmem_allocations</structname></title>
+
+ <indexterm zone="view-pg-shmem-allocations">
+ <primary>pg_shmem_allocations</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_shmem_allocations</structname> view shows what the
+ server's shared memory is being used for. That includes memory allocated by
+ <productname>postgres</> itself and memory used by extensions using the
+ mechanisms detailed in <xref linkend="xfunc-shared-addin">.
+ </para>
+
+ <para>
+ Note that this view does not include memory allocated using the dynamic
+ shared memory infrastructure. That is shown in
+ the <link linkend="view-pg-dynamic-shmem-allocations"><structname>pg_dynamic_shmem_allocations</structname></link>
+ view.
+ </para>
+
+ <table>
+ <title><structname>pg_shmem_allocations</structname> Columns</title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry><structfield>name</structfield></entry>
+ <entry><type>text</type></entry>
+ <entry>The name of the shared memory allocation. NULL for unused memory
+ and <anonymous> for anonymous allocations.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>off</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>The offset at which the allocation starts. NULL for anonymous
+ allocations.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>size</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Size of the allocation</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ Anonymous allocations are allocations that have been made
+ with <literal>ShmemAlloc()</> and not <literal>ShmemInitStruct()</literal>.
+ </para>
+
+ <para>
+ The <structname>pg_shmem_allocations</structname> view is read only.
+ </para>
+
+ </sect1>
+
<sect1 id="view-pg-stats">
<title><structname>pg_stats</structname></title>
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index 941b101..adf0d9d 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3275,7 +3275,7 @@ CREATE FUNCTION make_array(anyelement) RETURNS anyarray
</para>
</sect2>
- <sect2>
+ <sect2 id="xfunc-shared-addin">
<title>Shared Memory and LWLocks</title>
<para>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 42a4c00..4041ec3 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -387,6 +387,12 @@ CREATE VIEW pg_timezone_abbrevs AS
CREATE VIEW pg_timezone_names AS
SELECT * FROM pg_timezone_names();
+CREATE VIEW pg_shmem_allocations AS
+ SELECT * FROM pg_get_shmem_allocations();
+
+CREATE VIEW pg_dynamic_shmem_allocations AS
+ SELECT * FROM pg_get_dynamic_shmem_allocations();
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index c8fdf6e..a564645 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -34,12 +34,15 @@
#endif
#include <sys/stat.h>
+#include "fmgr.h"
+#include "funcapi.h"
#include "lib/ilist.h"
#include "miscadmin.h"
#include "storage/dsm.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
+#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/memutils.h"
#include "utils/resowner_private.h"
@@ -1022,3 +1025,80 @@ dsm_control_bytes_needed(uint32 nitems)
return offsetof(dsm_control_header, item)
+sizeof(dsm_control_item) * (uint64) nitems;
}
+
+/* SQL SRF showing allocated shared memory */
+Datum
+pg_get_dynamic_shmem_allocations(PG_FUNCTION_ARGS)
+{
+#define PG_GET_SHMEM_SIZES_COLS 4
+
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ int i;
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not " \
+ "allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ LWLockAcquire(DynamicSharedMemoryControlLock, LW_SHARED);
+
+ for (i = 0; i < dsm_control->nitems; ++i)
+ {
+ dsm_control_item *item;
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* don't look at unused or about to be destroyed items */
+ if (dsm_control->item[i].refcnt < 2)
+ continue;
+
+ item = &dsm_control->item[i];
+
+ /* handle */
+ values[0] = Int64GetDatum(item->handle);
+ nulls[0] = false;
+
+ /* name */
+ values[1] = CStringGetTextDatum(item->name);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(item->size);
+ nulls[2] = false;
+
+ /* refcnt */
+ values[3] = Int64GetDatum(item->refcnt - 2);
+ nulls[3] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+ LWLockRelease(DynamicSharedMemoryControlLock);
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 2ea2216..9c47959 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -66,11 +66,14 @@
#include "postgres.h"
#include "access/transam.h"
+#include "fmgr.h"
+#include "funcapi.h"
#include "miscadmin.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
#include "storage/shmem.h"
#include "storage/spin.h"
+#include "utils/builtins.h"
/* shared memory global variables */
@@ -459,3 +462,113 @@ mul_size(Size s1, Size s2)
errmsg("requested shared memory size overflows size_t")));
return result;
}
+
+/* SQL SRF showing allocated shared memory */
+Datum
+pg_get_shmem_allocations(PG_FUNCTION_ARGS)
+{
+#define PG_GET_SHMEM_SIZES_COLS 3
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ HASH_SEQ_STATUS hstat;
+ ShmemIndexEnt *ent;
+ Size named_allocated = 0;
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not " \
+ "allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ LWLockAcquire(ShmemIndexLock, LW_SHARED);
+
+ hash_seq_init(&hstat, ShmemIndex);
+
+ /* output all allocated entries */
+ while ((ent = (ShmemIndexEnt *) hash_seq_search(&hstat)) != NULL)
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key */
+ values[0] = CStringGetTextDatum(ent->key);
+ nulls[0] = false;
+
+ /* off */
+ values[1] = Int64GetDatum((char *) ent->location - (char *) ShmemSegHdr);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(ent->size);
+ nulls[2] = false;
+ named_allocated += ent->size;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* output shared memory allocated but not counted via the shmem index */
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key */
+ values[0] = CStringGetTextDatum("<anonymous>");
+ nulls[0] = false;
+
+ /* off */
+ nulls[1] = true;
+
+ /* size */
+ values[2] = Int64GetDatum(ShmemSegHdr->freeoffset - named_allocated);
+ nulls[2] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* output as-of-yet unused shared memory */
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key, show unallocated as NULL */
+ nulls[0] = true;
+
+ /* off */
+ values[1] = Int64GetDatum(ShmemSegHdr->freeoffset);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(ShmemSegHdr->totalsize - ShmemSegHdr->freeoffset);
+ nulls[2] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ LWLockRelease(ShmemIndexLock);
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index e601ccd..16502b0 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -3899,6 +3899,10 @@ DATA(insert OID = 3035 ( pg_listening_channels PGNSP PGUID 12 1 10 0 0 f f f f
DESCR("get the channels that the current backend listens to");
DATA(insert OID = 3036 ( pg_notify PGNSP PGUID 12 1 0 0 0 f f f f f f v 2 0 2278 "25 25" _null_ _null_ _null_ _null_ pg_notify _null_ _null_ _null_ ));
DESCR("send a notification event");
+DATA(insert OID = 86 ( pg_get_shmem_allocations PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{25,20,20}" "{o,o,o}" "{key, off, size}" _null_ pg_get_shmem_allocations _null_ _null_ _null_ ));
+DESCR("show shared memory allocations");
+DATA(insert OID = 87 ( pg_get_dynamic_shmem_allocations PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{20,25,20,20}" "{o,o,o,o}" "{handle, name, size, refcnt}" _null_ pg_get_dynamic_shmem_allocations _null_ _null_ _null_ ));
+DESCR("show dynamic shared memory allocations");
/* non-persistent series generator */
DATA(insert OID = 1066 ( generate_series PGNSP PGUID 12 1 1000 0 0 f f f f t t i 3 0 23 "23 23 23" _null_ _null_ _null_ _null_ generate_series_step_int4 _null_ _null_ _null_ ));
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index bbb5d39..b1e37cf 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -1209,4 +1209,9 @@ extern Datum pg_prepared_statement(PG_FUNCTION_ARGS);
/* utils/mmgr/portalmem.c */
extern Datum pg_cursor(PG_FUNCTION_ARGS);
+/* backend/storage/ipc/shmem.c */
+extern Datum pg_get_shmem_allocations(PG_FUNCTION_ARGS);
+/* backend/storage/ipc/dsm.c */
+extern Datum pg_get_dynamic_shmem_allocations(PG_FUNCTION_ARGS);
+
#endif /* BUILTINS_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 87870cf..97aec3f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1308,6 +1308,11 @@ pg_cursors| SELECT c.name,
c.is_scrollable,
c.creation_time
FROM pg_cursor() c(name, statement, is_holdable, is_binary, is_scrollable, creation_time);
+pg_dynamic_shmem_allocations| SELECT pg_get_dynamic_shmem_allocations.handle,
+ pg_get_dynamic_shmem_allocations.name,
+ pg_get_dynamic_shmem_allocations.size,
+ pg_get_dynamic_shmem_allocations.refcnt
+ FROM pg_get_dynamic_shmem_allocations() pg_get_dynamic_shmem_allocations(handle, name, size, refcnt);
pg_group| SELECT pg_authid.rolname AS groname,
pg_authid.oid AS grosysid,
ARRAY( SELECT pg_auth_members.member
@@ -1591,6 +1596,10 @@ pg_shadow| SELECT pg_authid.rolname AS usename,
FROM (pg_authid
LEFT JOIN pg_db_role_setting s ON (((pg_authid.oid = s.setrole) AND (s.setdatabase = (0)::oid))))
WHERE pg_authid.rolcanlogin;
+pg_shmem_allocations| SELECT pg_get_shmem_allocations.key,
+ pg_get_shmem_allocations.off,
+ pg_get_shmem_allocations.size
+ FROM pg_get_shmem_allocations() pg_get_shmem_allocations(key, off, size);
pg_stat_activity| SELECT s.datid,
d.datname,
s.pid,
--
1.8.3.251.g1462b67
At 2014-05-08 15:28:22 +0200, andres@2ndquadrant.com wrote:
Hm. Not sure what you're ACKing here ;).
The idea of giving the unallocated memory a NULL key.
Ok. A new version of the patches implementing that are attached.
Including a couple of small fixups and docs. The latter aren't
extensive, but that doesn't seem to be warranted anyway.
I realise now that this email didn't actually have an attachment. Could
you please repost the latest version of this patch?
Thanks.
-- Abhijit
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Jul 14, 2014 at 6:20 AM, Abhijit Menon-Sen <ams@2ndquadrant.com> wrote:
At 2014-05-08 15:28:22 +0200, andres@2ndquadrant.com wrote:
Hm. Not sure what you're ACKing here ;).
The idea of giving the unallocated memory a NULL key.
Ok. A new version of the patches implementing that are attached.
Including a couple of small fixups and docs. The latter aren't
extensive, but that doesn't seem to be warranted anyway.I realise now that this email didn't actually have an attachment. Could
you please repost the latest version of this patch?
That's odd. I received two attachments with that email. Of course, I
was copied directly, but why would the archives have lost the
attachments?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas wrote:
On Mon, Jul 14, 2014 at 6:20 AM, Abhijit Menon-Sen <ams@2ndquadrant.com> wrote:
At 2014-05-08 15:28:22 +0200, andres@2ndquadrant.com wrote:
Hm. Not sure what you're ACKing here ;).
The idea of giving the unallocated memory a NULL key.
Ok. A new version of the patches implementing that are attached.
Including a couple of small fixups and docs. The latter aren't
extensive, but that doesn't seem to be warranted anyway.I realise now that this email didn't actually have an attachment. Could
you please repost the latest version of this patch?That's odd. I received two attachments with that email. Of course, I
was copied directly, but why would the archives have lost the
attachments?
The attachments are there on the archives, and also on my mbox -- and
unlike Robert, I was not copied. I think this is a problem on Abhijit's
end.
--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
At 2014-07-14 16:48:09 -0400, alvherre@2ndquadrant.com wrote:
The attachments are there on the archives, and also on my mbox -- and
unlike Robert, I was not copied. I think this is a problem on
Abhijit's end.
Yes, it is. I apologise for the noise.
-- Abhijit
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, May 8, 2014 at 10:28 PM, Andres Freund <andres@2ndquadrant.com> wrote:
Well, we have to live with it for now :)
I just had a look at the first patch and got some comments:
1) Instead of using an assertion here, wouldn't it be better to error
out if name is NULL, and truncate the name if it is longer than
SHMEM_INDEX_KEYSIZE - 1 (including '\0')?
scanstr in scansup.c?
Assert(IsUnderPostmaster);
+ Assert(name != NULL && strlen(name) > 0 &&
+ strlen(name) < SHMEM_INDEX_KEYSIZE - 1);
2) The addition of a field to track the size of a dsm should be
explicitly mentioned, this is useful for the 2nd patch.
3) The refactoring done in dsm_create to find an unused slot should be
done as a separate patch for clarity.
4) Using '\0' here would be more adapted:
+ item->name[SHMEM_INDEX_KEYSIZE - 1] = 0;
Regards,
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
And here are some comments about patch 2:
- Patch applies with some hunks.
- Some typos are present
s#memory segments..#memory segments. (double dots)
s#NULL#<literal>NULL</> (in the docs as this refers to a value)
- Your thoughts about providing separate patches for each view? What
this patch does is straight-forward, but pg_shmem_allocations does not
actually depend on the first patch adding size and name to the dsm
fields. So IMO it makes sense to separate each feature properly.
- off should be renamed to offset for pg_get_shmem_allocations.
- Is it really worth showing unused shared memory? I'd rather rip out
the last portion of pg_get_shmem_allocations.
- For refcnt in pg_get_dynamic_shmem_allocations, could you add a
comment mentioning that refcnt = 1 means that the item is moribund and
0 is unused, and that reference count for active dsm segments only
begins from 2? I would imagine that this is enough, instead of using
some define's defining the ID from which a dsm item is considered as
active.
Regards,
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Hi
On Thu, May 8, 2014 at 4:28 PM, Andres Freund <andres@2ndquadrant.com> wrote:
Ok. A new version of the patches implementing that are
attached. Including a couple of small fixups and docs. The latter aren't
extensive, but that doesn't seem to be warranted anyway.
Is it really actually useful to expose the segment off(set) to users?
Seems to me like unnecessary internal details leaking out.
Regards,
Marti
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-08-15 11:12:11 +0300, Marti Raudsepp wrote:
Hi
On Thu, May 8, 2014 at 4:28 PM, Andres Freund <andres@2ndquadrant.com> wrote:
Ok. A new version of the patches implementing that are
attached. Including a couple of small fixups and docs. The latter aren't
extensive, but that doesn't seem to be warranted anyway.Is it really actually useful to expose the segment off(set) to users?
Seems to me like unnecessary internal details leaking out.
Yes. This is clearly developer oriented and I'd more than once wished I
could see where some stray pointer is pointing to... That's not really
possible without something like this.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-08-14 22:16:31 -0700, Michael Paquier wrote:
And here are some comments about patch 2:
- Patch applies with some hunks.
- Some typos are present
s#memory segments..#memory segments. (double dots)
s#NULL#<literal>NULL</> (in the docs as this refers to a value)
Will fix.
- Your thoughts about providing separate patches for each view? What
this patch does is straight-forward, but pg_shmem_allocations does not
actually depend on the first patch adding size and name to the dsm
fields. So IMO it makes sense to separate each feature properly.
I don't know, seems a bit like busywork to me. Postgres doesn't really
very granular commits...
- off should be renamed to offset for pg_get_shmem_allocations.
ok.
- Is it really worth showing unused shared memory? I'd rather rip out
the last portion of pg_get_shmem_allocations.
It's actually really helpful. There's a couple situations where you
possibly can run out of that spare memory and into troubles. Which
currently aren't diagnosable. Similarly we currently can't diagnose
whether we're superflously allocate too much 'reserve' shared memory.
- For refcnt in pg_get_dynamic_shmem_allocations, could you add a
comment mentioning that refcnt = 1 means that the item is moribund and
0 is unused, and that reference count for active dsm segments only
begins from 2? I would imagine that this is enough, instead of using
some define's defining the ID from which a dsm item is considered as
active.
Ok.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Aug 15, 2014 at 4:20 AM, Andres Freund <andres@2ndquadrant.com> wrote:
On 2014-08-15 11:12:11 +0300, Marti Raudsepp wrote:
Hi
On Thu, May 8, 2014 at 4:28 PM, Andres Freund <andres@2ndquadrant.com> wrote:Ok. A new version of the patches implementing that are
attached. Including a couple of small fixups and docs. The latter aren't
extensive, but that doesn't seem to be warranted anyway.Is it really actually useful to expose the segment off(set) to users?
Seems to me like unnecessary internal details leaking out.Yes. This is clearly developer oriented and I'd more than once wished I
could see where some stray pointer is pointing to... That's not really
possible without something like this.
Unfortunately, that information also has some security implications.
I'm sure someone trying to exploit any future stack-overrun
vulnerability will be very happy to have more rather than less
information about the layout of the process address space.
I fully agree with the idea of exposing the amount of free memory in
the shared memory segment (as discussed in other emails); that's
critical information. But I think exposing address space layout
information is of much less general utility and, really, far too
risky.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-08-18 11:56:44 -0400, Robert Haas wrote:
On Fri, Aug 15, 2014 at 4:20 AM, Andres Freund <andres@2ndquadrant.com> wrote:
On 2014-08-15 11:12:11 +0300, Marti Raudsepp wrote:
Hi
On Thu, May 8, 2014 at 4:28 PM, Andres Freund <andres@2ndquadrant.com> wrote:Ok. A new version of the patches implementing that are
attached. Including a couple of small fixups and docs. The latter aren't
extensive, but that doesn't seem to be warranted anyway.Is it really actually useful to expose the segment off(set) to users?
Seems to me like unnecessary internal details leaking out.Yes. This is clearly developer oriented and I'd more than once wished I
could see where some stray pointer is pointing to... That's not really
possible without something like this.Unfortunately, that information also has some security implications.
I'm sure someone trying to exploit any future stack-overrun
vulnerability will be very happy to have more rather than less
information about the layout of the process address space.I fully agree with the idea of exposing the amount of free memory in
the shared memory segment (as discussed in other emails); that's
critical information. But I think exposing address space layout
information is of much less general utility and, really, far too
risky.
Meh. For one it's just the offsets, not the actual addresses. It's also
something you can relatively easily compute at home by looking at a
couple of settings everyone can see. For another, I'd be perfectly
content making this superuser only. And if somebody can execute queries
as superuser, address layout information really isn't needed anymore to
execute arbitrary code.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund <andres@2ndquadrant.com> writes:
On 2014-08-18 11:56:44 -0400, Robert Haas wrote:
I fully agree with the idea of exposing the amount of free memory in
the shared memory segment (as discussed in other emails); that's
critical information. But I think exposing address space layout
information is of much less general utility and, really, far too
risky.
Meh. For one it's just the offsets, not the actual addresses. It's also
something you can relatively easily compute at home by looking at a
couple of settings everyone can see. For another, I'd be perfectly
content making this superuser only. And if somebody can execute queries
as superuser, address layout information really isn't needed anymore to
execute arbitrary code.
I agree that this has to be superuser-only if it's there at all.
Should we consider putting it into an extension rather than having
it in the core system? That would offer some additional protection
for production systems, which really shouldn't have much need for
this IMO.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-08-18 12:27:12 -0400, Tom Lane wrote:
Andres Freund <andres@2ndquadrant.com> writes:
On 2014-08-18 11:56:44 -0400, Robert Haas wrote:
I fully agree with the idea of exposing the amount of free memory in
the shared memory segment (as discussed in other emails); that's
critical information. But I think exposing address space layout
information is of much less general utility and, really, far too
risky.Meh. For one it's just the offsets, not the actual addresses. It's also
something you can relatively easily compute at home by looking at a
couple of settings everyone can see. For another, I'd be perfectly
content making this superuser only. And if somebody can execute queries
as superuser, address layout information really isn't needed anymore to
execute arbitrary code.I agree that this has to be superuser-only if it's there at all.
Should we consider putting it into an extension rather than having
it in the core system? That would offer some additional protection
for production systems, which really shouldn't have much need for
this IMO.
I'd considered that somewhere upthread and decided that it'd require
exposing to much internals from shmem.c/dsm.c without a corresponding
benefit.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund <andres@2ndquadrant.com> writes:
On 2014-08-18 12:27:12 -0400, Tom Lane wrote:
Should we consider putting it into an extension rather than having
it in the core system? That would offer some additional protection
for production systems, which really shouldn't have much need for
this IMO.
I'd considered that somewhere upthread and decided that it'd require
exposing to much internals from shmem.c/dsm.c without a corresponding
benefit.
Well, we could have the implementation code in those modules but not
provide any SQL-level access to it without installing an extension.
The only extra thing visible in the .h files would be a function or two.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-08-18 12:33:44 -0400, Tom Lane wrote:
Andres Freund <andres@2ndquadrant.com> writes:
On 2014-08-18 12:27:12 -0400, Tom Lane wrote:
Should we consider putting it into an extension rather than having
it in the core system? That would offer some additional protection
for production systems, which really shouldn't have much need for
this IMO.I'd considered that somewhere upthread and decided that it'd require
exposing to much internals from shmem.c/dsm.c without a corresponding
benefit.Well, we could have the implementation code in those modules but not
provide any SQL-level access to it without installing an extension.
The only extra thing visible in the .h files would be a function or two.
That'd require wrapper functions in the extension afaics. Not that that
is prohibitive, but a bit inconvenient. At least I don't see another way
to create a sql function referring to a builtin C implementation. I
don't think PG_FUNCTION_INFO_V1() can reliably made work. We could have
the underlying function in pg_proc, but not create the view...
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund <andres@2ndquadrant.com> writes:
On 2014-08-18 12:33:44 -0400, Tom Lane wrote:
Well, we could have the implementation code in those modules but not
provide any SQL-level access to it without installing an extension.
The only extra thing visible in the .h files would be a function or two.
That'd require wrapper functions in the extension afaics.
Sure. I'd want to put as much of the logic as possible in the extension,
anyway.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Aug 18, 2014 at 12:00 PM, Andres Freund <andres@2ndquadrant.com> wrote:
Unfortunately, that information also has some security implications.
I'm sure someone trying to exploit any future stack-overrun
vulnerability will be very happy to have more rather than less
information about the layout of the process address space.Meh. For one it's just the offsets, not the actual addresses. It's also
something you can relatively easily compute at home by looking at a
couple of settings everyone can see. For another, I'd be perfectly
content making this superuser only. And if somebody can execute queries
as superuser, address layout information really isn't needed anymore to
execute arbitrary code.
I'm just not sure it should be in there at all. If we punt this off
into an extension, it won't be available in a lot of situations where
it's really needed. But although the basic functionality would have
been quite useful to me on any number of occasions, I can't recall a
single occasion upon which I would have cared about the offset at all.
I wouldn't mind having a MemoryContextStats()-type function that could
be used to print this information out by attaching to the backend with
gdb, but the utility of exposing it at the SQL level seems very
marginal to me.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-08-18 12:41:58 -0400, Robert Haas wrote:
On Mon, Aug 18, 2014 at 12:00 PM, Andres Freund <andres@2ndquadrant.com> wrote:
Unfortunately, that information also has some security implications.
I'm sure someone trying to exploit any future stack-overrun
vulnerability will be very happy to have more rather than less
information about the layout of the process address space.Meh. For one it's just the offsets, not the actual addresses. It's also
something you can relatively easily compute at home by looking at a
couple of settings everyone can see. For another, I'd be perfectly
content making this superuser only. And if somebody can execute queries
as superuser, address layout information really isn't needed anymore to
execute arbitrary code.I'm just not sure it should be in there at all.
You realize that you can pretty much recompute the offsets from the
sizes of the individual allocations anyway? Yes, you need to add some
rounding, but that's about it. We could randomize the returned elements,
but that'd be rather annoying because it'd loose information.
If we punt this off
into an extension, it won't be available in a lot of situations where
it's really needed. But although the basic functionality would have
been quite useful to me on any number of occasions, I can't recall a
single occasion upon which I would have cared about the offset at all.
I wouldn't mind having a MemoryContextStats()-type function that could
be used to print this information out by attaching to the backend with
gdb, but the utility of exposing it at the SQL level seems very
marginal to me.
I'd use for it in the past when trying to figure out what some pointer
pointed to. It's easy enough to figure out that it's in the main shared
memory segment, but after that it get's rather hard. And even if you
don't count that, it gives a sensible order to the returned rows from
the SRF.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Aug 18, 2014 at 12:46 PM, Andres Freund <andres@2ndquadrant.com> wrote:
On 2014-08-18 12:41:58 -0400, Robert Haas wrote:
On Mon, Aug 18, 2014 at 12:00 PM, Andres Freund <andres@2ndquadrant.com> wrote:
Unfortunately, that information also has some security implications.
I'm sure someone trying to exploit any future stack-overrun
vulnerability will be very happy to have more rather than less
information about the layout of the process address space.Meh. For one it's just the offsets, not the actual addresses. It's also
something you can relatively easily compute at home by looking at a
couple of settings everyone can see. For another, I'd be perfectly
content making this superuser only. And if somebody can execute queries
as superuser, address layout information really isn't needed anymore to
execute arbitrary code.I'm just not sure it should be in there at all.
You realize that you can pretty much recompute the offsets from the
sizes of the individual allocations anyway?
Sure, if you know the segment base. Do you?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-08-18 12:50:27 -0400, Robert Haas wrote:
On Mon, Aug 18, 2014 at 12:46 PM, Andres Freund <andres@2ndquadrant.com> wrote:
You realize that you can pretty much recompute the offsets from the
sizes of the individual allocations anyway?Sure, if you know the segment base. Do you?
Err? The offset doesn't give you the base either?
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
I wouldn't mind having a MemoryContextStats()-type function that could
be used to print this information out by attaching to the backend with
gdb, but the utility of exposing it at the SQL level seems very
marginal to me.
+1 for doing it like that.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Aug 18, 2014 at 12:51 PM, Andres Freund <andres@2ndquadrant.com> wrote:
On 2014-08-18 12:50:27 -0400, Robert Haas wrote:
On Mon, Aug 18, 2014 at 12:46 PM, Andres Freund <andres@2ndquadrant.com> wrote:
You realize that you can pretty much recompute the offsets from the
sizes of the individual allocations anyway?Sure, if you know the segment base. Do you?
Err? The offset doesn't give you the base either?
Oh!
I thought you were printing actual pointer addresses. If you're just
printing offsets relative to wherever the segment happens to be
mapped, I don't care about that.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
I thought you were printing actual pointer addresses. If you're just
printing offsets relative to wherever the segment happens to be
mapped, I don't care about that.
Well, that just means that it's not an *obvious* security risk.
I still like the idea of providing something comparable to
MemoryContextStats, rather than creating a SQL interface. The problem
with a SQL interface is you can't interrogate it unless (1) you are not
already inside a query and (2) the client is interactive and under your
control. Something you can call easily from gdb is likely to be much
more useful in practice.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-08-18 13:27:07 -0400, Tom Lane wrote:
I still like the idea of providing something comparable to
MemoryContextStats, rather than creating a SQL interface. The problem
with a SQL interface is you can't interrogate it unless (1) you are not
already inside a query and (2) the client is interactive and under your
control. Something you can call easily from gdb is likely to be much
more useful in practice.
That might be true in some cases, but in many cases interfaces that can
only be used via gdb *SUCK*. A good reason to use the interface proposed
here is to investigate which extensions are allocating how much shared
memory. A pretty normal question to ask as a sysadmin/DBA. And DBA type
of stuff should never have to involve gdb.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Aug 18, 2014 at 1:27 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
I thought you were printing actual pointer addresses. If you're just
printing offsets relative to wherever the segment happens to be
mapped, I don't care about that.Well, that just means that it's not an *obvious* security risk.
I still like the idea of providing something comparable to
MemoryContextStats, rather than creating a SQL interface. The problem
with a SQL interface is you can't interrogate it unless (1) you are not
already inside a query and (2) the client is interactive and under your
control. Something you can call easily from gdb is likely to be much
more useful in practice.
Since the shared memory segment isn't changing at runtime, I don't see
this as being a big problem. It could possibly be an issue for
dynamic shared memory segments, though.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Aug 18, 2014 at 1:12 PM, Robert Haas <robertmhaas@gmail.com> wrote:
On Mon, Aug 18, 2014 at 1:27 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
I thought you were printing actual pointer addresses. If you're just
printing offsets relative to wherever the segment happens to be
mapped, I don't care about that.Well, that just means that it's not an *obvious* security risk.
I still like the idea of providing something comparable to
MemoryContextStats, rather than creating a SQL interface. The problem
with a SQL interface is you can't interrogate it unless (1) you are not
already inside a query and (2) the client is interactive and under your
control. Something you can call easily from gdb is likely to be much
more useful in practice.Since the shared memory segment isn't changing at runtime, I don't see
this as being a big problem. It could possibly be an issue for
dynamic shared memory segments, though.
Patch has been reviewed some time ago, extra ideas as well as
potential security risks discussed as well but no new version has been
sent, hence marking it as returned with feedback.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-09-19 23:07:07 -0500, Michael Paquier wrote:
On Mon, Aug 18, 2014 at 1:12 PM, Robert Haas <robertmhaas@gmail.com> wrote:
On Mon, Aug 18, 2014 at 1:27 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
I thought you were printing actual pointer addresses. If you're just
printing offsets relative to wherever the segment happens to be
mapped, I don't care about that.Well, that just means that it's not an *obvious* security risk.
I still like the idea of providing something comparable to
MemoryContextStats, rather than creating a SQL interface. The problem
with a SQL interface is you can't interrogate it unless (1) you are not
already inside a query and (2) the client is interactive and under your
control. Something you can call easily from gdb is likely to be much
more useful in practice.Since the shared memory segment isn't changing at runtime, I don't see
this as being a big problem. It could possibly be an issue for
dynamic shared memory segments, though.Patch has been reviewed some time ago, extra ideas as well as
potential security risks discussed as well but no new version has been
sent, hence marking it as returned with feedback.
Here's a rebased version. I remember why I didn't call the column
"offset" (as Michael complained about upthread), it's a keyword...
Regards,
Andres
Attachments:
0001-Associate-names-to-created-dynamic-shared-memory-seg.patchtext/x-patch; charset=us-asciiDownload
From 719f7e2493832564c58c3ab319344f31abef1653 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 6 May 2014 19:42:36 +0200
Subject: [PATCH 1/2] Associate names to created dynamic shared memory
segments.
This unfortunately implies an API breakage.
---
src/backend/access/transam/parallel.c | 3 +-
src/backend/storage/ipc/dsm.c | 61 ++++++++++++++++++++++-------------
src/include/storage/dsm.h | 2 +-
src/test/modules/test_shm_mq/setup.c | 2 +-
4 files changed, 42 insertions(+), 26 deletions(-)
diff --git a/src/backend/access/transam/parallel.c b/src/backend/access/transam/parallel.c
index 0bba9a7..c1473c1 100644
--- a/src/backend/access/transam/parallel.c
+++ b/src/backend/access/transam/parallel.c
@@ -268,7 +268,8 @@ InitializeParallelDSM(ParallelContext *pcxt)
*/
segsize = shm_toc_estimate(&pcxt->estimator);
if (pcxt->nworkers != 0)
- pcxt->seg = dsm_create(segsize, DSM_CREATE_NULL_IF_MAXSEGMENTS);
+ pcxt->seg = dsm_create("parallel worker", segsize,
+ DSM_CREATE_NULL_IF_MAXSEGMENTS);
if (pcxt->seg != NULL)
pcxt->toc = shm_toc_create(PARALLEL_MAGIC,
dsm_segment_address(pcxt->seg),
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index 573c54d..7166328 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -80,8 +80,10 @@ struct dsm_segment
/* Shared-memory state for a dynamic shared memory segment. */
typedef struct dsm_control_item
{
- dsm_handle handle;
+ dsm_handle handle; /* segment identifier */
uint32 refcnt; /* 2+ = active, 1 = moribund, 0 = gone */
+ Size size; /* current size */
+ char name[SHMEM_INDEX_KEYSIZE]; /* informational name */
} dsm_control_item;
/* Layout of the dynamic shared memory control segment. */
@@ -454,15 +456,18 @@ dsm_set_control_handle(dsm_handle h)
* Create a new dynamic shared memory segment.
*/
dsm_segment *
-dsm_create(Size size, int flags)
+dsm_create(const char *name, Size size, int flags)
{
dsm_segment *seg;
- uint32 i;
- uint32 nitems;
+ dsm_control_item *item;
+ uint32 slot;
/* Unsafe in postmaster (and pointless in a stand-alone backend). */
Assert(IsUnderPostmaster);
+ Assert(name != NULL && strlen(name) > 0 &&
+ strlen(name) < SHMEM_INDEX_KEYSIZE - 1);
+
if (!dsm_init_done)
dsm_backend_startup();
@@ -482,23 +487,18 @@ dsm_create(Size size, int flags)
/* Lock the control segment so we can register the new segment. */
LWLockAcquire(DynamicSharedMemoryControlLock, LW_EXCLUSIVE);
- /* Search the control segment for an unused slot. */
- nitems = dsm_control->nitems;
- for (i = 0; i < nitems; ++i)
+ /*
+ * Search the control segment for an unused slot that's previously been
+ * used. If we don't find one initialize a new one if there's still space.
+ */
+ for (slot = 0; slot < dsm_control->nitems; ++slot)
{
- if (dsm_control->item[i].refcnt == 0)
- {
- dsm_control->item[i].handle = seg->handle;
- /* refcnt of 1 triggers destruction, so start at 2 */
- dsm_control->item[i].refcnt = 2;
- seg->control_slot = i;
- LWLockRelease(DynamicSharedMemoryControlLock);
- return seg;
- }
+ if (dsm_control->item[slot].refcnt == 0)
+ break;
}
- /* Verify that we can support an additional mapping. */
- if (nitems >= dsm_control->maxitems)
+ /* Verify that we can support the mapping. */
+ if (slot >= dsm_control->maxitems)
{
if ((flags & DSM_CREATE_NULL_IF_MAXSEGMENTS) != 0)
{
@@ -516,12 +516,23 @@ dsm_create(Size size, int flags)
errmsg("too many dynamic shared memory segments")));
}
- /* Enter the handle into a new array slot. */
- dsm_control->item[nitems].handle = seg->handle;
+ item = &dsm_control->item[slot];
+ item->handle = seg->handle;
/* refcnt of 1 triggers destruction, so start at 2 */
- dsm_control->item[nitems].refcnt = 2;
- seg->control_slot = nitems;
- dsm_control->nitems++;
+ item->refcnt = 2;
+ item->size = size;
+ strncpy(item->name, name, SHMEM_INDEX_KEYSIZE);
+ item->name[SHMEM_INDEX_KEYSIZE - 1] = 0;
+
+ seg->control_slot = slot;
+
+ /*
+ * Increase number of initilized slots if we didn't reuse a previously
+ * used one.
+ */
+ if (slot >= dsm_control->nitems)
+ dsm_control->nitems++;
+
LWLockRelease(DynamicSharedMemoryControlLock);
return seg;
@@ -674,6 +685,10 @@ dsm_resize(dsm_segment *seg, Size size)
Assert(seg->control_slot != INVALID_CONTROL_SLOT);
dsm_impl_op(DSM_OP_RESIZE, seg->handle, size, &seg->impl_private,
&seg->mapped_address, &seg->mapped_size, ERROR);
+
+ /* persist the changed size */
+ dsm_control->item[seg->control_slot].size = size;
+
return seg->mapped_address;
}
diff --git a/src/include/storage/dsm.h b/src/include/storage/dsm.h
index 86ede7a..278a1ac 100644
--- a/src/include/storage/dsm.h
+++ b/src/include/storage/dsm.h
@@ -31,7 +31,7 @@ extern void dsm_set_control_handle(dsm_handle h);
#endif
/* Functions that create, update, or remove mappings. */
-extern dsm_segment *dsm_create(Size size, int flags);
+extern dsm_segment *dsm_create(const char *name, Size size, int flags);
extern dsm_segment *dsm_attach(dsm_handle h);
extern void *dsm_resize(dsm_segment *seg, Size size);
extern void *dsm_remap(dsm_segment *seg);
diff --git a/src/test/modules/test_shm_mq/setup.c b/src/test/modules/test_shm_mq/setup.c
index 5bd2820..13ad00c 100644
--- a/src/test/modules/test_shm_mq/setup.c
+++ b/src/test/modules/test_shm_mq/setup.c
@@ -125,7 +125,7 @@ setup_dynamic_shared_memory(int64 queue_size, int nworkers,
segsize = shm_toc_estimate(&e);
/* Create the shared memory segment and establish a table of contents. */
- seg = dsm_create(shm_toc_estimate(&e), 0);
+ seg = dsm_create("test_shm_mq", shm_toc_estimate(&e), 0);
toc = shm_toc_create(PG_TEST_SHM_MQ_MAGIC, dsm_segment_address(seg),
segsize);
--
2.7.0.229.g701fa7f
0002-Add-views-to-see-shared-memory-allocations.patchtext/x-patch; charset=us-asciiDownload
From e5ff0836922f988a76b73aa95e70debfb16c4d82 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 4 May 2014 13:37:20 +0200
Subject: [PATCH 2/2] Add views to see shared memory allocations.
Add the pg_shmem_allocations and pg_dynamic_shmem_allocations
views. These are useful to see what memory is being used for.
---
doc/src/sgml/catalogs.sgml | 135 +++++++++++++++++++++++++++++++++++
doc/src/sgml/xfunc.sgml | 2 +-
src/backend/catalog/system_views.sql | 6 ++
src/backend/storage/ipc/dsm.c | 80 +++++++++++++++++++++
src/backend/storage/ipc/shmem.c | 113 +++++++++++++++++++++++++++++
src/include/catalog/pg_proc.h | 4 ++
src/include/utils/builtins.h | 5 ++
src/test/regress/expected/rules.out | 9 +++
8 files changed, 353 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index eac6671..e1ca97d 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7835,6 +7835,73 @@
</sect1>
+ <sect1 id="view-pg-dynamic-shmem-allocations">
+ <title><structname>pg_dynamic_shmem_allocations</structname></title>
+
+ <indexterm zone="view-pg-dynamic-shmem-allocations">
+ <primary>pg_dynamic_shmem_allocations</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_dynamic_shmem_allocations</structname> view shows
+ information about the currently existing dynamic shared memory segments.
+ </para>
+
+ <para>
+ Note that this view does not include shared memory allocated at server
+ startup. That is shown in
+ the <link linkend="view-pg-shmem-allocations"><structname>pg_shmem_allocations</structname></link>
+ view.
+ </para>
+
+ <table>
+ <title><structname>pg_dynamic_shmem_allocations</structname> Columns</title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry><structfield>handle</structfield></entry>
+ <entry><type>int64</type></entry>
+ <entry>The identifier used to refer to this specific segment.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>name</structfield></entry>
+ <entry><type>text</type></entry>
+ <entry>Informational name for this segment.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>size</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Size of the segment.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>refcnt</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>
+ Number of backends attached to this segment (1 signals a moribund
+ entry, 2 signals one user, ...).
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ The <structname>pg_shmem_allocations</structname> view is read only.
+ </para>
+ </sect1>
+
<sect1 id="view-pg-group">
<title><structname>pg_group</structname></title>
@@ -9467,6 +9534,74 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</sect1>
+ <sect1 id="view-pg-shmem-allocations">
+ <title><structname>pg_shmem_allocations</structname></title>
+
+ <indexterm zone="view-pg-shmem-allocations">
+ <primary>pg_shmem_allocations</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_shmem_allocations</structname> view shows what the
+ server's shared memory is being used for. That includes memory allocated by
+ <productname>postgres</> itself and memory used by extensions using the
+ mechanisms detailed in <xref linkend="xfunc-shared-addin">.
+ </para>
+
+ <para>
+ Note that this view does not include memory allocated using the dynamic
+ shared memory infrastructure. That is shown in
+ the <link linkend="view-pg-dynamic-shmem-allocations"><structname>pg_dynamic_shmem_allocations</structname></link>
+ view.
+ </para>
+
+ <table>
+ <title><structname>pg_shmem_allocations</structname> Columns</title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry><structfield>name</structfield></entry>
+ <entry><type>text</type></entry>
+ <entry>The name of the shared memory allocation. NULL for unused memory
+ and <anonymous> for anonymous allocations.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>off</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>The offset at which the allocation starts. NULL for anonymous
+ allocations.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>size</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Size of the allocation</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ Anonymous allocations are allocations that have been made
+ with <literal>ShmemAlloc()</> and not <literal>ShmemInitStruct()</literal>.
+ </para>
+
+ <para>
+ The <structname>pg_shmem_allocations</structname> view is read only.
+ </para>
+
+ </sect1>
+
<sect1 id="view-pg-stats">
<title><structname>pg_stats</structname></title>
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index d8d2e9e..b59512f 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3318,7 +3318,7 @@ CREATE FUNCTION make_array(anyelement) RETURNS anyarray
</para>
</sect2>
- <sect2>
+ <sect2 id="xfunc-shared-addin">
<title>Shared Memory and LWLocks</title>
<para>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 272c02f..8f55801 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -439,6 +439,12 @@ CREATE VIEW pg_config AS
REVOKE ALL on pg_config FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_config() FROM PUBLIC;
+CREATE VIEW pg_shmem_allocations AS
+ SELECT * FROM pg_get_shmem_allocations();
+
+CREATE VIEW pg_dynamic_shmem_allocations AS
+ SELECT * FROM pg_get_dynamic_shmem_allocations();
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index 7166328..ce8d1e9 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -34,12 +34,15 @@
#endif
#include <sys/stat.h>
+#include "fmgr.h"
+#include "funcapi.h"
#include "lib/ilist.h"
#include "miscadmin.h"
#include "storage/dsm.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
+#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/memutils.h"
#include "utils/resowner_private.h"
@@ -1057,3 +1060,80 @@ dsm_control_bytes_needed(uint32 nitems)
return offsetof(dsm_control_header, item)
+sizeof(dsm_control_item) * (uint64) nitems;
}
+
+/* SQL SRF showing allocated shared memory */
+Datum
+pg_get_dynamic_shmem_allocations(PG_FUNCTION_ARGS)
+{
+#define PG_GET_SHMEM_SIZES_COLS 4
+
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ int i;
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not " \
+ "allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ LWLockAcquire(DynamicSharedMemoryControlLock, LW_SHARED);
+
+ for (i = 0; i < dsm_control->nitems; ++i)
+ {
+ dsm_control_item *item;
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* don't look at unused or about to be destroyed items */
+ if (dsm_control->item[i].refcnt < 2)
+ continue;
+
+ item = &dsm_control->item[i];
+
+ /* handle */
+ values[0] = Int64GetDatum(item->handle);
+ nulls[0] = false;
+
+ /* name */
+ values[1] = CStringGetTextDatum(item->name);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(item->size);
+ nulls[2] = false;
+
+ /* refcnt */
+ values[3] = Int64GetDatum(item->refcnt - 2);
+ nulls[3] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+ LWLockRelease(DynamicSharedMemoryControlLock);
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 1efe020..f21bdfc 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -66,11 +66,14 @@
#include "postgres.h"
#include "access/transam.h"
+#include "fmgr.h"
+#include "funcapi.h"
#include "miscadmin.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
#include "storage/shmem.h"
#include "storage/spin.h"
+#include "utils/builtins.h"
/* shared memory global variables */
@@ -471,3 +474,113 @@ mul_size(Size s1, Size s2)
errmsg("requested shared memory size overflows size_t")));
return result;
}
+
+/* SQL SRF showing allocated shared memory */
+Datum
+pg_get_shmem_allocations(PG_FUNCTION_ARGS)
+{
+#define PG_GET_SHMEM_SIZES_COLS 3
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ HASH_SEQ_STATUS hstat;
+ ShmemIndexEnt *ent;
+ Size named_allocated = 0;
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not " \
+ "allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ LWLockAcquire(ShmemIndexLock, LW_SHARED);
+
+ hash_seq_init(&hstat, ShmemIndex);
+
+ /* output all allocated entries */
+ while ((ent = (ShmemIndexEnt *) hash_seq_search(&hstat)) != NULL)
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key */
+ values[0] = CStringGetTextDatum(ent->key);
+ nulls[0] = false;
+
+ /* off */
+ values[1] = Int64GetDatum((char *) ent->location - (char *) ShmemSegHdr);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(ent->size);
+ nulls[2] = false;
+ named_allocated += ent->size;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* output shared memory allocated but not counted via the shmem index */
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key */
+ values[0] = CStringGetTextDatum("<anonymous>");
+ nulls[0] = false;
+
+ /* off */
+ nulls[1] = true;
+
+ /* size */
+ values[2] = Int64GetDatum(ShmemSegHdr->freeoffset - named_allocated);
+ nulls[2] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* output as-of-yet unused shared memory */
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key, show unallocated as NULL */
+ nulls[0] = true;
+
+ /* off */
+ values[1] = Int64GetDatum(ShmemSegHdr->freeoffset);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(ShmemSegHdr->totalsize - ShmemSegHdr->freeoffset);
+ nulls[2] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ LWLockRelease(ShmemIndexLock);
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 021ba50..465342b 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -4054,6 +4054,10 @@ DATA(insert OID = 3036 ( pg_notify PGNSP PGUID 12 1 0 0 0 f f f f f f v r 2
DESCR("send a notification event");
DATA(insert OID = 3296 ( pg_notification_queue_usage PGNSP PGUID 12 1 0 0 0 f f f f t f v s 0 0 701 "" _null_ _null_ _null_ _null_ _null_ pg_notification_queue_usage _null_ _null_ _null_ ));
DESCR("get the fraction of the asynchronous notification queue currently in use");
+DATA(insert OID = 441 ( pg_get_shmem_allocations PGNSP PGUID 12 1 10 0 0 f f f f f t s s 0 0 2249 "" "{25,20,20}" "{o,o,o}" "{key, off, size}" _null_ _null_ pg_get_shmem_allocations _null_ _null_ _null_ ));
+DESCR("show shared memory allocations");
+DATA(insert OID = 448 ( pg_get_dynamic_shmem_allocations PGNSP PGUID 12 1 10 0 0 f f f f f t s s 0 0 2249 "" "{20,25,20,20}" "{o,o,o,o}" "{handle, name, size, refcnt}" _null_ _null_ pg_get_dynamic_shmem_allocations _null_ _null_ _null_ ));
+DESCR("show dynamic shared memory allocations");
/* non-persistent series generator */
DATA(insert OID = 1066 ( generate_series PGNSP PGUID 12 1 1000 0 0 f f f f t t i s 3 0 23 "23 23 23" _null_ _null_ _null_ _null_ _null_ generate_series_step_int4 _null_ _null_ _null_ ));
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index 01976a1..30eedd5 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -1322,4 +1322,9 @@ extern Datum pg_prepared_statement(PG_FUNCTION_ARGS);
/* utils/mmgr/portalmem.c */
extern Datum pg_cursor(PG_FUNCTION_ARGS);
+/* backend/storage/ipc/shmem.c */
+extern Datum pg_get_shmem_allocations(PG_FUNCTION_ARGS);
+/* backend/storage/ipc/dsm.c */
+extern Datum pg_get_dynamic_shmem_allocations(PG_FUNCTION_ARGS);
+
#endif /* BUILTINS_H */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 5521a16..ae985a7 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1323,6 +1323,11 @@ pg_cursors| SELECT c.name,
c.is_scrollable,
c.creation_time
FROM pg_cursor() c(name, statement, is_holdable, is_binary, is_scrollable, creation_time);
+pg_dynamic_shmem_allocations| SELECT pg_get_dynamic_shmem_allocations.handle,
+ pg_get_dynamic_shmem_allocations.name,
+ pg_get_dynamic_shmem_allocations.size,
+ pg_get_dynamic_shmem_allocations.refcnt
+ FROM pg_get_dynamic_shmem_allocations() pg_get_dynamic_shmem_allocations(handle, name, size, refcnt);
pg_file_settings| SELECT a.sourcefile,
a.sourceline,
a.seqno,
@@ -1645,6 +1650,10 @@ pg_shadow| SELECT pg_authid.rolname AS usename,
FROM (pg_authid
LEFT JOIN pg_db_role_setting s ON (((pg_authid.oid = s.setrole) AND (s.setdatabase = (0)::oid))))
WHERE pg_authid.rolcanlogin;
+pg_shmem_allocations| SELECT pg_get_shmem_allocations.key,
+ pg_get_shmem_allocations.off,
+ pg_get_shmem_allocations.size
+ FROM pg_get_shmem_allocations() pg_get_shmem_allocations(key, off, size);
pg_stat_activity| SELECT s.datid,
d.datname,
s.pid,
--
2.7.0.229.g701fa7f
On Fri, May 6, 2016 at 8:01 AM, Andres Freund <andres@anarazel.de> wrote:
Here's a rebased version. I remember why I didn't call the column
"offset" (as Michael complained about upthread), it's a keyword...
Oh, an old patch resurrected from the dead... Reading again the patch
+ * Increase number of initilized slots if we didn't reuse a previously
+ * used one.
s/initilized/initialized
+ Number of backends attached to this segment (1 signals a moribund
+ entry, 2 signals one user, ...).
moribund? Or target for removal.
REVOKE ALL .. FROM PUBLIC;
REVOKE EXECUTE .. ON FUNCTION FROM PUBLIC;
Revoking he execution of those views and their underlying functions
would be a good thing I think, this can give hints on the system
activity, particularly for DSM segments.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, May 5, 2016 at 7:01 PM Andres Freund <andres@anarazel.de> wrote:
Here's a rebased version. I remember why I didn't call the column
"offset" (as Michael complained about upthread), it's a keyword...
This never got applied, and that annoyed me again today, so here's a
new version that I've whacked around somewhat and propose to commit. I
ripped out the stuff pertaining to dynamic shared memory segments,
both because I think it might need some more thought and discussion,
and because the part the pertains to the main shared memory segment is
the part I keep wishing we had. We can add that other part later if
we're all agreed on it, but let's go ahead and add this part now.
Other things I changed:
- Doc edits.
- Added REVOKE statements as proposed by Michael (and I agree).
- Can't patch pg_proc.h any more, gotta patch pg_proc.dat.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
0001-Add-a-pg_shmem_allocations-view.patchapplication/octet-stream; name=0001-Add-a-pg_shmem_allocations-view.patchDownload
From bcd5bdfee416c175f58612d587f7b6fbe7435fd8 Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Fri, 15 Nov 2019 14:40:23 -0500
Subject: [PATCH] Add a pg_shmem_allocations view.
It tells you how the main shared memory segment is being used.
Andres Freund, with modifications by me, reviewed (several
years ago) by Michael Paquier, Marti Raudsepp, and Tom Lane.
Discussion: http://postgr.es/m/20140504114417.GM12715@awork2.anarazel.de
---
doc/src/sgml/catalogs.sgml | 64 +++++++++++++++
doc/src/sgml/xfunc.sgml | 2 +-
src/backend/catalog/system_views.sql | 6 ++
src/backend/storage/ipc/shmem.c | 113 +++++++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 9 +++
src/test/regress/expected/rules.out | 4 +
6 files changed, 197 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 55694c4368..070ea0e93a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -10748,6 +10748,70 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</sect1>
+ <sect1 id="view-pg-shmem-allocations">
+ <title><structname>pg_shmem_allocations</structname></title>
+
+ <indexterm zone="view-pg-shmem-allocations">
+ <primary>pg_shmem_allocations</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_shmem_allocations</structname> view shows allocations
+ made from the server's main shared memory segment. This includes both
+ memory allocated by <productname>postgres</productname> itself and memory
+ allocated by extensions using the mechanisms detailed in
+ <xref linkend="xfunc-shared-addin" />.
+ </para>
+
+ <para>
+ Note that this view does not include memory allocated using the dynamic
+ shared memory infrastructure.
+ </para>
+
+ <table>
+ <title><structname>pg_shmem_allocations</structname> Columns</title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry><structfield>name</structfield></entry>
+ <entry><type>text</type></entry>
+ <entry>The name of the shared memory allocation. NULL for unused memory
+ and <anonymous> for anonymous allocations.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>off</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>The offset at which the allocation starts. NULL for anonymous
+ allocations.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>size</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Size of the allocation</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ Anonymous allocations are allocations that have been made
+ with <literal>ShmemAlloc()</literal> directly, rather than via
+ <literal>ShmemInitStruct()</literal> or
+ <literal>ShmemInitHash()</literal> or.
+ </para>
+ </sect1>
+
<sect1 id="view-pg-stats">
<title><structname>pg_stats</structname></title>
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index d9afd3be4d..bb8468553c 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3241,7 +3241,7 @@ CREATE FUNCTION make_array(anyelement) RETURNS anyarray
</para>
</sect2>
- <sect2>
+ <sect2 id="xfunc-shared-addin">
<title>Shared Memory and LWLocks</title>
<para>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 4456fefb38..cfa3b752c7 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -547,6 +547,12 @@ CREATE VIEW pg_config AS
REVOKE ALL on pg_config FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_config() FROM PUBLIC;
+CREATE VIEW pg_shmem_allocations AS
+ SELECT * FROM pg_get_shmem_allocations();
+
+REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index a1567d3559..57d6bf1c7b 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -66,11 +66,14 @@
#include "postgres.h"
#include "access/transam.h"
+#include "fmgr.h"
+#include "funcapi.h"
#include "miscadmin.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
#include "storage/shmem.h"
#include "storage/spin.h"
+#include "utils/builtins.h"
/* shared memory global variables */
@@ -503,3 +506,113 @@ mul_size(Size s1, Size s2)
errmsg("requested shared memory size overflows size_t")));
return result;
}
+
+/* SQL SRF showing allocated shared memory */
+Datum
+pg_get_shmem_allocations(PG_FUNCTION_ARGS)
+{
+#define PG_GET_SHMEM_SIZES_COLS 3
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ HASH_SEQ_STATUS hstat;
+ ShmemIndexEnt *ent;
+ Size named_allocated = 0;
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not " \
+ "allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ LWLockAcquire(ShmemIndexLock, LW_SHARED);
+
+ hash_seq_init(&hstat, ShmemIndex);
+
+ /* output all allocated entries */
+ while ((ent = (ShmemIndexEnt *) hash_seq_search(&hstat)) != NULL)
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key */
+ values[0] = CStringGetTextDatum(ent->key);
+ nulls[0] = false;
+
+ /* off */
+ values[1] = Int64GetDatum((char *) ent->location - (char *) ShmemSegHdr);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(ent->size);
+ nulls[2] = false;
+ named_allocated += ent->size;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* output shared memory allocated but not counted via the shmem index */
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key */
+ values[0] = CStringGetTextDatum("<anonymous>");
+ nulls[0] = false;
+
+ /* off */
+ nulls[1] = true;
+
+ /* size */
+ values[2] = Int64GetDatum(ShmemSegHdr->freeoffset - named_allocated);
+ nulls[2] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* output as-of-yet unused shared memory */
+ {
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* key, show unallocated as NULL */
+ nulls[0] = true;
+
+ /* off */
+ values[1] = Int64GetDatum(ShmemSegHdr->freeoffset);
+ nulls[1] = false;
+
+ /* size */
+ values[2] = Int64GetDatum(ShmemSegHdr->totalsize - ShmemSegHdr->freeoffset);
+ nulls[2] = false;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ LWLockRelease(ShmemIndexLock);
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58ea5b982b..29f093b87a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7656,6 +7656,15 @@
prorettype => 'float8', proargtypes => '',
prosrc => 'pg_notification_queue_usage' },
+# shared memory usage
+{ oid => '8613',
+ descr => 'allocations from the main shared memory segment',
+ proname => 'pg_get_shmem_allocations', 'prorows' => 10, 'proretset' => 't',
+ provolatile => 's', 'prorettype' => 'record', 'proargtypes' => '',
+ proallargtypes => '{text,int8,int8}', proargmodes => '{o,o,o}',
+ proargnames => '{name,off,size}',
+ prosrc => 'pg_get_shmem_allocations' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 14e7214346..4c1bb88c8d 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1721,6 +1721,10 @@ pg_shadow| SELECT pg_authid.rolname AS usename,
FROM (pg_authid
LEFT JOIN pg_db_role_setting s ON (((pg_authid.oid = s.setrole) AND (s.setdatabase = (0)::oid))))
WHERE pg_authid.rolcanlogin;
+pg_shmem_allocations| SELECT pg_get_shmem_allocations.name,
+ pg_get_shmem_allocations.off,
+ pg_get_shmem_allocations.size
+ FROM pg_get_shmem_allocations() pg_get_shmem_allocations(name, off, size);
pg_stat_activity| SELECT s.datid,
d.datname,
s.pid,
--
2.17.2 (Apple Git-113)
Robert Haas <robertmhaas@gmail.com> writes:
This never got applied, and that annoyed me again today, so here's a
new version that I've whacked around somewhat and propose to commit.
...
Other things I changed:
- Doc edits.
- Added REVOKE statements as proposed by Michael (and I agree).
- Can't patch pg_proc.h any more, gotta patch pg_proc.dat.
If we're disallowing public access to the view (which I agree on),
doesn't that need to be mentioned in the docs? I think there's
standard boilerplate we use for other such views.
Also, there's an introductory section in catalogs.sgml that
should have an entry for this view.
Also, likely the function should be volatile not stable. I'm
not sure that it makes any difference in the view's usage,
but in principle the answers could change intraquery.
I didn't really read the patch in any detail, but those things
hopped out at me.
regards, tom lane
Hi,
On 2019-11-15 14:43:09 -0500, Robert Haas wrote:
On Thu, May 5, 2016 at 7:01 PM Andres Freund <andres@anarazel.de> wrote:
Here's a rebased version. I remember why I didn't call the column
"offset" (as Michael complained about upthread), it's a keyword...This never got applied, and that annoyed me again today, so here's a
new version that I've whacked around somewhat and propose to commit. I
ripped out the stuff pertaining to dynamic shared memory segments,
both because I think it might need some more thought and discussion,
and because the part the pertains to the main shared memory segment is
the part I keep wishing we had. We can add that other part later if
we're all agreed on it, but let's go ahead and add this part now.
Oh, nice! Makes sense to me to split off the dsm part.
Mildly related: I really wish we had a postmaster option that'd parse
the config file, and exit after computing the amount of shared memory
that'll be required. Would be really useful to reliably compute the
number of huge pages needed. Right now one basically needs to start pg
and parse error messages, to do so.
+ <sect1 id="view-pg-shmem-allocations"> + <title><structname>pg_shmem_allocations</structname></title> + + <indexterm zone="view-pg-shmem-allocations"> + <primary>pg_shmem_allocations</primary> + </indexterm> + + <para> + The <structname>pg_shmem_allocations</structname> view shows allocations + made from the server's main shared memory segment. This includes both + memory allocated by <productname>postgres</productname> itself and memory + allocated by extensions using the mechanisms detailed in + <xref linkend="xfunc-shared-addin" />. + </para> + + <para> + Note that this view does not include memory allocated using the dynamic + shared memory infrastructure. + </para>
Perhaps add the equivalent of
<para>
By default, the <structname>pg_config</structname> view can be read
only by superusers.
</para>
now that you've added the revoke (with which I agree).
+ <tbody> + <row> + <entry><structfield>name</structfield></entry> + <entry><type>text</type></entry> + <entry>The name of the shared memory allocation. NULL for unused memory + and <anonymous> for anonymous allocations.</entry> + </row>
+ <row> + <entry><structfield>off</structfield></entry> + <entry><type>bigint</type></entry> + <entry>The offset at which the allocation starts. NULL for anonymous + allocations.</entry> + </row> + + <row> + <entry><structfield>size</structfield></entry> + <entry><type>bigint</type></entry> + <entry>Size of the allocation</entry> + </row> + </tbody> + </tgroup> + </table>
Should we note that there can be only one entry for unused and anonymous
memory? And that off will be NULL for anonymous memory?
+ /* output all allocated entries */ + while ((ent = (ShmemIndexEnt *) hash_seq_search(&hstat)) != NULL) + { + Datum values[PG_GET_SHMEM_SIZES_COLS]; + bool nulls[PG_GET_SHMEM_SIZES_COLS];
It's very mildly odd to have three of these values/nulls arrays...
+# shared memory usage +{ oid => '8613', + descr => 'allocations from the main shared memory segment', + proname => 'pg_get_shmem_allocations', 'prorows' => 10, 'proretset' => 't', + provolatile => 's', 'prorettype' => 'record', 'proargtypes' => '', + proallargtypes => '{text,int8,int8}', proargmodes => '{o,o,o}', + proargnames => '{name,off,size}', + prosrc => 'pg_get_shmem_allocations' },
Hm. I think the function is actually volatile, rather than stable?
Queries can trigger shmem allocations internally, right?
Greetings,
Andres Freund
On Fri, Nov 15, 2019 at 11:59:34AM -0800, Andres Freund wrote:
On 2019-11-15 14:43:09 -0500, Robert Haas wrote:
This never got applied, and that annoyed me again today, so here's a
new version that I've whacked around somewhat and propose to commit. I
ripped out the stuff pertaining to dynamic shared memory segments,
both because I think it might need some more thought and discussion,
and because the part the pertains to the main shared memory segment is
the part I keep wishing we had. We can add that other part later if
we're all agreed on it, but let's go ahead and add this part now.Oh, nice! Makes sense to me to split off the dsm part.
last Friday we had a conference in Tokyo, and this has been actually
mentioned when we had an after-dinner with a couple of other hackers.
Then a couple of hours later this thread rises from the ashes.
+/* SQL SRF showing allocated shared memory */
+Datum
+pg_get_shmem_allocations(PG_FUNCTION_ARGS)
This could be more talkative.
+# shared memory usage +{ oid => '8613', + descr => 'allocations from the main shared memory segment', + proname => 'pg_get_shmem_allocations', 'prorows' => 10, 'proretset' => 't', + provolatile => 's', 'prorettype' => 'record', 'proargtypes' => '', + proallargtypes => '{text,int8,int8}', proargmodes => '{o,o,o}', + proargnames => '{name,off,size}', + prosrc => 'pg_get_shmem_allocations' },Hm. I think the function is actually volatile, rather than stable?
Queries can trigger shmem allocations internally, right?
+1.
--
Michael
Hi,
On 2019-11-18 21:49:55 +0900, Michael Paquier wrote:
+/* SQL SRF showing allocated shared memory */ +Datum +pg_get_shmem_allocations(PG_FUNCTION_ARGS) This could be more talkative.
I don't really see what it'd say, except restate the function name as a
sentence? I think that kind of comment has negative, not positive value.
- Andres
On Fri, Nov 15, 2019 at 2:59 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
I didn't really read the patch in any detail, but those things
hopped out at me.
Thanks for the reviews. Here is a new version. Changes:
- doc: Add an entry to the table of system views, per Tom.
- doc: Wrap reference to "<anonymous>" in <literal> tags, per self-review.
- doc: Note that off is NULL for <anonymous> allocations, per Andres.
- doc: Remove extra "or" at the end of a sentence, per self-review.
- doc: Note that the view is by default super-user only, per both Tom
and Andres.
- code: Declare values/nulls arrays only once at function scope
instead of 3x, and tighten up code, per Andres and self-review.
- dat: Mark as volatile, per Tom and Andres and Michael.
Non-changes:
- I didn't add a comment saying that there would be only one entry for
anonymous and free memory, as requested by Andres, because it seemed
like that's what users would expect anyway and I couldn't think of a
way of saying it that didn't sound dumb.
- I didn't expand the comment for the C function
pg_get_shmem_allocations, as requested by Michael, because I agree
with Andres that there doesn't seem to be anything particularly
helpful to say there.
I'll go ahead and commit this if nobody has further comments.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
v2-0001-Add-a-pg_shmem_allocations-view.patchapplication/octet-stream; name=v2-0001-Add-a-pg_shmem_allocations-view.patchDownload
From a3d3faa60cf7d98bf3c506bc0f4b7a98e99f48bf Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Fri, 15 Nov 2019 14:40:23 -0500
Subject: [PATCH v2] Add a pg_shmem_allocations view.
It tells you how the main shared memory segment is being used.
Andres Freund, with modifications by me, reviewed by Michael
Paquier, Tom Lane, and (several years ago) Marti Raudsepp.
Discussion: http://postgr.es/m/20140504114417.GM12715@awork2.anarazel.de
---
doc/src/sgml/catalogs.sgml | 75 ++++++++++++++++++++++++++
doc/src/sgml/xfunc.sgml | 2 +-
src/backend/catalog/system_views.sql | 6 +++
src/backend/storage/ipc/shmem.c | 80 ++++++++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 9 ++++
src/test/regress/expected/rules.out | 4 ++
6 files changed, 175 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 55694c4368..fa044dbb3b 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8362,6 +8362,11 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>planner statistics</entry>
</row>
+ <row>
+ <entry><link linkend="view-pg-shmem-allocations"><structname>pg_shmem_allocations</structname></link></entry>
+ <entry>shared memory allocations</entry>
+ </row>
+
<row>
<entry><link linkend="view-pg-stats-ext"><structname>pg_stats_ext</structname></link></entry>
<entry>extended planner statistics</entry>
@@ -10748,6 +10753,76 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</sect1>
+ <sect1 id="view-pg-shmem-allocations">
+ <title><structname>pg_shmem_allocations</structname></title>
+
+ <indexterm zone="view-pg-shmem-allocations">
+ <primary>pg_shmem_allocations</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_shmem_allocations</structname> view shows allocations
+ made from the server's main shared memory segment. This includes both
+ memory allocated by <productname>postgres</productname> itself and memory
+ allocated by extensions using the mechanisms detailed in
+ <xref linkend="xfunc-shared-addin" />.
+ </para>
+
+ <para>
+ Note that this view does not include memory allocated using the dynamic
+ shared memory infrastructure.
+ </para>
+
+ <table>
+ <title><structname>pg_shmem_allocations</structname> Columns</title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry><structfield>name</structfield></entry>
+ <entry><type>text</type></entry>
+ <entry>The name of the shared memory allocation. NULL for unused memory
+ and <literal><anonymous></literal> for anonymous
+ allocations.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>off</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>The offset at which the allocation starts. NULL for anonymous
+ allocations and unused memory.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>size</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Size of the allocation</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ Anonymous allocations are allocations that have been made
+ with <literal>ShmemAlloc()</literal> directly, rather than via
+ <literal>ShmemInitStruct()</literal> or
+ <literal>ShmemInitHash()</literal>.
+ </para>
+
+ <para>
+ By default, the <structname>pg_shmem_allocations</structname> view can be
+ read only by superusers.
+ </para>
+ </sect1>
+
<sect1 id="view-pg-stats">
<title><structname>pg_stats</structname></title>
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index d9afd3be4d..bb8468553c 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3241,7 +3241,7 @@ CREATE FUNCTION make_array(anyelement) RETURNS anyarray
</para>
</sect2>
- <sect2>
+ <sect2 id="xfunc-shared-addin">
<title>Shared Memory and LWLocks</title>
<para>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f01a6..42a58f56ea 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -547,6 +547,12 @@ CREATE VIEW pg_config AS
REVOKE ALL on pg_config FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_config() FROM PUBLIC;
+CREATE VIEW pg_shmem_allocations AS
+ SELECT * FROM pg_get_shmem_allocations();
+
+REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index a1567d3559..aefdd6083d 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -66,11 +66,14 @@
#include "postgres.h"
#include "access/transam.h"
+#include "fmgr.h"
+#include "funcapi.h"
#include "miscadmin.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
#include "storage/shmem.h"
#include "storage/spin.h"
+#include "utils/builtins.h"
/* shared memory global variables */
@@ -503,3 +506,80 @@ mul_size(Size s1, Size s2)
errmsg("requested shared memory size overflows size_t")));
return result;
}
+
+/* SQL SRF showing allocated shared memory */
+Datum
+pg_get_shmem_allocations(PG_FUNCTION_ARGS)
+{
+#define PG_GET_SHMEM_SIZES_COLS 3
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ HASH_SEQ_STATUS hstat;
+ ShmemIndexEnt *ent;
+ Size named_allocated = 0;
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not " \
+ "allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ LWLockAcquire(ShmemIndexLock, LW_SHARED);
+
+ hash_seq_init(&hstat, ShmemIndex);
+
+ /* output all allocated entries */
+ memset(nulls, 0, sizeof(nulls));
+ while ((ent = (ShmemIndexEnt *) hash_seq_search(&hstat)) != NULL)
+ {
+ values[0] = CStringGetTextDatum(ent->key);
+ values[1] = Int64GetDatum((char *) ent->location - (char *) ShmemSegHdr);
+ values[2] = Int64GetDatum(ent->size);
+ named_allocated += ent->size;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* output shared memory allocated but not counted via the shmem index */
+ values[0] = CStringGetTextDatum("<anonymous>");
+ nulls[1] = true;
+ values[2] = Int64GetDatum(ShmemSegHdr->freeoffset - named_allocated);
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /* output as-of-yet unused shared memory */
+ nulls[0] = true;
+ values[1] = Int64GetDatum(ShmemSegHdr->freeoffset);
+ nulls[1] = false;
+ values[2] = Int64GetDatum(ShmemSegHdr->totalsize - ShmemSegHdr->freeoffset);
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ LWLockRelease(ShmemIndexLock);
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index ac8f64b219..f8d1e4fcff 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7656,6 +7656,15 @@
proparallel => 'r', prorettype => 'float8', proargtypes => '',
prosrc => 'pg_notification_queue_usage' },
+# shared memory usage
+{ oid => '8613',
+ descr => 'allocations from the main shared memory segment',
+ proname => 'pg_get_shmem_allocations', 'prorows' => 10, 'proretset' => 't',
+ provolatile => 'v', 'prorettype' => 'record', 'proargtypes' => '',
+ proallargtypes => '{text,int8,int8}', proargmodes => '{o,o,o}',
+ proargnames => '{name,off,size}',
+ prosrc => 'pg_get_shmem_allocations' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 80a07825b9..98a0ed222e 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1721,6 +1721,10 @@ pg_shadow| SELECT pg_authid.rolname AS usename,
FROM (pg_authid
LEFT JOIN pg_db_role_setting s ON (((pg_authid.oid = s.setrole) AND (s.setdatabase = (0)::oid))))
WHERE pg_authid.rolcanlogin;
+pg_shmem_allocations| SELECT pg_get_shmem_allocations.name,
+ pg_get_shmem_allocations.off,
+ pg_get_shmem_allocations.size
+ FROM pg_get_shmem_allocations() pg_get_shmem_allocations(name, off, size);
pg_stat_activity| SELECT s.datid,
d.datname,
s.pid,
--
2.17.2 (Apple Git-113)
On 2019-Dec-18, Robert Haas wrote:
- code: Declare values/nulls arrays only once at function scope
instead of 3x, and tighten up code, per Andres and self-review.
Really small nit: I'd rather have the "nulls" assignment in all cases
even when the previous value is still valid. This tight coding seems
weirdly asymmetrical.
Can we please stop splitting this error message in two?
+ errmsg("materialize mode required, but it is not " \
+ "allowed in this context")));
(What's with the newline escape there anyway?)
Shmem structs are cacheline-aligned; the padding space is not AFAICS
considered in ShmemIndexEnt->size. The reported size would be
under-reported (or reported as "anonymous"?) I think we should include
the alignment padding, for clarity.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Wed, Dec 18, 2019 at 10:59 AM Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
On 2019-Dec-18, Robert Haas wrote:
- code: Declare values/nulls arrays only once at function scope
instead of 3x, and tighten up code, per Andres and self-review.Really small nit: I'd rather have the "nulls" assignment in all cases
even when the previous value is still valid. This tight coding seems
weirdly asymmetrical.
I agree that the asymmetry of it is a bit bothersome, but I think
having extra code setting things to null is worse. It makes the code
significantly bigger, which to me is more problematic than the
asymmetry.
Can we please stop splitting this error message in two?
+ errmsg("materialize mode required, but it is not " \ + "allowed in this context")));(What's with the newline escape there anyway?)
That message is like that everywhere in the tree, including the
escape, except for a couple of instances in contrib which deviate. If
you want to go change them all, feel free, and I'll adjust this to
match the then-prevailing style.
Shmem structs are cacheline-aligned; the padding space is not AFAICS
considered in ShmemIndexEnt->size. The reported size would be
under-reported (or reported as "anonymous"?) I think we should include
the alignment padding, for clarity.
It seems to me that you could plausibly define this view to show
either (a) the amount of space that the caller actually tried to
allocate or (b) the amount of space that the allocator decided to
allocate, after padding, and it's not obvious that (b) is a better
definition than (a).
That having been said, you're correct that the padding space is
currently reported as <anonymous>, and that does seem wrong.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
On Wed, Dec 18, 2019 at 10:59 AM Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:Can we please stop splitting this error message in two?
+ errmsg("materialize mode required, but it is not " \ + "allowed in this context")));(What's with the newline escape there anyway?)
That message is like that everywhere in the tree, including the
escape, except for a couple of instances in contrib which deviate. If
you want to go change them all, feel free, and I'll adjust this to
match the then-prevailing style.
I agree with Alvaro that that is *not* project style, particularly not
the newline escape. Like Robert, I'm not quite fussed enough to go
change it, but +1 if Alvaro wants to.
It seems to me that you could plausibly define this view to show
either (a) the amount of space that the caller actually tried to
allocate or (b) the amount of space that the allocator decided to
allocate, after padding, and it's not obvious that (b) is a better
definition than (a).
That having been said, you're correct that the padding space is
currently reported as <anonymous>, and that does seem wrong.
It seems like it'd be worth subdividing "<anonymous>" into the actual
anonymous allocations and the allocator overhead (which is both
padding and whatever the shmem allocator itself eats). Maybe call
the latter "<overhead>". After which, I'd be tempted to call the
free space "<free>" rather than giving it a null name.
regards, tom lane
On Wed, Dec 18, 2019 at 12:06 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
It seems like it'd be worth subdividing "<anonymous>" into the actual
anonymous allocations and the allocator overhead (which is both
padding and whatever the shmem allocator itself eats). Maybe call
the latter "<overhead>". After which, I'd be tempted to call the
free space "<free>" rather than giving it a null name.
Here's a new version that takes a slightly different approach: it adds
an additional column to the view for "allocated_size," so that you can
see both what was requested and what you actually got. With this
approach, you can do interesting things like:
select *, off - lag(off + allocated_size, 1) over () as hole_size from
(select * from pg_shmem_allocations order by 2) x;
...which didn't work very well with the previous version.
All of this makes me think that we might want to do some followup to
(1) convert anonymous allocations to non-anonymous allocations, for
inspectability, and (2) do some renaming to get better stylistic
agreement between the names of various shared memory chunks. But I
think that work is properly done after this patch is committed, not
before.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
v3-0001-Add-a-pg_shmem_allocations-view.patchapplication/octet-stream; name=v3-0001-Add-a-pg_shmem_allocations-view.patchDownload
From 560b5b58b62d8929f8a88aec54d6f5053552568e Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Fri, 15 Nov 2019 14:40:23 -0500
Subject: [PATCH v3] Add a pg_shmem_allocations view.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
It tells you how the main shared memory segment is being used.
Andres Freund, with modifications by me, reviewed by Michael
Paquier, Tom Lane, Álvaro Herrera, and (several years ago) Marti
Raudsepp.
Discussion: http://postgr.es/m/20140504114417.GM12715@awork2.anarazel.de
---
doc/src/sgml/catalogs.sgml | 85 +++++++++++++++++++++
doc/src/sgml/xfunc.sgml | 2 +-
src/backend/catalog/system_views.sql | 6 ++
src/backend/storage/ipc/shmem.c | 107 ++++++++++++++++++++++++++-
src/include/catalog/pg_proc.dat | 9 +++
src/include/storage/shmem.h | 3 +-
src/test/regress/expected/rules.out | 5 ++
7 files changed, 213 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 55694c4368..85ac79f07e 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8362,6 +8362,11 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>planner statistics</entry>
</row>
+ <row>
+ <entry><link linkend="view-pg-shmem-allocations"><structname>pg_shmem_allocations</structname></link></entry>
+ <entry>shared memory allocations</entry>
+ </row>
+
<row>
<entry><link linkend="view-pg-stats-ext"><structname>pg_stats_ext</structname></link></entry>
<entry>extended planner statistics</entry>
@@ -10748,6 +10753,86 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</sect1>
+ <sect1 id="view-pg-shmem-allocations">
+ <title><structname>pg_shmem_allocations</structname></title>
+
+ <indexterm zone="view-pg-shmem-allocations">
+ <primary>pg_shmem_allocations</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_shmem_allocations</structname> view shows allocations
+ made from the server's main shared memory segment. This includes both
+ memory allocated by <productname>postgres</productname> itself and memory
+ allocated by extensions using the mechanisms detailed in
+ <xref linkend="xfunc-shared-addin" />.
+ </para>
+
+ <para>
+ Note that this view does not include memory allocated using the dynamic
+ shared memory infrastructure.
+ </para>
+
+ <table>
+ <title><structname>pg_shmem_allocations</structname> Columns</title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry><structfield>name</structfield></entry>
+ <entry><type>text</type></entry>
+ <entry>The name of the shared memory allocation. NULL for unused memory
+ and <literal><anonymous></literal> for anonymous
+ allocations.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>off</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>The offset at which the allocation starts. NULL for anonymous
+ allocations and unused memory.</entry>
+ </row>
+
+ <row>
+ <entry><structfield>size</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Size of the allocation</entry>
+ </row>
+
+ <row>
+ <entry><structfield>allocated_size</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Size of the allocation including padding. For anonymous
+ allocations, no information about padding is available, so the
+ <literal>size</literal> and <literal>allocated_size</literal> columns
+ will always be equal. Padding is not meaningful for free memory, so
+ the columns will be equal in that case also.</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ Anonymous allocations are allocations that have been made
+ with <literal>ShmemAlloc()</literal> directly, rather than via
+ <literal>ShmemInitStruct()</literal> or
+ <literal>ShmemInitHash()</literal>.
+ </para>
+
+ <para>
+ By default, the <structname>pg_shmem_allocations</structname> view can be
+ read only by superusers.
+ </para>
+ </sect1>
+
<sect1 id="view-pg-stats">
<title><structname>pg_stats</structname></title>
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index d9afd3be4d..bb8468553c 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3241,7 +3241,7 @@ CREATE FUNCTION make_array(anyelement) RETURNS anyarray
</para>
</sect2>
- <sect2>
+ <sect2 id="xfunc-shared-addin">
<title>Shared Memory and LWLocks</title>
<para>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f01a6..42a58f56ea 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -547,6 +547,12 @@ CREATE VIEW pg_config AS
REVOKE ALL on pg_config FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_config() FROM PUBLIC;
+CREATE VIEW pg_shmem_allocations AS
+ SELECT * FROM pg_get_shmem_allocations();
+
+REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index a1567d3559..c3427fbda2 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -66,12 +66,16 @@
#include "postgres.h"
#include "access/transam.h"
+#include "fmgr.h"
+#include "funcapi.h"
#include "miscadmin.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
#include "storage/shmem.h"
#include "storage/spin.h"
+#include "utils/builtins.h"
+static void *ShmemAllocRaw(Size size, Size *allocated_size);
/* shared memory global variables */
@@ -157,8 +161,9 @@ void *
ShmemAlloc(Size size)
{
void *newSpace;
+ Size allocated_size;
- newSpace = ShmemAllocNoError(size);
+ newSpace = ShmemAllocRaw(size, &allocated_size);
if (!newSpace)
ereport(ERROR,
(errcode(ERRCODE_OUT_OF_MEMORY),
@@ -174,6 +179,20 @@ ShmemAlloc(Size size)
*/
void *
ShmemAllocNoError(Size size)
+{
+ Size allocated_size;
+
+ return ShmemAllocRaw(size, &allocated_size);
+}
+
+/*
+ * ShmemAllocRaw -- allocate align chunk and return allocated size
+ *
+ * Also sets *allocated_size to the number of bytes allocated, which will
+ * be equal to the number requested plus any padding we choose to add.
+ */
+static void *
+ShmemAllocRaw(Size size, Size *allocated_size)
{
Size newStart;
Size newFree;
@@ -191,6 +210,7 @@ ShmemAllocNoError(Size size)
* won't be sufficient.
*/
size = CACHELINEALIGN(size);
+ *allocated_size = size;
Assert(ShmemSegHdr != NULL);
@@ -441,8 +461,10 @@ ShmemInitStruct(const char *name, Size size, bool *foundPtr)
}
else
{
+ Size allocated_size;
+
/* It isn't in the table yet. allocate and initialize it */
- structPtr = ShmemAllocNoError(size);
+ structPtr = ShmemAllocRaw(size, &allocated_size);
if (structPtr == NULL)
{
/* out of memory; remove the failed ShmemIndex entry */
@@ -455,6 +477,7 @@ ShmemInitStruct(const char *name, Size size, bool *foundPtr)
name, size)));
}
result->size = size;
+ result->allocated_size = allocated_size;
result->location = structPtr;
}
@@ -503,3 +526,83 @@ mul_size(Size s1, Size s2)
errmsg("requested shared memory size overflows size_t")));
return result;
}
+
+/* SQL SRF showing allocated shared memory */
+Datum
+pg_get_shmem_allocations(PG_FUNCTION_ARGS)
+{
+#define PG_GET_SHMEM_SIZES_COLS 4
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ HASH_SEQ_STATUS hstat;
+ ShmemIndexEnt *ent;
+ Size named_allocated = 0;
+ Datum values[PG_GET_SHMEM_SIZES_COLS];
+ bool nulls[PG_GET_SHMEM_SIZES_COLS];
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not " \
+ "allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ LWLockAcquire(ShmemIndexLock, LW_SHARED);
+
+ hash_seq_init(&hstat, ShmemIndex);
+
+ /* output all allocated entries */
+ memset(nulls, 0, sizeof(nulls));
+ while ((ent = (ShmemIndexEnt *) hash_seq_search(&hstat)) != NULL)
+ {
+ values[0] = CStringGetTextDatum(ent->key);
+ values[1] = Int64GetDatum((char *) ent->location - (char *) ShmemSegHdr);
+ values[2] = Int64GetDatum(ent->size);
+ values[3] = Int64GetDatum(ent->allocated_size);
+ named_allocated += ent->allocated_size;
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* output shared memory allocated but not counted via the shmem index */
+ values[0] = CStringGetTextDatum("<anonymous>");
+ nulls[1] = true;
+ values[2] = Int64GetDatum(ShmemSegHdr->freeoffset - named_allocated);
+ values[3] = values[2];
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ /* output as-of-yet unused shared memory */
+ nulls[0] = true;
+ values[1] = Int64GetDatum(ShmemSegHdr->freeoffset);
+ nulls[1] = false;
+ values[2] = Int64GetDatum(ShmemSegHdr->totalsize - ShmemSegHdr->freeoffset);
+ values[3] = values[2];
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+
+ LWLockRelease(ShmemIndexLock);
+
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index ac8f64b219..c403dc5b48 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -7656,6 +7656,15 @@
proparallel => 'r', prorettype => 'float8', proargtypes => '',
prosrc => 'pg_notification_queue_usage' },
+# shared memory usage
+{ oid => '8613',
+ descr => 'allocations from the main shared memory segment',
+ proname => 'pg_get_shmem_allocations', 'prorows' => 50, 'proretset' => 't',
+ provolatile => 'v', 'prorettype' => 'record', 'proargtypes' => '',
+ proallargtypes => '{text,int8,int8,int8}', proargmodes => '{o,o,o,o}',
+ proargnames => '{name,off,size,allocated_size}',
+ prosrc => 'pg_get_shmem_allocations' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/storage/shmem.h b/src/include/storage/shmem.h
index 876e908611..b75027539e 100644
--- a/src/include/storage/shmem.h
+++ b/src/include/storage/shmem.h
@@ -59,7 +59,8 @@ typedef struct
{
char key[SHMEM_INDEX_KEYSIZE]; /* string name */
void *location; /* location in shared mem */
- Size size; /* # bytes allocated for the structure */
+ Size size; /* # bytes requested for the structure */
+ Size allocated_size; /* # bytes actually allocated */
} ShmemIndexEnt;
/*
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 80a07825b9..62eaf90a0f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1721,6 +1721,11 @@ pg_shadow| SELECT pg_authid.rolname AS usename,
FROM (pg_authid
LEFT JOIN pg_db_role_setting s ON (((pg_authid.oid = s.setrole) AND (s.setdatabase = (0)::oid))))
WHERE pg_authid.rolcanlogin;
+pg_shmem_allocations| SELECT pg_get_shmem_allocations.name,
+ pg_get_shmem_allocations.off,
+ pg_get_shmem_allocations.size,
+ pg_get_shmem_allocations.allocated_size
+ FROM pg_get_shmem_allocations() pg_get_shmem_allocations(name, off, size, allocated_size);
pg_stat_activity| SELECT s.datid,
d.datname,
s.pid,
--
2.17.2 (Apple Git-113)
At Wed, 18 Dec 2019 12:30:51 -0500, Robert Haas <robertmhaas@gmail.com> wrote in
On Wed, Dec 18, 2019 at 12:06 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
It seems like it'd be worth subdividing "<anonymous>" into the actual
anonymous allocations and the allocator overhead (which is both
padding and whatever the shmem allocator itself eats). Maybe call
the latter "<overhead>". After which, I'd be tempted to call the
free space "<free>" rather than giving it a null name.Here's a new version that takes a slightly different approach: it adds
an additional column to the view for "allocated_size," so that you can
see both what was requested and what you actually got. With this
The doc is saying that "size" is "Size of the allocation" and
"allocated_size" is "size including padding". It seems somewhat
confusing to me. I'm not sure what wording is best but I think people
use net/gross wordings to describe like that.
approach, you can do interesting things like:
select *, off - lag(off + allocated_size, 1) over () as hole_size from
(select * from pg_shmem_allocations order by 2) x;...which didn't work very well with the previous version.
All of this makes me think that we might want to do some followup to
(1) convert anonymous allocations to non-anonymous allocations, for
inspectability, and (2) do some renaming to get better stylistic
agreement between the names of various shared memory chunks. But I
think that work is properly done after this patch is committed, not
before.
I agree to (2), but regarding (1), most or perhaps all of the
anonymous allocations seems belonging to one of the shared hashes but
are recognized as holes shown by the above statement. So the current
output of the view is wrong in that sense. I think (1) should be
resolved before adding the view.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Wed, Dec 18, 2019 at 9:53 PM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:
The doc is saying that "size" is "Size of the allocation" and
"allocated_size" is "size including padding". It seems somewhat
confusing to me. I'm not sure what wording is best but I think people
use net/gross wordings to describe like that.
I don't think I'd find that particularly clear. It seems to me that if
the second size includes padding, then the first one differs in not
including padding, so I'm not really sure where the confusion is. I
thought about writing, for the first one, that is the requested size
of the allocation, but that seemed like it might confuse someone by
suggesting that the actual size of the allocation might be less than
what was requested. I also thought about describing the first one as
the size excluding padding, but that seems redundant. Maybe it would
be good to change the second one to say "Size of the allocation
including padding added by the allocator itself."
All of this makes me think that we might want to do some followup to
(1) convert anonymous allocations to non-anonymous allocations, for
inspectability, and (2) do some renaming to get better stylistic
agreement between the names of various shared memory chunks. But I
think that work is properly done after this patch is committed, not
before.I agree to (2), but regarding (1), most or perhaps all of the
anonymous allocations seems belonging to one of the shared hashes but
are recognized as holes shown by the above statement. So the current
output of the view is wrong in that sense. I think (1) should be
resolved before adding the view.
I guess I don't understand how this makes the output wrong. Either the
allocations have a name, or they are anonymous. This dumps everything
that has a name. What would you suggest? It seems to me that it's more
appropriate for this patch to just tell us about what's in shared
memory, not change what's in shared memory. If we want to do the
latter, that's a job for another patch.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
At Mon, 30 Dec 2019 13:52:50 -0500, Robert Haas <robertmhaas@gmail.com> wrote in
On Wed, Dec 18, 2019 at 9:53 PM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:The doc is saying that "size" is "Size of the allocation" and
"allocated_size" is "size including padding". It seems somewhat
confusing to me. I'm not sure what wording is best but I think people
use net/gross wordings to describe like that.I don't think I'd find that particularly clear. It seems to me that if
the second size includes padding, then the first one differs in not
including padding, so I'm not really sure where the confusion is. I
thought about writing, for the first one, that is the requested size
of the allocation, but that seemed like it might confuse someone by
suggesting that the actual size of the allocation might be less than
what was requested. I also thought about describing the first one as
the size excluding padding, but that seems redundant. Maybe it would
be good to change the second one to say "Size of the allocation
including padding added by the allocator itself."
Ugh.. Thanks for the explanation and I'm convinced that the current
wording is the best.
All of this makes me think that we might want to do some followup to
(1) convert anonymous allocations to non-anonymous allocations, for
inspectability, and (2) do some renaming to get better stylistic
agreement between the names of various shared memory chunks. But I
think that work is properly done after this patch is committed, not
before.I agree to (2), but regarding (1), most or perhaps all of the
anonymous allocations seems belonging to one of the shared hashes but
are recognized as holes shown by the above statement. So the current
output of the view is wrong in that sense. I think (1) should be
resolved before adding the view.I guess I don't understand how this makes the output wrong. Either the
allocations have a name, or they are anonymous. This dumps everything
that has a name. What would you suggest? It seems to me that it's more
appropriate for this patch to just tell us about what's in shared
memory, not change what's in shared memory. If we want to do the
latter, that's a job for another patch.
Sorry for the strange sentense. "I agree to (2)" meant that "I agree
that (2) is done in later patch".
About (1), I perplexed by the word "hole", which seemed to me as a
region that is not allocated to anything. But no columns with the
name actually is not in the view, so the view is not wrong in the
first place.
I agree to the patch as-is. Thanks for the explanatin.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Wed, Jan 8, 2020 at 2:30 AM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:
I agree to the patch as-is. Thanks for the explanatin.
OK. Thanks for the review, and for your thoughtful consideration of my comments.
I went ahead and committed this.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Hi,
On 2020-01-09 11:13:46 -0500, Robert Haas wrote:
On Wed, Jan 8, 2020 at 2:30 AM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:I agree to the patch as-is. Thanks for the explanatin.
OK. Thanks for the review, and for your thoughtful consideration of my comments.
I went ahead and committed this.
Wee!
Greetings,
Andres Freund