logical decoding/replication: new functions pg_ls_logicaldir and pg_ls_replslotdir
Hi,
At times, users want to know what are the files (snapshot and mapping
files) that are available under pg_logical directory and also the
spill files that are under pg_replslot directory and how much space
they occupy. This will help to better know the storage usage pattern
of these directories. Can we have two new functions pg_ls_logicaldir
and pg_ls_replslotdir on the similar lines of pg_ls_logdir,
pg_ls_logdir,pg_ls_tmpdir, pg_ls_archive_statusdir [1]https://www.postgresql.org/docs/devel/functions-admin.html?
[1]: https://www.postgresql.org/docs/devel/functions-admin.html
Regards,
Bharath Rupireddy.
On Fri, Oct 8, 2021 at 4:39 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
Hi,
At times, users want to know what are the files (snapshot and mapping
files) that are available under pg_logical directory and also the
spill files that are under pg_replslot directory and how much space
they occupy.
Why can't you use pg_ls_dir to see the contents of pg_replslot? To
know the space taken by spilling, you might want to check
pg_stat_replication_slots[1]https://www.postgresql.org/docs/devel/monitoring-stats.html#MONITORING-PG-STAT-REPLICATION-SLOTS-VIEW as that gives information about
spill_bytes.
--
With Regards,
Amit Kapila.
On Fri, Oct 22, 2021 at 3:18 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Oct 8, 2021 at 4:39 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:Hi,
At times, users want to know what are the files (snapshot and mapping
files) that are available under pg_logical directory and also the
spill files that are under pg_replslot directory and how much space
they occupy.Why can't you use pg_ls_dir to see the contents of pg_replslot? To
know the space taken by spilling, you might want to check
pg_stat_replication_slots[1] as that gives information about
spill_bytes.
Thanks Amit!
pg_ls_dir gives the list of directories and files, but not their
sizes. And it looks like the spill_bytes from
pg_stat_replication_slots is the accumulated byte count (see [1]postgres=# select pg_ls_dir('/home/bharath/postgres/inst/bin/data/pg_replslot/mysub'); pg_ls_dir ----------- state (1 row)), not
the current size of the spills files, so it's not representing the
spill files and their size at that moment.
If we have pg_ls_logicaldir and pg_ls_replslotdir returning the
files, szies, and last modified times, it will be useful in production
environments to see the disk usage of those files at the current
moment. The data from these functions can be fed to an external
analytics tool invoking the functions at regular intervals of time and
report the disk usage of these folders. This will be super useful to
analyze the questions like: Was the disk usage more at time t1? What
happened to my database system at that time? etc. And, these
functions can run independent of the stats collector process which is
currently required for the pg_stat_replication_slots view.
Thoughts?
I plan to work on a patch if okay.
[1]: postgres=# select pg_ls_dir('/home/bharath/postgres/inst/bin/data/pg_replslot/mysub'); pg_ls_dir ----------- state (1 row)
postgres=# select
pg_ls_dir('/home/bharath/postgres/inst/bin/data/pg_replslot/mysub');
pg_ls_dir
-----------
state
(1 row)
postgres=# select * from pg_stat_replication_slots;
slot_name | spill_txns | spill_count | spill_bytes | stream_txns |
stream_count | stream_bytes | total_txns | total_bytes | stats_reset
-----------+------------+-------------+-------------+-------------+--------------+--------------+------------+-------------+-------------
mysub | 3 | 6 | 396000000 | 0 |
0 | 0 | 5 | 396001128 |
(1 row)
Regards,
Bharath Rupireddy.
On Fri, Oct 22, 2021 at 04:18:04PM +0530, Bharath Rupireddy wrote:
On Fri, Oct 22, 2021 at 3:18 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Oct 8, 2021 at 4:39 PM Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote:
At times, users want to know what are the files (snapshot and mapping
files) that are available under pg_logical directory and also the
spill files that are under pg_replslot directory and how much space
they occupy.Why can't you use pg_ls_dir to see the contents of pg_replslot? To
Thanks Amit!
pg_ls_dir gives the list of directories and files, but not their sizes.
Returning sizes is already possible by using pg_stat_file:
ts=# SELECT dd, a, ls, stat.* FROM (SELECT current_setting('data_directory') AS dd, 'pg_logical' AS a) AS a, pg_ls_dir(a) AS ls, pg_stat_file(dd ||'/'|| a ||'/'|| ls) AS stat ;
dd | a | ls | size | access | modification | change | creation | isdir
------------------------+------------+-----------------------+------+------------------------+------------------------+------------------------+----------+-------
/var/lib/pgsql/14/data | pg_logical | replorigin_checkpoint | 8 | 2021-10-22 08:20:30-06 | 2021-10-22 08:20:30-06 | 2021-10-22 08:20:30-06 | | f
/var/lib/pgsql/14/data | pg_logical | mappings | 4096 | 2021-10-21 19:54:19-06 | 2021-10-15 19:50:35-06 | 2021-10-15 19:50:35-06 | | t
/var/lib/pgsql/14/data | pg_logical | snapshots | 4096 | 2021-10-21 19:54:19-06 | 2021-10-15 19:50:35-06 | 2021-10-15 19:50:35-06 | | t
I agree that this isn't a very friendly query, so I had created a patch adding
pg_ls_dir_metadata():
https://commitfest.postgresql.org/33/2377/
postgres=# SELECT * FROM pg_ls_dir_metadata('pg_logical');
filename | size | access | modification | change | creation | type | path
-----------------------+------+------------------------+------------------------+------------------------+----------+------+----------------------------------
mappings | 4096 | 2021-10-22 09:15:29-05 | 2021-10-22 09:15:29-05 | 2021-10-22 09:15:29-05 | | d | pg_logical/mappings
replorigin_checkpoint | 8 | 2021-10-22 09:15:47-05 | 2021-10-22 09:15:45-05 | 2021-10-22 09:15:45-05 | | - | pg_logical/replorigin_checkpoint
. | 4096 | 2021-10-22 09:16:23-05 | 2021-10-22 09:15:45-05 | 2021-10-22 09:15:45-05 | | d | pg_logical/.
.. | 4096 | 2021-10-22 09:16:01-05 | 2021-10-22 09:15:47-05 | 2021-10-22 09:15:47-05 | | d | pg_logical/..
snapshots | 4096 | 2021-10-22 09:15:29-05 | 2021-10-22 09:15:29-05 | 2021-10-22 09:15:29-05 | | d | pg_logical/snapshots
(5 rows)
I concluded that it's better to add a function to list metadata of an arbitrary
dir, rather than adding more functions to handle specific, hardcoded dirs:
/messages/by-id/20191227170220.GE12890@telsasoft.com
--
Justin
On Fri, Oct 22, 2021 at 9:26 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
I concluded that it's better to add a function to list metadata of an arbitrary
dir, rather than adding more functions to handle specific, hardcoded dirs:
/messages/by-id/20191227170220.GE12890@telsasoft.com
I just had a quick look at the pg_ls_dir_metadata() patch(I didn't
look at the other patches). While it's a good idea to have a single
function for all the PGDATA directories, I'm not sure if one would
ever need the info like type, change, creation path etc. If we do
this, the function will become the linux equivalent command. I don't
see the difference between modification and change time stamps. For
debugging or analytical purposes in production environments, one would
majorly look at the file name, it's size on the disk, modification
time (to decide whether the file is stale or not, creation time (to
decide how old is the file), file/directory(maybe?). I'm not sure if
your patch has a recursive option for pg_ls_dir_metadata(), if it has,
I think it's more complex from a usability perspective.
And the functions like pg_ls_tmpdir, pg_ls_tmpdir, pg_ls_waldir etc.
(existing) and pg_ls_logicaldir, pg_ls_replslotdir (yet to have) will
provide the better usability compared to a generic function. Having
said this, I don't oppose having a generic function returning the file
name, file size, modification time, creation time, but not other info,
please. If one is interested in knowing the other information file
type, path etc. they can go run linux/windows/OS commands.
In summary what I think at this point is:
1) pg_ls_logicaldir, pg_ls_replslotdir - better for usability and
serving special purpose like their peers
2) modify pg_ls_dir such that it returns the file name, file size,
modification time, creation time, for directories, to be simple, it
shouldn't go recursively over all the directories, it should just
return the directory name, size, modification time, creation time.
If okay, I'm ready to spend time implementing them.
Thoughts?
Regards,
Bharath Rupireddy.
On Sat, Oct 23, 2021 at 11:10 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
On Fri, Oct 22, 2021 at 9:26 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
I concluded that it's better to add a function to list metadata of an arbitrary
dir, rather than adding more functions to handle specific, hardcoded dirs:
/messages/by-id/20191227170220.GE12890@telsasoft.comI just had a quick look at the pg_ls_dir_metadata() patch(I didn't
look at the other patches). While it's a good idea to have a single
function for all the PGDATA directories, I'm not sure if one would
ever need the info like type, change, creation path etc. If we do
this, the function will become the linux equivalent command. I don't
see the difference between modification and change time stamps. For
debugging or analytical purposes in production environments, one would
majorly look at the file name, it's size on the disk, modification
time (to decide whether the file is stale or not, creation time (to
decide how old is the file), file/directory(maybe?). I'm not sure if
your patch has a recursive option for pg_ls_dir_metadata(), if it has,
I think it's more complex from a usability perspective.And the functions like pg_ls_tmpdir, pg_ls_tmpdir, pg_ls_waldir etc.
(existing) and pg_ls_logicaldir, pg_ls_replslotdir (yet to have) will
provide the better usability compared to a generic function. Having
said this, I don't oppose having a generic function returning the file
name, file size, modification time, creation time, but not other info,
please. If one is interested in knowing the other information file
type, path etc. they can go run linux/windows/OS commands.In summary what I think at this point is:
1) pg_ls_logicaldir, pg_ls_replslotdir - better for usability and
serving special purpose like their peers
I've added 3 functions pg_ls_logicalsnapdir, pg_ls_logicalmapdir,
pg_ls_replslotdir, and attached the patch. The sample output looks
like [1]postgres=# select pg_ls_logicalsnapdir(); pg_ls_logicalsnapdir ----------------------------------------------- (0-14A50C0.snap,128,"2021-10-30 09:15:56+00") (0-14C46D8.snap,128,"2021-10-30 09:16:05+00") (0-14C97C8.snap,132,"2021-10-30 09:16:20+00"). Please review it further.
Here's the CF entry - https://commitfest.postgresql.org/35/3390/
[1]: postgres=# select pg_ls_logicalsnapdir(); pg_ls_logicalsnapdir ----------------------------------------------- (0-14A50C0.snap,128,"2021-10-30 09:15:56+00") (0-14C46D8.snap,128,"2021-10-30 09:16:05+00") (0-14C97C8.snap,132,"2021-10-30 09:16:20+00")
postgres=# select pg_ls_logicalsnapdir();
pg_ls_logicalsnapdir
-----------------------------------------------
(0-14A50C0.snap,128,"2021-10-30 09:15:56+00")
(0-14C46D8.snap,128,"2021-10-30 09:16:05+00")
(0-14C97C8.snap,132,"2021-10-30 09:16:20+00")
postgres=# select pg_ls_logicalmapdir();
pg_ls_logicalmapdir
---------------------------------------------------------------
(map-31d5-4eb-0_CDDDE88-2d9-2db,108,"2021-10-30 09:24:34+00")
(map-31d5-4eb-0_CDDDE88-2da-2db,108,"2021-10-30 09:24:34+00")
(map-31d5-4eb-0_CE48038-2dc-2de,108,"2021-10-30 09:24:35+00")
(map-31d5-4eb-0_CE6BAF0-2dd-2df,108,"2021-10-30 09:24:35+00")
(map-31d5-4eb-0_CD97DE0-2d9-2d9,36,"2021-10-30 09:24:30+00")
(map-31d5-4eb-0_CE24808-2da-2dd,108,"2021-10-30 09:24:35+00")
(map-31d5-4eb-0_CE01200-2dc-2dc,36,"2021-10-30 09:24:34+00")
(map-31d5-4eb-0_CDDDE88-2db-2db,36,"2021-10-30 09:24:34+00")
(map-31d5-4eb-0_CE6BAF0-2dc-2df,108,"2021-10-30 09:24:35+00")
(map-31d5-4eb-0_CDBA920-2d9-2da,108,"2021-10-30 09:24:32+00")
(map-31d5-4eb-0_CE01200-2da-2dc,108,"2021-10-30 09:24:34+00")
(map-31d5-4eb-0_CE6BAF0-2d9-2df,108,"2021-10-30 09:24:35+00")
(map-31d5-4eb-0_CE24808-2db-2dd,108,"2021-10-30 09:24:35+00")
(map-31d5-4eb-0_CE6BAF0-2db-2df,108,"2021-10-30 09:24:35+00")
(map-31d5-4eb-0_CE24808-2dd-2dd,36,"2021-10-30 09:24:35+00")
(map-31d5-4eb-0_CE24808-2dc-2dd,108,"2021-10-30 09:24:35+00")
(map-31d5-4eb-0_CD74E48-2d8-2d8,36,"2021-10-30 09:24:25+00")
(map-31d5-4eb-0_CE24808-2d9-2dd,108,"2021-10-30 09:24:35+00")
postgres=# select pg_ls_replslotdir('mysub');
pg_ls_replslotdir
-----------------------------------------------------------------
(xid-722-lsn-0-2000000.spill,36592640,"2021-10-30 09:18:29+00")
(xid-722-lsn-0-5000000.spill,4577860,"2021-10-30 09:18:32+00")
(state,200,"2021-10-30 09:18:25+00")
(xid-722-lsn-0-1000000.spill,25644220,"2021-10-30 09:18:29+00")
(xid-722-lsn-0-4000000.spill,36592640,"2021-10-30 09:18:32+00")
(xid-722-lsn-0-3000000.spill,36592640,"2021-10-30 09:18:32+00")
Regards,
Bharath Rupireddy.
Attachments:
v1-0001-Add-pg_ls_logicalsnapdir-pg_ls_logicalmapdir-and-.patchapplication/octet-stream; name=v1-0001-Add-pg_ls_logicalsnapdir-pg_ls_logicalmapdir-and-.patchDownload
From 428681ee0a66b98136445d607bd51026b8a204b4 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Sat, 30 Oct 2021 09:27:36 +0000
Subject: [PATCH v1] Add pg_ls_logicalsnapdir, pg_ls_logicalmapdir and
pg_ls_replslotdir functions
These functions lists the contents of the respective directories,
and are intended to be used by monitoring tools. Unlike pg_ls_dir(),
access to it can be granted to non-superusers so that those monitoring
tools can observe the principle of least privilege. Access is also
given by default to members of pg_monitor.
---
doc/src/sgml/func.sgml | 72 ++++++++++++++++++++++++
src/backend/catalog/system_functions.sql | 12 ++++
src/backend/utils/adt/genfile.c | 34 +++++++++++
src/include/catalog/catversion.h | 2 +-
src/include/catalog/pg_proc.dat | 15 +++++
5 files changed, 134 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 4b49dff2ff..51825edb01 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -27407,6 +27407,78 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
can be granted EXECUTE to run the function.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_ls_logicalsnapdir</primary>
+ </indexterm>
+ <function>pg_ls_logicalsnapdir</function> ()
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>size</parameter> <type>bigint</type>,
+ <parameter>modification</parameter> <type>timestamp with time zone</type> )
+ </para>
+ <para>
+ Returns the name, size, and last modification time (mtime) of each
+ ordinary file in the server's pg_logical/snapshots directory.
+ Filenames beginning with a dot, directories, and other special files
+ are excluded.
+ </para>
+ <para>
+ This function is restricted to superusers and members of
+ the <literal>pg_monitor</literal> role by default, but other users can
+ be granted EXECUTE to run the function.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_ls_logicalmapdir</primary>
+ </indexterm>
+ <function>pg_ls_logicalmapdir</function> ()
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>size</parameter> <type>bigint</type>,
+ <parameter>modification</parameter> <type>timestamp with time zone</type> )
+ </para>
+ <para>
+ Returns the name, size, and last modification time (mtime) of each
+ ordinary file in the server's pg_logical/mappings directory.
+ Filenames beginning with a dot, directories, and other special files
+ are excluded.
+ </para>
+ <para>
+ This function is restricted to superusers and members of
+ the <literal>pg_monitor</literal> role by default, but other users can
+ be granted EXECUTE to run the function.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_ls_replslotdir</primary>
+ </indexterm>
+ <function>pg_ls_replslotdir</function> ( <parameter>slot_name</parameter> <type>text</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>size</parameter> <type>bigint</type>,
+ <parameter>modification</parameter> <type>timestamp with time zone</type> )
+ </para>
+ <para>
+ Returns the name, size, and last modification time (mtime) of each
+ ordinary file in the server's pg_replslot/<parameter>slot_name</parameter> directory.
+ Filenames beginning with a dot, directories, and other special files
+ are excluded.
+ </para>
+ <para>
+ This function is restricted to superusers and members of
+ the <literal>pg_monitor</literal> role by default, but other users can
+ be granted EXECUTE to run the function.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 54c93b16c4..f6789025a5 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -701,6 +701,12 @@ REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public;
REVOKE EXECUTE ON FUNCTION pg_log_backend_memory_contexts(integer) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_ls_logicalsnapdir() FROM PUBLIC;
+
+REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
+
+REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+
--
-- We also set up some things as accessible to standard roles.
--
@@ -715,6 +721,12 @@ GRANT EXECUTE ON FUNCTION pg_ls_tmpdir() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_ls_tmpdir(oid) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_ls_logicalsnapdir() TO pg_monitor;
+
+GRANT EXECUTE ON FUNCTION pg_ls_logicalmapdir() TO pg_monitor;
+
+GRANT EXECUTE ON FUNCTION pg_ls_replslotdir(text) TO pg_monitor;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index c436d9318b..a83700c576 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -720,3 +720,37 @@ pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
{
return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
}
+
+/*
+ * Function to return the list of files in the pg_logical/snapshots directory.
+ */
+Datum
+pg_ls_logicalsnapdir(PG_FUNCTION_ARGS)
+{
+ return pg_ls_dir_files(fcinfo, "pg_logical/snapshots", false);
+}
+
+/*
+ * Function to return the list of files in the pg_logical/mappings directory.
+ */
+Datum
+pg_ls_logicalmapdir(PG_FUNCTION_ARGS)
+{
+ return pg_ls_dir_files(fcinfo, "pg_logical/mappings", false);
+}
+
+/*
+ * Function to return the list of files in the pg_replslot/<replication_slot>
+ * directory.
+ */
+Datum
+pg_ls_replslotdir(PG_FUNCTION_ARGS)
+{
+ text *filename_t = PG_GETARG_TEXT_PP(0);
+ char path[MAXPGPATH + 11];
+ char *filename;
+
+ filename = text_to_cstring(filename_t);
+ snprintf(path, sizeof(path), "%s/%s", "pg_replslot", filename);
+ return pg_ls_dir_files(fcinfo, path, false);
+}
diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h
index 9faf017457..f1c2c4d1e9 100644
--- a/src/include/catalog/catversion.h
+++ b/src/include/catalog/catversion.h
@@ -53,6 +53,6 @@
*/
/* yyyymmddN */
-#define CATALOG_VERSION_NO 202110272
+#define CATALOG_VERSION_NO 202110300
#endif
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index d068d6532e..ca0d6ca1d1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11619,6 +11619,21 @@
proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
proargnames => '{tablespace,name,size,modification}',
prosrc => 'pg_ls_tmpdir_1arg' },
+{ oid => '4642', descr => 'list of files in the pg_logical/snapshots directory',
+ proname => 'pg_ls_logicalsnapdir', procost => '10', prorows => '20', proretset => 't',
+ provolatile => 'v', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
+ proargnames => '{name,size,modification}', prosrc => 'pg_ls_logicalsnapdir' },
+{ oid => '4643', descr => 'list of files in the pg_logical/mappings directory',
+ proname => 'pg_ls_logicalmapdir', procost => '10', prorows => '20', proretset => 't',
+ provolatile => 'v', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
+ proargnames => '{name,size,modification}', prosrc => 'pg_ls_logicalmapdir' },
+{ oid => '4644', descr => 'list of files in the pg_logical/mappings directory',
+ proname => 'pg_ls_replslotdir', procost => '10', prorows => '20', proretset => 't',
+ provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+ proallargtypes => '{text,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
+ proargnames => '{slot_name,name,size,modification}', prosrc => 'pg_ls_replslotdir' },
# hash partitioning constraint function
{ oid => '5028', descr => 'hash partition CHECK constraint',
--
2.25.1
On 10/30/21, 2:36 AM, "Bharath Rupireddy" <bharath.rupireddyforpostgres@gmail.com> wrote:
I've added 3 functions pg_ls_logicalsnapdir, pg_ls_logicalmapdir,
pg_ls_replslotdir, and attached the patch. The sample output looks
like [1]. Please review it further.
I took a look at the patch.
+ char path[MAXPGPATH + 11];
Why are you adding 11 to MAXPGPATH here? I would think that MAXPGPATH
is sufficient.
+ filename = text_to_cstring(filename_t);
+ snprintf(path, sizeof(path), "%s/%s", "pg_replslot", filename);
+ return pg_ls_dir_files(fcinfo, path, false);
I think we need to do some additional input validation here. It's
pretty easy to use this to see the contents of other directories.
postgres=# SELECT * FROM pg_ls_replslotdir('../');
name | size | modification
----------------------+-------+------------------------
postgresql.conf | 28995 | 2021-11-17 18:40:33+00
pg_hba.conf | 4789 | 2021-11-17 18:40:33+00
postmaster.opts | 39 | 2021-11-17 18:43:07+00
postgresql.auto.conf | 88 | 2021-11-17 18:40:33+00
pg_ident.conf | 1636 | 2021-11-17 18:40:33+00
postmaster.pid | 95 | 2021-11-17 18:43:07+00
PG_VERSION | 3 | 2021-11-17 18:40:33+00
(7 rows)
Nathan
On Wed, Nov 17, 2021 at 06:46:47PM +0000, Bossart, Nathan wrote:
On 10/30/21, 2:36 AM, "Bharath Rupireddy" <bharath.rupireddyforpostgres@gmail.com> wrote:
I've added 3 functions pg_ls_logicalsnapdir, pg_ls_logicalmapdir,
pg_ls_replslotdir, and attached the patch. The sample output looks
like [1]. Please review it further.I took a look at the patch.
+ char path[MAXPGPATH + 11]; + filename = text_to_cstring(filename_t); + snprintf(path, sizeof(path), "%s/%s", "pg_replslot", filename); + return pg_ls_dir_files(fcinfo, path, false);Why are you adding 11 to MAXPGPATH here? I would think that MAXPGPATH
is sufficient.
I suppose it's for "pg_replslot" - but it forgot about the "/".
MAXPGPATH isn't sufficient (even if you add 12), since it's a user-supplied
string. snprintf keeps it from overflowing the buffer, but its return value
isn't checked, so it could (hypothetically) return a result for the wrong slot,
if the slot name were very long, or MAXPGPATH were very short..
+ text *filename_t = PG_GETARG_TEXT_PP(0);
I think we need to do some additional input validation here. It's
pretty easy to use this to see the contents of other directories.
Actually, limiting the dir seems like a valid reason to add this function,
since it would allow GRANTing privileges for just those directories.
So now I agree that this patch should be included. But I suggest to add it
after my "ls" patches, which change the output fields, and include directories.
Directories might normally not be present, but an extension might put them
there. (And it may be important to show things that aren't supposed to be
there, too).
--
Justin
On Thu, Nov 18, 2021 at 12:16 AM Bossart, Nathan <bossartn@amazon.com> wrote:
On 10/30/21, 2:36 AM, "Bharath Rupireddy" <bharath.rupireddyforpostgres@gmail.com> wrote:
I've added 3 functions pg_ls_logicalsnapdir, pg_ls_logicalmapdir,
pg_ls_replslotdir, and attached the patch. The sample output looks
like [1]. Please review it further.I took a look at the patch.
+ char path[MAXPGPATH + 11];
Why are you adding 11 to MAXPGPATH here? I would think that MAXPGPATH
is sufficient.
Yeah, MAXPGPATH is sufficient. Note that the replication slot name be
at max NAMEDATALEN(64 bytes) size
(ReplicationSlotPersistentData->name) and what we pass to the
pg_ls_dir_files is
"pg_replslot/<<user_entered_slot_name_with_max_64_bytes>>", so it can
never cross MAXPGPATH (1024).
+ filename = text_to_cstring(filename_t); + snprintf(path, sizeof(path), "%s/%s", "pg_replslot", filename); + return pg_ls_dir_files(fcinfo, path, false);I think we need to do some additional input validation here. It's
pretty easy to use this to see the contents of other directories.
Done. Checking if the entered slot exists or not, if not throwing an
error, see [1]postgres=# select * from pg_ls_replslotdir(''); ERROR: replication slot "" does not exist postgres=# select * from pg_ls_replslotdir('../'); ERROR: replication slot "../" does not exist postgres=# select pg_ls_replslotdir('mysub'); pg_ls_replslotdir ----------------------------------------------------------------- (xid-722-lsn-0-2000000.spill,36592640,"2021-11-18 07:34:40+00") (xid-722-lsn-0-5000000.spill,36592640,"2021-11-18 07:34:43+00") (xid-722-lsn-0-A000000.spill,29910720,"2021-11-18 07:34:48+00") (xid-722-lsn-0-7000000.spill,36592640,"2021-11-18 07:34:45+00") (xid-722-lsn-0-9000000.spill,36592640,"2021-11-18 07:34:47+00") (state,200,"2021-11-18 07:34:36+00") (xid-722-lsn-0-8000000.spill,36592500,"2021-11-18 07:34:46+00") (xid-722-lsn-0-6000000.spill,36592640,"2021-11-18 07:34:44+00") (xid-722-lsn-0-1000000.spill,11171300,"2021-11-18 07:34:39+00") (xid-722-lsn-0-4000000.spill,36592500,"2021-11-18 07:34:42+00") (xid-722-lsn-0-3000000.spill,36592640,"2021-11-18 07:34:42+00") (11 rows).
Please review the attached v2.
[1]: postgres=# select * from pg_ls_replslotdir(''); ERROR: replication slot "" does not exist postgres=# select * from pg_ls_replslotdir('../'); ERROR: replication slot "../" does not exist postgres=# select pg_ls_replslotdir('mysub'); pg_ls_replslotdir ----------------------------------------------------------------- (xid-722-lsn-0-2000000.spill,36592640,"2021-11-18 07:34:40+00") (xid-722-lsn-0-5000000.spill,36592640,"2021-11-18 07:34:43+00") (xid-722-lsn-0-A000000.spill,29910720,"2021-11-18 07:34:48+00") (xid-722-lsn-0-7000000.spill,36592640,"2021-11-18 07:34:45+00") (xid-722-lsn-0-9000000.spill,36592640,"2021-11-18 07:34:47+00") (state,200,"2021-11-18 07:34:36+00") (xid-722-lsn-0-8000000.spill,36592500,"2021-11-18 07:34:46+00") (xid-722-lsn-0-6000000.spill,36592640,"2021-11-18 07:34:44+00") (xid-722-lsn-0-1000000.spill,11171300,"2021-11-18 07:34:39+00") (xid-722-lsn-0-4000000.spill,36592500,"2021-11-18 07:34:42+00") (xid-722-lsn-0-3000000.spill,36592640,"2021-11-18 07:34:42+00") (11 rows)
postgres=# select * from pg_ls_replslotdir('');
ERROR: replication slot "" does not exist
postgres=# select * from pg_ls_replslotdir('../');
ERROR: replication slot "../" does not exist
postgres=# select pg_ls_replslotdir('mysub');
pg_ls_replslotdir
-----------------------------------------------------------------
(xid-722-lsn-0-2000000.spill,36592640,"2021-11-18 07:34:40+00")
(xid-722-lsn-0-5000000.spill,36592640,"2021-11-18 07:34:43+00")
(xid-722-lsn-0-A000000.spill,29910720,"2021-11-18 07:34:48+00")
(xid-722-lsn-0-7000000.spill,36592640,"2021-11-18 07:34:45+00")
(xid-722-lsn-0-9000000.spill,36592640,"2021-11-18 07:34:47+00")
(state,200,"2021-11-18 07:34:36+00")
(xid-722-lsn-0-8000000.spill,36592500,"2021-11-18 07:34:46+00")
(xid-722-lsn-0-6000000.spill,36592640,"2021-11-18 07:34:44+00")
(xid-722-lsn-0-1000000.spill,11171300,"2021-11-18 07:34:39+00")
(xid-722-lsn-0-4000000.spill,36592500,"2021-11-18 07:34:42+00")
(xid-722-lsn-0-3000000.spill,36592640,"2021-11-18 07:34:42+00")
(11 rows)
Regards,
Bharath Rupireddy.
Attachments:
v2-0001-Add-pg_ls_logicalsnapdir-pg_ls_logicalmapdir-pg_l.patchapplication/octet-stream; name=v2-0001-Add-pg_ls_logicalsnapdir-pg_ls_logicalmapdir-pg_l.patchDownload
From b8ecfbf1a1987b57d4735a2fea6b8f4f8dfd5b5d Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Thu, 18 Nov 2021 07:35:51 +0000
Subject: [PATCH v2] Add pg_ls_logicalsnapdir, pg_ls_logicalmapdir,
pg_ls_replslotdir functions
These functions lists the contents of the respective directories,
and are intended to be used by monitoring tools. Unlike pg_ls_dir(),
access to it can be granted to non-superusers so that those monitoring
tools can observe the principle of least privilege. Access is also
given by default to members of pg_monitor.
Note: Haven't changed the CATALOG_VERSION_NO as it is creating the
conflicts with the committed code, so the committer can take care
of it while committing this patch.
---
doc/src/sgml/func.sgml | 72 ++++++++++++++++++++++++
src/backend/catalog/system_functions.sql | 12 ++++
src/backend/utils/adt/genfile.c | 47 ++++++++++++++++
src/include/catalog/pg_proc.dat | 15 +++++
4 files changed, 146 insertions(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 24447c0017..5e7eb76023 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -27417,6 +27417,78 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
can be granted EXECUTE to run the function.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_ls_logicalsnapdir</primary>
+ </indexterm>
+ <function>pg_ls_logicalsnapdir</function> ()
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>size</parameter> <type>bigint</type>,
+ <parameter>modification</parameter> <type>timestamp with time zone</type> )
+ </para>
+ <para>
+ Returns the name, size, and last modification time (mtime) of each
+ ordinary file in the server's pg_logical/snapshots directory.
+ Filenames beginning with a dot, directories, and other special files
+ are excluded.
+ </para>
+ <para>
+ This function is restricted to superusers and members of
+ the <literal>pg_monitor</literal> role by default, but other users can
+ be granted EXECUTE to run the function.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_ls_logicalmapdir</primary>
+ </indexterm>
+ <function>pg_ls_logicalmapdir</function> ()
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>size</parameter> <type>bigint</type>,
+ <parameter>modification</parameter> <type>timestamp with time zone</type> )
+ </para>
+ <para>
+ Returns the name, size, and last modification time (mtime) of each
+ ordinary file in the server's pg_logical/mappings directory.
+ Filenames beginning with a dot, directories, and other special files
+ are excluded.
+ </para>
+ <para>
+ This function is restricted to superusers and members of
+ the <literal>pg_monitor</literal> role by default, but other users can
+ be granted EXECUTE to run the function.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_ls_replslotdir</primary>
+ </indexterm>
+ <function>pg_ls_replslotdir</function> ( <parameter>slot_name</parameter> <type>text</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>size</parameter> <type>bigint</type>,
+ <parameter>modification</parameter> <type>timestamp with time zone</type> )
+ </para>
+ <para>
+ Returns the name, size, and last modification time (mtime) of each
+ ordinary file in the server's pg_replslot/<parameter>slot_name</parameter> directory.
+ Filenames beginning with a dot, directories, and other special files
+ are excluded.
+ </para>
+ <para>
+ This function is restricted to superusers and members of
+ the <literal>pg_monitor</literal> role by default, but other users can
+ be granted EXECUTE to run the function.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 54c93b16c4..f6789025a5 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -701,6 +701,12 @@ REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public;
REVOKE EXECUTE ON FUNCTION pg_log_backend_memory_contexts(integer) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_ls_logicalsnapdir() FROM PUBLIC;
+
+REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
+
+REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+
--
-- We also set up some things as accessible to standard roles.
--
@@ -715,6 +721,12 @@ GRANT EXECUTE ON FUNCTION pg_ls_tmpdir() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_ls_tmpdir(oid) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_ls_logicalsnapdir() TO pg_monitor;
+
+GRANT EXECUTE ON FUNCTION pg_ls_logicalmapdir() TO pg_monitor;
+
+GRANT EXECUTE ON FUNCTION pg_ls_replslotdir(text) TO pg_monitor;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index c436d9318b..04666589d3 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -29,6 +29,7 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "postmaster/syslogger.h"
+#include "replication/slot.h"
#include "storage/fd.h"
#include "utils/acl.h"
#include "utils/builtins.h"
@@ -720,3 +721,49 @@ pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
{
return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
}
+
+/*
+ * Function to return the list of files in the pg_logical/snapshots directory.
+ */
+Datum
+pg_ls_logicalsnapdir(PG_FUNCTION_ARGS)
+{
+ return pg_ls_dir_files(fcinfo, "pg_logical/snapshots", false);
+}
+
+/*
+ * Function to return the list of files in the pg_logical/mappings directory.
+ */
+Datum
+pg_ls_logicalmapdir(PG_FUNCTION_ARGS)
+{
+ return pg_ls_dir_files(fcinfo, "pg_logical/mappings", false);
+}
+
+/*
+ * Function to return the list of files in the pg_replslot/<replication_slot>
+ * directory.
+ */
+Datum
+pg_ls_replslotdir(PG_FUNCTION_ARGS)
+{
+ text *slotname_t;
+ char path[MAXPGPATH];
+ char *slotname;
+ ReplicationSlot *slot;
+
+ slotname_t = PG_GETARG_TEXT_PP(0);
+
+ slotname = text_to_cstring(slotname_t);
+
+ slot = SearchNamedReplicationSlot(slotname, true);
+
+ if (!slot)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("replication slot \"%s\" does not exist",
+ slotname)));
+
+ snprintf(path, sizeof(path), "%s/%s", "pg_replslot", slotname);
+ return pg_ls_dir_files(fcinfo, path, false);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6412f369f1..509b9824fd 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11623,6 +11623,21 @@
proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
proargnames => '{tablespace,name,size,modification}',
prosrc => 'pg_ls_tmpdir_1arg' },
+{ oid => '4642', descr => 'list of files in the pg_logical/snapshots directory',
+ proname => 'pg_ls_logicalsnapdir', procost => '10', prorows => '20', proretset => 't',
+ provolatile => 'v', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
+ proargnames => '{name,size,modification}', prosrc => 'pg_ls_logicalsnapdir' },
+{ oid => '4643', descr => 'list of files in the pg_logical/mappings directory',
+ proname => 'pg_ls_logicalmapdir', procost => '10', prorows => '20', proretset => 't',
+ provolatile => 'v', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
+ proargnames => '{name,size,modification}', prosrc => 'pg_ls_logicalmapdir' },
+{ oid => '4644', descr => 'list of files in the pg_logical/mappings directory',
+ proname => 'pg_ls_replslotdir', procost => '10', prorows => '20', proretset => 't',
+ provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+ proallargtypes => '{text,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
+ proargnames => '{slot_name,name,size,modification}', prosrc => 'pg_ls_replslotdir' },
# hash partitioning constraint function
{ oid => '5028', descr => 'hash partition CHECK constraint',
--
2.25.1
On 11/17/21, 11:39 PM, "Bharath Rupireddy" <bharath.rupireddyforpostgres@gmail.com> wrote:
Please review the attached v2.
LGTM. I've marked this one as ready-for-committer.
Nathan
On Sat, Nov 20, 2021 at 12:29:51AM +0000, Bossart, Nathan wrote:
On 11/17/21, 11:39 PM, "Bharath Rupireddy" <bharath.rupireddyforpostgres@gmail.com> wrote:
Please review the attached v2.
LGTM. I've marked this one as ready-for-committer.
One issue that I have with this patch is that there are zero
regression tests. Could you add a couple of things in
misc_functions.sql (for the negative tests perhaps) or
contrib/test_decoding/, taking advantage of places where slots are
already created? You may want to look after the non-superuser case
where the calls should fail, and the second case where a role is part
of pg_monitor where the call succeeds. Note that any roles created in
the tests have to be prefixed with "regress_".
+ snprintf(path, sizeof(path), "%s/%s", "pg_replslot", slotname);
+ return pg_ls_dir_files(fcinfo, path, false);
"pg_replslot" could be part of the third argument here. There is no
need to separate it.
+ ordinary file in the server's pg_logical/mappings directory.
Paths had better have <filename> markups around them, no?
--
Michael
On Sun, Nov 21, 2021 at 6:58 AM Michael Paquier <michael@paquier.xyz> wrote:
On Sat, Nov 20, 2021 at 12:29:51AM +0000, Bossart, Nathan wrote:
On 11/17/21, 11:39 PM, "Bharath Rupireddy" <bharath.rupireddyforpostgres@gmail.com> wrote:
Please review the attached v2.
LGTM. I've marked this one as ready-for-committer.
One issue that I have with this patch is that there are zero
regression tests. Could you add a couple of things in
misc_functions.sql (for the negative tests perhaps) or
contrib/test_decoding/, taking advantage of places where slots are
already created? You may want to look after the non-superuser case
where the calls should fail, and the second case where a role is part
of pg_monitor where the call succeeds. Note that any roles created in
the tests have to be prefixed with "regress_".
I don't think we need to go far to contrib/test_decoding/, even if we
add it there we can't test it for the outputs of these functions, so
I've added the tests in misc_functinos.sql itself.
+ snprintf(path, sizeof(path), "%s/%s", "pg_replslot", slotname); + return pg_ls_dir_files(fcinfo, path, false); "pg_replslot" could be part of the third argument here. There is no need to separate it.
Done.
+ ordinary file in the server's pg_logical/mappings directory.
Paths had better have <filename> markups around them, no?
Done.
Attached v3 patch, please review it further.
Regards,
Bharath Rupireddy.
Attachments:
v3-0001-Add-pg_ls_logicalsnapdir-pg_ls_logicalmapdir-pg_l.patchapplication/octet-stream; name=v3-0001-Add-pg_ls_logicalsnapdir-pg_ls_logicalmapdir-pg_l.patchDownload
From 73f756b9b961588c4dbd732cc4fc0493e8e0f2a3 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Sun, 21 Nov 2021 03:13:18 +0000
Subject: [PATCH v3] Add pg_ls_logicalsnapdir, pg_ls_logicalmapdir,
pg_ls_replslotdir functions
These functions lists the contents of the respective directories,
and are intended to be used by monitoring tools. Unlike pg_ls_dir(),
access to it can be granted to non-superusers so that those monitoring
tools can observe the principle of least privilege. Access is also
given by default to members of pg_monitor.
Note: Bump the CATALOG_VERSION_NO
---
doc/src/sgml/func.sgml | 73 ++++++++++++++++++++
src/backend/catalog/system_functions.sql | 12 ++++
src/backend/utils/adt/genfile.c | 47 +++++++++++++
src/include/catalog/pg_proc.dat | 15 ++++
src/test/regress/expected/misc_functions.out | 65 +++++++++++++++++
src/test/regress/sql/misc_functions.sql | 31 +++++++++
6 files changed, 243 insertions(+)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 24447c0017..5e2a7753a5 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -27417,6 +27417,79 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');
can be granted EXECUTE to run the function.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_ls_logicalsnapdir</primary>
+ </indexterm>
+ <function>pg_ls_logicalsnapdir</function> ()
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>size</parameter> <type>bigint</type>,
+ <parameter>modification</parameter> <type>timestamp with time zone</type> )
+ </para>
+ <para>
+ Returns the name, size, and last modification time (mtime) of each
+ ordinary file in the server's <filename>pg_logical/snapshots</filename>
+ directory. Filenames beginning with a dot, directories, and other
+ special files are excluded.
+ </para>
+ <para>
+ This function is restricted to superusers and members of
+ the <literal>pg_monitor</literal> role by default, but other users can
+ be granted EXECUTE to run the function.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_ls_logicalmapdir</primary>
+ </indexterm>
+ <function>pg_ls_logicalmapdir</function> ()
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>size</parameter> <type>bigint</type>,
+ <parameter>modification</parameter> <type>timestamp with time zone</type> )
+ </para>
+ <para>
+ Returns the name, size, and last modification time (mtime) of each
+ ordinary file in the server's <filename>pg_logical/mappings</filename>
+ directory. Filenames beginning with a dot, directories, and other
+ special files are excluded.
+ </para>
+ <para>
+ This function is restricted to superusers and members of
+ the <literal>pg_monitor</literal> role by default, but other users can
+ be granted EXECUTE to run the function.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_ls_replslotdir</primary>
+ </indexterm>
+ <function>pg_ls_replslotdir</function> ( <parameter>slot_name</parameter> <type>text</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>size</parameter> <type>bigint</type>,
+ <parameter>modification</parameter> <type>timestamp with time zone</type> )
+ </para>
+ <para>
+ Returns the name, size, and last modification time (mtime) of each
+ ordinary file in the server's <filename>pg_replslot/slot_name</filename>
+ (where <parameter>slot_name</parameter> is name of the replication slot
+ provided as an input to the function) directory. Filenames beginning
+ with a dot, directories, and other special files are excluded.
+ </para>
+ <para>
+ This function is restricted to superusers and members of
+ the <literal>pg_monitor</literal> role by default, but other users can
+ be granted EXECUTE to run the function.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 54c93b16c4..f6789025a5 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -701,6 +701,12 @@ REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public;
REVOKE EXECUTE ON FUNCTION pg_log_backend_memory_contexts(integer) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_ls_logicalsnapdir() FROM PUBLIC;
+
+REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
+
+REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+
--
-- We also set up some things as accessible to standard roles.
--
@@ -715,6 +721,12 @@ GRANT EXECUTE ON FUNCTION pg_ls_tmpdir() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_ls_tmpdir(oid) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_ls_logicalsnapdir() TO pg_monitor;
+
+GRANT EXECUTE ON FUNCTION pg_ls_logicalmapdir() TO pg_monitor;
+
+GRANT EXECUTE ON FUNCTION pg_ls_replslotdir(text) TO pg_monitor;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index c436d9318b..6805a00d2c 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -29,6 +29,7 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "postmaster/syslogger.h"
+#include "replication/slot.h"
#include "storage/fd.h"
#include "utils/acl.h"
#include "utils/builtins.h"
@@ -720,3 +721,49 @@ pg_ls_archive_statusdir(PG_FUNCTION_ARGS)
{
return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true);
}
+
+/*
+ * Function to return the list of files in the pg_logical/snapshots directory.
+ */
+Datum
+pg_ls_logicalsnapdir(PG_FUNCTION_ARGS)
+{
+ return pg_ls_dir_files(fcinfo, "pg_logical/snapshots", false);
+}
+
+/*
+ * Function to return the list of files in the pg_logical/mappings directory.
+ */
+Datum
+pg_ls_logicalmapdir(PG_FUNCTION_ARGS)
+{
+ return pg_ls_dir_files(fcinfo, "pg_logical/mappings", false);
+}
+
+/*
+ * Function to return the list of files in the pg_replslot/<replication_slot>
+ * directory.
+ */
+Datum
+pg_ls_replslotdir(PG_FUNCTION_ARGS)
+{
+ text *slotname_t;
+ char path[MAXPGPATH];
+ char *slotname;
+ ReplicationSlot *slot;
+
+ slotname_t = PG_GETARG_TEXT_PP(0);
+
+ slotname = text_to_cstring(slotname_t);
+
+ slot = SearchNamedReplicationSlot(slotname, true);
+
+ if (!slot)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("replication slot \"%s\" does not exist",
+ slotname)));
+
+ snprintf(path, sizeof(path), "pg_replslot/%s", slotname);
+ return pg_ls_dir_files(fcinfo, path, false);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6412f369f1..509b9824fd 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11623,6 +11623,21 @@
proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
proargnames => '{tablespace,name,size,modification}',
prosrc => 'pg_ls_tmpdir_1arg' },
+{ oid => '4642', descr => 'list of files in the pg_logical/snapshots directory',
+ proname => 'pg_ls_logicalsnapdir', procost => '10', prorows => '20', proretset => 't',
+ provolatile => 'v', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
+ proargnames => '{name,size,modification}', prosrc => 'pg_ls_logicalsnapdir' },
+{ oid => '4643', descr => 'list of files in the pg_logical/mappings directory',
+ proname => 'pg_ls_logicalmapdir', procost => '10', prorows => '20', proretset => 't',
+ provolatile => 'v', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}',
+ proargnames => '{name,size,modification}', prosrc => 'pg_ls_logicalmapdir' },
+{ oid => '4644', descr => 'list of files in the pg_logical/mappings directory',
+ proname => 'pg_ls_replslotdir', procost => '10', prorows => '20', proretset => 't',
+ provolatile => 'v', prorettype => 'record', proargtypes => 'text',
+ proallargtypes => '{text,text,int8,timestamptz}', proargmodes => '{i,o,o,o}',
+ proargnames => '{slot_name,name,size,modification}', prosrc => 'pg_ls_replslotdir' },
# hash partitioning constraint function
{ oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index 71d316cad3..dbe9e57ddb 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -243,6 +243,71 @@ select count(*) > 0 from
t
(1 row)
+--
+-- Test replication slot directory functions
+--
+-- The outputs of these are variable, so we can't just print their results
+-- directly, but we can at least verify that the code doesn't fail, and that
+-- the permissions are set properly.
+--
+CREATE ROLE regress_slot_dir_funcs;
+SELECT has_function_privilege('regress_slot_dir_funcs',
+ 'pg_ls_logicalsnapdir()', 'EXECUTE'); -- no
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+SELECT has_function_privilege('regress_slot_dir_funcs',
+ 'pg_ls_logicalmapdir()', 'EXECUTE'); -- no
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+SELECT has_function_privilege('regress_slot_dir_funcs',
+ 'pg_ls_replslotdir(text)', 'EXECUTE'); -- no
+ has_function_privilege
+------------------------
+ f
+(1 row)
+
+GRANT pg_monitor TO regress_slot_dir_funcs;
+SELECT 'init' FROM pg_create_physical_replication_slot('slot_dir_funcs');
+ ?column?
+----------
+ init
+(1 row)
+
+SET ROLE regress_slot_dir_funcs;
+SELECT COUNT(*) >= 0 AS OK FROM pg_ls_logicalsnapdir();
+ ok
+----
+ t
+(1 row)
+
+SELECT COUNT(*) >= 0 AS OK FROM pg_ls_logicalmapdir();
+ ok
+----
+ t
+(1 row)
+
+SELECT COUNT(*) >= 0 AS OK FROM pg_ls_replslotdir('slot_dir_funcs');
+ ok
+----
+ t
+(1 row)
+
+SELECT pg_ls_replslotdir('non_existent_slot'); -- ERROR
+ERROR: replication slot "non_existent_slot" does not exist
+RESET ROLE;
+SELECT pg_drop_replication_slot('slot_dir_funcs');
+ pg_drop_replication_slot
+--------------------------
+
+(1 row)
+
+DROP ROLE regress_slot_dir_funcs;
--
-- Test adding a support function to a subject function
--
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 8c23874b3f..c47d013624 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -91,6 +91,37 @@ select count(*) > 0 from
where spcname = 'pg_default') pts
join pg_database db on pts.pts = db.oid;
+--
+-- Test replication slot directory functions
+--
+-- The outputs of these are variable, so we can't just print their results
+-- directly, but we can at least verify that the code doesn't fail, and that
+-- the permissions are set properly.
+--
+
+CREATE ROLE regress_slot_dir_funcs;
+
+SELECT has_function_privilege('regress_slot_dir_funcs',
+ 'pg_ls_logicalsnapdir()', 'EXECUTE'); -- no
+SELECT has_function_privilege('regress_slot_dir_funcs',
+ 'pg_ls_logicalmapdir()', 'EXECUTE'); -- no
+SELECT has_function_privilege('regress_slot_dir_funcs',
+ 'pg_ls_replslotdir(text)', 'EXECUTE'); -- no
+
+GRANT pg_monitor TO regress_slot_dir_funcs;
+
+SELECT 'init' FROM pg_create_physical_replication_slot('slot_dir_funcs');
+
+SET ROLE regress_slot_dir_funcs;
+SELECT COUNT(*) >= 0 AS OK FROM pg_ls_logicalsnapdir();
+SELECT COUNT(*) >= 0 AS OK FROM pg_ls_logicalmapdir();
+SELECT COUNT(*) >= 0 AS OK FROM pg_ls_replslotdir('slot_dir_funcs');
+SELECT pg_ls_replslotdir('non_existent_slot'); -- ERROR
+RESET ROLE;
+
+SELECT pg_drop_replication_slot('slot_dir_funcs');
+DROP ROLE regress_slot_dir_funcs;
+
--
-- Test adding a support function to a subject function
--
--
2.25.1
On Sun, Nov 21, 2021 at 08:45:52AM +0530, Bharath Rupireddy wrote:
I don't think we need to go far to contrib/test_decoding/, even if we
add it there we can't test it for the outputs of these functions, so
I've added the tests in misc_functinos.sql itself.
+SELECT COUNT(*) >= 0 AS OK FROM pg_ls_replslotdir('slot_dir_funcs');
+ ok
+----
+ t
+(1 row)
Creating a slot within the main regression test suite is something we
should avoid as it impacts the portability of the tests (note that we
don't have tests creating slots in src/test/regress/, and we'd require
max_replication_slots > 0 with this version of the patch). This was
the point I was trying to make upthread about using test_decoding/
where we already have slots.
A second thing I have noticed is the set of OIDs used by the patch
which was incorrect. On a development branch, we require new features
to use OIDs between 8000-9999 (unused_oids would recommend a random
range of them). A third thing was that pg_proc.dat had an incorrect
description for pg_ls_replslotdir(), and that it was in need of
indentation.
I have tweaked a bit the tests and the docs, and the result looked
fine at the end. Hence, applied.
--
Michael