pg_replslotdata - a tool for displaying replication slot information

Started by Bharath Rupireddyabout 4 years ago17 messages
#1Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
2 attachment(s)

Hi,

The replication slots data is stored in binary format on the disk under the
pg_replslot/<<slot_name>> directory which isn't human readable. If the
server is crashed/down (for whatever reasons) and unable to come up,
currently there's no way for the user/admin/developer to know what were all
the replication slots available at the time of server crash/down to figure
out what's the restart lsn, xid, two phase info or types of slots etc.

pg_replslotdata is a tool that interprets the replication slots information
and displays it onto the stdout even if the server is crashed/down. The
design of this tool is similar to other tools available in the core today
i.e. pg_controldata, pg_waldump.

Attaching initial patch herewith. I will improve it with documentation and
other stuff a bit later.

Please see the attached picture for the sample output.

Thoughts?

Regards,
Bharath Rupireddy.

Attachments:

pg_replslotdata_sample_output.PNGimage/png; name=pg_replslotdata_sample_output.PNGDownload
v1-0001-pg_replslotdata.patchapplication/x-patch; name=v1-0001-pg_replslotdata.patchDownload
From dd31d7ff207b5ac98adb0bac18fdcaa5bf9883ff Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Sun, 21 Nov 2021 13:47:56 +0000
Subject: [PATCH v1] pg_replslotdata

---
 src/backend/replication/slot.c            |  47 +--
 src/bin/Makefile                          |   1 +
 src/bin/pg_replslotdata/.gitignore        |   2 +
 src/bin/pg_replslotdata/Makefile          |  44 +++
 src/bin/pg_replslotdata/nls.mk            |   6 +
 src/bin/pg_replslotdata/pg_replslotdata.c | 364 ++++++++++++++++++++++
 src/bin/pg_replslotdata/t/001_basic.pl    |  11 +
 src/include/replication/slot.h            |  82 +----
 src/include/replication/slot_common.h     | 132 ++++++++
 9 files changed, 565 insertions(+), 124 deletions(-)
 create mode 100644 src/bin/pg_replslotdata/.gitignore
 create mode 100644 src/bin/pg_replslotdata/Makefile
 create mode 100644 src/bin/pg_replslotdata/nls.mk
 create mode 100644 src/bin/pg_replslotdata/pg_replslotdata.c
 create mode 100644 src/bin/pg_replslotdata/t/001_basic.pl
 create mode 100644 src/include/replication/slot_common.h

diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index e4f5c6fb36..236ebe999a 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -50,45 +50,6 @@
 #include "storage/procarray.h"
 #include "utils/builtins.h"
 
-/*
- * Replication slot on-disk data structure.
- */
-typedef struct ReplicationSlotOnDisk
-{
-	/* first part of this struct needs to be version independent */
-
-	/* data not covered by checksum */
-	uint32		magic;
-	pg_crc32c	checksum;
-
-	/* data covered by checksum */
-	uint32		version;
-	uint32		length;
-
-	/*
-	 * The actual data in the slot that follows can differ based on the above
-	 * 'version'.
-	 */
-
-	ReplicationSlotPersistentData slotdata;
-} ReplicationSlotOnDisk;
-
-/* size of version independent data */
-#define ReplicationSlotOnDiskConstantSize \
-	offsetof(ReplicationSlotOnDisk, slotdata)
-/* size of the part of the slot not covered by the checksum */
-#define SnapBuildOnDiskNotChecksummedSize \
-	offsetof(ReplicationSlotOnDisk, version)
-/* size of the part covered by the checksum */
-#define SnapBuildOnDiskChecksummedSize \
-	sizeof(ReplicationSlotOnDisk) - SnapBuildOnDiskNotChecksummedSize
-/* size of the slot data that is version dependent */
-#define ReplicationSlotOnDiskV2Size \
-	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskConstantSize
-
-#define SLOT_MAGIC		0x1051CA1	/* format identifier */
-#define SLOT_VERSION	2		/* version for new files */
-
 /* Control array for replication slot management */
 ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
 
@@ -1571,8 +1532,8 @@ SaveSlotToPath(ReplicationSlot *slot, const char *dir, int elevel)
 	SpinLockRelease(&slot->mutex);
 
 	COMP_CRC32C(cp.checksum,
-				(char *) (&cp) + SnapBuildOnDiskNotChecksummedSize,
-				SnapBuildOnDiskChecksummedSize);
+				(char *) (&cp) + ReplicationSlotOnDiskNotChecksummedSize,
+				ReplicationSlotOnDiskChecksummedSize);
 	FIN_CRC32C(cp.checksum);
 
 	errno = 0;
@@ -1787,8 +1748,8 @@ RestoreSlotFromDisk(const char *name)
 	/* now verify the CRC */
 	INIT_CRC32C(checksum);
 	COMP_CRC32C(checksum,
-				(char *) &cp + SnapBuildOnDiskNotChecksummedSize,
-				SnapBuildOnDiskChecksummedSize);
+				(char *) &cp + ReplicationSlotOnDiskNotChecksummedSize,
+				ReplicationSlotOnDiskChecksummedSize);
 	FIN_CRC32C(checksum);
 
 	if (!EQ_CRC32C(checksum, cp.checksum))
diff --git a/src/bin/Makefile b/src/bin/Makefile
index 2fe0ae6652..5e20009beb 100644
--- a/src/bin/Makefile
+++ b/src/bin/Makefile
@@ -23,6 +23,7 @@ SUBDIRS = \
 	pg_controldata \
 	pg_ctl \
 	pg_dump \
+	pg_replslotdata \
 	pg_resetwal \
 	pg_rewind \
 	pg_test_fsync \
diff --git a/src/bin/pg_replslotdata/.gitignore b/src/bin/pg_replslotdata/.gitignore
new file mode 100644
index 0000000000..13a4afb8ef
--- /dev/null
+++ b/src/bin/pg_replslotdata/.gitignore
@@ -0,0 +1,2 @@
+/pg_replslotdata
+/tmp_check/
diff --git a/src/bin/pg_replslotdata/Makefile b/src/bin/pg_replslotdata/Makefile
new file mode 100644
index 0000000000..69518ee53b
--- /dev/null
+++ b/src/bin/pg_replslotdata/Makefile
@@ -0,0 +1,44 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/bin/pg_replslotdata
+#
+# Copyright (c) 1998-2021, PostgreSQL Global Development Group
+#
+# src/bin/pg_replslotdata/Makefile
+#
+#-------------------------------------------------------------------------
+
+PGFILEDESC = "pg_replslotdata - provides information about the replication slots from $PGDATA/pg_replslot/<slot_name> $PGDATA/pg_replslot/<slot_name>"
+PGAPPICON=win32
+
+subdir = src/bin/pg_replslotdata
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = \
+	$(WIN32RES) \
+	pg_replslotdata.o
+
+all: pg_replslotdata
+
+pg_replslotdata: $(OBJS) | submake-libpgport
+	$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
+install: all installdirs
+	$(INSTALL_PROGRAM) pg_replslotdata$(X) '$(DESTDIR)$(bindir)/pg_replslotdata$(X)'
+
+installdirs:
+	$(MKDIR_P) '$(DESTDIR)$(bindir)'
+
+uninstall:
+	rm -f '$(DESTDIR)$(bindir)/pg_replslotdata$(X)'
+
+clean distclean maintainer-clean:
+	rm -f pg_replslotdata$(X) $(OBJS)
+	rm -rf tmp_check
+
+check:
+	$(prove_check)
+
+installcheck:
+	$(prove_installcheck)
diff --git a/src/bin/pg_replslotdata/nls.mk b/src/bin/pg_replslotdata/nls.mk
new file mode 100644
index 0000000000..74bee593c9
--- /dev/null
+++ b/src/bin/pg_replslotdata/nls.mk
@@ -0,0 +1,6 @@
+# src/bin/pg_replslotdata/nls.mk
+CATALOG_NAME     = pg_replslotdata
+AVAIL_LANGUAGES  = cs de el es fr ja ko pl ru sv tr uk vi zh_CN
+GETTEXT_FILES    = $(FRONTEND_COMMON_GETTEXT_FILES) pg_replslotdata.c
+GETTEXT_TRIGGERS = $(FRONTEND_COMMON_GETTEXT_TRIGGERS)
+GETTEXT_FLAGS    = $(FRONTEND_COMMON_GETTEXT_FLAGS)
diff --git a/src/bin/pg_replslotdata/pg_replslotdata.c b/src/bin/pg_replslotdata/pg_replslotdata.c
new file mode 100644
index 0000000000..f9b4b56b0e
--- /dev/null
+++ b/src/bin/pg_replslotdata/pg_replslotdata.c
@@ -0,0 +1,364 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_replslotdata.c - provides information about the replication slots
+ * from $PGDATA/pg_replslot/<slot_name>.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ *		  src/bin/pg_replslotdata/pg_replslotdata.c
+ *-------------------------------------------------------------------------
+ */
+/*
+ * We have to use postgres.h not postgres_fe.h here, because there's so much
+ * backend-only stuff in the XLOG include files we need.  But we need a
+ * frontend-ish environment otherwise.  Hence this ugly hack.
+ */
+#define FRONTEND 1
+
+#include "postgres.h"
+
+#include <sys/stat.h>
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "getopt_long.h"
+#include "pg_getopt.h"
+#include "replication/slot_common.h"
+
+static bool verbose = false;
+
+static void process_replslots(void);
+static void read_and_display_repl_slot(const char *name);
+
+static void
+usage(const char *progname)
+{
+	printf(_("%s displays information about the replication slots from $PGDATA/pg_replslot/<slot_name>.\n\n"), progname);
+	printf(_("Usage:\n"));
+	printf(_("  %s [OPTION] [DATADIR]\n"), progname);
+	printf(_("\nOptions:\n"));
+	printf(_(" [-D, --pgdata=]DATADIR  data directory\n"));
+	printf(_("  -V, --version          output version information, then exit\n"));
+	printf(_("  -v, --verbose          write a lot of output\n"));
+	printf(_("  -?, --help             show this help, then exit\n"));
+	printf(_("\nIf no data directory (DATADIR) is specified, "
+			 "the environment variable PGDATA\nis used.\n\n"));
+	printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+	printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+static void
+process_replslots(void)
+{
+	DIR		   *rsdir;
+	struct dirent *rsde;
+	uint32		cnt = 0;
+
+	rsdir = opendir("pg_replslot");
+	if (rsdir == NULL)
+	{
+		pg_log_error("could not open directory \"%s\": %m", "pg_replslot");
+		exit(1);
+	}
+
+	/* XXX: comment here about the format spefiiers */
+	printf("%-64s %9s %10s %11s %10s %12s %21s %21s %21s %21s %10s %20s\n"
+		   "%-64s %9s %10s %11s %10s %12s %21s %21s %21s %21s %10s %20s\n",
+		   "slot_name", "slot_type", "datoid", "persistency", "xmin", "catalog_xmin", "restart_lsn", "invalidated_at", "confirmed_flush", "two_phase_at", "two_phase", "plugin",
+		   "---------", "---------", "------", "-----------", "----", "------------", "-----------", "--------------", "---------------", "------------", "---------", "------");
+
+	while (errno = 0, (rsde = readdir(rsdir)) != NULL)
+	{
+		struct stat statbuf;
+		char		path[MAXPGPATH];
+
+		if (strcmp(rsde->d_name, ".") == 0 ||
+			strcmp(rsde->d_name, "..") == 0)
+			continue;
+
+		snprintf(path, sizeof(path), "pg_replslot/%s", rsde->d_name);
+
+		/* we're only creating directories here, skip if it's not our's */
+		if (lstat(path, &statbuf) == 0 && !S_ISDIR(statbuf.st_mode))
+			continue;
+
+		/* we crashed while a slot was being setup or deleted, clean up */
+		if (pg_str_endswith(rsde->d_name, ".tmp"))
+		{
+			pg_log_warning("server was crashed while the slot \"%s\" was being setup or deleted",
+						   rsde->d_name);
+			continue;
+		}
+
+		/* looks like a slot in a normal state, restore */
+		read_and_display_repl_slot(rsde->d_name);
+		cnt++;
+	}
+
+	if (errno)
+	{
+		pg_log_error("could not read directory \"%s\": %m", "pg_replslot");
+		exit(1);
+	}
+
+	if (cnt == 0)
+	{
+		pg_log_info("no replication slots were found");
+		exit(0);
+	}
+
+	if (closedir(rsdir))
+	{
+		pg_log_error("could not close directory \"%s\": %m", "pg_replslot");
+		exit(1);
+	}
+}
+
+static void
+read_and_display_repl_slot(const char *name)
+{
+	ReplicationSlotOnDisk cp;
+	char		slotdir[MAXPGPATH];
+	char		path[MAXPGPATH];
+	char		restart_lsn[NAMEDATALEN];
+	char		invalidated_at[NAMEDATALEN];
+	char		confirmed_flush[NAMEDATALEN];
+	char		two_phase_at[NAMEDATALEN];
+	char		persistency[NAMEDATALEN];
+	int			fd;
+	int			readBytes;
+	pg_crc32c	checksum;
+
+	/* delete temp file if it exists */
+	sprintf(slotdir, "pg_replslot/%s", name);
+	sprintf(path, "%s/state.tmp", slotdir);
+
+	fd = open(path, O_RDONLY | PG_BINARY);
+
+	if (fd > 0)
+	{
+		pg_log_error("found temporary state file \"%s\": %m", path);
+		exit(1);
+	}
+
+	sprintf(path, "%s/state", slotdir);
+
+	if (verbose)
+		pg_log_info("reading replication slot from \"%s\"", path);
+
+	fd = open(path, O_RDONLY | PG_BINARY);
+
+	/*
+	 * We do not need to handle this as we are rename()ing the directory into
+	 * place only after we fsync()ed the state file.
+	 */
+	if (fd < 0)
+	{
+		pg_log_error("could not open file \"%s\": %m", path);
+		exit(1);
+	}
+
+	if (verbose)
+		pg_log_info("reading version independent replication slot state file");
+
+	/* read part of statefile that's guaranteed to be version independent */
+	readBytes = read(fd, &cp, ReplicationSlotOnDiskConstantSize);
+	if (readBytes != ReplicationSlotOnDiskConstantSize)
+	{
+		if (readBytes < 0)
+		{
+			pg_log_error("could not read file \"%s\": %m", path);
+			exit(1);
+		}
+		else
+		{
+			pg_log_error("could not read file \"%s\": read %d of %zu",
+						 path, readBytes,
+						 (Size) ReplicationSlotOnDiskConstantSize);
+			exit(1);
+		}
+	}
+
+	/* verify magic */
+	if (cp.magic != SLOT_MAGIC)
+	{
+		pg_log_error("replication slot file \"%s\" has wrong magic number: %u instead of %u",
+					 path, cp.magic, SLOT_MAGIC);
+		exit(1);
+	}
+
+	/* verify version */
+	if (cp.version != SLOT_VERSION)
+	{
+		pg_log_error("replication slot file \"%s\" has unsupported version %u",
+					 path, cp.version);
+		exit(1);
+	}
+
+	/* boundary check on length */
+	if (cp.length != ReplicationSlotOnDiskV2Size)
+	{
+		pg_log_error("replication slot file \"%s\" has corrupted length %u",
+					 path, cp.length);
+		exit(1);
+	}
+
+	if (verbose)
+		pg_log_info("reading the entire replication slot state file");
+
+	/* now that we know the size, read the entire file */
+	readBytes = read(fd,
+					 (char *) &cp + ReplicationSlotOnDiskConstantSize,
+					 cp.length);
+	if (readBytes != cp.length)
+	{
+		if (readBytes < 0)
+		{
+			pg_log_error("could not read file \"%s\": %m", path);
+			exit(1);
+		}
+		else
+		{
+			pg_log_error("could not read file \"%s\": read %d of %zu",
+						 path, readBytes, (Size) cp.length);
+			exit(1);
+		}
+	}
+
+	if (close(fd) != 0)
+	{
+		pg_log_error("could not close file \"%s\": %m", path);
+		exit(1);
+	}
+
+	/* now verify the CRC */
+	INIT_CRC32C(checksum);
+	COMP_CRC32C(checksum,
+				(char *) &cp + ReplicationSlotOnDiskNotChecksummedSize,
+				ReplicationSlotOnDiskChecksummedSize);
+	FIN_CRC32C(checksum);
+
+	if (!EQ_CRC32C(checksum, cp.checksum))
+	{
+		pg_log_error("checksum mismatch for replication slot file \"%s\": is %u, should be %u",
+					 path, checksum, cp.checksum);
+		exit(1);
+	}
+
+	sprintf(restart_lsn, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.restart_lsn));
+	sprintf(invalidated_at, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.invalidated_at));
+	sprintf(confirmed_flush, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.confirmed_flush));
+	sprintf(two_phase_at, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.two_phase_at));
+
+	if (cp.slotdata.persistency == RS_PERSISTENT)
+		sprintf(persistency, "persistent");
+	else if (cp.slotdata.persistency == RS_EPHEMERAL)
+		sprintf(persistency, "ephemeral");
+	else if (cp.slotdata.persistency == RS_TEMPORARY)
+		sprintf(persistency, "temporary");
+
+	/* display the slot information */
+	printf("%-64s %9s %10u %11s %10u %12u %21s %21s %21s %21s %10d %20s\n",
+		   NameStr(cp.slotdata.name),
+		   cp.slotdata.database == InvalidOid ? "physical" : "logical",
+		   cp.slotdata.database,
+		   persistency,
+		   cp.slotdata.xmin,
+		   cp.slotdata.catalog_xmin,
+		   restart_lsn,
+		   invalidated_at,
+		   confirmed_flush,
+		   two_phase_at,
+		   cp.slotdata.two_phase,
+		   NameStr(cp.slotdata.plugin));
+}
+
+int
+main(int argc, char *argv[])
+{
+	static struct option long_options[] = {
+		{"pgdata", required_argument, NULL, 'D'},
+		{"verbose", no_argument, NULL, 'v'},
+		{NULL, 0, NULL, 0}
+	};
+
+	char	   *DataDir = NULL;
+	const char *progname;
+	int			c;
+
+	pg_logging_init(argv[0]);
+	set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_replslotdata"));
+	progname = get_progname(argv[0]);
+
+	if (argc > 1)
+	{
+		if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+		{
+			usage(progname);
+			exit(0);
+		}
+		if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0)
+		{
+			puts("pg_replslotdata (PostgreSQL) " PG_VERSION);
+			exit(0);
+		}
+	}
+
+	while ((c = getopt_long(argc, argv, "D:", long_options, NULL)) != -1)
+	{
+		switch (c)
+		{
+			case 'D':
+				DataDir = optarg;
+				break;
+			case 'v':
+				verbose = true;
+				break;
+			default:
+				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+				exit(1);
+		}
+	}
+
+	if (DataDir == NULL)
+	{
+		if (optind < argc)
+			DataDir = argv[optind++];
+		else
+			DataDir = getenv("PGDATA");
+	}
+
+	/* Complain if any arguments remain */
+	if (optind < argc)
+	{
+		pg_log_error("too many command-line arguments (first is \"%s\")",
+					 argv[optind]);
+		fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+				progname);
+		exit(1);
+	}
+
+	if (DataDir == NULL)
+	{
+		pg_log_error("no data directory specified");
+		fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+		exit(1);
+	}
+
+	if (verbose)
+		pg_log_info("data directory: \"%s\"", DataDir);
+
+	if (chdir(DataDir) < 0)
+	{
+		pg_log_error("could not change directory to \"%s\": %m",
+					 DataDir);
+		exit(1);
+	}
+
+	process_replslots();
+
+	return 0;
+}
diff --git a/src/bin/pg_replslotdata/t/001_basic.pl b/src/bin/pg_replslotdata/t/001_basic.pl
new file mode 100644
index 0000000000..d6830dc2ac
--- /dev/null
+++ b/src/bin/pg_replslotdata/t/001_basic.pl
@@ -0,0 +1,11 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 8;
+
+program_help_ok('pg_replslotdata');
+program_version_ok('pg_replslotdata');
+program_options_handling_ok('pg_replslotdata');
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 53d773ccff..479d504a68 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -15,89 +15,9 @@
 #include "storage/lwlock.h"
 #include "storage/shmem.h"
 #include "storage/spin.h"
+#include "replication/slot_common.h"
 #include "replication/walreceiver.h"
 
-/*
- * Behaviour of replication slots, upon release or crash.
- *
- * Slots marked as PERSISTENT are crash-safe and will not be dropped when
- * released. Slots marked as EPHEMERAL will be dropped when released or after
- * restarts.  Slots marked TEMPORARY will be dropped at the end of a session
- * or on error.
- *
- * EPHEMERAL is used as a not-quite-ready state when creating persistent
- * slots.  EPHEMERAL slots can be made PERSISTENT by calling
- * ReplicationSlotPersist().  For a slot that goes away at the end of a
- * session, TEMPORARY is the appropriate choice.
- */
-typedef enum ReplicationSlotPersistency
-{
-	RS_PERSISTENT,
-	RS_EPHEMERAL,
-	RS_TEMPORARY
-} ReplicationSlotPersistency;
-
-/*
- * On-Disk data of a replication slot, preserved across restarts.
- */
-typedef struct ReplicationSlotPersistentData
-{
-	/* The slot's identifier */
-	NameData	name;
-
-	/* database the slot is active on */
-	Oid			database;
-
-	/*
-	 * The slot's behaviour when being dropped (or restored after a crash).
-	 */
-	ReplicationSlotPersistency persistency;
-
-	/*
-	 * xmin horizon for data
-	 *
-	 * NB: This may represent a value that hasn't been written to disk yet;
-	 * see notes for effective_xmin, below.
-	 */
-	TransactionId xmin;
-
-	/*
-	 * xmin horizon for catalog tuples
-	 *
-	 * NB: This may represent a value that hasn't been written to disk yet;
-	 * see notes for effective_xmin, below.
-	 */
-	TransactionId catalog_xmin;
-
-	/* oldest LSN that might be required by this replication slot */
-	XLogRecPtr	restart_lsn;
-
-	/* restart_lsn is copied here when the slot is invalidated */
-	XLogRecPtr	invalidated_at;
-
-	/*
-	 * Oldest LSN that the client has acked receipt for.  This is used as the
-	 * start_lsn point in case the client doesn't specify one, and also as a
-	 * safety measure to jump forwards in case the client specifies a
-	 * start_lsn that's further in the past than this value.
-	 */
-	XLogRecPtr	confirmed_flush;
-
-	/*
-	 * LSN at which we enabled two_phase commit for this slot or LSN at which
-	 * we found a consistent point at the time of slot creation.
-	 */
-	XLogRecPtr	two_phase_at;
-
-	/*
-	 * Allow decoding of prepared transactions?
-	 */
-	bool		two_phase;
-
-	/* plugin name */
-	NameData	plugin;
-} ReplicationSlotPersistentData;
-
 /*
  * Shared memory state of a single replication slot.
  *
diff --git a/src/include/replication/slot_common.h b/src/include/replication/slot_common.h
new file mode 100644
index 0000000000..2d221637bb
--- /dev/null
+++ b/src/include/replication/slot_common.h
@@ -0,0 +1,132 @@
+/*-------------------------------------------------------------------------
+ * slot_common.h
+ *	   Replication slot management.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef SLOT_COMMON_H
+#define SLOT_COMMON_H
+
+/*
+ * Behaviour of replication slots, upon release or crash.
+ *
+ * Slots marked as PERSISTENT are crash-safe and will not be dropped when
+ * released. Slots marked as EPHEMERAL will be dropped when released or after
+ * restarts.  Slots marked TEMPORARY will be dropped at the end of a session
+ * or on error.
+ *
+ * EPHEMERAL is used as a not-quite-ready state when creating persistent
+ * slots.  EPHEMERAL slots can be made PERSISTENT by calling
+ * ReplicationSlotPersist().  For a slot that goes away at the end of a
+ * session, TEMPORARY is the appropriate choice.
+ */
+typedef enum ReplicationSlotPersistency
+{
+	RS_PERSISTENT,
+	RS_EPHEMERAL,
+	RS_TEMPORARY
+} ReplicationSlotPersistency;
+
+/*
+ * On-Disk data of a replication slot, preserved across restarts.
+ */
+typedef struct ReplicationSlotPersistentData
+{
+	/* The slot's identifier */
+	NameData	name;
+
+	/* database the slot is active on */
+	Oid			database;
+
+	/*
+	 * The slot's behaviour when being dropped (or restored after a crash).
+	 */
+	ReplicationSlotPersistency persistency;
+
+	/*
+	 * xmin horizon for data
+	 *
+	 * NB: This may represent a value that hasn't been written to disk yet;
+	 * see notes for effective_xmin, below.
+	 */
+	TransactionId xmin;
+
+	/*
+	 * xmin horizon for catalog tuples
+	 *
+	 * NB: This may represent a value that hasn't been written to disk yet;
+	 * see notes for effective_xmin, below.
+	 */
+	TransactionId catalog_xmin;
+
+	/* oldest LSN that might be required by this replication slot */
+	XLogRecPtr	restart_lsn;
+
+	/* restart_lsn is copied here when the slot is invalidated */
+	XLogRecPtr	invalidated_at;
+
+	/*
+	 * Oldest LSN that the client has acked receipt for.  This is used as the
+	 * start_lsn point in case the client doesn't specify one, and also as a
+	 * safety measure to jump forwards in case the client specifies a
+	 * start_lsn that's further in the past than this value.
+	 */
+	XLogRecPtr	confirmed_flush;
+
+	/*
+	 * LSN at which we enabled two_phase commit for this slot or LSN at which
+	 * we found a consistent point at the time of slot creation.
+	 */
+	XLogRecPtr	two_phase_at;
+
+	/*
+	 * Allow decoding of prepared transactions?
+	 */
+	bool		two_phase;
+
+	/* plugin name */
+	NameData	plugin;
+} ReplicationSlotPersistentData;
+
+/*
+ * Replication slot on-disk data structure.
+ */
+typedef struct ReplicationSlotOnDisk
+{
+	/* first part of this struct needs to be version independent */
+
+	/* data not covered by checksum */
+	uint32		magic;
+	pg_crc32c	checksum;
+
+	/* data covered by checksum */
+	uint32		version;
+	uint32		length;
+
+	/*
+	 * The actual data in the slot that follows can differ based on the above
+	 * 'version'.
+	 */
+
+	ReplicationSlotPersistentData slotdata;
+} ReplicationSlotOnDisk;
+
+/* size of version independent data */
+#define ReplicationSlotOnDiskConstantSize \
+	offsetof(ReplicationSlotOnDisk, slotdata)
+/* size of the part of the slot not covered by the checksum */
+#define ReplicationSlotOnDiskNotChecksummedSize \
+	offsetof(ReplicationSlotOnDisk, version)
+/* size of the part covered by the checksum */
+#define ReplicationSlotOnDiskChecksummedSize \
+	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskNotChecksummedSize
+/* size of the slot data that is version dependent */
+#define ReplicationSlotOnDiskV2Size \
+	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskConstantSize
+
+#define SLOT_MAGIC		0x1051CA1	/* format identifier */
+#define SLOT_VERSION	2		/* version for new files */
+
+#endif							/* SLOT_COMMON_H */
-- 
2.25.1

#2Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Bharath Rupireddy (#1)
1 attachment(s)
Re: pg_replslotdata - a tool for displaying replication slot information

On Tue, Nov 23, 2021 at 10:39 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

Hi,

The replication slots data is stored in binary format on the disk under the pg_replslot/<<slot_name>> directory which isn't human readable. If the server is crashed/down (for whatever reasons) and unable to come up, currently there's no way for the user/admin/developer to know what were all the replication slots available at the time of server crash/down to figure out what's the restart lsn, xid, two phase info or types of slots etc.

pg_replslotdata is a tool that interprets the replication slots information and displays it onto the stdout even if the server is crashed/down. The design of this tool is similar to other tools available in the core today i.e. pg_controldata, pg_waldump.

Attaching initial patch herewith. I will improve it with documentation and other stuff a bit later.

Please see the attached picture for the sample output.

Thoughts?

Attaching the rebased v2 patch.

Regards,
Bharath Rupireddy.

Attachments:

v2-0001-pg_replslotdata.patchapplication/octet-stream; name=v2-0001-pg_replslotdata.patchDownload
From d352425ff3fc58319519cf969e885126be3de3ef Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Wed, 24 Nov 2021 15:37:22 +0000
Subject: [PATCH v2] pg_replslotdata

---
 src/backend/replication/slot.c            |  39 ---
 src/bin/Makefile                          |   1 +
 src/bin/pg_replslotdata/.gitignore        |   2 +
 src/bin/pg_replslotdata/Makefile          |  44 +++
 src/bin/pg_replslotdata/nls.mk            |   6 +
 src/bin/pg_replslotdata/pg_replslotdata.c | 364 ++++++++++++++++++++++
 src/bin/pg_replslotdata/t/001_basic.pl    |  11 +
 src/include/replication/slot.h            |  82 +----
 src/include/replication/slot_common.h     | 132 ++++++++
 9 files changed, 561 insertions(+), 120 deletions(-)
 create mode 100644 src/bin/pg_replslotdata/.gitignore
 create mode 100644 src/bin/pg_replslotdata/Makefile
 create mode 100644 src/bin/pg_replslotdata/nls.mk
 create mode 100644 src/bin/pg_replslotdata/pg_replslotdata.c
 create mode 100644 src/bin/pg_replslotdata/t/001_basic.pl
 create mode 100644 src/include/replication/slot_common.h

diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 90ba9b417d..236ebe999a 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -50,45 +50,6 @@
 #include "storage/procarray.h"
 #include "utils/builtins.h"
 
-/*
- * Replication slot on-disk data structure.
- */
-typedef struct ReplicationSlotOnDisk
-{
-	/* first part of this struct needs to be version independent */
-
-	/* data not covered by checksum */
-	uint32		magic;
-	pg_crc32c	checksum;
-
-	/* data covered by checksum */
-	uint32		version;
-	uint32		length;
-
-	/*
-	 * The actual data in the slot that follows can differ based on the above
-	 * 'version'.
-	 */
-
-	ReplicationSlotPersistentData slotdata;
-} ReplicationSlotOnDisk;
-
-/* size of version independent data */
-#define ReplicationSlotOnDiskConstantSize \
-	offsetof(ReplicationSlotOnDisk, slotdata)
-/* size of the part of the slot not covered by the checksum */
-#define ReplicationSlotOnDiskNotChecksummedSize  \
-	offsetof(ReplicationSlotOnDisk, version)
-/* size of the part covered by the checksum */
-#define ReplicationSlotOnDiskChecksummedSize \
-	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskNotChecksummedSize
-/* size of the slot data that is version dependent */
-#define ReplicationSlotOnDiskV2Size \
-	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskConstantSize
-
-#define SLOT_MAGIC		0x1051CA1	/* format identifier */
-#define SLOT_VERSION	2		/* version for new files */
-
 /* Control array for replication slot management */
 ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
 
diff --git a/src/bin/Makefile b/src/bin/Makefile
index 2fe0ae6652..5e20009beb 100644
--- a/src/bin/Makefile
+++ b/src/bin/Makefile
@@ -23,6 +23,7 @@ SUBDIRS = \
 	pg_controldata \
 	pg_ctl \
 	pg_dump \
+	pg_replslotdata \
 	pg_resetwal \
 	pg_rewind \
 	pg_test_fsync \
diff --git a/src/bin/pg_replslotdata/.gitignore b/src/bin/pg_replslotdata/.gitignore
new file mode 100644
index 0000000000..13a4afb8ef
--- /dev/null
+++ b/src/bin/pg_replslotdata/.gitignore
@@ -0,0 +1,2 @@
+/pg_replslotdata
+/tmp_check/
diff --git a/src/bin/pg_replslotdata/Makefile b/src/bin/pg_replslotdata/Makefile
new file mode 100644
index 0000000000..69518ee53b
--- /dev/null
+++ b/src/bin/pg_replslotdata/Makefile
@@ -0,0 +1,44 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/bin/pg_replslotdata
+#
+# Copyright (c) 1998-2021, PostgreSQL Global Development Group
+#
+# src/bin/pg_replslotdata/Makefile
+#
+#-------------------------------------------------------------------------
+
+PGFILEDESC = "pg_replslotdata - provides information about the replication slots from $PGDATA/pg_replslot/<slot_name> $PGDATA/pg_replslot/<slot_name>"
+PGAPPICON=win32
+
+subdir = src/bin/pg_replslotdata
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = \
+	$(WIN32RES) \
+	pg_replslotdata.o
+
+all: pg_replslotdata
+
+pg_replslotdata: $(OBJS) | submake-libpgport
+	$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
+install: all installdirs
+	$(INSTALL_PROGRAM) pg_replslotdata$(X) '$(DESTDIR)$(bindir)/pg_replslotdata$(X)'
+
+installdirs:
+	$(MKDIR_P) '$(DESTDIR)$(bindir)'
+
+uninstall:
+	rm -f '$(DESTDIR)$(bindir)/pg_replslotdata$(X)'
+
+clean distclean maintainer-clean:
+	rm -f pg_replslotdata$(X) $(OBJS)
+	rm -rf tmp_check
+
+check:
+	$(prove_check)
+
+installcheck:
+	$(prove_installcheck)
diff --git a/src/bin/pg_replslotdata/nls.mk b/src/bin/pg_replslotdata/nls.mk
new file mode 100644
index 0000000000..74bee593c9
--- /dev/null
+++ b/src/bin/pg_replslotdata/nls.mk
@@ -0,0 +1,6 @@
+# src/bin/pg_replslotdata/nls.mk
+CATALOG_NAME     = pg_replslotdata
+AVAIL_LANGUAGES  = cs de el es fr ja ko pl ru sv tr uk vi zh_CN
+GETTEXT_FILES    = $(FRONTEND_COMMON_GETTEXT_FILES) pg_replslotdata.c
+GETTEXT_TRIGGERS = $(FRONTEND_COMMON_GETTEXT_TRIGGERS)
+GETTEXT_FLAGS    = $(FRONTEND_COMMON_GETTEXT_FLAGS)
diff --git a/src/bin/pg_replslotdata/pg_replslotdata.c b/src/bin/pg_replslotdata/pg_replslotdata.c
new file mode 100644
index 0000000000..f9b4b56b0e
--- /dev/null
+++ b/src/bin/pg_replslotdata/pg_replslotdata.c
@@ -0,0 +1,364 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_replslotdata.c - provides information about the replication slots
+ * from $PGDATA/pg_replslot/<slot_name>.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ *		  src/bin/pg_replslotdata/pg_replslotdata.c
+ *-------------------------------------------------------------------------
+ */
+/*
+ * We have to use postgres.h not postgres_fe.h here, because there's so much
+ * backend-only stuff in the XLOG include files we need.  But we need a
+ * frontend-ish environment otherwise.  Hence this ugly hack.
+ */
+#define FRONTEND 1
+
+#include "postgres.h"
+
+#include <sys/stat.h>
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "getopt_long.h"
+#include "pg_getopt.h"
+#include "replication/slot_common.h"
+
+static bool verbose = false;
+
+static void process_replslots(void);
+static void read_and_display_repl_slot(const char *name);
+
+static void
+usage(const char *progname)
+{
+	printf(_("%s displays information about the replication slots from $PGDATA/pg_replslot/<slot_name>.\n\n"), progname);
+	printf(_("Usage:\n"));
+	printf(_("  %s [OPTION] [DATADIR]\n"), progname);
+	printf(_("\nOptions:\n"));
+	printf(_(" [-D, --pgdata=]DATADIR  data directory\n"));
+	printf(_("  -V, --version          output version information, then exit\n"));
+	printf(_("  -v, --verbose          write a lot of output\n"));
+	printf(_("  -?, --help             show this help, then exit\n"));
+	printf(_("\nIf no data directory (DATADIR) is specified, "
+			 "the environment variable PGDATA\nis used.\n\n"));
+	printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+	printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+static void
+process_replslots(void)
+{
+	DIR		   *rsdir;
+	struct dirent *rsde;
+	uint32		cnt = 0;
+
+	rsdir = opendir("pg_replslot");
+	if (rsdir == NULL)
+	{
+		pg_log_error("could not open directory \"%s\": %m", "pg_replslot");
+		exit(1);
+	}
+
+	/* XXX: comment here about the format spefiiers */
+	printf("%-64s %9s %10s %11s %10s %12s %21s %21s %21s %21s %10s %20s\n"
+		   "%-64s %9s %10s %11s %10s %12s %21s %21s %21s %21s %10s %20s\n",
+		   "slot_name", "slot_type", "datoid", "persistency", "xmin", "catalog_xmin", "restart_lsn", "invalidated_at", "confirmed_flush", "two_phase_at", "two_phase", "plugin",
+		   "---------", "---------", "------", "-----------", "----", "------------", "-----------", "--------------", "---------------", "------------", "---------", "------");
+
+	while (errno = 0, (rsde = readdir(rsdir)) != NULL)
+	{
+		struct stat statbuf;
+		char		path[MAXPGPATH];
+
+		if (strcmp(rsde->d_name, ".") == 0 ||
+			strcmp(rsde->d_name, "..") == 0)
+			continue;
+
+		snprintf(path, sizeof(path), "pg_replslot/%s", rsde->d_name);
+
+		/* we're only creating directories here, skip if it's not our's */
+		if (lstat(path, &statbuf) == 0 && !S_ISDIR(statbuf.st_mode))
+			continue;
+
+		/* we crashed while a slot was being setup or deleted, clean up */
+		if (pg_str_endswith(rsde->d_name, ".tmp"))
+		{
+			pg_log_warning("server was crashed while the slot \"%s\" was being setup or deleted",
+						   rsde->d_name);
+			continue;
+		}
+
+		/* looks like a slot in a normal state, restore */
+		read_and_display_repl_slot(rsde->d_name);
+		cnt++;
+	}
+
+	if (errno)
+	{
+		pg_log_error("could not read directory \"%s\": %m", "pg_replslot");
+		exit(1);
+	}
+
+	if (cnt == 0)
+	{
+		pg_log_info("no replication slots were found");
+		exit(0);
+	}
+
+	if (closedir(rsdir))
+	{
+		pg_log_error("could not close directory \"%s\": %m", "pg_replslot");
+		exit(1);
+	}
+}
+
+static void
+read_and_display_repl_slot(const char *name)
+{
+	ReplicationSlotOnDisk cp;
+	char		slotdir[MAXPGPATH];
+	char		path[MAXPGPATH];
+	char		restart_lsn[NAMEDATALEN];
+	char		invalidated_at[NAMEDATALEN];
+	char		confirmed_flush[NAMEDATALEN];
+	char		two_phase_at[NAMEDATALEN];
+	char		persistency[NAMEDATALEN];
+	int			fd;
+	int			readBytes;
+	pg_crc32c	checksum;
+
+	/* delete temp file if it exists */
+	sprintf(slotdir, "pg_replslot/%s", name);
+	sprintf(path, "%s/state.tmp", slotdir);
+
+	fd = open(path, O_RDONLY | PG_BINARY);
+
+	if (fd > 0)
+	{
+		pg_log_error("found temporary state file \"%s\": %m", path);
+		exit(1);
+	}
+
+	sprintf(path, "%s/state", slotdir);
+
+	if (verbose)
+		pg_log_info("reading replication slot from \"%s\"", path);
+
+	fd = open(path, O_RDONLY | PG_BINARY);
+
+	/*
+	 * We do not need to handle this as we are rename()ing the directory into
+	 * place only after we fsync()ed the state file.
+	 */
+	if (fd < 0)
+	{
+		pg_log_error("could not open file \"%s\": %m", path);
+		exit(1);
+	}
+
+	if (verbose)
+		pg_log_info("reading version independent replication slot state file");
+
+	/* read part of statefile that's guaranteed to be version independent */
+	readBytes = read(fd, &cp, ReplicationSlotOnDiskConstantSize);
+	if (readBytes != ReplicationSlotOnDiskConstantSize)
+	{
+		if (readBytes < 0)
+		{
+			pg_log_error("could not read file \"%s\": %m", path);
+			exit(1);
+		}
+		else
+		{
+			pg_log_error("could not read file \"%s\": read %d of %zu",
+						 path, readBytes,
+						 (Size) ReplicationSlotOnDiskConstantSize);
+			exit(1);
+		}
+	}
+
+	/* verify magic */
+	if (cp.magic != SLOT_MAGIC)
+	{
+		pg_log_error("replication slot file \"%s\" has wrong magic number: %u instead of %u",
+					 path, cp.magic, SLOT_MAGIC);
+		exit(1);
+	}
+
+	/* verify version */
+	if (cp.version != SLOT_VERSION)
+	{
+		pg_log_error("replication slot file \"%s\" has unsupported version %u",
+					 path, cp.version);
+		exit(1);
+	}
+
+	/* boundary check on length */
+	if (cp.length != ReplicationSlotOnDiskV2Size)
+	{
+		pg_log_error("replication slot file \"%s\" has corrupted length %u",
+					 path, cp.length);
+		exit(1);
+	}
+
+	if (verbose)
+		pg_log_info("reading the entire replication slot state file");
+
+	/* now that we know the size, read the entire file */
+	readBytes = read(fd,
+					 (char *) &cp + ReplicationSlotOnDiskConstantSize,
+					 cp.length);
+	if (readBytes != cp.length)
+	{
+		if (readBytes < 0)
+		{
+			pg_log_error("could not read file \"%s\": %m", path);
+			exit(1);
+		}
+		else
+		{
+			pg_log_error("could not read file \"%s\": read %d of %zu",
+						 path, readBytes, (Size) cp.length);
+			exit(1);
+		}
+	}
+
+	if (close(fd) != 0)
+	{
+		pg_log_error("could not close file \"%s\": %m", path);
+		exit(1);
+	}
+
+	/* now verify the CRC */
+	INIT_CRC32C(checksum);
+	COMP_CRC32C(checksum,
+				(char *) &cp + ReplicationSlotOnDiskNotChecksummedSize,
+				ReplicationSlotOnDiskChecksummedSize);
+	FIN_CRC32C(checksum);
+
+	if (!EQ_CRC32C(checksum, cp.checksum))
+	{
+		pg_log_error("checksum mismatch for replication slot file \"%s\": is %u, should be %u",
+					 path, checksum, cp.checksum);
+		exit(1);
+	}
+
+	sprintf(restart_lsn, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.restart_lsn));
+	sprintf(invalidated_at, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.invalidated_at));
+	sprintf(confirmed_flush, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.confirmed_flush));
+	sprintf(two_phase_at, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.two_phase_at));
+
+	if (cp.slotdata.persistency == RS_PERSISTENT)
+		sprintf(persistency, "persistent");
+	else if (cp.slotdata.persistency == RS_EPHEMERAL)
+		sprintf(persistency, "ephemeral");
+	else if (cp.slotdata.persistency == RS_TEMPORARY)
+		sprintf(persistency, "temporary");
+
+	/* display the slot information */
+	printf("%-64s %9s %10u %11s %10u %12u %21s %21s %21s %21s %10d %20s\n",
+		   NameStr(cp.slotdata.name),
+		   cp.slotdata.database == InvalidOid ? "physical" : "logical",
+		   cp.slotdata.database,
+		   persistency,
+		   cp.slotdata.xmin,
+		   cp.slotdata.catalog_xmin,
+		   restart_lsn,
+		   invalidated_at,
+		   confirmed_flush,
+		   two_phase_at,
+		   cp.slotdata.two_phase,
+		   NameStr(cp.slotdata.plugin));
+}
+
+int
+main(int argc, char *argv[])
+{
+	static struct option long_options[] = {
+		{"pgdata", required_argument, NULL, 'D'},
+		{"verbose", no_argument, NULL, 'v'},
+		{NULL, 0, NULL, 0}
+	};
+
+	char	   *DataDir = NULL;
+	const char *progname;
+	int			c;
+
+	pg_logging_init(argv[0]);
+	set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_replslotdata"));
+	progname = get_progname(argv[0]);
+
+	if (argc > 1)
+	{
+		if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+		{
+			usage(progname);
+			exit(0);
+		}
+		if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0)
+		{
+			puts("pg_replslotdata (PostgreSQL) " PG_VERSION);
+			exit(0);
+		}
+	}
+
+	while ((c = getopt_long(argc, argv, "D:", long_options, NULL)) != -1)
+	{
+		switch (c)
+		{
+			case 'D':
+				DataDir = optarg;
+				break;
+			case 'v':
+				verbose = true;
+				break;
+			default:
+				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+				exit(1);
+		}
+	}
+
+	if (DataDir == NULL)
+	{
+		if (optind < argc)
+			DataDir = argv[optind++];
+		else
+			DataDir = getenv("PGDATA");
+	}
+
+	/* Complain if any arguments remain */
+	if (optind < argc)
+	{
+		pg_log_error("too many command-line arguments (first is \"%s\")",
+					 argv[optind]);
+		fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+				progname);
+		exit(1);
+	}
+
+	if (DataDir == NULL)
+	{
+		pg_log_error("no data directory specified");
+		fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+		exit(1);
+	}
+
+	if (verbose)
+		pg_log_info("data directory: \"%s\"", DataDir);
+
+	if (chdir(DataDir) < 0)
+	{
+		pg_log_error("could not change directory to \"%s\": %m",
+					 DataDir);
+		exit(1);
+	}
+
+	process_replslots();
+
+	return 0;
+}
diff --git a/src/bin/pg_replslotdata/t/001_basic.pl b/src/bin/pg_replslotdata/t/001_basic.pl
new file mode 100644
index 0000000000..d6830dc2ac
--- /dev/null
+++ b/src/bin/pg_replslotdata/t/001_basic.pl
@@ -0,0 +1,11 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 8;
+
+program_help_ok('pg_replslotdata');
+program_version_ok('pg_replslotdata');
+program_options_handling_ok('pg_replslotdata');
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 53d773ccff..479d504a68 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -15,89 +15,9 @@
 #include "storage/lwlock.h"
 #include "storage/shmem.h"
 #include "storage/spin.h"
+#include "replication/slot_common.h"
 #include "replication/walreceiver.h"
 
-/*
- * Behaviour of replication slots, upon release or crash.
- *
- * Slots marked as PERSISTENT are crash-safe and will not be dropped when
- * released. Slots marked as EPHEMERAL will be dropped when released or after
- * restarts.  Slots marked TEMPORARY will be dropped at the end of a session
- * or on error.
- *
- * EPHEMERAL is used as a not-quite-ready state when creating persistent
- * slots.  EPHEMERAL slots can be made PERSISTENT by calling
- * ReplicationSlotPersist().  For a slot that goes away at the end of a
- * session, TEMPORARY is the appropriate choice.
- */
-typedef enum ReplicationSlotPersistency
-{
-	RS_PERSISTENT,
-	RS_EPHEMERAL,
-	RS_TEMPORARY
-} ReplicationSlotPersistency;
-
-/*
- * On-Disk data of a replication slot, preserved across restarts.
- */
-typedef struct ReplicationSlotPersistentData
-{
-	/* The slot's identifier */
-	NameData	name;
-
-	/* database the slot is active on */
-	Oid			database;
-
-	/*
-	 * The slot's behaviour when being dropped (or restored after a crash).
-	 */
-	ReplicationSlotPersistency persistency;
-
-	/*
-	 * xmin horizon for data
-	 *
-	 * NB: This may represent a value that hasn't been written to disk yet;
-	 * see notes for effective_xmin, below.
-	 */
-	TransactionId xmin;
-
-	/*
-	 * xmin horizon for catalog tuples
-	 *
-	 * NB: This may represent a value that hasn't been written to disk yet;
-	 * see notes for effective_xmin, below.
-	 */
-	TransactionId catalog_xmin;
-
-	/* oldest LSN that might be required by this replication slot */
-	XLogRecPtr	restart_lsn;
-
-	/* restart_lsn is copied here when the slot is invalidated */
-	XLogRecPtr	invalidated_at;
-
-	/*
-	 * Oldest LSN that the client has acked receipt for.  This is used as the
-	 * start_lsn point in case the client doesn't specify one, and also as a
-	 * safety measure to jump forwards in case the client specifies a
-	 * start_lsn that's further in the past than this value.
-	 */
-	XLogRecPtr	confirmed_flush;
-
-	/*
-	 * LSN at which we enabled two_phase commit for this slot or LSN at which
-	 * we found a consistent point at the time of slot creation.
-	 */
-	XLogRecPtr	two_phase_at;
-
-	/*
-	 * Allow decoding of prepared transactions?
-	 */
-	bool		two_phase;
-
-	/* plugin name */
-	NameData	plugin;
-} ReplicationSlotPersistentData;
-
 /*
  * Shared memory state of a single replication slot.
  *
diff --git a/src/include/replication/slot_common.h b/src/include/replication/slot_common.h
new file mode 100644
index 0000000000..2d221637bb
--- /dev/null
+++ b/src/include/replication/slot_common.h
@@ -0,0 +1,132 @@
+/*-------------------------------------------------------------------------
+ * slot_common.h
+ *	   Replication slot management.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef SLOT_COMMON_H
+#define SLOT_COMMON_H
+
+/*
+ * Behaviour of replication slots, upon release or crash.
+ *
+ * Slots marked as PERSISTENT are crash-safe and will not be dropped when
+ * released. Slots marked as EPHEMERAL will be dropped when released or after
+ * restarts.  Slots marked TEMPORARY will be dropped at the end of a session
+ * or on error.
+ *
+ * EPHEMERAL is used as a not-quite-ready state when creating persistent
+ * slots.  EPHEMERAL slots can be made PERSISTENT by calling
+ * ReplicationSlotPersist().  For a slot that goes away at the end of a
+ * session, TEMPORARY is the appropriate choice.
+ */
+typedef enum ReplicationSlotPersistency
+{
+	RS_PERSISTENT,
+	RS_EPHEMERAL,
+	RS_TEMPORARY
+} ReplicationSlotPersistency;
+
+/*
+ * On-Disk data of a replication slot, preserved across restarts.
+ */
+typedef struct ReplicationSlotPersistentData
+{
+	/* The slot's identifier */
+	NameData	name;
+
+	/* database the slot is active on */
+	Oid			database;
+
+	/*
+	 * The slot's behaviour when being dropped (or restored after a crash).
+	 */
+	ReplicationSlotPersistency persistency;
+
+	/*
+	 * xmin horizon for data
+	 *
+	 * NB: This may represent a value that hasn't been written to disk yet;
+	 * see notes for effective_xmin, below.
+	 */
+	TransactionId xmin;
+
+	/*
+	 * xmin horizon for catalog tuples
+	 *
+	 * NB: This may represent a value that hasn't been written to disk yet;
+	 * see notes for effective_xmin, below.
+	 */
+	TransactionId catalog_xmin;
+
+	/* oldest LSN that might be required by this replication slot */
+	XLogRecPtr	restart_lsn;
+
+	/* restart_lsn is copied here when the slot is invalidated */
+	XLogRecPtr	invalidated_at;
+
+	/*
+	 * Oldest LSN that the client has acked receipt for.  This is used as the
+	 * start_lsn point in case the client doesn't specify one, and also as a
+	 * safety measure to jump forwards in case the client specifies a
+	 * start_lsn that's further in the past than this value.
+	 */
+	XLogRecPtr	confirmed_flush;
+
+	/*
+	 * LSN at which we enabled two_phase commit for this slot or LSN at which
+	 * we found a consistent point at the time of slot creation.
+	 */
+	XLogRecPtr	two_phase_at;
+
+	/*
+	 * Allow decoding of prepared transactions?
+	 */
+	bool		two_phase;
+
+	/* plugin name */
+	NameData	plugin;
+} ReplicationSlotPersistentData;
+
+/*
+ * Replication slot on-disk data structure.
+ */
+typedef struct ReplicationSlotOnDisk
+{
+	/* first part of this struct needs to be version independent */
+
+	/* data not covered by checksum */
+	uint32		magic;
+	pg_crc32c	checksum;
+
+	/* data covered by checksum */
+	uint32		version;
+	uint32		length;
+
+	/*
+	 * The actual data in the slot that follows can differ based on the above
+	 * 'version'.
+	 */
+
+	ReplicationSlotPersistentData slotdata;
+} ReplicationSlotOnDisk;
+
+/* size of version independent data */
+#define ReplicationSlotOnDiskConstantSize \
+	offsetof(ReplicationSlotOnDisk, slotdata)
+/* size of the part of the slot not covered by the checksum */
+#define ReplicationSlotOnDiskNotChecksummedSize \
+	offsetof(ReplicationSlotOnDisk, version)
+/* size of the part covered by the checksum */
+#define ReplicationSlotOnDiskChecksummedSize \
+	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskNotChecksummedSize
+/* size of the slot data that is version dependent */
+#define ReplicationSlotOnDiskV2Size \
+	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskConstantSize
+
+#define SLOT_MAGIC		0x1051CA1	/* format identifier */
+#define SLOT_VERSION	2		/* version for new files */
+
+#endif							/* SLOT_COMMON_H */
-- 
2.25.1

#3Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Bharath Rupireddy (#2)
1 attachment(s)
Re: pg_replslotdata - a tool for displaying replication slot information

On Wed, Nov 24, 2021 at 9:09 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

On Tue, Nov 23, 2021 at 10:39 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

Hi,

The replication slots data is stored in binary format on the disk under the pg_replslot/<<slot_name>> directory which isn't human readable. If the server is crashed/down (for whatever reasons) and unable to come up, currently there's no way for the user/admin/developer to know what were all the replication slots available at the time of server crash/down to figure out what's the restart lsn, xid, two phase info or types of slots etc.

pg_replslotdata is a tool that interprets the replication slots information and displays it onto the stdout even if the server is crashed/down. The design of this tool is similar to other tools available in the core today i.e. pg_controldata, pg_waldump.

Attaching initial patch herewith. I will improve it with documentation and other stuff a bit later.

Please see the attached picture for the sample output.

Thoughts?

Attaching the rebased v2 patch.

On windows the previous patches were failing, fixed that in the v3
patch. I'm really sorry for the noise.

Regards,
Bharath Rupireddy.

Attachments:

v3-0001-pg_replslotdata.patchapplication/octet-stream; name=v3-0001-pg_replslotdata.patchDownload
From 53d2dadc21443decd0f4e4a58f1cbbf952e1ff1c Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforwork@gmail.com>
Date: Wed, 24 Nov 2021 21:25:53 +0530
Subject: [PATCH v3] pg_replslotdata

---
 src/backend/replication/slot.c            |  39 ---
 src/bin/Makefile                          |   1 +
 src/bin/pg_replslotdata/.gitignore        |   2 +
 src/bin/pg_replslotdata/Makefile          |  44 +++
 src/bin/pg_replslotdata/nls.mk            |   6 +
 src/bin/pg_replslotdata/pg_replslotdata.c | 364 ++++++++++++++++++++++
 src/bin/pg_replslotdata/t/001_basic.pl    |  11 +
 src/include/replication/slot.h            |  82 +----
 src/include/replication/slot_common.h     | 132 ++++++++
 9 files changed, 561 insertions(+), 120 deletions(-)
 create mode 100644 src/bin/pg_replslotdata/.gitignore
 create mode 100644 src/bin/pg_replslotdata/Makefile
 create mode 100644 src/bin/pg_replslotdata/nls.mk
 create mode 100644 src/bin/pg_replslotdata/pg_replslotdata.c
 create mode 100644 src/bin/pg_replslotdata/t/001_basic.pl
 create mode 100644 src/include/replication/slot_common.h

diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 90ba9b417d..236ebe999a 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -50,45 +50,6 @@
 #include "storage/procarray.h"
 #include "utils/builtins.h"
 
-/*
- * Replication slot on-disk data structure.
- */
-typedef struct ReplicationSlotOnDisk
-{
-	/* first part of this struct needs to be version independent */
-
-	/* data not covered by checksum */
-	uint32		magic;
-	pg_crc32c	checksum;
-
-	/* data covered by checksum */
-	uint32		version;
-	uint32		length;
-
-	/*
-	 * The actual data in the slot that follows can differ based on the above
-	 * 'version'.
-	 */
-
-	ReplicationSlotPersistentData slotdata;
-} ReplicationSlotOnDisk;
-
-/* size of version independent data */
-#define ReplicationSlotOnDiskConstantSize \
-	offsetof(ReplicationSlotOnDisk, slotdata)
-/* size of the part of the slot not covered by the checksum */
-#define ReplicationSlotOnDiskNotChecksummedSize  \
-	offsetof(ReplicationSlotOnDisk, version)
-/* size of the part covered by the checksum */
-#define ReplicationSlotOnDiskChecksummedSize \
-	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskNotChecksummedSize
-/* size of the slot data that is version dependent */
-#define ReplicationSlotOnDiskV2Size \
-	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskConstantSize
-
-#define SLOT_MAGIC		0x1051CA1	/* format identifier */
-#define SLOT_VERSION	2		/* version for new files */
-
 /* Control array for replication slot management */
 ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
 
diff --git a/src/bin/Makefile b/src/bin/Makefile
index 2fe0ae6652..5e20009beb 100644
--- a/src/bin/Makefile
+++ b/src/bin/Makefile
@@ -23,6 +23,7 @@ SUBDIRS = \
 	pg_controldata \
 	pg_ctl \
 	pg_dump \
+	pg_replslotdata \
 	pg_resetwal \
 	pg_rewind \
 	pg_test_fsync \
diff --git a/src/bin/pg_replslotdata/.gitignore b/src/bin/pg_replslotdata/.gitignore
new file mode 100644
index 0000000000..13a4afb8ef
--- /dev/null
+++ b/src/bin/pg_replslotdata/.gitignore
@@ -0,0 +1,2 @@
+/pg_replslotdata
+/tmp_check/
diff --git a/src/bin/pg_replslotdata/Makefile b/src/bin/pg_replslotdata/Makefile
new file mode 100644
index 0000000000..69518ee53b
--- /dev/null
+++ b/src/bin/pg_replslotdata/Makefile
@@ -0,0 +1,44 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/bin/pg_replslotdata
+#
+# Copyright (c) 1998-2021, PostgreSQL Global Development Group
+#
+# src/bin/pg_replslotdata/Makefile
+#
+#-------------------------------------------------------------------------
+
+PGFILEDESC = "pg_replslotdata - provides information about the replication slots from $PGDATA/pg_replslot/<slot_name> $PGDATA/pg_replslot/<slot_name>"
+PGAPPICON=win32
+
+subdir = src/bin/pg_replslotdata
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = \
+	$(WIN32RES) \
+	pg_replslotdata.o
+
+all: pg_replslotdata
+
+pg_replslotdata: $(OBJS) | submake-libpgport
+	$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
+install: all installdirs
+	$(INSTALL_PROGRAM) pg_replslotdata$(X) '$(DESTDIR)$(bindir)/pg_replslotdata$(X)'
+
+installdirs:
+	$(MKDIR_P) '$(DESTDIR)$(bindir)'
+
+uninstall:
+	rm -f '$(DESTDIR)$(bindir)/pg_replslotdata$(X)'
+
+clean distclean maintainer-clean:
+	rm -f pg_replslotdata$(X) $(OBJS)
+	rm -rf tmp_check
+
+check:
+	$(prove_check)
+
+installcheck:
+	$(prove_installcheck)
diff --git a/src/bin/pg_replslotdata/nls.mk b/src/bin/pg_replslotdata/nls.mk
new file mode 100644
index 0000000000..74bee593c9
--- /dev/null
+++ b/src/bin/pg_replslotdata/nls.mk
@@ -0,0 +1,6 @@
+# src/bin/pg_replslotdata/nls.mk
+CATALOG_NAME     = pg_replslotdata
+AVAIL_LANGUAGES  = cs de el es fr ja ko pl ru sv tr uk vi zh_CN
+GETTEXT_FILES    = $(FRONTEND_COMMON_GETTEXT_FILES) pg_replslotdata.c
+GETTEXT_TRIGGERS = $(FRONTEND_COMMON_GETTEXT_TRIGGERS)
+GETTEXT_FLAGS    = $(FRONTEND_COMMON_GETTEXT_FLAGS)
diff --git a/src/bin/pg_replslotdata/pg_replslotdata.c b/src/bin/pg_replslotdata/pg_replslotdata.c
new file mode 100644
index 0000000000..1c3420c922
--- /dev/null
+++ b/src/bin/pg_replslotdata/pg_replslotdata.c
@@ -0,0 +1,364 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_replslotdata.c - provides information about the replication slots
+ * from $PGDATA/pg_replslot/<slot_name>.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ *		  src/bin/pg_replslotdata/pg_replslotdata.c
+ *-------------------------------------------------------------------------
+ */
+/*
+ * We have to use postgres.h not postgres_fe.h here, because there's so much
+ * backend-only stuff in the XLOG include files we need.  But we need a
+ * frontend-ish environment otherwise.  Hence this ugly hack.
+ */
+#define FRONTEND 1
+
+#include "postgres.h"
+
+#include <sys/stat.h>
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "getopt_long.h"
+#include "pg_getopt.h"
+#include "replication/slot_common.h"
+
+static bool verbose = false;
+
+static void process_replslots(void);
+static void read_and_display_repl_slot(const char *name);
+
+static void
+usage(const char *progname)
+{
+	printf(_("%s displays information about the replication slots from $PGDATA/pg_replslot/<slot_name>.\n\n"), progname);
+	printf(_("Usage:\n"));
+	printf(_("  %s [OPTION] [DATADIR]\n"), progname);
+	printf(_("\nOptions:\n"));
+	printf(_(" [-D, --pgdata=]DATADIR  data directory\n"));
+	printf(_("  -V, --version          output version information, then exit\n"));
+	printf(_("  -v, --verbose          write a lot of output\n"));
+	printf(_("  -?, --help             show this help, then exit\n"));
+	printf(_("\nIf no data directory (DATADIR) is specified, "
+			 "the environment variable PGDATA\nis used.\n\n"));
+	printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+	printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+static void
+process_replslots(void)
+{
+	DIR		   *rsdir;
+	struct dirent *rsde;
+	uint32		cnt = 0;
+
+	rsdir = opendir("pg_replslot");
+	if (rsdir == NULL)
+	{
+		pg_log_error("could not open directory \"%s\": %m", "pg_replslot");
+		exit(1);
+	}
+
+	/* XXX: comment here about the format spefiiers */
+	printf("%-64s %9s %10s %11s %10s %12s %21s %21s %21s %21s %10s %20s\n"
+		   "%-64s %9s %10s %11s %10s %12s %21s %21s %21s %21s %10s %20s\n",
+		   "slot_name", "slot_type", "datoid", "persistency", "xmin", "catalog_xmin", "restart_lsn", "invalidated_at", "confirmed_flush", "two_phase_at", "two_phase", "plugin",
+		   "---------", "---------", "------", "-----------", "----", "------------", "-----------", "--------------", "---------------", "------------", "---------", "------");
+
+	while (errno = 0, (rsde = readdir(rsdir)) != NULL)
+	{
+		struct stat statbuf;
+		char		path[MAXPGPATH];
+
+		if (strcmp(rsde->d_name, ".") == 0 ||
+			strcmp(rsde->d_name, "..") == 0)
+			continue;
+
+		snprintf(path, sizeof(path), "pg_replslot/%s", rsde->d_name);
+
+		/* we're only creating directories here, skip if it's not our's */
+		if (lstat(path, &statbuf) == 0 && !S_ISDIR(statbuf.st_mode))
+			continue;
+
+		/* we crashed while a slot was being setup or deleted, clean up */
+		if (pg_str_endswith(rsde->d_name, ".tmp"))
+		{
+			pg_log_warning("server was crashed while the slot \"%s\" was being setup or deleted",
+						   rsde->d_name);
+			continue;
+		}
+
+		/* looks like a slot in a normal state, restore */
+		read_and_display_repl_slot(rsde->d_name);
+		cnt++;
+	}
+
+	if (errno)
+	{
+		pg_log_error("could not read directory \"%s\": %m", "pg_replslot");
+		exit(1);
+	}
+
+	if (cnt == 0)
+	{
+		pg_log_info("no replication slots were found");
+		exit(0);
+	}
+
+	if (closedir(rsdir))
+	{
+		pg_log_error("could not close directory \"%s\": %m", "pg_replslot");
+		exit(1);
+	}
+}
+
+static void
+read_and_display_repl_slot(const char *name)
+{
+	ReplicationSlotOnDisk cp;
+	char		slotdir[MAXPGPATH];
+	char		path[MAXPGPATH];
+	char		restart_lsn[NAMEDATALEN];
+	char		invalidated_at[NAMEDATALEN];
+	char		confirmed_flush[NAMEDATALEN];
+	char		two_phase_at[NAMEDATALEN];
+	char		persistency[NAMEDATALEN];
+	int			fd;
+	int			readBytes;
+	pg_crc32c	checksum;
+
+	/* delete temp file if it exists */
+	sprintf(slotdir, "pg_replslot/%s", name);
+	sprintf(path, "%s/state.tmp", slotdir);
+
+	fd = open(path, O_RDONLY | PG_BINARY, 0);
+
+	if (fd > 0)
+	{
+		pg_log_error("found temporary state file \"%s\": %m", path);
+		exit(1);
+	}
+
+	sprintf(path, "%s/state", slotdir);
+
+	if (verbose)
+		pg_log_info("reading replication slot from \"%s\"", path);
+
+	fd = open(path, O_RDONLY | PG_BINARY, 0);
+
+	/*
+	 * We do not need to handle this as we are rename()ing the directory into
+	 * place only after we fsync()ed the state file.
+	 */
+	if (fd < 0)
+	{
+		pg_log_error("could not open file \"%s\": %m", path);
+		exit(1);
+	}
+
+	if (verbose)
+		pg_log_info("reading version independent replication slot state file");
+
+	/* read part of statefile that's guaranteed to be version independent */
+	readBytes = read(fd, &cp, ReplicationSlotOnDiskConstantSize);
+	if (readBytes != ReplicationSlotOnDiskConstantSize)
+	{
+		if (readBytes < 0)
+		{
+			pg_log_error("could not read file \"%s\": %m", path);
+			exit(1);
+		}
+		else
+		{
+			pg_log_error("could not read file \"%s\": read %d of %zu",
+						 path, readBytes,
+						 (Size) ReplicationSlotOnDiskConstantSize);
+			exit(1);
+		}
+	}
+
+	/* verify magic */
+	if (cp.magic != SLOT_MAGIC)
+	{
+		pg_log_error("replication slot file \"%s\" has wrong magic number: %u instead of %u",
+					 path, cp.magic, SLOT_MAGIC);
+		exit(1);
+	}
+
+	/* verify version */
+	if (cp.version != SLOT_VERSION)
+	{
+		pg_log_error("replication slot file \"%s\" has unsupported version %u",
+					 path, cp.version);
+		exit(1);
+	}
+
+	/* boundary check on length */
+	if (cp.length != ReplicationSlotOnDiskV2Size)
+	{
+		pg_log_error("replication slot file \"%s\" has corrupted length %u",
+					 path, cp.length);
+		exit(1);
+	}
+
+	if (verbose)
+		pg_log_info("reading the entire replication slot state file");
+
+	/* now that we know the size, read the entire file */
+	readBytes = read(fd,
+					 (char *) &cp + ReplicationSlotOnDiskConstantSize,
+					 cp.length);
+	if (readBytes != cp.length)
+	{
+		if (readBytes < 0)
+		{
+			pg_log_error("could not read file \"%s\": %m", path);
+			exit(1);
+		}
+		else
+		{
+			pg_log_error("could not read file \"%s\": read %d of %zu",
+						 path, readBytes, (Size) cp.length);
+			exit(1);
+		}
+	}
+
+	if (close(fd) != 0)
+	{
+		pg_log_error("could not close file \"%s\": %m", path);
+		exit(1);
+	}
+
+	/* now verify the CRC */
+	INIT_CRC32C(checksum);
+	COMP_CRC32C(checksum,
+				(char *) &cp + ReplicationSlotOnDiskNotChecksummedSize,
+				ReplicationSlotOnDiskChecksummedSize);
+	FIN_CRC32C(checksum);
+
+	if (!EQ_CRC32C(checksum, cp.checksum))
+	{
+		pg_log_error("checksum mismatch for replication slot file \"%s\": is %u, should be %u",
+					 path, checksum, cp.checksum);
+		exit(1);
+	}
+
+	sprintf(restart_lsn, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.restart_lsn));
+	sprintf(invalidated_at, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.invalidated_at));
+	sprintf(confirmed_flush, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.confirmed_flush));
+	sprintf(two_phase_at, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.two_phase_at));
+
+	if (cp.slotdata.persistency == RS_PERSISTENT)
+		sprintf(persistency, "persistent");
+	else if (cp.slotdata.persistency == RS_EPHEMERAL)
+		sprintf(persistency, "ephemeral");
+	else if (cp.slotdata.persistency == RS_TEMPORARY)
+		sprintf(persistency, "temporary");
+
+	/* display the slot information */
+	printf("%-64s %9s %10u %11s %10u %12u %21s %21s %21s %21s %10d %20s\n",
+		   NameStr(cp.slotdata.name),
+		   cp.slotdata.database == InvalidOid ? "physical" : "logical",
+		   cp.slotdata.database,
+		   persistency,
+		   cp.slotdata.xmin,
+		   cp.slotdata.catalog_xmin,
+		   restart_lsn,
+		   invalidated_at,
+		   confirmed_flush,
+		   two_phase_at,
+		   cp.slotdata.two_phase,
+		   NameStr(cp.slotdata.plugin));
+}
+
+int
+main(int argc, char *argv[])
+{
+	static struct option long_options[] = {
+		{"pgdata", required_argument, NULL, 'D'},
+		{"verbose", no_argument, NULL, 'v'},
+		{NULL, 0, NULL, 0}
+	};
+
+	char	   *DataDir = NULL;
+	const char *progname;
+	int			c;
+
+	pg_logging_init(argv[0]);
+	set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_replslotdata"));
+	progname = get_progname(argv[0]);
+
+	if (argc > 1)
+	{
+		if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+		{
+			usage(progname);
+			exit(0);
+		}
+		if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0)
+		{
+			puts("pg_replslotdata (PostgreSQL) " PG_VERSION);
+			exit(0);
+		}
+	}
+
+	while ((c = getopt_long(argc, argv, "D:", long_options, NULL)) != -1)
+	{
+		switch (c)
+		{
+			case 'D':
+				DataDir = optarg;
+				break;
+			case 'v':
+				verbose = true;
+				break;
+			default:
+				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+				exit(1);
+		}
+	}
+
+	if (DataDir == NULL)
+	{
+		if (optind < argc)
+			DataDir = argv[optind++];
+		else
+			DataDir = getenv("PGDATA");
+	}
+
+	/* Complain if any arguments remain */
+	if (optind < argc)
+	{
+		pg_log_error("too many command-line arguments (first is \"%s\")",
+					 argv[optind]);
+		fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+				progname);
+		exit(1);
+	}
+
+	if (DataDir == NULL)
+	{
+		pg_log_error("no data directory specified");
+		fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+		exit(1);
+	}
+
+	if (verbose)
+		pg_log_info("data directory: \"%s\"", DataDir);
+
+	if (chdir(DataDir) < 0)
+	{
+		pg_log_error("could not change directory to \"%s\": %m",
+					 DataDir);
+		exit(1);
+	}
+
+	process_replslots();
+
+	return 0;
+}
diff --git a/src/bin/pg_replslotdata/t/001_basic.pl b/src/bin/pg_replslotdata/t/001_basic.pl
new file mode 100644
index 0000000000..d6830dc2ac
--- /dev/null
+++ b/src/bin/pg_replslotdata/t/001_basic.pl
@@ -0,0 +1,11 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 8;
+
+program_help_ok('pg_replslotdata');
+program_version_ok('pg_replslotdata');
+program_options_handling_ok('pg_replslotdata');
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 53d773ccff..479d504a68 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -15,89 +15,9 @@
 #include "storage/lwlock.h"
 #include "storage/shmem.h"
 #include "storage/spin.h"
+#include "replication/slot_common.h"
 #include "replication/walreceiver.h"
 
-/*
- * Behaviour of replication slots, upon release or crash.
- *
- * Slots marked as PERSISTENT are crash-safe and will not be dropped when
- * released. Slots marked as EPHEMERAL will be dropped when released or after
- * restarts.  Slots marked TEMPORARY will be dropped at the end of a session
- * or on error.
- *
- * EPHEMERAL is used as a not-quite-ready state when creating persistent
- * slots.  EPHEMERAL slots can be made PERSISTENT by calling
- * ReplicationSlotPersist().  For a slot that goes away at the end of a
- * session, TEMPORARY is the appropriate choice.
- */
-typedef enum ReplicationSlotPersistency
-{
-	RS_PERSISTENT,
-	RS_EPHEMERAL,
-	RS_TEMPORARY
-} ReplicationSlotPersistency;
-
-/*
- * On-Disk data of a replication slot, preserved across restarts.
- */
-typedef struct ReplicationSlotPersistentData
-{
-	/* The slot's identifier */
-	NameData	name;
-
-	/* database the slot is active on */
-	Oid			database;
-
-	/*
-	 * The slot's behaviour when being dropped (or restored after a crash).
-	 */
-	ReplicationSlotPersistency persistency;
-
-	/*
-	 * xmin horizon for data
-	 *
-	 * NB: This may represent a value that hasn't been written to disk yet;
-	 * see notes for effective_xmin, below.
-	 */
-	TransactionId xmin;
-
-	/*
-	 * xmin horizon for catalog tuples
-	 *
-	 * NB: This may represent a value that hasn't been written to disk yet;
-	 * see notes for effective_xmin, below.
-	 */
-	TransactionId catalog_xmin;
-
-	/* oldest LSN that might be required by this replication slot */
-	XLogRecPtr	restart_lsn;
-
-	/* restart_lsn is copied here when the slot is invalidated */
-	XLogRecPtr	invalidated_at;
-
-	/*
-	 * Oldest LSN that the client has acked receipt for.  This is used as the
-	 * start_lsn point in case the client doesn't specify one, and also as a
-	 * safety measure to jump forwards in case the client specifies a
-	 * start_lsn that's further in the past than this value.
-	 */
-	XLogRecPtr	confirmed_flush;
-
-	/*
-	 * LSN at which we enabled two_phase commit for this slot or LSN at which
-	 * we found a consistent point at the time of slot creation.
-	 */
-	XLogRecPtr	two_phase_at;
-
-	/*
-	 * Allow decoding of prepared transactions?
-	 */
-	bool		two_phase;
-
-	/* plugin name */
-	NameData	plugin;
-} ReplicationSlotPersistentData;
-
 /*
  * Shared memory state of a single replication slot.
  *
diff --git a/src/include/replication/slot_common.h b/src/include/replication/slot_common.h
new file mode 100644
index 0000000000..2d221637bb
--- /dev/null
+++ b/src/include/replication/slot_common.h
@@ -0,0 +1,132 @@
+/*-------------------------------------------------------------------------
+ * slot_common.h
+ *	   Replication slot management.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef SLOT_COMMON_H
+#define SLOT_COMMON_H
+
+/*
+ * Behaviour of replication slots, upon release or crash.
+ *
+ * Slots marked as PERSISTENT are crash-safe and will not be dropped when
+ * released. Slots marked as EPHEMERAL will be dropped when released or after
+ * restarts.  Slots marked TEMPORARY will be dropped at the end of a session
+ * or on error.
+ *
+ * EPHEMERAL is used as a not-quite-ready state when creating persistent
+ * slots.  EPHEMERAL slots can be made PERSISTENT by calling
+ * ReplicationSlotPersist().  For a slot that goes away at the end of a
+ * session, TEMPORARY is the appropriate choice.
+ */
+typedef enum ReplicationSlotPersistency
+{
+	RS_PERSISTENT,
+	RS_EPHEMERAL,
+	RS_TEMPORARY
+} ReplicationSlotPersistency;
+
+/*
+ * On-Disk data of a replication slot, preserved across restarts.
+ */
+typedef struct ReplicationSlotPersistentData
+{
+	/* The slot's identifier */
+	NameData	name;
+
+	/* database the slot is active on */
+	Oid			database;
+
+	/*
+	 * The slot's behaviour when being dropped (or restored after a crash).
+	 */
+	ReplicationSlotPersistency persistency;
+
+	/*
+	 * xmin horizon for data
+	 *
+	 * NB: This may represent a value that hasn't been written to disk yet;
+	 * see notes for effective_xmin, below.
+	 */
+	TransactionId xmin;
+
+	/*
+	 * xmin horizon for catalog tuples
+	 *
+	 * NB: This may represent a value that hasn't been written to disk yet;
+	 * see notes for effective_xmin, below.
+	 */
+	TransactionId catalog_xmin;
+
+	/* oldest LSN that might be required by this replication slot */
+	XLogRecPtr	restart_lsn;
+
+	/* restart_lsn is copied here when the slot is invalidated */
+	XLogRecPtr	invalidated_at;
+
+	/*
+	 * Oldest LSN that the client has acked receipt for.  This is used as the
+	 * start_lsn point in case the client doesn't specify one, and also as a
+	 * safety measure to jump forwards in case the client specifies a
+	 * start_lsn that's further in the past than this value.
+	 */
+	XLogRecPtr	confirmed_flush;
+
+	/*
+	 * LSN at which we enabled two_phase commit for this slot or LSN at which
+	 * we found a consistent point at the time of slot creation.
+	 */
+	XLogRecPtr	two_phase_at;
+
+	/*
+	 * Allow decoding of prepared transactions?
+	 */
+	bool		two_phase;
+
+	/* plugin name */
+	NameData	plugin;
+} ReplicationSlotPersistentData;
+
+/*
+ * Replication slot on-disk data structure.
+ */
+typedef struct ReplicationSlotOnDisk
+{
+	/* first part of this struct needs to be version independent */
+
+	/* data not covered by checksum */
+	uint32		magic;
+	pg_crc32c	checksum;
+
+	/* data covered by checksum */
+	uint32		version;
+	uint32		length;
+
+	/*
+	 * The actual data in the slot that follows can differ based on the above
+	 * 'version'.
+	 */
+
+	ReplicationSlotPersistentData slotdata;
+} ReplicationSlotOnDisk;
+
+/* size of version independent data */
+#define ReplicationSlotOnDiskConstantSize \
+	offsetof(ReplicationSlotOnDisk, slotdata)
+/* size of the part of the slot not covered by the checksum */
+#define ReplicationSlotOnDiskNotChecksummedSize \
+	offsetof(ReplicationSlotOnDisk, version)
+/* size of the part covered by the checksum */
+#define ReplicationSlotOnDiskChecksummedSize \
+	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskNotChecksummedSize
+/* size of the slot data that is version dependent */
+#define ReplicationSlotOnDiskV2Size \
+	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskConstantSize
+
+#define SLOT_MAGIC		0x1051CA1	/* format identifier */
+#define SLOT_VERSION	2		/* version for new files */
+
+#endif							/* SLOT_COMMON_H */
-- 
2.33.0.windows.2

#4Japin Li
japinli@hotmail.com
In reply to: Bharath Rupireddy (#3)
Re: pg_replslotdata - a tool for displaying replication slot information

On Wed, 24 Nov 2021 at 23:59, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote:

On Wed, Nov 24, 2021 at 9:09 PM Bharath Rupireddy

Thoughts?

Attaching the rebased v2 patch.

On windows the previous patches were failing, fixed that in the v3
patch. I'm really sorry for the noise.

Cool! When I try to use it, there is an error for -v, --verbose option.

px@ubuntu:~/Codes/postgres/Debug$ pg_replslotdata -v
pg_replslotdata: invalid option -- 'v'
Try "pg_replslotdata --help" for more information.

This is because the getopt_long() missing 'v' in the third parameter.

while ((c = getopt_long(argc, argv, "D:v", long_options, NULL)) != -1)

--
Regrads,
Japin Li.
ChengDu WenWu Information Technology Co.,Ltd.

#5Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Japin Li (#4)
1 attachment(s)
Re: pg_replslotdata - a tool for displaying replication slot information

On Wed, Nov 24, 2021 at 9:40 PM Japin Li <japinli@hotmail.com> wrote:

On Wed, 24 Nov 2021 at 23:59, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote:

On Wed, Nov 24, 2021 at 9:09 PM Bharath Rupireddy

Thoughts?

Attaching the rebased v2 patch.

On windows the previous patches were failing, fixed that in the v3
patch. I'm really sorry for the noise.

Cool! When I try to use it, there is an error for -v, --verbose option.

px@ubuntu:~/Codes/postgres/Debug$ pg_replslotdata -v
pg_replslotdata: invalid option -- 'v'
Try "pg_replslotdata --help" for more information.

This is because the getopt_long() missing 'v' in the third parameter.

while ((c = getopt_long(argc, argv, "D:v", long_options, NULL)) != -1)

Thanks for taking a look at the patch, attaching v4.

There are many things that I could do in the patch, for instance, more
comments, documentation, code improvements etc. I would like to first
know what hackers think about this tool, and then start spending more
time on it.

Regards,
Bharath Rupireddy.

Attachments:

v4-0001-pg_replslotdata.patchapplication/octet-stream; name=v4-0001-pg_replslotdata.patchDownload
From 0d787e51dca4ea2edc9be77b9cd6d0f6e3777614 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Thu, 25 Nov 2021 04:45:37 +0000
Subject: [PATCH v4] pg_replslotdata

---
 src/backend/replication/slot.c            |  39 ---
 src/bin/Makefile                          |   1 +
 src/bin/pg_replslotdata/.gitignore        |   2 +
 src/bin/pg_replslotdata/Makefile          |  44 +++
 src/bin/pg_replslotdata/nls.mk            |   6 +
 src/bin/pg_replslotdata/pg_replslotdata.c | 364 ++++++++++++++++++++++
 src/bin/pg_replslotdata/t/001_basic.pl    |  11 +
 src/include/replication/slot.h            |  82 +----
 src/include/replication/slot_common.h     | 132 ++++++++
 9 files changed, 561 insertions(+), 120 deletions(-)
 create mode 100644 src/bin/pg_replslotdata/.gitignore
 create mode 100644 src/bin/pg_replslotdata/Makefile
 create mode 100644 src/bin/pg_replslotdata/nls.mk
 create mode 100644 src/bin/pg_replslotdata/pg_replslotdata.c
 create mode 100644 src/bin/pg_replslotdata/t/001_basic.pl
 create mode 100644 src/include/replication/slot_common.h

diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 90ba9b417d..236ebe999a 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -50,45 +50,6 @@
 #include "storage/procarray.h"
 #include "utils/builtins.h"
 
-/*
- * Replication slot on-disk data structure.
- */
-typedef struct ReplicationSlotOnDisk
-{
-	/* first part of this struct needs to be version independent */
-
-	/* data not covered by checksum */
-	uint32		magic;
-	pg_crc32c	checksum;
-
-	/* data covered by checksum */
-	uint32		version;
-	uint32		length;
-
-	/*
-	 * The actual data in the slot that follows can differ based on the above
-	 * 'version'.
-	 */
-
-	ReplicationSlotPersistentData slotdata;
-} ReplicationSlotOnDisk;
-
-/* size of version independent data */
-#define ReplicationSlotOnDiskConstantSize \
-	offsetof(ReplicationSlotOnDisk, slotdata)
-/* size of the part of the slot not covered by the checksum */
-#define ReplicationSlotOnDiskNotChecksummedSize  \
-	offsetof(ReplicationSlotOnDisk, version)
-/* size of the part covered by the checksum */
-#define ReplicationSlotOnDiskChecksummedSize \
-	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskNotChecksummedSize
-/* size of the slot data that is version dependent */
-#define ReplicationSlotOnDiskV2Size \
-	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskConstantSize
-
-#define SLOT_MAGIC		0x1051CA1	/* format identifier */
-#define SLOT_VERSION	2		/* version for new files */
-
 /* Control array for replication slot management */
 ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
 
diff --git a/src/bin/Makefile b/src/bin/Makefile
index 2fe0ae6652..5e20009beb 100644
--- a/src/bin/Makefile
+++ b/src/bin/Makefile
@@ -23,6 +23,7 @@ SUBDIRS = \
 	pg_controldata \
 	pg_ctl \
 	pg_dump \
+	pg_replslotdata \
 	pg_resetwal \
 	pg_rewind \
 	pg_test_fsync \
diff --git a/src/bin/pg_replslotdata/.gitignore b/src/bin/pg_replslotdata/.gitignore
new file mode 100644
index 0000000000..13a4afb8ef
--- /dev/null
+++ b/src/bin/pg_replslotdata/.gitignore
@@ -0,0 +1,2 @@
+/pg_replslotdata
+/tmp_check/
diff --git a/src/bin/pg_replslotdata/Makefile b/src/bin/pg_replslotdata/Makefile
new file mode 100644
index 0000000000..69518ee53b
--- /dev/null
+++ b/src/bin/pg_replslotdata/Makefile
@@ -0,0 +1,44 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/bin/pg_replslotdata
+#
+# Copyright (c) 1998-2021, PostgreSQL Global Development Group
+#
+# src/bin/pg_replslotdata/Makefile
+#
+#-------------------------------------------------------------------------
+
+PGFILEDESC = "pg_replslotdata - provides information about the replication slots from $PGDATA/pg_replslot/<slot_name> $PGDATA/pg_replslot/<slot_name>"
+PGAPPICON=win32
+
+subdir = src/bin/pg_replslotdata
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = \
+	$(WIN32RES) \
+	pg_replslotdata.o
+
+all: pg_replslotdata
+
+pg_replslotdata: $(OBJS) | submake-libpgport
+	$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
+install: all installdirs
+	$(INSTALL_PROGRAM) pg_replslotdata$(X) '$(DESTDIR)$(bindir)/pg_replslotdata$(X)'
+
+installdirs:
+	$(MKDIR_P) '$(DESTDIR)$(bindir)'
+
+uninstall:
+	rm -f '$(DESTDIR)$(bindir)/pg_replslotdata$(X)'
+
+clean distclean maintainer-clean:
+	rm -f pg_replslotdata$(X) $(OBJS)
+	rm -rf tmp_check
+
+check:
+	$(prove_check)
+
+installcheck:
+	$(prove_installcheck)
diff --git a/src/bin/pg_replslotdata/nls.mk b/src/bin/pg_replslotdata/nls.mk
new file mode 100644
index 0000000000..74bee593c9
--- /dev/null
+++ b/src/bin/pg_replslotdata/nls.mk
@@ -0,0 +1,6 @@
+# src/bin/pg_replslotdata/nls.mk
+CATALOG_NAME     = pg_replslotdata
+AVAIL_LANGUAGES  = cs de el es fr ja ko pl ru sv tr uk vi zh_CN
+GETTEXT_FILES    = $(FRONTEND_COMMON_GETTEXT_FILES) pg_replslotdata.c
+GETTEXT_TRIGGERS = $(FRONTEND_COMMON_GETTEXT_TRIGGERS)
+GETTEXT_FLAGS    = $(FRONTEND_COMMON_GETTEXT_FLAGS)
diff --git a/src/bin/pg_replslotdata/pg_replslotdata.c b/src/bin/pg_replslotdata/pg_replslotdata.c
new file mode 100644
index 0000000000..d55c1291ce
--- /dev/null
+++ b/src/bin/pg_replslotdata/pg_replslotdata.c
@@ -0,0 +1,364 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_replslotdata.c - provides information about the replication slots
+ * from $PGDATA/pg_replslot/<slot_name>.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ *		  src/bin/pg_replslotdata/pg_replslotdata.c
+ *-------------------------------------------------------------------------
+ */
+/*
+ * We have to use postgres.h not postgres_fe.h here, because there's so much
+ * backend-only stuff in the XLOG include files we need.  But we need a
+ * frontend-ish environment otherwise.  Hence this ugly hack.
+ */
+#define FRONTEND 1
+
+#include "postgres.h"
+
+#include <sys/stat.h>
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "getopt_long.h"
+#include "pg_getopt.h"
+#include "replication/slot_common.h"
+
+static bool verbose = false;
+
+static void process_replslots(void);
+static void read_and_display_repl_slot(const char *name);
+
+static void
+usage(const char *progname)
+{
+	printf(_("%s displays information about the replication slots from $PGDATA/pg_replslot/<slot_name>.\n\n"), progname);
+	printf(_("Usage:\n"));
+	printf(_("  %s [OPTION] [DATADIR]\n"), progname);
+	printf(_("\nOptions:\n"));
+	printf(_(" [-D, --pgdata=]DATADIR  data directory\n"));
+	printf(_("  -V, --version          output version information, then exit\n"));
+	printf(_("  -v, --verbose          write a lot of output\n"));
+	printf(_("  -?, --help             show this help, then exit\n"));
+	printf(_("\nIf no data directory (DATADIR) is specified, "
+			 "the environment variable PGDATA\nis used.\n\n"));
+	printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+	printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+static void
+process_replslots(void)
+{
+	DIR		   *rsdir;
+	struct dirent *rsde;
+	uint32		cnt = 0;
+
+	rsdir = opendir("pg_replslot");
+	if (rsdir == NULL)
+	{
+		pg_log_error("could not open directory \"%s\": %m", "pg_replslot");
+		exit(1);
+	}
+
+	/* XXX: comment here about the format spefiiers */
+	printf("%-64s %9s %10s %11s %10s %12s %21s %21s %21s %21s %10s %20s\n"
+		   "%-64s %9s %10s %11s %10s %12s %21s %21s %21s %21s %10s %20s\n",
+		   "slot_name", "slot_type", "datoid", "persistency", "xmin", "catalog_xmin", "restart_lsn", "invalidated_at", "confirmed_flush", "two_phase_at", "two_phase", "plugin",
+		   "---------", "---------", "------", "-----------", "----", "------------", "-----------", "--------------", "---------------", "------------", "---------", "------");
+
+	while (errno = 0, (rsde = readdir(rsdir)) != NULL)
+	{
+		struct stat statbuf;
+		char		path[MAXPGPATH];
+
+		if (strcmp(rsde->d_name, ".") == 0 ||
+			strcmp(rsde->d_name, "..") == 0)
+			continue;
+
+		snprintf(path, sizeof(path), "pg_replslot/%s", rsde->d_name);
+
+		/* we're only creating directories here, skip if it's not our's */
+		if (lstat(path, &statbuf) == 0 && !S_ISDIR(statbuf.st_mode))
+			continue;
+
+		/* we crashed while a slot was being setup or deleted, clean up */
+		if (pg_str_endswith(rsde->d_name, ".tmp"))
+		{
+			pg_log_warning("server was crashed while the slot \"%s\" was being setup or deleted",
+						   rsde->d_name);
+			continue;
+		}
+
+		/* looks like a slot in a normal state, restore */
+		read_and_display_repl_slot(rsde->d_name);
+		cnt++;
+	}
+
+	if (errno)
+	{
+		pg_log_error("could not read directory \"%s\": %m", "pg_replslot");
+		exit(1);
+	}
+
+	if (cnt == 0)
+	{
+		pg_log_info("no replication slots were found");
+		exit(0);
+	}
+
+	if (closedir(rsdir))
+	{
+		pg_log_error("could not close directory \"%s\": %m", "pg_replslot");
+		exit(1);
+	}
+}
+
+static void
+read_and_display_repl_slot(const char *name)
+{
+	ReplicationSlotOnDisk cp;
+	char		slotdir[MAXPGPATH];
+	char		path[MAXPGPATH];
+	char		restart_lsn[NAMEDATALEN];
+	char		invalidated_at[NAMEDATALEN];
+	char		confirmed_flush[NAMEDATALEN];
+	char		two_phase_at[NAMEDATALEN];
+	char		persistency[NAMEDATALEN];
+	int			fd;
+	int			readBytes;
+	pg_crc32c	checksum;
+
+	/* delete temp file if it exists */
+	sprintf(slotdir, "pg_replslot/%s", name);
+	sprintf(path, "%s/state.tmp", slotdir);
+
+	fd = open(path, O_RDONLY | PG_BINARY, 0);
+
+	if (fd > 0)
+	{
+		pg_log_error("found temporary state file \"%s\": %m", path);
+		exit(1);
+	}
+
+	sprintf(path, "%s/state", slotdir);
+
+	if (verbose)
+		pg_log_info("reading replication slot from \"%s\"", path);
+
+	fd = open(path, O_RDONLY | PG_BINARY, 0);
+
+	/*
+	 * We do not need to handle this as we are rename()ing the directory into
+	 * place only after we fsync()ed the state file.
+	 */
+	if (fd < 0)
+	{
+		pg_log_error("could not open file \"%s\": %m", path);
+		exit(1);
+	}
+
+	if (verbose)
+		pg_log_info("reading version independent replication slot state file");
+
+	/* read part of statefile that's guaranteed to be version independent */
+	readBytes = read(fd, &cp, ReplicationSlotOnDiskConstantSize);
+	if (readBytes != ReplicationSlotOnDiskConstantSize)
+	{
+		if (readBytes < 0)
+		{
+			pg_log_error("could not read file \"%s\": %m", path);
+			exit(1);
+		}
+		else
+		{
+			pg_log_error("could not read file \"%s\": read %d of %zu",
+						 path, readBytes,
+						 (Size) ReplicationSlotOnDiskConstantSize);
+			exit(1);
+		}
+	}
+
+	/* verify magic */
+	if (cp.magic != SLOT_MAGIC)
+	{
+		pg_log_error("replication slot file \"%s\" has wrong magic number: %u instead of %u",
+					 path, cp.magic, SLOT_MAGIC);
+		exit(1);
+	}
+
+	/* verify version */
+	if (cp.version != SLOT_VERSION)
+	{
+		pg_log_error("replication slot file \"%s\" has unsupported version %u",
+					 path, cp.version);
+		exit(1);
+	}
+
+	/* boundary check on length */
+	if (cp.length != ReplicationSlotOnDiskV2Size)
+	{
+		pg_log_error("replication slot file \"%s\" has corrupted length %u",
+					 path, cp.length);
+		exit(1);
+	}
+
+	if (verbose)
+		pg_log_info("reading the entire replication slot state file");
+
+	/* now that we know the size, read the entire file */
+	readBytes = read(fd,
+					 (char *) &cp + ReplicationSlotOnDiskConstantSize,
+					 cp.length);
+	if (readBytes != cp.length)
+	{
+		if (readBytes < 0)
+		{
+			pg_log_error("could not read file \"%s\": %m", path);
+			exit(1);
+		}
+		else
+		{
+			pg_log_error("could not read file \"%s\": read %d of %zu",
+						 path, readBytes, (Size) cp.length);
+			exit(1);
+		}
+	}
+
+	if (close(fd) != 0)
+	{
+		pg_log_error("could not close file \"%s\": %m", path);
+		exit(1);
+	}
+
+	/* now verify the CRC */
+	INIT_CRC32C(checksum);
+	COMP_CRC32C(checksum,
+				(char *) &cp + ReplicationSlotOnDiskNotChecksummedSize,
+				ReplicationSlotOnDiskChecksummedSize);
+	FIN_CRC32C(checksum);
+
+	if (!EQ_CRC32C(checksum, cp.checksum))
+	{
+		pg_log_error("checksum mismatch for replication slot file \"%s\": is %u, should be %u",
+					 path, checksum, cp.checksum);
+		exit(1);
+	}
+
+	sprintf(restart_lsn, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.restart_lsn));
+	sprintf(invalidated_at, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.invalidated_at));
+	sprintf(confirmed_flush, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.confirmed_flush));
+	sprintf(two_phase_at, "%X/%X", LSN_FORMAT_ARGS(cp.slotdata.two_phase_at));
+
+	if (cp.slotdata.persistency == RS_PERSISTENT)
+		sprintf(persistency, "persistent");
+	else if (cp.slotdata.persistency == RS_EPHEMERAL)
+		sprintf(persistency, "ephemeral");
+	else if (cp.slotdata.persistency == RS_TEMPORARY)
+		sprintf(persistency, "temporary");
+
+	/* display the slot information */
+	printf("%-64s %9s %10u %11s %10u %12u %21s %21s %21s %21s %10d %20s\n",
+		   NameStr(cp.slotdata.name),
+		   cp.slotdata.database == InvalidOid ? "physical" : "logical",
+		   cp.slotdata.database,
+		   persistency,
+		   cp.slotdata.xmin,
+		   cp.slotdata.catalog_xmin,
+		   restart_lsn,
+		   invalidated_at,
+		   confirmed_flush,
+		   two_phase_at,
+		   cp.slotdata.two_phase,
+		   NameStr(cp.slotdata.plugin));
+}
+
+int
+main(int argc, char *argv[])
+{
+	static struct option long_options[] = {
+		{"pgdata", required_argument, NULL, 'D'},
+		{"verbose", no_argument, NULL, 'v'},
+		{NULL, 0, NULL, 0}
+	};
+
+	char	   *DataDir = NULL;
+	const char *progname;
+	int			c;
+
+	pg_logging_init(argv[0]);
+	set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_replslotdata"));
+	progname = get_progname(argv[0]);
+
+	if (argc > 1)
+	{
+		if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+		{
+			usage(progname);
+			exit(0);
+		}
+		if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0)
+		{
+			puts("pg_replslotdata (PostgreSQL) " PG_VERSION);
+			exit(0);
+		}
+	}
+
+	while ((c = getopt_long(argc, argv, "D:v", long_options, NULL)) != -1)
+	{
+		switch (c)
+		{
+			case 'D':
+				DataDir = optarg;
+				break;
+			case 'v':
+				verbose = true;
+				break;
+			default:
+				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+				exit(1);
+		}
+	}
+
+	if (DataDir == NULL)
+	{
+		if (optind < argc)
+			DataDir = argv[optind++];
+		else
+			DataDir = getenv("PGDATA");
+	}
+
+	/* complain if any arguments remain */
+	if (optind < argc)
+	{
+		pg_log_error("too many command-line arguments (first is \"%s\")",
+					 argv[optind]);
+		fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+				progname);
+		exit(1);
+	}
+
+	if (DataDir == NULL)
+	{
+		pg_log_error("no data directory specified");
+		fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+		exit(1);
+	}
+
+	if (verbose)
+		pg_log_info("data directory: \"%s\"", DataDir);
+
+	if (chdir(DataDir) < 0)
+	{
+		pg_log_error("could not change directory to \"%s\": %m",
+					 DataDir);
+		exit(1);
+	}
+
+	process_replslots();
+
+	return 0;
+}
diff --git a/src/bin/pg_replslotdata/t/001_basic.pl b/src/bin/pg_replslotdata/t/001_basic.pl
new file mode 100644
index 0000000000..d6830dc2ac
--- /dev/null
+++ b/src/bin/pg_replslotdata/t/001_basic.pl
@@ -0,0 +1,11 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 8;
+
+program_help_ok('pg_replslotdata');
+program_version_ok('pg_replslotdata');
+program_options_handling_ok('pg_replslotdata');
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 53d773ccff..479d504a68 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -15,89 +15,9 @@
 #include "storage/lwlock.h"
 #include "storage/shmem.h"
 #include "storage/spin.h"
+#include "replication/slot_common.h"
 #include "replication/walreceiver.h"
 
-/*
- * Behaviour of replication slots, upon release or crash.
- *
- * Slots marked as PERSISTENT are crash-safe and will not be dropped when
- * released. Slots marked as EPHEMERAL will be dropped when released or after
- * restarts.  Slots marked TEMPORARY will be dropped at the end of a session
- * or on error.
- *
- * EPHEMERAL is used as a not-quite-ready state when creating persistent
- * slots.  EPHEMERAL slots can be made PERSISTENT by calling
- * ReplicationSlotPersist().  For a slot that goes away at the end of a
- * session, TEMPORARY is the appropriate choice.
- */
-typedef enum ReplicationSlotPersistency
-{
-	RS_PERSISTENT,
-	RS_EPHEMERAL,
-	RS_TEMPORARY
-} ReplicationSlotPersistency;
-
-/*
- * On-Disk data of a replication slot, preserved across restarts.
- */
-typedef struct ReplicationSlotPersistentData
-{
-	/* The slot's identifier */
-	NameData	name;
-
-	/* database the slot is active on */
-	Oid			database;
-
-	/*
-	 * The slot's behaviour when being dropped (or restored after a crash).
-	 */
-	ReplicationSlotPersistency persistency;
-
-	/*
-	 * xmin horizon for data
-	 *
-	 * NB: This may represent a value that hasn't been written to disk yet;
-	 * see notes for effective_xmin, below.
-	 */
-	TransactionId xmin;
-
-	/*
-	 * xmin horizon for catalog tuples
-	 *
-	 * NB: This may represent a value that hasn't been written to disk yet;
-	 * see notes for effective_xmin, below.
-	 */
-	TransactionId catalog_xmin;
-
-	/* oldest LSN that might be required by this replication slot */
-	XLogRecPtr	restart_lsn;
-
-	/* restart_lsn is copied here when the slot is invalidated */
-	XLogRecPtr	invalidated_at;
-
-	/*
-	 * Oldest LSN that the client has acked receipt for.  This is used as the
-	 * start_lsn point in case the client doesn't specify one, and also as a
-	 * safety measure to jump forwards in case the client specifies a
-	 * start_lsn that's further in the past than this value.
-	 */
-	XLogRecPtr	confirmed_flush;
-
-	/*
-	 * LSN at which we enabled two_phase commit for this slot or LSN at which
-	 * we found a consistent point at the time of slot creation.
-	 */
-	XLogRecPtr	two_phase_at;
-
-	/*
-	 * Allow decoding of prepared transactions?
-	 */
-	bool		two_phase;
-
-	/* plugin name */
-	NameData	plugin;
-} ReplicationSlotPersistentData;
-
 /*
  * Shared memory state of a single replication slot.
  *
diff --git a/src/include/replication/slot_common.h b/src/include/replication/slot_common.h
new file mode 100644
index 0000000000..2d221637bb
--- /dev/null
+++ b/src/include/replication/slot_common.h
@@ -0,0 +1,132 @@
+/*-------------------------------------------------------------------------
+ * slot_common.h
+ *	   Replication slot management.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef SLOT_COMMON_H
+#define SLOT_COMMON_H
+
+/*
+ * Behaviour of replication slots, upon release or crash.
+ *
+ * Slots marked as PERSISTENT are crash-safe and will not be dropped when
+ * released. Slots marked as EPHEMERAL will be dropped when released or after
+ * restarts.  Slots marked TEMPORARY will be dropped at the end of a session
+ * or on error.
+ *
+ * EPHEMERAL is used as a not-quite-ready state when creating persistent
+ * slots.  EPHEMERAL slots can be made PERSISTENT by calling
+ * ReplicationSlotPersist().  For a slot that goes away at the end of a
+ * session, TEMPORARY is the appropriate choice.
+ */
+typedef enum ReplicationSlotPersistency
+{
+	RS_PERSISTENT,
+	RS_EPHEMERAL,
+	RS_TEMPORARY
+} ReplicationSlotPersistency;
+
+/*
+ * On-Disk data of a replication slot, preserved across restarts.
+ */
+typedef struct ReplicationSlotPersistentData
+{
+	/* The slot's identifier */
+	NameData	name;
+
+	/* database the slot is active on */
+	Oid			database;
+
+	/*
+	 * The slot's behaviour when being dropped (or restored after a crash).
+	 */
+	ReplicationSlotPersistency persistency;
+
+	/*
+	 * xmin horizon for data
+	 *
+	 * NB: This may represent a value that hasn't been written to disk yet;
+	 * see notes for effective_xmin, below.
+	 */
+	TransactionId xmin;
+
+	/*
+	 * xmin horizon for catalog tuples
+	 *
+	 * NB: This may represent a value that hasn't been written to disk yet;
+	 * see notes for effective_xmin, below.
+	 */
+	TransactionId catalog_xmin;
+
+	/* oldest LSN that might be required by this replication slot */
+	XLogRecPtr	restart_lsn;
+
+	/* restart_lsn is copied here when the slot is invalidated */
+	XLogRecPtr	invalidated_at;
+
+	/*
+	 * Oldest LSN that the client has acked receipt for.  This is used as the
+	 * start_lsn point in case the client doesn't specify one, and also as a
+	 * safety measure to jump forwards in case the client specifies a
+	 * start_lsn that's further in the past than this value.
+	 */
+	XLogRecPtr	confirmed_flush;
+
+	/*
+	 * LSN at which we enabled two_phase commit for this slot or LSN at which
+	 * we found a consistent point at the time of slot creation.
+	 */
+	XLogRecPtr	two_phase_at;
+
+	/*
+	 * Allow decoding of prepared transactions?
+	 */
+	bool		two_phase;
+
+	/* plugin name */
+	NameData	plugin;
+} ReplicationSlotPersistentData;
+
+/*
+ * Replication slot on-disk data structure.
+ */
+typedef struct ReplicationSlotOnDisk
+{
+	/* first part of this struct needs to be version independent */
+
+	/* data not covered by checksum */
+	uint32		magic;
+	pg_crc32c	checksum;
+
+	/* data covered by checksum */
+	uint32		version;
+	uint32		length;
+
+	/*
+	 * The actual data in the slot that follows can differ based on the above
+	 * 'version'.
+	 */
+
+	ReplicationSlotPersistentData slotdata;
+} ReplicationSlotOnDisk;
+
+/* size of version independent data */
+#define ReplicationSlotOnDiskConstantSize \
+	offsetof(ReplicationSlotOnDisk, slotdata)
+/* size of the part of the slot not covered by the checksum */
+#define ReplicationSlotOnDiskNotChecksummedSize \
+	offsetof(ReplicationSlotOnDisk, version)
+/* size of the part covered by the checksum */
+#define ReplicationSlotOnDiskChecksummedSize \
+	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskNotChecksummedSize
+/* size of the slot data that is version dependent */
+#define ReplicationSlotOnDiskV2Size \
+	sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskConstantSize
+
+#define SLOT_MAGIC		0x1051CA1	/* format identifier */
+#define SLOT_VERSION	2		/* version for new files */
+
+#endif							/* SLOT_COMMON_H */
-- 
2.25.1

#6Peter Eisentraut
peter.eisentraut@enterprisedb.com
In reply to: Bharath Rupireddy (#1)
Re: pg_replslotdata - a tool for displaying replication slot information

On 23.11.21 06:09, Bharath Rupireddy wrote:

The replication slots data is stored in binary format on the disk under
the pg_replslot/<<slot_name>> directory which isn't human readable. If
the server is crashed/down (for whatever reasons) and unable to come up,
currently there's no way for the user/admin/developer to know what were
all the replication slots available at the time of server crash/down to
figure out what's the restart lsn, xid, two phase info or types of slots
etc.

What do you need that for? You can't do anything with a replication
slot while the server is down.

#7Bossart, Nathan
bossartn@amazon.com
In reply to: Peter Eisentraut (#6)
Re: pg_replslotdata - a tool for displaying replication slot information

On 11/30/21, 6:14 AM, "Peter Eisentraut" <peter.eisentraut@enterprisedb.com> wrote:

On 23.11.21 06:09, Bharath Rupireddy wrote:

The replication slots data is stored in binary format on the disk under
the pg_replslot/<<slot_name>> directory which isn't human readable. If
the server is crashed/down (for whatever reasons) and unable to come up,
currently there's no way for the user/admin/developer to know what were
all the replication slots available at the time of server crash/down to
figure out what's the restart lsn, xid, two phase info or types of slots
etc.

What do you need that for? You can't do anything with a replication
slot while the server is down.

One use-case might be to discover the value you need to set for
max_replication_slots, although it's pretty trivial to discover the
number of replication slots by looking at the folder directly.
However, you also need to know how many replication origins there are,
and AFAIK there isn't an easy way to read the replorigin_checkpoint
file at the moment. IMO a utility like this should also show details
for the replication origins. I don't have any other compelling use-
cases at the moment, but I will say that it is typically nice from an
administrative standpoint to be able to inspect things like this
without logging into a running server.

Nathan

#8Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Bossart, Nathan (#7)
Re: pg_replslotdata - a tool for displaying replication slot information

On Wed, Dec 1, 2021 at 12:13 AM Bossart, Nathan <bossartn@amazon.com> wrote:

On 11/30/21, 6:14 AM, "Peter Eisentraut" <peter.eisentraut@enterprisedb.com> wrote:

On 23.11.21 06:09, Bharath Rupireddy wrote:

The replication slots data is stored in binary format on the disk under
the pg_replslot/<<slot_name>> directory which isn't human readable. If
the server is crashed/down (for whatever reasons) and unable to come up,
currently there's no way for the user/admin/developer to know what were
all the replication slots available at the time of server crash/down to
figure out what's the restart lsn, xid, two phase info or types of slots
etc.

What do you need that for? You can't do anything with a replication
slot while the server is down.

One use-case might be to discover the value you need to set for
max_replication_slots, although it's pretty trivial to discover the
number of replication slots by looking at the folder directly.

Apart from the above use-case, one can do some exploratory analysis on
the replication slot information after the server crash, this may be
useful for RCA or debugging purposes, for instance:
1) to look at the restart_lsn of the slots to get to know why there
were many WAL files filled up on the disk (because of the restart_lsn
being low)
2) to know how many replication slots available at the time of crash,
if required, one can choose to drop selective replication slots or the
ones that were falling behind to make the server up
3) if we persist active_pid info of the replication slot to the
disk(currently we don't have this info in the disk), one can get to
know the inactive replication slots at the time of crash
4) if the primary server is down and failover were to happen on to the
standby, by looking at the replication slot information on the
primary, one can easily recreate the slots on the standby

However, you also need to know how many replication origins there are,
and AFAIK there isn't an easy way to read the replorigin_checkpoint
file at the moment. IMO a utility like this should also show details
for the replication origins. I don't have any other compelling use-
cases at the moment, but I will say that it is typically nice from an
administrative standpoint to be able to inspect things like this
without logging into a running server.

Yeah, this can be added too, probably as an extra option to the
proposed pg_replslotdata tool. But for now, let's deal with the
replication slot information alone and once this gets committed, we
can extend it further for replication origin info.

Regards,
Bharath Rupireddy.

#9SATYANARAYANA NARLAPURAM
satyanarlapuram@gmail.com
In reply to: Bharath Rupireddy (#8)
Re: pg_replslotdata - a tool for displaying replication slot information

On Tue, Nov 30, 2021 at 9:47 PM Bharath Rupireddy <
bharath.rupireddyforpostgres@gmail.com> wrote:

On Wed, Dec 1, 2021 at 12:13 AM Bossart, Nathan <bossartn@amazon.com>
wrote:

On 11/30/21, 6:14 AM, "Peter Eisentraut" <

peter.eisentraut@enterprisedb.com> wrote:

On 23.11.21 06:09, Bharath Rupireddy wrote:

The replication slots data is stored in binary format on the disk

under

the pg_replslot/<<slot_name>> directory which isn't human readable. If
the server is crashed/down (for whatever reasons) and unable to come

up,

currently there's no way for the user/admin/developer to know what

were

all the replication slots available at the time of server crash/down

to

figure out what's the restart lsn, xid, two phase info or types of

slots

etc.

What do you need that for? You can't do anything with a replication
slot while the server is down.

One use-case might be to discover the value you need to set for
max_replication_slots, although it's pretty trivial to discover the
number of replication slots by looking at the folder directly.

Apart from the above use-case, one can do some exploratory analysis on
the replication slot information after the server crash, this may be
useful for RCA or debugging purposes, for instance:
1) to look at the restart_lsn of the slots to get to know why there
were many WAL files filled up on the disk (because of the restart_lsn
being low)

In a disk full scenario because of WAL, this tool comes handy identifying
which WAL files to delete to free up the space and also help assess the
accidental delete of the WAL files. I am not sure if there is a tool to
help cleanup the WAL (may be invoking the archive_command too?) without
impacting physical / logical slots, and respecting last checkpoint location
but if one exist that will be handy

#10Andres Freund
andres@anarazel.de
In reply to: Bossart, Nathan (#7)
Re: pg_replslotdata - a tool for displaying replication slot information

Hi,

On 2021-11-30 18:43:23 +0000, Bossart, Nathan wrote:

On 11/30/21, 6:14 AM, "Peter Eisentraut" <peter.eisentraut@enterprisedb.com> wrote:

On 23.11.21 06:09, Bharath Rupireddy wrote:

The replication slots data is stored in binary format on the disk under
the pg_replslot/<<slot_name>> directory which isn't human readable. If
the server is crashed/down (for whatever reasons) and unable to come up,
currently there's no way for the user/admin/developer to know what were
all the replication slots available at the time of server crash/down to
figure out what's the restart lsn, xid, two phase info or types of slots
etc.

What do you need that for? You can't do anything with a replication
slot while the server is down.

Yes, I don't think there's sufficient need for this.

I don't have any other compelling use- cases at the moment, but I will say
that it is typically nice from an administrative standpoint to be able to
inspect things like this without logging into a running server.

Weighed against the cost of maintaining (including documentation) a separate
tool this doesn't seem sufficient reason.

Greetings,

Andres Freund

#11Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Andres Freund (#10)
Re: pg_replslotdata - a tool for displaying replication slot information

On Thu, Dec 2, 2021 at 4:22 AM Andres Freund <andres@anarazel.de> wrote:

Hi,

On 2021-11-30 18:43:23 +0000, Bossart, Nathan wrote:

On 11/30/21, 6:14 AM, "Peter Eisentraut" <peter.eisentraut@enterprisedb.com> wrote:

On 23.11.21 06:09, Bharath Rupireddy wrote:

The replication slots data is stored in binary format on the disk under
the pg_replslot/<<slot_name>> directory which isn't human readable. If
the server is crashed/down (for whatever reasons) and unable to come up,
currently there's no way for the user/admin/developer to know what were
all the replication slots available at the time of server crash/down to
figure out what's the restart lsn, xid, two phase info or types of slots
etc.

What do you need that for? You can't do anything with a replication
slot while the server is down.

Yes, I don't think there's sufficient need for this.

Thanks. The idea of the pg_replslotdata is emanated from the real-time
working experience with the customer issues and answering some of
their questions. Given the fact that replication slots are used in
almost every major production servers, and they are likely to cause
problems, postgres having a core tool like pg_replslotdata to
interpret the replication slot info without contacting the server,
will definitely be useful for all the other postgres vendors out
there. Having some important tool in the core, can avoid duplicate
efforts.

I don't have any other compelling use- cases at the moment, but I will say
that it is typically nice from an administrative standpoint to be able to
inspect things like this without logging into a running server.

Weighed against the cost of maintaining (including documentation) a separate
tool this doesn't seem sufficient reason.

IMHO, this shouldn't be a reason to not get something useful (useful
IMO and few others in this thread) into the core. The maintenance of
the tools generally is low compared to the core server features once
they get reviewed and committed.

Having said that, other hackers may have better thoughts.

Regards,
Bharath Rupireddy.

#12Michael Paquier
michael@paquier.xyz
In reply to: Bharath Rupireddy (#11)
Re: pg_replslotdata - a tool for displaying replication slot information

On Thu, Dec 02, 2021 at 08:32:08AM +0530, Bharath Rupireddy wrote:

On Thu, Dec 2, 2021 at 4:22 AM Andres Freund <andres@anarazel.de> wrote:

I don't have any other compelling use- cases at the moment, but I will say
that it is typically nice from an administrative standpoint to be able to
inspect things like this without logging into a running server.

Weighed against the cost of maintaining (including documentation) a separate
tool this doesn't seem sufficient reason.

IMHO, this shouldn't be a reason to not get something useful (useful
IMO and few others in this thread) into the core. The maintenance of
the tools generally is low compared to the core server features once
they get reviewed and committed.

Well, a bit less maintenance is always better than more maintenance.
An extra cost that you may be missing is related to the translation of
the documentation, as well as the translation of any new strings this
would require. FWIW, I don't directly see a use for this tool that
could not be solved with an online server.
--
Michael

#13Bossart, Nathan
bossartn@amazon.com
In reply to: Michael Paquier (#12)
Re: pg_replslotdata - a tool for displaying replication slot information

On 12/5/21, 11:10 PM, "Michael Paquier" <michael@paquier.xyz> wrote:

On Thu, Dec 02, 2021 at 08:32:08AM +0530, Bharath Rupireddy wrote:

On Thu, Dec 2, 2021 at 4:22 AM Andres Freund <andres@anarazel.de> wrote:

I don't have any other compelling use- cases at the moment, but I will say
that it is typically nice from an administrative standpoint to be able to
inspect things like this without logging into a running server.

Weighed against the cost of maintaining (including documentation) a separate
tool this doesn't seem sufficient reason.

IMHO, this shouldn't be a reason to not get something useful (useful
IMO and few others in this thread) into the core. The maintenance of
the tools generally is low compared to the core server features once
they get reviewed and committed.

Well, a bit less maintenance is always better than more maintenance.
An extra cost that you may be missing is related to the translation of
the documentation, as well as the translation of any new strings this
would require. FWIW, I don't directly see a use for this tool that
could not be solved with an online server.

Bharath, perhaps you should maintain this outside of core PostgreSQL
for now. If some compelling use-cases ever surface that make it seem
worth the added maintenance burden, this thread could probably be
revisited.

Nathan

#14Julien Rouhaud
rjuju123@gmail.com
In reply to: Bossart, Nathan (#13)
Re: pg_replslotdata - a tool for displaying replication slot information

Hi,

On Mon, Dec 06, 2021 at 07:16:12PM +0000, Bossart, Nathan wrote:

On 12/5/21, 11:10 PM, "Michael Paquier" <michael@paquier.xyz> wrote:

On Thu, Dec 02, 2021 at 08:32:08AM +0530, Bharath Rupireddy wrote:

On Thu, Dec 2, 2021 at 4:22 AM Andres Freund <andres@anarazel.de> wrote:

I don't have any other compelling use- cases at the moment, but I will say
that it is typically nice from an administrative standpoint to be able to
inspect things like this without logging into a running server.

Weighed against the cost of maintaining (including documentation) a separate
tool this doesn't seem sufficient reason.

IMHO, this shouldn't be a reason to not get something useful (useful
IMO and few others in this thread) into the core. The maintenance of
the tools generally is low compared to the core server features once
they get reviewed and committed.

Well, a bit less maintenance is always better than more maintenance.
An extra cost that you may be missing is related to the translation of
the documentation, as well as the translation of any new strings this
would require. FWIW, I don't directly see a use for this tool that
could not be solved with an online server.

Bharath, perhaps you should maintain this outside of core PostgreSQL
for now. If some compelling use-cases ever surface that make it seem
worth the added maintenance burden, this thread could probably be
revisited.

Ironically, the patch is currently failing due to translation problem:

https://cirrus-ci.com/task/5467034313031680
[19:12:28.179] su postgres -c "make -s -j${BUILD_JOBS} world-bin"
[19:12:44.270] make[3]: *** No rule to make target 'po/cs.po', needed by 'po/cs.mo'. Stop.
[19:12:44.270] make[2]: *** [Makefile:44: all-pg_replslotdata-recurse] Error 2
[19:12:44.270] make[2]: *** Waiting for unfinished jobs....
[19:12:44.499] make[1]: *** [Makefile:42: all-bin-recurse] Error 2
[19:12:44.499] make: *** [GNUmakefile:21: world-bin-src-recurse] Error 2

Looking at the thread, I see support from 3 people:

- Bharath
- Japin
- Satyanarayana

while 3 committers think that the extra maintenance effort isn't worth the
usage:

- Peter E.
- Andres
- Michael

and a +0.5 from Nathan IIUC.

I also personally don't think that this worth the maintenance effort. This
tool being entirely client side, there's no problem with maintaining it on a
separate repository, as mentioned by Nathan, including using it on the cloud
providers that provides access to at least the data file. Another pro of the
external repo is that the tool can be made available immediately and for older
releases.

Since 3 committers voted against it I think that the patch should be closed
as "Rejected". I will do that in a few days unless there's some compelling
objection by then.

#15Bharath Rupireddy
bharath.rupireddyforpostgres@gmail.com
In reply to: Julien Rouhaud (#14)
Re: pg_replslotdata - a tool for displaying replication slot information

On Sat, Jan 15, 2022 at 2:20 PM Julien Rouhaud <rjuju123@gmail.com> wrote:

Bharath, perhaps you should maintain this outside of core PostgreSQL
for now. If some compelling use-cases ever surface that make it seem
worth the added maintenance burden, this thread could probably be
revisited.

Ironically, the patch is currently failing due to translation problem:

https://cirrus-ci.com/task/5467034313031680
[19:12:28.179] su postgres -c "make -s -j${BUILD_JOBS} world-bin"
[19:12:44.270] make[3]: *** No rule to make target 'po/cs.po', needed by 'po/cs.mo'. Stop.
[19:12:44.270] make[2]: *** [Makefile:44: all-pg_replslotdata-recurse] Error 2
[19:12:44.270] make[2]: *** Waiting for unfinished jobs....
[19:12:44.499] make[1]: *** [Makefile:42: all-bin-recurse] Error 2
[19:12:44.499] make: *** [GNUmakefile:21: world-bin-src-recurse] Error 2

Thanks Juilen. I'm okay if the patch gets dropped. But, I'm curious to
know why the above error occurred. Is it because I included the nls.mk
file in the patch which I'm not supposed to? Are these nls.mk files
generated as part of the commit that does translation changes?

Regards,
Bharath Rupireddy.

#16Julien Rouhaud
rjuju123@gmail.com
In reply to: Bharath Rupireddy (#15)
Re: pg_replslotdata - a tool for displaying replication slot information

Hi,

On Mon, Jan 17, 2022 at 04:10:13PM +0530, Bharath Rupireddy wrote:

Thanks Juilen. I'm okay if the patch gets dropped.

Ok, I will take care of that soon.

But, I'm curious to
know why the above error occurred. Is it because I included the nls.mk
file in the patch which I'm not supposed to? Are these nls.mk files
generated as part of the commit that does translation changes?

Not exactly. I think it's a good thing to take care of the translatability in
the initial submission, at least for the infrastructure part. So the nlk.mk
and the _() function are fine, but you should have added an empty
AVAIL_LANGUAGES in your nlk.mk to avoid those errors. The translation is being
done at a later stage by the various teams on babel ([1]https://babel.postgresql.org/) and then synced
periodically (usually by PeterE, thanks a lot to him!) in the tree.

[1]: https://babel.postgresql.org/

#17Gurjeet
gurjeet@singh.im
In reply to: Julien Rouhaud (#16)
Re: pg_replslotdata - a tool for displaying replication slot information

On Mon Jan 17, 2022 at 5:11 AM PST, Julien Rouhaud wrote:

On Mon, Jan 17, 2022 at 04:10:13PM +0530, Bharath Rupireddy wrote:

Thanks Juilen. I'm okay if the patch gets dropped.

Ok, I will take care of that soon.

I find this utility interesting and useful, especially for the reason
that it can provide information about the replication slots without
consuming a connection. I would be willing to continue the work on it.

Just checking here if, a year later, anyone has seen any more, or
interesting use-cases that would make it a candidate for its inclusion
in Postgres.

Best regards,
Gurjeet, http://Gurje.et
Postgres Contributors Team, https://aws.amazon.com/opensource