Coverage improvements of src/bin/pg_basebackup and pg_receivewal --endpos

Started by Michael Paquierover 8 years ago8 messages
#1Michael Paquier
michael.paquier@gmail.com

Hi all,

The coverage of pg_basebackup is reaching 50%, which is not bad:
https://coverage.postgresql.org/src/bin/pg_basebackup/index.html
In this set pg_receivewal.c is the bad student with less than 20%.

There are a couple of causes behind that, with no tests like:
- option interactions like --create-slot and --drop-slot.
- replication slot drop.
- end of streaming.
- check handling of history files.
- looking at compressed and uncompressed WAL data.
- restart of pg_receivewal on an existing folder.
- pg_basebackup's tests don't use the progress report.

While it is possible to tackle some of those issues independently,
like pg_basebackup stuff, it seems to me that it would be helpful to
make the tests more deterministic by having an --endpos option for
pg_receivewal, similarly to what exists already in pg_recvlogical.

Thoughts?
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#2Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#1)
1 attachment(s)
Re: Coverage improvements of src/bin/pg_basebackup and pg_receivewal --endpos

On Wed, Jun 7, 2017 at 9:20 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

While it is possible to tackle some of those issues independently,
like pg_basebackup stuff, it seems to me that it would be helpful to
make the tests more deterministic by having an --endpos option for
pg_receivewal, similarly to what exists already in pg_recvlogical.

Thoughts?

I have just played with that, and attached is a patch to implement the
so-said option with a basic set of tests, increasing code coverage of
pg_receivewal.c from 15% to 60%. I'll park that in the next CF of
September.
--
Michael

Attachments:

0001-Implement-pg_receivewal-endpos.patchapplication/octet-stream; name=0001-Implement-pg_receivewal-endpos.patchDownload
From 420059da8bd03c8e1ee6cda6991e7745385caf29 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Fri, 9 Jun 2017 13:39:37 +0900
Subject: [PATCH] Implement pg_receivewal --endpos

This is primarily useful in making any regression test using this utility
more deterministic as pg_receivewal cannot be started as a deamon in TAP
tests.

While this is less useful than the equivalent of pg_recvlogical, users
can as well use it for example to enforce WAL streaming up to a
end-of-backup position, to save only a minimal amount of WAL, though the
end position of WAL is only known once the backup is finished, but some
users may want to get that done as a two-step process with one single WAL
receiver in use all the time.

Use this new option to stream WAL data in a deterministic way within a
new set of TAP tests.
---
 doc/src/sgml/ref/pg_receivewal.sgml          | 16 ++++++++
 src/bin/pg_basebackup/pg_receivewal.c        | 35 +++++++++++++++-
 src/bin/pg_basebackup/t/020_pg_receivewal.pl | 61 +++++++++++++++++++++++++++-
 3 files changed, 110 insertions(+), 2 deletions(-)

diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index a17082bb11..80995c115b 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -93,6 +93,22 @@ PostgreSQL documentation
      </varlistentry>
 
      <varlistentry>
+      <term><option>-E <replaceable>lsn</replaceable></option></term>
+      <term><option>--endpos=<replaceable>lsn</replaceable></option></term>
+      <listitem>
+       <para>
+        If specified, automatically stop replication and exit with normal
+        exit status 0 when receiving reaches the specified LSN.
+       </para>
+
+       <para>
+        If there's a record with LSN exactly equal to <replaceable>lsn</>,
+        the record will be considered.
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>--if-not-exists</option></term>
       <listitem>
        <para>
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 370d871660..8a272c9ef1 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -42,6 +42,8 @@ static bool slot_exists_ok = false;
 static bool do_drop_slot = false;
 static bool synchronous = false;
 static char *replication_slot = NULL;
+static XLogRecPtr endpos = InvalidXLogRecPtr;
+static bool exit_on_stop = false;
 
 
 static void usage(void);
@@ -78,6 +80,7 @@ usage(void)
 	printf(_("\nOptions:\n"));
 	printf(_("  -D, --directory=DIR    receive write-ahead log files into this directory\n"));
 	printf(_("      --if-not-exists    do not error if slot already exists when creating a slot\n"));
+	printf(_("  -E, --endpos=LSN       exit after receiving the specified LSN\n"));
 	printf(_("  -n, --no-loop          do not loop on connection lost\n"));
 	printf(_("  -s, --status-interval=SECS\n"
 			 "                         time between status packets sent to server (default: %d)\n"), (standby_message_timeout / 1000));
@@ -112,6 +115,16 @@ stop_streaming(XLogRecPtr xlogpos, uint32 timeline, bool segment_finished)
 				progname, (uint32) (xlogpos >> 32), (uint32) xlogpos,
 				timeline);
 
+	if (!XLogRecPtrIsInvalid(endpos) && endpos < xlogpos)
+	{
+		if (verbose)
+			fprintf(stderr, _("%s: stopped streaming at %X/%X (timeline %u)\n"),
+					progname, (uint32) (xlogpos >> 32), (uint32) xlogpos,
+					timeline);
+		exit_on_stop = true;
+		return true;
+	}
+
 	/*
 	 * Note that we report the previous, not current, position here. After a
 	 * timeline switch, xlogpos points to the beginning of the segment because
@@ -459,6 +472,7 @@ main(int argc, char **argv)
 		{"version", no_argument, NULL, 'V'},
 		{"directory", required_argument, NULL, 'D'},
 		{"dbname", required_argument, NULL, 'd'},
+		{"endpos", required_argument, NULL, 'E'},
 		{"host", required_argument, NULL, 'h'},
 		{"port", required_argument, NULL, 'p'},
 		{"username", required_argument, NULL, 'U'},
@@ -480,6 +494,7 @@ main(int argc, char **argv)
 	int			c;
 	int			option_index;
 	char	   *db_name;
+	uint32		hi, lo;
 
 	progname = get_progname(argv[0]);
 	set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_basebackup"));
@@ -499,7 +514,7 @@ main(int argc, char **argv)
 		}
 	}
 
-	while ((c = getopt_long(argc, argv, "D:d:h:p:U:s:S:nwWvZ:",
+	while ((c = getopt_long(argc, argv, "D:d:E:h:p:U:s:S:nwWvZ:",
 							long_options, &option_index)) != -1)
 	{
 		switch (c)
@@ -543,6 +558,16 @@ main(int argc, char **argv)
 			case 'S':
 				replication_slot = pg_strdup(optarg);
 				break;
+			case 'E':
+				if (sscanf(optarg, "%X/%X", &hi, &lo) != 2)
+				{
+					fprintf(stderr,
+							_("%s: could not parse end position \"%s\"\n"),
+							progname, optarg);
+					exit(1);
+				}
+				endpos = ((uint64) hi) << 32 | lo;
+				break;
 			case 'n':
 				noloop = 1;
 				break;
@@ -721,6 +746,14 @@ main(int argc, char **argv)
 			 */
 			exit(0);
 		}
+		else if (exit_on_stop)
+		{
+			/*
+			 * End of streaming position has been willingly reached, so exit
+			 * without an error code.
+			 */
+			exit(0);
+		}
 		else if (noloop)
 		{
 			fprintf(stderr, _("%s: disconnected\n"), progname);
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index b4cb6f729d..db060cf4f5 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -1,8 +1,67 @@
 use strict;
 use warnings;
 use TestLib;
-use Test::More tests => 8;
+use PostgresNode;
+use Test::More tests => 15;
 
 program_help_ok('pg_receivewal');
 program_version_ok('pg_receivewal');
 program_options_handling_ok('pg_receivewal');
+
+my $primary = get_new_node('primary');
+$primary->init(allows_streaming => 1);
+$primary->start;
+
+my $stream_dir = $primary->basedir . '/archive_wal';
+mkdir($stream_dir);
+
+# Sanity checks for command line options.
+command_fails(['pg_receivewal'],
+	'pg_receivewal needs target directory specified');
+command_fails(['pg_receivewal', '-D', $stream_dir, '--create-slot',
+			   '--drop-slot'],
+	'failure if both --create-slot and --drop-slot specified');
+command_fails(['pg_receivewal', '-D', $stream_dir, '--create-slot'],
+	'failure if --create-slot defined without --slot');
+
+# Slot creation and drop
+my $slot_name = 'test';
+command_ok(['pg_receivewal', '--slot', $slot_name,
+	'-d', $primary->connstr(), '--create-slot' ],
+	'creation of replication slot');
+command_ok(['pg_receivewal', '--slot', $slot_name,
+	'-d', $primary->connstr(), '--drop-slot' ],
+	'drop of replication slot');
+
+# Generate some WAL using non-compression mode. Use --synchronous
+# at the same time to add more code coverage. Switch to the next
+# segment first so as subsequent restarts of pg_receivewal will
+# see this segment as full and non-compressed.
+$primary->psql('postgres', 'CREATE TABLE test_table(x integer);');
+$primary->psql('postgres', 'SELECT pg_switch_wal();');
+my $nextlsn =
+	$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+chomp($nextlsn);
+$primary->psql('postgres',
+	'INSERT INTO test_table VALUES (generate_series(1,100));');
+
+# Stream up to the given position.
+$primary->command_ok(
+	[   'pg_receivewal', '-d', $primary->connstr(), '-D', $stream_dir,
+		'--verbose', '--endpos', $nextlsn, '--synchronous', '--no-loop' ],
+	'streaming some WAL with --synchronous and without compression');
+
+# Now generate more WAL, switch to a new segment and stream
+# changes using the compression mode.
+$primary->psql('postgres',
+	'INSERT INTO test_table VALUES (generate_series(1,100));');
+$primary->psql('postgres', 'SELECT pg_switch_wal();');
+$nextlsn =
+	$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+chomp($nextlsn);
+$primary->psql('postgres',
+	'INSERT INTO test_table VALUES (generate_series(1,100));');
+$primary->command_ok(
+	[   'pg_receivewal', '-d', $primary->connstr(), '-D', $stream_dir,
+		'--verbose', '--endpos', $nextlsn, '--compress', '1', '--no-loop' ],
+	'streaming some WAL with with compression');
-- 
2.13.1

#3Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Michael Paquier (#2)
Re: Coverage improvements of src/bin/pg_basebackup and pg_receivewal --endpos

On 6/9/17 02:08, Michael Paquier wrote:

On Wed, Jun 7, 2017 at 9:20 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

While it is possible to tackle some of those issues independently,
like pg_basebackup stuff, it seems to me that it would be helpful to
make the tests more deterministic by having an --endpos option for
pg_receivewal, similarly to what exists already in pg_recvlogical.

Thoughts?

I have just played with that, and attached is a patch to implement the
so-said option with a basic set of tests, increasing code coverage of
pg_receivewal.c from 15% to 60%. I'll park that in the next CF of
September.

Looks like a good idea. A couple of thoughts:

- I think the tests should be written in the style of
$node->command_fails instead of just command_fails. At least, that's
how it's done in the pg_basebackup tests.

- I think the tests will fail if libz support is not built. Again,
pg_basebackup doesn't have tests for that. So maybe omit that and
address it later.

- The variable exit_on_stop seems redundant with time_to_abort. They do
the same thing, so maybe your patch could reuse it.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#4Michael Paquier
michael.paquier@gmail.com
In reply to: Peter Eisentraut (#3)
1 attachment(s)
Re: Coverage improvements of src/bin/pg_basebackup and pg_receivewal --endpos

On Tue, Sep 5, 2017 at 10:19 PM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

On 6/9/17 02:08, Michael Paquier wrote:

I have just played with that, and attached is a patch to implement the
so-said option with a basic set of tests, increasing code coverage of
pg_receivewal.c from 15% to 60%. I'll park that in the next CF of
September.

Looks like a good idea. A couple of thoughts:

Thanks.

- I think the tests should be written in the style of
$node->command_fails instead of just command_fails. At least, that's
how it's done in the pg_basebackup tests.

Good idea.

- I think the tests will fail if libz support is not built. Again,
pg_basebackup doesn't have tests for that. So maybe omit that and
address it later.

Let's address it now. This can be countered by querying pg_config()
before running the command of pg_receivexwal which uses --compress.

- The variable exit_on_stop seems redundant with time_to_abort. They do
the same thing, so maybe your patch could reuse it.

Yes, that's doable. time_to_abort does not seem a variable name
adapted to me though if both are combined, so I have renamed that to
time_to_stop, which maps more with the fact that stop can be also
willingly wanted without a SIGINT.

Attached is a new version of the patch.
--
Michael

Attachments:

0001-Implement-pg_receivewal-endpos.patchtext/x-patch; charset=US-ASCII; name=0001-Implement-pg_receivewal-endpos.patchDownload
From 7987b412b3b16f8995ccbe39c7de0e6762aa964d Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Wed, 6 Sep 2017 13:39:30 +0900
Subject: [PATCH] Implement pg_receivewal --endpos

This is primarily useful in making any regression test using this utility
more deterministic as pg_receivewal cannot be started as a deamon in TAP
tests.

While this is less useful than the equivalent of pg_recvlogical, users
can as well use it for example to enforce WAL streaming up to a
end-of-backup position, to save only a minimal amount of WAL, though the
end position of WAL is only known once the backup is finished, but some
users may want to get that done as a two-step process with one single WAL
receiver in use all the time.

Use this new option to stream WAL data in a deterministic way within a
new set of TAP tests.
---
 doc/src/sgml/ref/pg_receivewal.sgml          | 16 ++++++
 src/bin/pg_basebackup/pg_receivewal.c        | 38 +++++++++++---
 src/bin/pg_basebackup/t/020_pg_receivewal.pl | 75 +++++++++++++++++++++++++++-
 3 files changed, 121 insertions(+), 8 deletions(-)

diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml
index 7c82e36c7c..6d192bd606 100644
--- a/doc/src/sgml/ref/pg_receivewal.sgml
+++ b/doc/src/sgml/ref/pg_receivewal.sgml
@@ -98,6 +98,22 @@ PostgreSQL documentation
       </listitem>
      </varlistentry>
 
+     <varlistentry>
+      <term><option>-E <replaceable>lsn</replaceable></option></term>
+      <term><option>--endpos=<replaceable>lsn</replaceable></option></term>
+      <listitem>
+       <para>
+        If specified, automatically stop replication and exit with normal
+        exit status 0 when receiving reaches the specified LSN.
+       </para>
+
+       <para>
+        If there is a record with LSN exactly equal to <replaceable>lsn</>,
+        the record will be considered.
+       </para>
+      </listitem>
+     </varlistentry>
+
      <varlistentry>
       <term><option>--if-not-exists</option></term>
       <listitem>
diff --git a/src/bin/pg_basebackup/pg_receivewal.c b/src/bin/pg_basebackup/pg_receivewal.c
index 4a1a5658fb..93b396c10f 100644
--- a/src/bin/pg_basebackup/pg_receivewal.c
+++ b/src/bin/pg_basebackup/pg_receivewal.c
@@ -36,12 +36,13 @@ static int	verbose = 0;
 static int	compresslevel = 0;
 static int	noloop = 0;
 static int	standby_message_timeout = 10 * 1000;	/* 10 sec = default */
-static volatile bool time_to_abort = false;
+static volatile bool time_to_stop = false;
 static bool do_create_slot = false;
 static bool slot_exists_ok = false;
 static bool do_drop_slot = false;
 static bool synchronous = false;
 static char *replication_slot = NULL;
+static XLogRecPtr endpos = InvalidXLogRecPtr;
 
 
 static void usage(void);
@@ -78,6 +79,7 @@ usage(void)
 	printf(_("\nOptions:\n"));
 	printf(_("  -D, --directory=DIR    receive write-ahead log files into this directory\n"));
 	printf(_("      --if-not-exists    do not error if slot already exists when creating a slot\n"));
+	printf(_("  -E, --endpos=LSN       exit after receiving the specified LSN\n"));
 	printf(_("  -n, --no-loop          do not loop on connection lost\n"));
 	printf(_("  -s, --status-interval=SECS\n"
 			 "                         time between status packets sent to server (default: %d)\n"), (standby_message_timeout / 1000));
@@ -112,6 +114,16 @@ stop_streaming(XLogRecPtr xlogpos, uint32 timeline, bool segment_finished)
 				progname, (uint32) (xlogpos >> 32), (uint32) xlogpos,
 				timeline);
 
+	if (!XLogRecPtrIsInvalid(endpos) && endpos < xlogpos)
+	{
+		if (verbose)
+			fprintf(stderr, _("%s: stopped streaming at %X/%X (timeline %u)\n"),
+					progname, (uint32) (xlogpos >> 32), (uint32) xlogpos,
+					timeline);
+		time_to_stop = true;
+		return true;
+	}
+
 	/*
 	 * Note that we report the previous, not current, position here. After a
 	 * timeline switch, xlogpos points to the beginning of the segment because
@@ -128,7 +140,7 @@ stop_streaming(XLogRecPtr xlogpos, uint32 timeline, bool segment_finished)
 	prevtimeline = timeline;
 	prevpos = xlogpos;
 
-	if (time_to_abort)
+	if (time_to_stop)
 	{
 		if (verbose)
 			fprintf(stderr, _("%s: received interrupt signal, exiting\n"),
@@ -448,7 +460,7 @@ StreamLog(void)
 static void
 sigint_handler(int signum)
 {
-	time_to_abort = true;
+	time_to_stop = true;
 }
 #endif
 
@@ -460,6 +472,7 @@ main(int argc, char **argv)
 		{"version", no_argument, NULL, 'V'},
 		{"directory", required_argument, NULL, 'D'},
 		{"dbname", required_argument, NULL, 'd'},
+		{"endpos", required_argument, NULL, 'E'},
 		{"host", required_argument, NULL, 'h'},
 		{"port", required_argument, NULL, 'p'},
 		{"username", required_argument, NULL, 'U'},
@@ -481,6 +494,7 @@ main(int argc, char **argv)
 	int			c;
 	int			option_index;
 	char	   *db_name;
+	uint32		hi, lo;
 
 	progname = get_progname(argv[0]);
 	set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_basebackup"));
@@ -500,7 +514,7 @@ main(int argc, char **argv)
 		}
 	}
 
-	while ((c = getopt_long(argc, argv, "D:d:h:p:U:s:S:nwWvZ:",
+	while ((c = getopt_long(argc, argv, "D:d:E:h:p:U:s:S:nwWvZ:",
 							long_options, &option_index)) != -1)
 	{
 		switch (c)
@@ -544,6 +558,16 @@ main(int argc, char **argv)
 			case 'S':
 				replication_slot = pg_strdup(optarg);
 				break;
+			case 'E':
+				if (sscanf(optarg, "%X/%X", &hi, &lo) != 2)
+				{
+					fprintf(stderr,
+							_("%s: could not parse end position \"%s\"\n"),
+							progname, optarg);
+					exit(1);
+				}
+				endpos = ((uint64) hi) << 32 | lo;
+				break;
 			case 'n':
 				noloop = 1;
 				break;
@@ -714,11 +738,11 @@ main(int argc, char **argv)
 	while (true)
 	{
 		StreamLog();
-		if (time_to_abort)
+		if (time_to_stop)
 		{
 			/*
-			 * We've been Ctrl-C'ed. That's not an error, so exit without an
-			 * errorcode.
+			 * We've been Ctrl-C'ed or end of streaming position has been
+			 * willingly reached, so exit without an error code.
 			 */
 			exit(0);
 		}
diff --git a/src/bin/pg_basebackup/t/020_pg_receivewal.pl b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
index b4cb6f729d..2d14b8242f 100644
--- a/src/bin/pg_basebackup/t/020_pg_receivewal.pl
+++ b/src/bin/pg_basebackup/t/020_pg_receivewal.pl
@@ -1,8 +1,81 @@
 use strict;
 use warnings;
 use TestLib;
-use Test::More tests => 8;
+use PostgresNode;
+use Test::More tests => 15;
 
 program_help_ok('pg_receivewal');
 program_version_ok('pg_receivewal');
 program_options_handling_ok('pg_receivewal');
+
+my $primary = get_new_node('primary');
+$primary->init(allows_streaming => 1);
+$primary->start;
+
+my $stream_dir = $primary->basedir . '/archive_wal';
+mkdir($stream_dir);
+
+# Sanity checks for command line options.
+$primary->command_fails(['pg_receivewal'],
+	'pg_receivewal needs target directory specified');
+$primary->command_fails(['pg_receivewal', '-D', $stream_dir, '--create-slot',
+			   '--drop-slot'],
+	'failure if both --create-slot and --drop-slot specified');
+$primary->command_fails(['pg_receivewal', '-D', $stream_dir, '--create-slot'],
+	'failure if --create-slot defined without --slot');
+
+# Slot creation and drop
+my $slot_name = 'test';
+$primary->command_ok(['pg_receivewal', '--slot', $slot_name, '--create-slot' ],
+	'creation of replication slot');
+$primary->command_ok(['pg_receivewal', '--slot', $slot_name, '--drop-slot' ],
+	'drop of replication slot');
+
+# Generate some WAL using non-compression mode. Use --synchronous
+# at the same time to add more code coverage. Switch to the next
+# segment first so as subsequent restarts of pg_receivewal will
+# see this segment as full and non-compressed.
+$primary->psql('postgres', 'CREATE TABLE test_table(x integer);');
+$primary->psql('postgres', 'SELECT pg_switch_wal();');
+my $nextlsn =
+	$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+chomp($nextlsn);
+$primary->psql('postgres',
+	'INSERT INTO test_table VALUES (generate_series(1,100));');
+
+# Stream up to the given position.
+$primary->command_ok(
+	[   'pg_receivewal', '-D', $stream_dir, '--verbose', '--endpos',
+		$nextlsn, '--synchronous', '--no-loop' ],
+	'streaming some WAL with --synchronous and without compression');
+
+# Check if build supports compression with libz, in which case the following
+# command would pass or fail depending on its support.
+my $lz_support = $primary->safe_psql('postgres',
+	"SELECT setting ~ '-lz ' from pg_config WHERE name = 'LIBS'");
+chomp($lz_support);
+if ($lz_support eq 't')
+{
+	# Now generate more WAL, switch to a new segment and stream
+	# changes using the compression mode.
+	$primary->psql('postgres',
+		'INSERT INTO test_table VALUES (generate_series(1,100));');
+	$primary->psql('postgres', 'SELECT pg_switch_wal();');
+	$nextlsn =
+		$primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn();');
+	chomp($nextlsn);
+	$primary->psql('postgres',
+		'INSERT INTO test_table VALUES (generate_series(1,100));');
+
+	$primary->command_ok(
+		[   'pg_receivewal', '-D', $stream_dir, '--verbose', '--endpos',
+			$nextlsn, '--compress', '1', '--no-loop' ],
+		'streaming some WAL with compression');
+}
+else
+{
+	$primary->command_fails(
+		[   'pg_receivewal', '-D', $stream_dir, '--verbose', '--endpos',
+			$nextlsn, '--compress', '1', '--no-loop' ],
+		'compression not supported');
+}
-- 
2.14.1

#5Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Michael Paquier (#4)
Re: Coverage improvements of src/bin/pg_basebackup and pg_receivewal --endpos

On 9/6/17 00:54, Michael Paquier wrote:

- I think the tests will fail if libz support is not built. Again,
pg_basebackup doesn't have tests for that. So maybe omit that and
address it later.

Let's address it now. This can be countered by querying pg_config()
before running the command of pg_receivexwal which uses --compress.

I have committed this patch, after a perltidy run, but the way the libz
detection was implemented was a bit too hackish for me, so I have
omitted that for now. I think a more robust way would be to parse
Makefile.global, perhaps by a function in TestLib, so it can be reused
in other tests. (Even more robust would be to write out an actual Perl
file with configuration information somewhere, but maybe we don't need
to go there yet.) Or maybe have the respective make variables exported
when make check is called (could be included in the prove_check
variable?). Anyway, some more pondering could lead to a more general
solution.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#6Michael Paquier
michael.paquier@gmail.com
In reply to: Peter Eisentraut (#5)
Re: Coverage improvements of src/bin/pg_basebackup and pg_receivewal --endpos

On Tue, Sep 12, 2017 at 5:57 AM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

I have committed this patch, after a perltidy run, but the way the libz
detection was implemented was a bit too hackish for me, so I have
omitted that for now.

Thanks.

I think a more robust way would be to parse
Makefile.global, perhaps by a function in TestLib, so it can be reused
in other tests. (Even more robust would be to write out an actual Perl
file with configuration information somewhere, but maybe we don't need
to go there yet.) Or maybe have the respective make variables exported
when make check is called (could be included in the prove_check
variable?). Anyway, some more pondering could lead to a more general
solution.

There is always room for improvement,
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#7Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Michael Paquier (#6)
Re: Coverage improvements of src/bin/pg_basebackup and pg_receivewal --endpos

On 9/11/17 18:21, Michael Paquier wrote:

On Tue, Sep 12, 2017 at 5:57 AM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

I have committed this patch, after a perltidy run, but the way the libz
detection was implemented was a bit too hackish for me, so I have
omitted that for now.

Thanks.

I think a more robust way would be to parse
Makefile.global, perhaps by a function in TestLib, so it can be reused
in other tests. (Even more robust would be to write out an actual Perl
file with configuration information somewhere, but maybe we don't need
to go there yet.) Or maybe have the respective make variables exported
when make check is called (could be included in the prove_check
variable?). Anyway, some more pondering could lead to a more general
solution.

There is always room for improvement,

I interpret that as that you are not working on that right now, so I've
closed this patch.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#8Michael Paquier
michael.paquier@gmail.com
In reply to: Peter Eisentraut (#7)
Re: Coverage improvements of src/bin/pg_basebackup and pg_receivewal --endpos

On Tue, Sep 12, 2017 at 9:19 PM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

On 9/11/17 18:21, Michael Paquier wrote:

On Tue, Sep 12, 2017 at 5:57 AM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

I think a more robust way would be to parse
Makefile.global, perhaps by a function in TestLib, so it can be reused
in other tests. (Even more robust would be to write out an actual Perl
file with configuration information somewhere, but maybe we don't need
to go there yet.) Or maybe have the respective make variables exported
when make check is called (could be included in the prove_check
variable?). Anyway, some more pondering could lead to a more general
solution.

There is always room for improvement,

I interpret that as that you are not working on that right now, so I've
closed this patch.

Indeed, I have no plans for that now. There are enough patches to look
at these days. My apologies for not making that clear.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers