Add tablespace tap test to pg_rewind

Started by Shaoqi Baialmost 7 years ago20 messages
#1Shaoqi Bai
sbai@pivotal.io
1 attachment(s)

Hi hackers,

There is already a databases tap tests in pg_rewind, wonder if there is a
need for tablespace tap tests in pg_rewind.
Attached is a initial patch from me.

Here is a patch for runing pg_rewind, it is very similar to
src/bin/pg_rewind/t/002_databases.pl, but there is no master_psql("CREATE
TABLESPACE beforepromotion LOCATION '$tempdir/beforepromotion'"); after
create_standby() and before promote_standby(), because pg_rewind will error
out :
could not create directory
"/Users/sbai/work/postgres/src/bin/pg_rewind/tmp_check/t_006_tablespace_master_local_data/pgdata/pg_tblspc/24576/PG_12_201903063":
File exists
Failure, exiting

The patch is created on top of the
commit e1e0e8d58c5c70da92e36cb9d59c2f7ecf839e00 (origin/master, origin/HEAD)
Author: Michael Paquier <michael@paquier.xyz>
Date: Fri Mar 8 15:10:14 2019 +0900

Fix function signatures of pageinspect in documentation

tuple_data_split() lacked the type of the first argument, and
heap_page_item_attrs() has reversed the first and second argument,
with the bytea argument using an incorrect name.

Author: Laurenz Albe
Discussion:
/messages/by-id/8f9ab7b16daf623e87eeef5203a4ffc0dece8dfd.camel@cybertec.at
Backpatch-through: 9.6

Attachments:

0001-Add-tablespace-tap-test-for-pg_rewind.patchapplication/octet-stream; name=0001-Add-tablespace-tap-test-for-pg_rewind.patchDownload
From 4280335c4b81aa04b967d93348ccc6874bc9c287 Mon Sep 17 00:00:00 2001
From: Shaoqi Bai <sbai@pivotal.io>
Date: Fri, 8 Mar 2019 18:04:20 +0800
Subject: [PATCH] Add tablespace tap test for pg_rewind

---
 src/bin/pg_rewind/t/006_tablespace.pl | 51 +++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100644 src/bin/pg_rewind/t/006_tablespace.pl

diff --git a/src/bin/pg_rewind/t/006_tablespace.pl b/src/bin/pg_rewind/t/006_tablespace.pl
new file mode 100644
index 0000000000..094041dd89
--- /dev/null
+++ b/src/bin/pg_rewind/t/006_tablespace.pl
@@ -0,0 +1,51 @@
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use RewindTest;
+
+my $tempdir = TestLib::tempdir;
+
+sub run_test
+{
+	my $test_mode = shift;
+
+	RewindTest::setup_cluster($test_mode, ['-g']);
+	RewindTest::start_master();
+
+	RewindTest::create_standby($test_mode);
+
+	RewindTest::promote_standby();
+
+	mkdir "$tempdir/master_afterpromotion";
+	mkdir "$tempdir/standby_afterpromotion";
+
+	# Create tablespaces in the old master and the new promoted standby.
+	master_psql("CREATE TABLESPACE master_afterpromotion LOCATION '$tempdir/master_afterpromotion'");
+	standby_psql("CREATE TABLESPACE standby_afterpromotion LOCATION '$tempdir/standby_afterpromotion'");
+
+	# The clusters are now diverged.
+
+	RewindTest::run_pg_rewind($test_mode);
+
+	# Check that the correct databases are present after pg_rewind.
+	check_query(
+		'SELECT spcname FROM pg_tablespace ORDER BY spcname',
+		qq(pg_default
+pg_global
+standby_afterpromotion
+),
+		'tablespace names');
+
+	RewindTest::clean_rewind_test();
+	return;
+}
+
+# Run the test in both modes.
+run_test('local');
+
+exit(0);
-- 
2.19.1

#2Shaoqi Bai
sbai@pivotal.io
In reply to: Shaoqi Bai (#1)
1 attachment(s)
Fwd: Add tablespace tap test to pg_rewind

Hi hackers,

There is already a databases tap tests in pg_rewind, wonder if there is a
need for tablespace tap tests in pg_rewind.
Attached is a initial patch from me.

Here is a patch for runing pg_rewind, it is very similar to
src/bin/pg_rewind/t/002_databases.pl, but there is no master_psql("CREATE
TABLESPACE beforepromotion LOCATION '$tempdir/beforepromotion'"); after
create_standby() and before promote_standby(), because pg_rewind will error
out :
could not create directory
"/Users/sbai/work/postgres/src/bin/pg_rewind/tmp_check/t_006_tablespace_master_local_data/pgdata/pg_tblspc/24576/PG_12_201903063":
File exists
Failure, exiting

The patch is created on top of the
commit e1e0e8d58c5c70da92e36cb9d59c2f7ecf839e00 (origin/master, origin/HEAD)
Author: Michael Paquier <michael@paquier.xyz>
Date: Fri Mar 8 15:10:14 2019 +0900

Fix function signatures of pageinspect in documentation

tuple_data_split() lacked the type of the first argument, and
heap_page_item_attrs() has reversed the first and second argument,
with the bytea argument using an incorrect name.

Author: Laurenz Albe
Discussion:
/messages/by-id/8f9ab7b16daf623e87eeef5203a4ffc0dece8dfd.camel@cybertec.at
Backpatch-through: 9.6

Attachments:

0001-Add-tablespace-tap-test-for-pg_rewind.patchapplication/x-patch; name=0001-Add-tablespace-tap-test-for-pg_rewind.patchDownload
From 4280335c4b81aa04b967d93348ccc6874bc9c287 Mon Sep 17 00:00:00 2001
From: Shaoqi Bai <sbai@pivotal.io>
Date: Fri, 8 Mar 2019 18:04:20 +0800
Subject: [PATCH] Add tablespace tap test for pg_rewind

---
 src/bin/pg_rewind/t/006_tablespace.pl | 51 +++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100644 src/bin/pg_rewind/t/006_tablespace.pl

diff --git a/src/bin/pg_rewind/t/006_tablespace.pl b/src/bin/pg_rewind/t/006_tablespace.pl
new file mode 100644
index 0000000000..094041dd89
--- /dev/null
+++ b/src/bin/pg_rewind/t/006_tablespace.pl
@@ -0,0 +1,51 @@
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use RewindTest;
+
+my $tempdir = TestLib::tempdir;
+
+sub run_test
+{
+	my $test_mode = shift;
+
+	RewindTest::setup_cluster($test_mode, ['-g']);
+	RewindTest::start_master();
+
+	RewindTest::create_standby($test_mode);
+
+	RewindTest::promote_standby();
+
+	mkdir "$tempdir/master_afterpromotion";
+	mkdir "$tempdir/standby_afterpromotion";
+
+	# Create tablespaces in the old master and the new promoted standby.
+	master_psql("CREATE TABLESPACE master_afterpromotion LOCATION '$tempdir/master_afterpromotion'");
+	standby_psql("CREATE TABLESPACE standby_afterpromotion LOCATION '$tempdir/standby_afterpromotion'");
+
+	# The clusters are now diverged.
+
+	RewindTest::run_pg_rewind($test_mode);
+
+	# Check that the correct databases are present after pg_rewind.
+	check_query(
+		'SELECT spcname FROM pg_tablespace ORDER BY spcname',
+		qq(pg_default
+pg_global
+standby_afterpromotion
+),
+		'tablespace names');
+
+	RewindTest::clean_rewind_test();
+	return;
+}
+
+# Run the test in both modes.
+run_test('local');
+
+exit(0);
-- 
2.19.1

#3Michael Paquier
michael@paquier.xyz
In reply to: Shaoqi Bai (#2)
Re: Fwd: Add tablespace tap test to pg_rewind

On Fri, Mar 08, 2019 at 09:42:29PM +0800, Shaoqi Bai wrote:

There is already a databases tap tests in pg_rewind, wonder if there is a
need for tablespace tap tests in pg_rewind. Attached is a initial
patch from me.

When working on the first version of pg_rewind for VMware with Heikki,
tablespace support has been added only after, so what you propose is
sensible I think.

+# Run the test in both modes.
+run_test('local');
Small nit: we should test for the remote mode here as well.
--
Michael
#4Michael Paquier
michael@paquier.xyz
In reply to: Michael Paquier (#3)
Re: Fwd: Add tablespace tap test to pg_rewind

On Sat, Mar 09, 2019 at 09:09:24AM +0900, Michael Paquier wrote:

When working on the first version of pg_rewind for VMware with Heikki,
tablespace support has been added only after, so what you propose is
sensible I think.

+# Run the test in both modes.
+run_test('local');
Small nit: we should test for the remote mode here as well.

I got to think more about this one a bit more, and I think that it
would be good to also check the consistency of a tablespace created
before promotion. If you copy the logic from 002_databases.pl, this
is not going to work because the primary and the standby would try to
create the tablespace in the same path, stepping on each other's
toes. So what you need to do is to create the tablespace on the
primary because creating the standby. This requires a bit more work
than what you propose here though as you basically need to extend
RewindTest::create_standby so as it is possible to pass extra
arguments to $node_master->backup. And in this case the extra option
to use is --tablespace-mapping to make sure that the primary and the
standby have the same tablespace, but defined on different paths.
--
Michael

#5Shaoqi Bai
sbai@pivotal.io
In reply to: Michael Paquier (#4)
Re: Fwd: Add tablespace tap test to pg_rewind

Thanks, will work on it as you suggested
Add pg_basebackup --T olddir=newdir to support check the consistency of a
tablespace created before promotion
Add run_test('remote');

On Mon, Mar 11, 2019 at 6:50 AM Michael Paquier <michael@paquier.xyz> wrote:

Show quoted text

On Sat, Mar 09, 2019 at 09:09:24AM +0900, Michael Paquier wrote:

When working on the first version of pg_rewind for VMware with Heikki,
tablespace support has been added only after, so what you propose is
sensible I think.

+# Run the test in both modes.
+run_test('local');
Small nit: we should test for the remote mode here as well.

I got to think more about this one a bit more, and I think that it
would be good to also check the consistency of a tablespace created
before promotion. If you copy the logic from 002_databases.pl, this
is not going to work because the primary and the standby would try to
create the tablespace in the same path, stepping on each other's
toes. So what you need to do is to create the tablespace on the
primary because creating the standby. This requires a bit more work
than what you propose here though as you basically need to extend
RewindTest::create_standby so as it is possible to pass extra
arguments to $node_master->backup. And in this case the extra option
to use is --tablespace-mapping to make sure that the primary and the
standby have the same tablespace, but defined on different paths.
--
Michael

#6Michael Paquier
michael@paquier.xyz
In reply to: Shaoqi Bai (#5)
Re: Fwd: Add tablespace tap test to pg_rewind

On Mon, Mar 11, 2019 at 07:49:11PM +0800, Shaoqi Bai wrote:

Thanks, will work on it as you suggested
Add pg_basebackup --T olddir=newdir to support check the consistency of a
tablespace created before promotion
Add run_test('remote');

Thanks for considering my input. Why don't you register your patch to
the next commit fest then so as it goes through a formal review once
you are able to provide a new version? The commit fest is here:
https://commitfest.postgresql.org/23/

We are currently in the process of wrapping up the last commit fest
for v12, so this stuff will have to wait a bit :(

It could be an idea to split the patch in two pieces:
- One patch which refactors the code for the new option in
PostgresNode.pm
- Second patch for the new test with integration in RewindTest.pm.
This should touch different parts of the code, so combining both would
be fine as well for me :)
--
Michael

#7Shaoqi Bai
sbai@pivotal.io
In reply to: Michael Paquier (#6)
1 attachment(s)
Re: Fwd: Add tablespace tap test to pg_rewind

On Tue, Mar 12, 2019 at 10:27 AM Michael Paquier <michael@paquier.xyz>
wrote:

On Mon, Mar 11, 2019 at 07:49:11PM +0800, Shaoqi Bai wrote:

Thanks, will work on it as you suggested
Add pg_basebackup --T olddir=newdir to support check the consistency of a
tablespace created before promotion
Add run_test('remote');

Thanks for considering my input. Why don't you register your patch to
the next commit fest then so as it goes through a formal review once
you are able to provide a new version? The commit fest is here:
https://commitfest.postgresql.org/23/

We are currently in the process of wrapping up the last commit fest
for v12, so this stuff will have to wait a bit :(

It could be an idea to split the patch in two pieces:
- One patch which refactors the code for the new option in
PostgresNode.pm
- Second patch for the new test with integration in RewindTest.pm.
This should touch different parts of the code, so combining both would
be fine as well for me :)

Thanks for your advice, sorry for taking so long to give update in the
thread, because I am stuck in modifing Perl script, knowing little about
Perl language.

I tried to do the following two things in this patch.
1. add pg_basebackup --T olddir=newdir to support check the consistency of
a tablespace created before promotion
2. run both run_test('local') and run_test('remote');

The code can pass make installcheck under src/bin/pg_rewind, but can not
pass make installcheck under src/bin/pg_basebackup.
Because the patch refactor is not done well.

The patch still need refactor, to make other tests pass, like tests under
src/bin/pg_basebackup.
Sending the letter is just to let you know the little progress on the
thread.

Attachments:

0001-Add-tablespace-tap-test-for-pg_rewind.patchapplication/octet-stream; name=0001-Add-tablespace-tap-test-for-pg_rewind.patchDownload
From 43172d6e025af48a7ea31a8700d25455b20453c5 Mon Sep 17 00:00:00 2001
From: Shaoqi Bai <sbai@pivotal.io>
Date: Fri, 8 Mar 2019 18:04:20 +0800
Subject: [PATCH] Add tablespace tap test for pg_rewind

---
 src/bin/pg_rewind/t/006_tablespace.pl | 83 +++++++++++++++++++++++++++
 src/bin/pg_rewind/t/RewindTest.pm     | 26 +++++++++
 src/test/perl/PostgresNode.pm         | 26 +++++++++
 src/test/perl/RecursiveCopy.pm        | 18 +++++-
 4 files changed, 150 insertions(+), 3 deletions(-)
 create mode 100644 src/bin/pg_rewind/t/006_tablespace.pl

diff --git a/src/bin/pg_rewind/t/006_tablespace.pl b/src/bin/pg_rewind/t/006_tablespace.pl
new file mode 100644
index 0000000000..19b3403c90
--- /dev/null
+++ b/src/bin/pg_rewind/t/006_tablespace.pl
@@ -0,0 +1,83 @@
+use strict;
+use warnings;
+use File::Path qw(rmtree);
+use TestLib;
+use Test::More tests => 6;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use RewindTest;
+
+my $tempdir = TestLib::tempdir;
+
+sub run_test
+{
+	my $test_mode = shift;
+
+	RewindTest::setup_cluster($test_mode, ['-g']);
+	RewindTest::start_master();
+
+	rmtree("$tempdir/inmaster");
+	rmtree("$tempdir/instandby");
+	rmtree("$tempdir/master_beforepromotion");
+	rmtree("$tempdir/master_afterpromotion");
+	rmtree("$tempdir/standby_afterpromotion");
+
+	mkdir "$tempdir/inmaster";
+	mkdir "$tempdir/instandby";
+
+	# Create a tablespace in master.
+	master_psql("CREATE TABLESPACE inmaster LOCATION '$tempdir/inmaster'");
+
+	RewindTest::create_standby_tbl_mapping($test_mode, "$tempdir/inmaster", "$tempdir/instandby");
+
+	mkdir "$tempdir/master_beforepromotion";
+
+	# Create a tablespace, it has to be droped before doing pg_rewind, or else pg_rewind will fail
+	master_psql("CREATE TABLESPACE master_beforepromotion LOCATION '$tempdir/master_beforepromotion'");
+
+	RewindTest::promote_standby();
+
+	mkdir "$tempdir/master_afterpromotion";
+	mkdir "$tempdir/standby_afterpromotion";
+
+	# Create tablespaces in the old master and the new promoted standby.
+	master_psql("CREATE TABLESPACE master_afterpromotion LOCATION '$tempdir/master_afterpromotion'");
+	standby_psql("CREATE TABLESPACE standby_afterpromotion LOCATION '$tempdir/standby_afterpromotion'");
+	# Drop tablespace in the new promoted standby, because pg_rewind can not handle this case.
+	standby_psql("DROP TABLESPACE standby_afterpromotion");
+
+	# The clusters are now diverged.
+
+	RewindTest::run_pg_rewind($test_mode);
+
+	# Check that the correct databases are present after pg_rewind.
+	check_query(
+		'SELECT spcname FROM pg_tablespace ORDER BY spcname',
+		qq(inmaster
+master_beforepromotion
+pg_default
+pg_global
+),
+		'tablespace names');
+
+	# Permissions on PGDATA should have group permissions
+  SKIP:
+	{
+		skip "unix-style permissions not supported on Windows", 1
+		  if ($windows_os);
+
+		ok(check_mode_recursive($node_master->data_dir(), 0750, 0640),
+			'check PGDATA permissions');
+	}
+
+	RewindTest::clean_rewind_test();
+	return;
+}
+
+# Run the test in both modes.
+run_test('local');
+run_test('remote');
+
+exit(0);
diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm
index 85cae7e47b..81bf4d0e47 100644
--- a/src/bin/pg_rewind/t/RewindTest.pm
+++ b/src/bin/pg_rewind/t/RewindTest.pm
@@ -174,6 +174,32 @@ primary_conninfo='$connstr_master application_name=rewind_standby'
 	return;
 }
 
+sub create_standby_tbl_mapping
+{
+	my $extra_name = shift;
+
+	$node_standby =
+	  get_new_node('standby' . ($extra_name ? "_${extra_name}" : ''));
+	my ($olddir, $newdir) = @_;
+	$node_master->backup_tbl_mapping('my_backup', $olddir, $newdir);
+	$node_standby->init_from_backup($node_master, 'my_backup');
+	my $connstr_master = $node_master->connstr();
+	$node_standby->append_conf(
+		"postgresql.conf", qq(
+primary_conninfo='$connstr_master application_name=rewind_standby'
+));
+
+	$node_standby->set_standby_mode();
+
+	# Start standby
+	$node_standby->start;
+
+	# The standby may have WAL to apply before it matches the primary.  That
+	# is fine, because no test examines the standby before promotion.
+
+	return;
+}
+
 sub promote_standby
 {
 	#### Now run the test-specific parts to run after standby has been started
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 0634aefd20..8d2f0c044d 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -546,6 +546,32 @@ sub backup
 	return;
 }
 
+=item $node->backup_tblspc_mapping(backup_name, olddir, newdir)
+
+Create a hot backup with B<pg_basebackup> in subdirectory B<backup_name> of
+B<< $node->backup_dir >>, with configuration --tablespace-mapping=B<olddir>=B<newdir>
+, including the WAL. WAL files
+fetched at the end of the backup, not streamed.
+
+You'll have to configure a suitable B<max_wal_senders> on the
+target server since it isn't done by default.
+
+=cut
+
+sub backup_tbl_mapping
+{
+	my ($self, $backup_name, $olddir, $newdir) = @_;
+	my $backup_path = $self->backup_dir . '/' . $backup_name;
+	my $port        = $self->port;
+	my $name        = $self->name;
+
+	print "# Taking pg_basebackup $backup_name from node \"$name\"\n";
+	TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-p', $port,
+		'-T', "$olddir=$newdir", '--no-sync');
+	print "# Backup finished\n";
+	return;
+}
+
 =item $node->backup_fs_hot(backup_name)
 
 Create a backup with a filesystem level copy in subdirectory B<backup_name> of
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index baf5d0ac63..1840de8c8e 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -99,7 +99,7 @@ sub _copypath_recurse
 
 	# Check for symlink -- needed only on source dir
 	# (note: this will fall through quietly if file is already gone)
-	croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
+	# croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
 
 	# Abort if destination path already exists.  Should we allow directories
 	# to exist already?
@@ -107,7 +107,7 @@ sub _copypath_recurse
 
 	# If this source path is a file, simply copy it to destination with the
 	# same name and we're done.
-	if (-f $srcpath)
+	if (-f $srcpath && !(-l $srcpath))
 	{
 		my $fh;
 		unless (open($fh, '<', $srcpath))
@@ -122,7 +122,7 @@ sub _copypath_recurse
 	}
 
 	# If it's a directory, create it on dest and recurse into it.
-	if (-d $srcpath)
+	if (-d $srcpath && !(-l $srcpath))
 	{
 		my $directory;
 		unless (opendir($directory, $srcpath))
@@ -145,6 +145,18 @@ sub _copypath_recurse
 		return 1;
 	}
 
+	if (-l $srcpath)
+	{
+         my $old_link = $srcpath;
+         my $new_link = $destpath;
+
+         my $dst = readlink($old_link);
+         my @stat = lstat($old_link);
+
+         symlink $dst, $new_link;
+         return 1;
+	}
+
 	# If it disappeared from sight, that's OK.
 	return 1 if !-e $srcpath;
 
-- 
2.19.1

#8Michael Paquier
michael@paquier.xyz
In reply to: Shaoqi Bai (#7)
Re: Fwd: Add tablespace tap test to pg_rewind

On Tue, Mar 19, 2019 at 08:16:21PM +0800, Shaoqi Bai wrote:

Thanks for your advice, sorry for taking so long to give update in the
thread, because I am stuck in modifing Perl script, knowing little about
Perl language.

No problem. It is true that using perl for the first time can be a
certain gap, but once you get used to it it becomes really nice to be
able to control how tests are run in the tree. I would have avoided
extra routines in the patch, like what you have done with
create_standby_tbl_mapping(), but instead do something like
init_from_backup() which is able to take extra parameters in a way
similar to has_streaming and has_restoring. However, the trick with
tablespace mapping is that the caller of backup() should be able to
pass down multiple tablespace mapping references to make that a
maximum portable.
--
Michael

#9Shaoqi Bai
sbai@pivotal.io
In reply to: Michael Paquier (#6)
2 attachment(s)
Re: Fwd: Add tablespace tap test to pg_rewind

On Tue, Mar 12, 2019 at 10:27 AM Michael Paquier <michael@paquier.xyz>
wrote:

It could be an idea to split the patch in two pieces:
- One patch which refactors the code for the new option in
PostgresNode.pm
- Second patch for the new test with integration in RewindTest.pm.
This should touch different parts of the code, so combining both would
be fine as well for me :)
--
Michael

Have updated the patch doing as you suggested

On Wed, Mar 20, 2019 at 7:45 AM Michael Paquier <michael@paquier.xyz> wrote:

I would have avoided
extra routines in the patch, like what you have done with
create_standby_tbl_mapping(), but instead do something like
init_from_backup() which is able to take extra parameters in a way
similar to has_streaming and has_restoring. However, the trick with
tablespace mapping is that the caller of backup() should be able to
pass down multiple tablespace mapping references to make that a
maximum portable.
--
Michael

Also updated the patch to achieve your suggestion.

Attachments:

0001-Refactors-the-code-for-the-new-option-in-PostgresNod.patchapplication/octet-stream; name=0001-Refactors-the-code-for-the-new-option-in-PostgresNod.patchDownload
From 8bd3742ddc872e8a5414c5ee74a8ba4aec6dca95 Mon Sep 17 00:00:00 2001
From: Shaoqi Bai <sbai@pivotal.io>
Date: Thu, 21 Mar 2019 23:20:12 +0800
Subject: [PATCH 1/2] Refactors the code for the new option in PostgresNode.pm

---
 src/test/perl/PostgresNode.pm  | 14 +++++++++++---
 src/test/perl/RecursiveCopy.pm | 20 ++++++++++++++++----
 2 files changed, 27 insertions(+), 7 deletions(-)

diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 0634aefd20..b339a7f4f0 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -534,14 +534,22 @@ target server since it isn't done by default.
 
 sub backup
 {
-	my ($self, $backup_name) = @_;
+	my ($self, $backup_name, %params) = @_;
 	my $backup_path = $self->backup_dir . '/' . $backup_name;
 	my $port        = $self->port;
 	my $name        = $self->name;
 
 	print "# Taking pg_basebackup $backup_name from node \"$name\"\n";
-	TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-p', $port,
-		'--no-sync');
+
+	if (defined $params{has_tablespace_mapping}) {
+		TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-p', $port,
+			"--tablespace-mapping=" . $params{has_tablespace_mapping},
+			'--no-sync');
+	} else {
+		TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-p', $port,
+			'--no-sync');
+	}
+
 	print "# Backup finished\n";
 	return;
 }
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index baf5d0ac63..d684463884 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -97,14 +97,26 @@ sub _copypath_recurse
 	# invoke the filter and skip all further operation if it returns false
 	return 1 unless &$filterfn($curr_path);
 
-	# Check for symlink -- needed only on source dir
-	# (note: this will fall through quietly if file is already gone)
-	croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
-
 	# Abort if destination path already exists.  Should we allow directories
 	# to exist already?
 	croak "Destination path \"$destpath\" already exists" if -e $destpath;
 
+	# Check for symlink -- needed only on source dir, only allow symlink
+	# when under pg_tblspc
+	# (note: this will fall through quietly if file is already gone)
+	if (-l $srcpath)
+	{
+		if (dirname($srcpath) =~ /pg_tblspc$/)
+		{
+			my $dst = readlink($srcpath);
+			symlink($dst, $destpath);
+			return 1;
+		}
+		else
+		{
+			croak "Cannot operate on symlink \"$srcpath\"";
+		}
+	}
 	# If this source path is a file, simply copy it to destination with the
 	# same name and we're done.
 	if (-f $srcpath)
-- 
2.19.1

0002-Add-new-test-with-integration-in-RewindTest.pm.patchapplication/octet-stream; name=0002-Add-new-test-with-integration-in-RewindTest.pm.patchDownload
From 4408c259c6ac09b32c9597386bc6c329165a6ca7 Mon Sep 17 00:00:00 2001
From: Shaoqi Bai <sbai@pivotal.io>
Date: Thu, 21 Mar 2019 23:20:57 +0800
Subject: [PATCH 2/2] Add new test with integration in RewindTest.pm

---
 src/bin/pg_rewind/t/006_tablespace.pl | 83 +++++++++++++++++++++++++++
 src/bin/pg_rewind/t/RewindTest.pm     |  4 +-
 2 files changed, 85 insertions(+), 2 deletions(-)
 create mode 100644 src/bin/pg_rewind/t/006_tablespace.pl

diff --git a/src/bin/pg_rewind/t/006_tablespace.pl b/src/bin/pg_rewind/t/006_tablespace.pl
new file mode 100644
index 0000000000..e4afd3be1d
--- /dev/null
+++ b/src/bin/pg_rewind/t/006_tablespace.pl
@@ -0,0 +1,83 @@
+use strict;
+use warnings;
+use File::Path qw(rmtree);
+use TestLib;
+use Test::More tests => 6;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use RewindTest;
+
+my $tempdir = TestLib::tempdir;
+
+sub run_test
+{
+	my $test_mode = shift;
+
+	RewindTest::setup_cluster($test_mode, ['-g']);
+	RewindTest::start_master();
+
+	rmtree("$tempdir/inmaster");
+	rmtree("$tempdir/instandby");
+	rmtree("$tempdir/master_beforepromotion");
+	rmtree("$tempdir/master_afterpromotion");
+	rmtree("$tempdir/standby_afterpromotion");
+
+	mkdir "$tempdir/inmaster";
+	mkdir "$tempdir/instandby";
+
+	# Create a tablespace in master.
+	master_psql("CREATE TABLESPACE inmaster LOCATION '$tempdir/inmaster'");
+
+	RewindTest::create_standby($test_mode, has_tablespace_mapping =>"$tempdir/inmaster=$tempdir/instandby");
+
+	mkdir "$tempdir/master_beforepromotion";
+
+	# Create a tablespace, it has to be droped before doing pg_rewind, or else pg_rewind will fail
+	master_psql("CREATE TABLESPACE master_beforepromotion LOCATION '$tempdir/master_beforepromotion'");
+
+	RewindTest::promote_standby();
+
+	mkdir "$tempdir/master_afterpromotion";
+	mkdir "$tempdir/standby_afterpromotion";
+
+	# Create tablespaces in the old master and the new promoted standby.
+	master_psql("CREATE TABLESPACE master_afterpromotion LOCATION '$tempdir/master_afterpromotion'");
+	standby_psql("CREATE TABLESPACE standby_afterpromotion LOCATION '$tempdir/standby_afterpromotion'");
+	# Drop tablespace in the new promoted standby, because pg_rewind can not handle this case.
+	standby_psql("DROP TABLESPACE standby_afterpromotion");
+
+	# The clusters are now diverged.
+
+	RewindTest::run_pg_rewind($test_mode);
+
+	# Check that the correct databases are present after pg_rewind.
+	check_query(
+		'SELECT spcname FROM pg_tablespace ORDER BY spcname',
+		qq(inmaster
+master_beforepromotion
+pg_default
+pg_global
+),
+		'tablespace names');
+
+	# Permissions on PGDATA should have group permissions
+  SKIP:
+	{
+		skip "unix-style permissions not supported on Windows", 1
+		  if ($windows_os);
+
+		ok(check_mode_recursive($node_master->data_dir(), 0750, 0640),
+			'check PGDATA permissions');
+	}
+
+	RewindTest::clean_rewind_test();
+	return;
+}
+
+# Run the test in both modes.
+run_test('local');
+run_test('remote');
+
+exit(0);
diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm
index 85cae7e47b..cd106a55dd 100644
--- a/src/bin/pg_rewind/t/RewindTest.pm
+++ b/src/bin/pg_rewind/t/RewindTest.pm
@@ -150,11 +150,11 @@ sub start_master
 
 sub create_standby
 {
-	my $extra_name = shift;
+	my ($extra_name, %params) = @_;
 
 	$node_standby =
 	  get_new_node('standby' . ($extra_name ? "_${extra_name}" : ''));
-	$node_master->backup('my_backup');
+	$node_master->backup('my_backup', %params);
 	$node_standby->init_from_backup($node_master, 'my_backup');
 	my $connstr_master = $node_master->connstr();
 
-- 
2.19.1

#10Michael Paquier
michael@paquier.xyz
In reply to: Shaoqi Bai (#9)
Re: Fwd: Add tablespace tap test to pg_rewind

On Thu, Mar 21, 2019 at 11:41:01PM +0800, Shaoqi Bai wrote:

Have updated the patch doing as you suggested

+   RewindTest::setup_cluster($test_mode, ['-g']);
+   RewindTest::start_master();

There is no need to test for group permissions here, 002_databases.pl
already looks after that.

+   # Check for symlink -- needed only on source dir, only allow symlink
+   # when under pg_tblspc
+   # (note: this will fall through quietly if file is already gone)
+   if (-l $srcpath)
So you need that but in RecursiveCopy.pm because of init_from_backup
when creating the standby, which makes sense when it comes to
pg_tblspc.  I am wondering about any side effects though, and if it
would make sense to just remove the restriction for symlinks in
_copypath_recurse().
--
Michael
#11Shaoqi Bai
sbai@pivotal.io
In reply to: Michael Paquier (#10)
2 attachment(s)
Re: Fwd: Add tablespace tap test to pg_rewind

On Fri, Mar 22, 2019 at 1:34 PM Michael Paquier <michael@paquier.xyz> wrote:

On Thu, Mar 21, 2019 at 11:41:01PM +0800, Shaoqi Bai wrote:

Have updated the patch doing as you suggested

+   RewindTest::setup_cluster($test_mode, ['-g']);
+   RewindTest::start_master();

There is no need to test for group permissions here, 002_databases.pl
already looks after that.

Deleted the test for group permissions in updated patch.

+   # Check for symlink -- needed only on source dir, only allow symlink
+   # when under pg_tblspc
+   # (note: this will fall through quietly if file is already gone)
+   if (-l $srcpath)
So you need that but in RecursiveCopy.pm because of init_from_backup
when creating the standby, which makes sense when it comes to
pg_tblspc.  I am wondering about any side effects though, and if it
would make sense to just remove the restriction for symlinks in
_copypath_recurse().
--
Michael

Checking the RecursiveCopy::copypath being called, only _backup_fs and
init_from_backup called it.
After runing cmd make -C src/bin check in updated patch, seeing no failure.

Attachments:

0002-Add-new-test-with-integration-in-RewindTest.pm.patchapplication/octet-stream; name=0002-Add-new-test-with-integration-in-RewindTest.pm.patchDownload
From 03fb187595c701c259485ab2a815d1f419cd2411 Mon Sep 17 00:00:00 2001
From: Shaoqi Bai <sbai@pivotal.io>
Date: Thu, 21 Mar 2019 23:20:57 +0800
Subject: [PATCH 2/2] Add new test with integration in RewindTest.pm

---
 src/bin/pg_rewind/t/006_tablespace.pl | 73 +++++++++++++++++++++++++++
 src/bin/pg_rewind/t/RewindTest.pm     |  4 +-
 2 files changed, 75 insertions(+), 2 deletions(-)
 create mode 100644 src/bin/pg_rewind/t/006_tablespace.pl

diff --git a/src/bin/pg_rewind/t/006_tablespace.pl b/src/bin/pg_rewind/t/006_tablespace.pl
new file mode 100644
index 0000000000..6912578ba3
--- /dev/null
+++ b/src/bin/pg_rewind/t/006_tablespace.pl
@@ -0,0 +1,73 @@
+use strict;
+use warnings;
+use File::Path qw(rmtree);
+use TestLib;
+use Test::More tests => 4;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+use RewindTest;
+
+my $tempdir = TestLib::tempdir;
+
+sub run_test
+{
+	my $test_mode = shift;
+
+	RewindTest::setup_cluster($test_mode);
+	RewindTest::start_master();
+
+	rmtree("$tempdir/inmaster");
+	rmtree("$tempdir/instandby");
+	rmtree("$tempdir/master_beforepromotion");
+	rmtree("$tempdir/master_afterpromotion");
+	rmtree("$tempdir/standby_afterpromotion");
+
+	mkdir "$tempdir/inmaster";
+	mkdir "$tempdir/instandby";
+
+	# Create a tablespace in master.
+	master_psql("CREATE TABLESPACE inmaster LOCATION '$tempdir/inmaster'");
+
+	RewindTest::create_standby($test_mode, has_tablespace_mapping =>"$tempdir/inmaster=$tempdir/instandby");
+
+	mkdir "$tempdir/master_beforepromotion";
+
+	# Create a tablespace, it has to be droped before doing pg_rewind, or else pg_rewind will fail
+	master_psql("CREATE TABLESPACE master_beforepromotion LOCATION '$tempdir/master_beforepromotion'");
+
+	RewindTest::promote_standby();
+
+	mkdir "$tempdir/master_afterpromotion";
+	mkdir "$tempdir/standby_afterpromotion";
+
+	# Create tablespaces in the old master and the new promoted standby.
+	master_psql("CREATE TABLESPACE master_afterpromotion LOCATION '$tempdir/master_afterpromotion'");
+	standby_psql("CREATE TABLESPACE standby_afterpromotion LOCATION '$tempdir/standby_afterpromotion'");
+	# Drop tablespace in the new promoted standby, because pg_rewind can not handle this case.
+	standby_psql("DROP TABLESPACE standby_afterpromotion");
+
+	# The clusters are now diverged.
+
+	RewindTest::run_pg_rewind($test_mode);
+
+	# Check that the correct databases are present after pg_rewind.
+	check_query(
+		'SELECT spcname FROM pg_tablespace ORDER BY spcname',
+		qq(inmaster
+master_beforepromotion
+pg_default
+pg_global
+),
+		'tablespace names');
+
+	RewindTest::clean_rewind_test();
+	return;
+}
+
+# Run the test in both modes.
+run_test('local');
+run_test('remote');
+
+exit(0);
diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm
index 85cae7e47b..cd106a55dd 100644
--- a/src/bin/pg_rewind/t/RewindTest.pm
+++ b/src/bin/pg_rewind/t/RewindTest.pm
@@ -150,11 +150,11 @@ sub start_master
 
 sub create_standby
 {
-	my $extra_name = shift;
+	my ($extra_name, %params) = @_;
 
 	$node_standby =
 	  get_new_node('standby' . ($extra_name ? "_${extra_name}" : ''));
-	$node_master->backup('my_backup');
+	$node_master->backup('my_backup', %params);
 	$node_standby->init_from_backup($node_master, 'my_backup');
 	my $connstr_master = $node_master->connstr();
 
-- 
2.19.1

0001-Refactors-the-code-for-the-new-option-in-PostgresNod.patchapplication/octet-stream; name=0001-Refactors-the-code-for-the-new-option-in-PostgresNod.patchDownload
From 01535b0047ba993ef471b15b917427637978bd1a Mon Sep 17 00:00:00 2001
From: Shaoqi Bai <sbai@pivotal.io>
Date: Thu, 21 Mar 2019 23:20:12 +0800
Subject: [PATCH 1/2] Refactors the code for the new option in PostgresNode.pm

---
 src/test/perl/PostgresNode.pm  | 14 +++++++++++---
 src/test/perl/RecursiveCopy.pm | 13 +++++++++----
 2 files changed, 20 insertions(+), 7 deletions(-)

diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 0634aefd20..b339a7f4f0 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -534,14 +534,22 @@ target server since it isn't done by default.
 
 sub backup
 {
-	my ($self, $backup_name) = @_;
+	my ($self, $backup_name, %params) = @_;
 	my $backup_path = $self->backup_dir . '/' . $backup_name;
 	my $port        = $self->port;
 	my $name        = $self->name;
 
 	print "# Taking pg_basebackup $backup_name from node \"$name\"\n";
-	TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-p', $port,
-		'--no-sync');
+
+	if (defined $params{has_tablespace_mapping}) {
+		TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-p', $port,
+			"--tablespace-mapping=" . $params{has_tablespace_mapping},
+			'--no-sync');
+	} else {
+		TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-p', $port,
+			'--no-sync');
+	}
+
 	print "# Backup finished\n";
 	return;
 }
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index baf5d0ac63..3920f06b34 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -97,14 +97,19 @@ sub _copypath_recurse
 	# invoke the filter and skip all further operation if it returns false
 	return 1 unless &$filterfn($curr_path);
 
-	# Check for symlink -- needed only on source dir
-	# (note: this will fall through quietly if file is already gone)
-	croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
-
 	# Abort if destination path already exists.  Should we allow directories
 	# to exist already?
 	croak "Destination path \"$destpath\" already exists" if -e $destpath;
 
+	# If this source path is a link, simply symlink it to destination with
+	# the same name and we're done.
+	if (-l $srcpath)
+	{
+		my $dst = readlink($srcpath);
+		symlink($dst, $destpath);
+		return 1;
+	}
+
 	# If this source path is a file, simply copy it to destination with the
 	# same name and we're done.
 	if (-f $srcpath)
-- 
2.19.1

#12Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Shaoqi Bai (#11)
Re: Fwd: Add tablespace tap test to pg_rewind

I just noticed this thread. What do we think of adding this test to
pg12? (The patch doesn't apply verbatim, but it's a small update to get
it to apply.)

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#13Michael Paquier
michael@paquier.xyz
In reply to: Alvaro Herrera (#12)
Re: Fwd: Add tablespace tap test to pg_rewind

On Fri, Apr 26, 2019 at 10:11:24AM -0400, Alvaro Herrera wrote:

I just noticed this thread. What do we think of adding this test to
pg12? (The patch doesn't apply verbatim, but it's a small update to get
it to apply.)

Could you let me have a look at it? I have not tested on Windows, but
I suspect that because of the symlink() part this would fail, so we
may need to skip the tests.
--
Michael

#14Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#13)
Re: Fwd: Add tablespace tap test to pg_rewind

On 2019-Apr-26, Michael Paquier wrote:

On Fri, Apr 26, 2019 at 10:11:24AM -0400, Alvaro Herrera wrote:

I just noticed this thread. What do we think of adding this test to
pg12? (The patch doesn't apply verbatim, but it's a small update to get
it to apply.)

Could you let me have a look at it?

Absolutely. I was just trying to get a sense of how frozen the water is
at this point.

--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#15Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#14)
Re: Fwd: Add tablespace tap test to pg_rewind

Alvaro Herrera <alvherre@2ndquadrant.com> writes:

On 2019-Apr-26, Michael Paquier wrote:

On Fri, Apr 26, 2019 at 10:11:24AM -0400, Alvaro Herrera wrote:

I just noticed this thread. What do we think of adding this test to
pg12? (The patch doesn't apply verbatim, but it's a small update to get
it to apply.)

Could you let me have a look at it?

Absolutely. I was just trying to get a sense of how frozen the water is
at this point.

I don't think feature freeze precludes adding new test cases.

regards, tom lane

#16Michael Paquier
michael@paquier.xyz
In reply to: Tom Lane (#15)
Re: Fwd: Add tablespace tap test to pg_rewind

On Fri, Apr 26, 2019 at 11:21:48AM -0400, Tom Lane wrote:

I don't think feature freeze precludes adding new test cases.

I think as well that adding this stuff into v12 would be fine. Now if
there is any objection let's wait for later.
--
Michael

#17Kyotaro HORIGUCHI
horiguchi.kyotaro@lab.ntt.co.jp
In reply to: Michael Paquier (#16)
Re: Add tablespace tap test to pg_rewind

At Sat, 27 Apr 2019 09:35:19 +0900, Michael Paquier <michael@paquier.xyz> wrote in <20190427003519.GC2032@paquier.xyz>

On Fri, Apr 26, 2019 at 11:21:48AM -0400, Tom Lane wrote:

I don't think feature freeze precludes adding new test cases.

I think as well that adding this stuff into v12 would be fine. Now if
there is any objection let's wait for later.

The patch seems to be using the tablespace directory in backups
directly from standbys. In other words, multiple standbys created
from A backup shares the tablespace directory in the backup.

Another nitpicking is it sound a bit strainge that the parameter
"has_tablespace_mapping" is not a boolean.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

#18Michael Paquier
michael@paquier.xyz
In reply to: Shaoqi Bai (#11)
Re: Fwd: Add tablespace tap test to pg_rewind

On Fri, Mar 22, 2019 at 04:25:53PM +0800, Shaoqi Bai wrote:

Deleted the test for group permissions in updated patch.

Well, there are a couple of things I am not really happy about in this
patch:
- There is not much point to have has_tablespace_mapping as it is not
extensible. Instead I'd rather use a single "extra" parameter which
can define a range of options. An example of that is within
PostgresNode::init for "extra" and "auth_extra".
- CREATE TABLESPACE is run once on the primary *before* promoting the
standby, which causes the tablespace paths to map between both of
them. This is not correct. Creating a tablespace on the primary
before creating the standby, and use --tablespace-map would be the way
to go... However per the next point...
- standby_afterpromotion is created on the promoted standby, and then
immediately dropped. pg_rewind is able to handle this case when
working on different hosts. But with this test we finish by having
the same problem as pg_basebackup: the source and the target server
finish by eating each other. I think that this could actually be an
interesting feature for pg_rewind.
- A comment at the end refers to databases, and not tablespaces.

You could work out the first problem with the backup by changing the
backup()/init_from_backup() in RewindTest::create_standby by a pure
call to pg_basebackup, but you still have the second problem, which we
should still be able to test, and this requires more facility in
pg_rewind so as it is basically possible to hijack
create_target_symlink() to create a symlink to a different path than
the initial one.

Checking the RecursiveCopy::copypath being called, only _backup_fs and
init_from_backup called it.
After runing cmd make -C src/bin check in updated patch, seeing no failure.

Yes, I can see that. The issue is that even if we do a backup with
--tablespace-mapping then we still need a tweak to allow the copy of
symlinks. I am not sure that this is completely what we are looking
for either, as it means that any test setting a primary with a
tablespace and two standbys initialized from the same base backup
would fail. That's not really portable.
--
Michael

#19Michael Paquier
michael@paquier.xyz
In reply to: Kyotaro HORIGUCHI (#17)
Re: Add tablespace tap test to pg_rewind

On Tue, May 07, 2019 at 10:31:59AM +0900, Kyotaro HORIGUCHI wrote:

The patch seems to be using the tablespace directory in backups
directly from standbys. In other words, multiple standbys created
from A backup shares the tablespace directory in the backup.

Yes, I noticed that, and I am not happy about that either. I'd like
to think that what we are looking for is an equivalent of the
tablespace mapping of pg_basebackup, but for init_from_backup(). At
least that sounds like a plausible solution.
--
Michael

#20Michael Paquier
michael@paquier.xyz
In reply to: Michael Paquier (#18)
Re: Fwd: Add tablespace tap test to pg_rewind

On Thu, May 09, 2019 at 02:36:48PM +0900, Michael Paquier wrote:

Yes, I can see that. The issue is that even if we do a backup with
--tablespace-mapping then we still need a tweak to allow the copy of
symlinks. I am not sure that this is completely what we are looking
for either, as it means that any test setting a primary with a
tablespace and two standbys initialized from the same base backup
would fail. That's not really portable.

So for now I am marking the patch as returned with feedback. I can
see a simple way to put us through that by having an equivalent of
--tablespace-mapping for the file-level backup routine which passes it
down to RecursiveCopy.pm as well as PostgresNode::init_from_backup.
At the end of the day, we would need to be able to point the path to
different locations for:
- the primary
- any backup tables
- any standbys which are restored from the previous backups.

And on top of that there is of course the argument of pg_rewind which
would need an option similar to pg_basebackup's --tablespace-mapping
so as a target instance does not finish by using the same tablespace
path as the source where there is a tablespace of difference during
the operation. And it does not prevent an overlap if a tablespace
needs to be created when the target instance replays WAL up to the
consistent point of the source. So that's a lot of work which may be
hard to justify.
--
Michael