In-core regression tests for replication, cascading, archiving, PITR, etc.

Started by Michael Paquierabout 12 years ago157 messages
#1Michael Paquier
michael.paquier@gmail.com

Hi all,

The data replication bug causing data corruption on hot slaves found
lately (http://wiki.postgresql.org/wiki/Nov2013ReplicationIssue) is
causing a certain amount of damage among the users of Postgres, either
companies or individuals, and impacts a lot of people. So perhaps it
would be a good time to start thinking about adding some dedicated
regression tests for replication, archiving, PITR, data integrity,
parameter reloading, etc. The possible number of things or use-cases
that could be applied to that is very vast, but let's say that it
involves the manipulation of multiple Postgres nodes and structures
(like a WAL archive) defining a Postgres cluster.

The main purpose of those test would be really simple: using a given
GIT repository, a buildfarm or developer should be able to run a
single "make check" command that runs those tests on a local machine
in a fashion similar to isolation or regression tests and validate
builds or patches.

I imagine that there would be roughly two ways to implement such a facility:
1) Use of a smart set of bash scripts. This would be easy to implement
but reduces pluggability of custom scripts (I am sure that each
user/company has already its own set of scenarios.). pg_upgrade uses
something similar with its test.sh.
2) Use a scripting language, in a way similar to how isolation tests
are done. This would make custom test more customizable.
Here is for example an approach that has been presented at the
unconference of PGcon 2013 (would be something different though as
this proposal does not include node manipulation *during* the tests
like promotion):
https://wiki.postgresql.org/images/1/14/Pg_testframework.pptx
3) Import (and improve) solutions that other projects based on
Postgres technology use for those things.

In all cases, here are the common primary actions that could be run for a test:
- Define and perform actions on a node: init, start, stop, promote,
create_archive, base_backup. So it is a sort of improved wrapper of
pg_ctl.
- Pass parameters to a configuration file, either postgresql.conf,
recovery.conf, or anything.
- Launch SQL commands to a node.
Things like the creation of folders for WAL archiving should be simply
harcoded to simplify the life of developer... As well, the facility
should be smart enough to be allow the use of custom commands that are
the combination of primary actions above, like for example the
possibility to define a sync slave, linked to a root node, is simply
1) create a base backup from a node, 2) pass parameters to
postgresql.conf and recovery.conf, 3) start the node.

Let me know your thoughts.
Regards,
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#2Heikki Linnakangas
hlinnakangas@vmware.com
In reply to: Michael Paquier (#1)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 12/02/2013 08:40 AM, Michael Paquier wrote:

The data replication bug causing data corruption on hot slaves found
lately (http://wiki.postgresql.org/wiki/Nov2013ReplicationIssue) is
causing a certain amount of damage among the users of Postgres, either
companies or individuals, and impacts a lot of people. So perhaps it
would be a good time to start thinking about adding some dedicated
regression tests for replication, archiving, PITR, data integrity,
parameter reloading, etc. The possible number of things or use-cases
that could be applied to that is very vast, but let's say that it
involves the manipulation of multiple Postgres nodes and structures
(like a WAL archive) defining a Postgres cluster.

The main purpose of those test would be really simple: using a given
GIT repository, a buildfarm or developer should be able to run a
single "make check" command that runs those tests on a local machine
in a fashion similar to isolation or regression tests and validate
builds or patches.

+1. The need for such a test suite has been mentioned every single time
that a bug or new feature related to replication, PITR or hot standby
has come up. So yes please! The only thing missing is someone to
actually write the thing. So if you have the time and energy, that'd be
great!

- Heikki

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#3Michael Paquier
michael.paquier@gmail.com
In reply to: Heikki Linnakangas (#2)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Mon, Dec 2, 2013 at 6:24 PM, Heikki Linnakangas
<hlinnakangas@vmware.com> wrote:

+1. The need for such a test suite has been mentioned every single time that
a bug or new feature related to replication, PITR or hot standby has come
up. So yes please! The only thing missing is someone to actually write the
thing. So if you have the time and energy, that'd be great!

I am sure you know who we need to convince in this case :)
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#4Andres Freund
andres@2ndquadrant.com
In reply to: Michael Paquier (#3)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 2013-12-02 18:45:37 +0900, Michael Paquier wrote:

On Mon, Dec 2, 2013 at 6:24 PM, Heikki Linnakangas
<hlinnakangas@vmware.com> wrote:

+1. The need for such a test suite has been mentioned every single time that
a bug or new feature related to replication, PITR or hot standby has come
up. So yes please! The only thing missing is someone to actually write the
thing. So if you have the time and energy, that'd be great!

I am sure you know who we need to convince in this case :)

If you're alluding to Tom, I'd guess he doesn't need to be convinced of
such a facility in general. I seem to remember him complaining about the
lack of testing that as well.
Maybe that it shouldn't be part of the main regression schedule...

+many from me as well. I think the big battle will be how to do it, not
if in general.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#5Michael Paquier
michael.paquier@gmail.com
In reply to: Andres Freund (#4)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Mon, Dec 2, 2013 at 7:07 PM, Andres Freund <andres@2ndquadrant.com> wrote:

Maybe that it shouldn't be part of the main regression schedule...

Yes, like isolation tests, it don't see those new tests in the main
flow as well.

+many from me as well. I think the big battle will be how to do it, not
if in general.

Yeah, that's why we should gather first feedback about the methods
that other projects (Slony, Londiste, Pgpool) are using as well before
heading to a solution or another. Having smth, whatever small for 9.4
would also be of great help.

I am however sure that getting a small prototype integrated with some
of my in-house scripts would not take that much time though...
Regards,
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#6Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andres Freund (#4)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Andres Freund <andres@2ndquadrant.com> writes:

Maybe that it shouldn't be part of the main regression schedule...

It *can't* be part of the main regression tests; those are supposed to
be runnable against an already-installed server, and fooling with that
server's configuration is off-limits too. But I agree that some
other facility to simplify running tests like this would be handy.

At the same time, I'm pretty skeptical that any simple regression-test
type facility would have caught the bugs we've fixed lately ...

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#7Andres Freund
andres@2ndquadrant.com
In reply to: Tom Lane (#6)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 2013-12-02 09:41:39 -0500, Tom Lane wrote:

At the same time, I'm pretty skeptical that any simple regression-test
type facility would have caught the bugs we've fixed lately ...

Agreed, but it would make reorganizing stuff to be more robust more
realistic. At the moment for everything you change you have to hand-test
everyting possibly affected which takes ages.

I think we also needs support for testing xid/multixid wraparound. It
currently isn't realistically testable because of the timeframes
involved.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#8Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andres Freund (#7)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Andres Freund <andres@2ndquadrant.com> writes:

I think we also needs support for testing xid/multixid wraparound. It
currently isn't realistically testable because of the timeframes
involved.

When I've wanted to do that in the past, I've used pg_resetxlog to
adjust a cluster's counters. It still requires some manual hacking
though because pg_resetxlog isn't bright enough to create the new
pg_clog files needed when you move the xid counter a long way.
We could fix that, or we could make the backend more forgiving of
not finding the initial clog segment present at startup ...

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#9Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#8)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Tom Lane escribi�:

When I've wanted to do that in the past, I've used pg_resetxlog to
adjust a cluster's counters. It still requires some manual hacking
though because pg_resetxlog isn't bright enough to create the new
pg_clog files needed when you move the xid counter a long way.
We could fix that, or we could make the backend more forgiving of
not finding the initial clog segment present at startup ...

FWIW we already have some new code that creates segments when not found.
It's currently used in multixact, and the submitted "commit timestamp"
module uses it too.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#10Andres Freund
andres@2ndquadrant.com
In reply to: Tom Lane (#8)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 2013-12-02 09:59:12 -0500, Tom Lane wrote:

Andres Freund <andres@2ndquadrant.com> writes:

I think we also needs support for testing xid/multixid wraparound. It
currently isn't realistically testable because of the timeframes
involved.

When I've wanted to do that in the past, I've used pg_resetxlog to
adjust a cluster's counters.

I've done that as well, but it's painful and not neccessarily testing
the right thing. E.g. I am far from sure we handle setting the
anti-wraparound limits correctly when promoting a standby - a restart to
adapt pg_control changes things and it might get rolled back because of
a already logged checkpoints.

What I'd love is a function that gives me the opportunity to
*efficiently* move forward pg_clog, pg_multixact/offset,members by large
chunks. So e.g. I could run a normal pgbench alongside another pgbench
moving clog forward in 500k chunks, but so it creates the necessary
files I could possibly need to access.

If you do it naivly you get into quite some fun with hot standby btw. I
can tell you that from experience :P

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#11Michael Paquier
michael.paquier@gmail.com
In reply to: Tom Lane (#6)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Mon, Dec 2, 2013 at 11:41 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andres Freund <andres@2ndquadrant.com> writes:
At the same time, I'm pretty skeptical that any simple regression-test
type facility would have caught the bugs we've fixed lately ...

The replication bug would have been reproducible at least, Heikki
produced a simple test case able to reproduce it. For the MultiXact
stuff... well some more infrastructure in core might be needed before
having a wrapper calling test scripts aimed to manipulate cluster of
nodes.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#12Michael Paquier
michael.paquier@gmail.com
In reply to: Andres Freund (#4)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Mon, Dec 2, 2013 at 7:07 PM, Andres Freund <andres@2ndquadrant.com> wrote:

On 2013-12-02 18:45:37 +0900, Michael Paquier wrote:

On Mon, Dec 2, 2013 at 6:24 PM, Heikki Linnakangas
<hlinnakangas@vmware.com> wrote:

+1. The need for such a test suite has been mentioned every single time that
a bug or new feature related to replication, PITR or hot standby has come
up. So yes please! The only thing missing is someone to actually write the
thing. So if you have the time and energy, that'd be great!

I am sure you know who we need to convince in this case :)

If you're alluding to Tom, I'd guess he doesn't need to be convinced of
such a facility in general. I seem to remember him complaining about the
lack of testing that as well.
Maybe that it shouldn't be part of the main regression schedule...

+many from me as well. I think the big battle will be how to do it, not
if in general.

(Reviving an old thread)
So I am planning to seriously focus soon on this stuff, basically
using the TAP tests as base infrastructure for this regression test
suite. First, does using the TAP tests sound fine?

On the top of my mind I got the following items that should be tested:
- WAL replay: from archive, from stream
- hot standby and read-only queries
- node promotion
- recovery targets and their interferences when multiple targets are
specified (XID, name, timestamp, immediate)
- timelines
- recovery_target_action
- recovery_min_apply_delay (check that WAL is fetch from a source at
some correct interval, can use a special restore_command for that)
- archive_cleanup_command (check that command is kicked at each restart point)
- recovery_end_command (check that command is kicked at the end of recovery)
- timeline jump of a standby after reconnecting to a promoted node

Regards,
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#13Jim Nasby
Jim.Nasby@BlueTreble.com
In reply to: Michael Paquier (#12)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 3/8/15 6:19 AM, Michael Paquier wrote:

On Mon, Dec 2, 2013 at 7:07 PM, Andres Freund <andres@2ndquadrant.com> wrote:

On 2013-12-02 18:45:37 +0900, Michael Paquier wrote:

On Mon, Dec 2, 2013 at 6:24 PM, Heikki Linnakangas
<hlinnakangas@vmware.com> wrote:

+1. The need for such a test suite has been mentioned every single time that
a bug or new feature related to replication, PITR or hot standby has come
up. So yes please! The only thing missing is someone to actually write the
thing. So if you have the time and energy, that'd be great!

I am sure you know who we need to convince in this case :)

If you're alluding to Tom, I'd guess he doesn't need to be convinced of
such a facility in general. I seem to remember him complaining about the
lack of testing that as well.
Maybe that it shouldn't be part of the main regression schedule...

+many from me as well. I think the big battle will be how to do it, not
if in general.

(Reviving an old thread)
So I am planning to seriously focus soon on this stuff, basically
using the TAP tests as base infrastructure for this regression test
suite. First, does using the TAP tests sound fine?

On the top of my mind I got the following items that should be tested:
- WAL replay: from archive, from stream
- hot standby and read-only queries
- node promotion
- recovery targets and their interferences when multiple targets are
specified (XID, name, timestamp, immediate)
- timelines
- recovery_target_action
- recovery_min_apply_delay (check that WAL is fetch from a source at
some correct interval, can use a special restore_command for that)
- archive_cleanup_command (check that command is kicked at each restart point)
- recovery_end_command (check that command is kicked at the end of recovery)
- timeline jump of a standby after reconnecting to a promoted node

If we're keeping a list, there's also hot_standby_feedback,
max_standby_archive_delay and max_standby_streaming_delay.
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#14Noah Misch
noah@leadboat.com
In reply to: Michael Paquier (#12)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sun, Mar 08, 2015 at 08:19:39PM +0900, Michael Paquier wrote:

So I am planning to seriously focus soon on this stuff, basically
using the TAP tests as base infrastructure for this regression test
suite. First, does using the TAP tests sound fine?

Yes.

On the top of my mind I got the following items that should be tested:
- WAL replay: from archive, from stream
- hot standby and read-only queries
- node promotion
- recovery targets and their interferences when multiple targets are
specified (XID, name, timestamp, immediate)
- timelines
- recovery_target_action
- recovery_min_apply_delay (check that WAL is fetch from a source at
some correct interval, can use a special restore_command for that)
- archive_cleanup_command (check that command is kicked at each restart point)
- recovery_end_command (check that command is kicked at the end of recovery)
- timeline jump of a standby after reconnecting to a promoted node

Those sound good. The TAP suites still lack support for any Windows target.
If you're inclined to fix that, it would be a great contribution. The more we
accrue tests before doing that, the harder it will be to dig out.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#15Michael Paquier
michael.paquier@gmail.com
In reply to: Noah Misch (#14)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Mar 11, 2015 at 2:47 PM, Noah Misch <noah@leadboat.com> wrote:

On Sun, Mar 08, 2015 at 08:19:39PM +0900, Michael Paquier wrote:

So I am planning to seriously focus soon on this stuff, basically
using the TAP tests as base infrastructure for this regression test
suite. First, does using the TAP tests sound fine?

Yes.

Check.

On the top of my mind I got the following items that should be tested:
- WAL replay: from archive, from stream
- hot standby and read-only queries
- node promotion
- recovery targets and their interferences when multiple targets are
specified (XID, name, timestamp, immediate)
- timelines
- recovery_target_action
- recovery_min_apply_delay (check that WAL is fetch from a source at
some correct interval, can use a special restore_command for that)
- archive_cleanup_command (check that command is kicked at each restart point)
- recovery_end_command (check that command is kicked at the end of recovery)
- timeline jump of a standby after reconnecting to a promoted node

Those sound good. The TAP suites still lack support for any Windows target.
If you're inclined to fix that, it would be a great contribution. The more we
accrue tests before doing that, the harder it will be to dig out.

Yeah, that's already on my TODO list.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#16Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#15)
1 attachment(s)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Mar 11, 2015 at 3:04 PM, Michael Paquier wrote:

On Wed, Mar 11, 2015 at 2:47 PM, Noah Misch <noah@leadboat.com> wrote:

On Sun, Mar 08, 2015 at 08:19:39PM +0900, Michael Paquier wrote:

So I am planning to seriously focus soon on this stuff, basically
using the TAP tests as base infrastructure for this regression test
suite. First, does using the TAP tests sound fine?

Yes.

Check.

On the top of my mind I got the following items that should be tested:
- WAL replay: from archive, from stream
- hot standby and read-only queries
- node promotion
- recovery targets and their interferences when multiple targets are
specified (XID, name, timestamp, immediate)
- timelines
- recovery_target_action
- recovery_min_apply_delay (check that WAL is fetch from a source at
some correct interval, can use a special restore_command for that)
- archive_cleanup_command (check that command is kicked at each restart point)
- recovery_end_command (check that command is kicked at the end of recovery)
- timeline jump of a standby after reconnecting to a promoted node

So, as long as I had a clear picture of what I wanted to do regarding
this stuff (even if this is a busy commit fest, sorry), I have been
toying around with perl and I have finished with the patch attached,
adding some base structure for a new test suite covering recovery.

This patch includes basic tests for the following items:
- node promotion, test of archiving, streaming, replication cascading
- recovery targets XID, name, timestamp, immediate and PITR
- Timeline jump of a standby when reconnecting to a newly-promoted standby
- Replay delay
Tests are located in src/test/recovery, and are not part of the main
test suite, similarly to the ssl stuff.
I have dropped recovery_target_action for the time being as long as
the matter on the other thread is not set
(/messages/by-id/20150315132707.GB19792@alap3.anarazel.de),
and I don't think that it would be complicated to create tests for
that btw.

The most important part of this patch is not the tests themselves, but
the base set of routines allowing to simply create nodes, take
backups, create standbys from backups, and set up nodes to do stuff
like streaming, archiving, or restoring from archives. There are many
configurations possible of course in recovery.conf, but the set of
routines that this patch present are made to be *simple* to not
overcomplicate the way tests can be written.

Feedback is of course welcome, but note that I am not seriously
expecting any until we get into 9.6 development cycle and I am adding
this patch to the next CF.

Regards,
--
Michael

Attachments:

20150318_recovery_regressions.patchtext/x-patch; charset=US-ASCII; name=20150318_recovery_regressions.patchDownload
diff --git a/src/test/Makefile b/src/test/Makefile
index b7cddc8..7174c2d 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -16,7 +16,7 @@ SUBDIRS = regress isolation modules
 
 # The SSL suite is not secure to run on a multi-user system, so don't run
 # it as part of global "check" target.
-ALWAYS_SUBDIRS = ssl
+ALWAYS_SUBDIRS = recovery ssl
 
 # We want to recurse to all subdirs for all standard targets, except that
 # installcheck and install should not recurse into the subdirectory "modules".
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 003cd9a..a035472 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -20,6 +20,7 @@ our @EXPORT = qw(
   program_version_ok
   program_options_handling_ok
   command_like
+  command_is
   issues_sql_like
 );
 
@@ -200,6 +201,16 @@ sub command_like
 	like($stdout, $expected_stdout, "$test_name: matches");
 }
 
+sub command_is
+{
+	my ($cmd, $expected_stdout, $test_name) = @_;
+	my ($stdout, $stderr);
+	my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+	ok($result, "@$cmd exit code 0");
+	is($stderr, '', "@$cmd no stderr");
+	is($stdout, $expected_stdout, "$test_name: matches");
+}
+
 sub issues_sql_like
 {
 	my ($cmd, $expected_sql, $test_name) = @_;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..c194297
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master of standby for the
+purpose of the tests.
diff --git a/src/test/recovery/RecoveryTest.pm b/src/test/recovery/RecoveryTest.pm
new file mode 100644
index 0000000..f401418
--- /dev/null
+++ b/src/test/recovery/RecoveryTest.pm
@@ -0,0 +1,345 @@
+package RecoveryTest;
+
+# Set of common routines for recovery regression tests for a PostgreSQL
+# cluster. This includes global variables and methods that can be used
+# by the various set of tests present to set up cluster nodes and
+# configure them according to the test scenario.
+#
+# Cluster nodes can be freely created using initdb or using the existing
+# base backup of another node, with minimum configuration done when the
+# node is created for the first time like having a proper port number.
+# It is then up to the test to decide what to do with the newly-created
+# node for flexibility though.
+#
+# Data folders of each node are available through the global variables
+# provided by this package, hashed depending on the port number:
+# - connstr_nodes to query the connection string of a given node
+# - datadir_nodes to get the data folder of a given node
+# - archive_nodes for the location of the WAL archives
+# - backup_nodes for the location of base backups
+#
+# Nodes are identified by their port number, which should be unique
+# for each node of the cluster as it is run locally.
+
+use Cwd;
+use TestLib;
+use Test::More;
+
+use Archive::Tar;
+use File::Copy;
+use File::Path qw(remove_tree);
+
+# TODO: Should try to remove this dependency
+use IPC::Run qw(run start);
+
+use Exporter 'import';
+
+our @EXPORT = qw(
+	%connstr_nodes
+	%datadir_nodes
+	%backup_nodes
+	%archive_nodes
+
+	append_to_file
+
+	backup_node
+	enable_archiving
+	get_free_port
+	init_node
+	init_node_from_backup
+	init_recovery_test
+	make_archiving_standby
+	make_streaming_standby
+	pgctl_node
+	psql_get
+	psql_node
+);
+
+# Global variables for node data
+%datadir_nodes = {};	# PGDATA folders
+%backup_nodes = {};	# Backup base folder
+%archive_nodes = {};	# Archive base folder
+%connstr_nodes = {};	# Connection strings
+
+# Adjust these paths for your environment
+my $testroot = "./tmp_check";
+
+# Location of log files
+my $logdir = "./regress_log";
+mkdir $logdir;
+my $log_path;
+
+$ENV{PGDATABASE} = "postgres";
+
+sub is_correct_port
+{
+	my $port = $shift;
+
+	if ( $port =~ m/^\d+$/ )
+	{
+		die "Port number specified is not an integer";
+	}
+
+	if (port > 65536)
+	{
+		die "Port number specified higher than 65536";
+	}
+}
+
+# init_recovery_test
+# Routine to call at the beginning of a test
+sub init_recovery_test
+{
+	$testname = shift;
+	$log_path="regress_log/recovery_log_${testname}.log";
+
+	remove_tree $log_path;
+}
+
+# Handy routine to report a message in the test's log file
+sub report_log
+{
+	my $msg = shift;
+
+	system_or_bail("echo '$msg' >> $log_path 2>&1 ");
+}
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_root }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+restore_command='cp -i $archive_nodes{ $port_root }/%f %p'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $port = shift;
+
+	is_correct_port($port);
+
+	# Enable archive_mode and archive_command on node
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+archive_mode = on
+archive_command = 'cp -i %p $archive_nodes{ $port }/%f'
+));
+}
+
+# Standby node initialization
+# Node only streaming.
+sub make_streaming_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_streaming($port_master, $port_standby);
+	pgctl_node($port_standby, 'start');
+
+	return $port_standby;
+}
+# Node getting only from archives
+sub make_archiving_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_restoring($port_master, $port_standby);
+	pgctl_node($port_standby, 'start');
+
+	return $port_standby;
+}
+
+sub configure_base_node
+{
+	my $port = shift;
+
+	is_correct_port($port);
+
+	# Make configuration somewhat generic to test recovery
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+port = $port
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+max_connections = 10
+wal_retrieve_retry_interval = '100ms'
+));
+
+	# Accept replication connections
+	append_to_file("$datadir_nodes{ $port }/pg_hba.conf", qq(
+local replication all trust
+));
+}
+
+# Get a port number not in use currently for a new node
+# As port number retrieval is based on the nodes currently running,
+# be sure that the node that is consuming this port number has already
+# been started.
+sub get_free_port
+{
+	my $found = 0;
+	# XXX: Should this part use PG_VERSION_NUM?
+	my $port = 90400 % 16384 + 49152;
+
+	while ($found == 0)
+	{
+		$port++;
+		my $ret = system("psql -X -p $port postgres </dev/null 2>/dev/null");
+		if ($ret != 0)
+		{
+			$found = 1;
+		}
+	}
+
+	report_log("Found free port $port");
+	return $port;
+}
+
+# Low-level routines to initialize a node
+# Initialize a node from scratch
+sub init_node
+{
+	my $port = shift;
+
+	is_correct_port($port);
+
+	# Save configuration information
+	$datadir_nodes{ $port } = TestLib::tempdir;
+	$backup_nodes{ $port } = TestLib::tempdir;
+	$connstr_nodes{ $port } = "port=$port";
+	$archive_nodes{ $port } =  TestLib::tempdir;
+
+	standard_initdb($datadir_nodes{ $port });
+	configure_base_node($port);
+}
+
+# Initialize a node from an existing base backup
+sub init_node_from_backup
+{
+	my ($port, $root_port, $backup_name) = @_;
+
+	is_correct_port($port);
+	is_correct_port($root_port);
+
+	my $backup_path =  "$backup_nodes{ $root_port }/$backup_name";
+	my $backup_file = "$backup_path/base.tar";
+
+	# Check existence of backup wanted
+	if ( ! -d $backup_path )
+	{
+		die "Backup $backup_path does not exist";
+	}
+
+	# Save configuration information
+	$datadir_nodes{ $port } = TestLib::tempdir;
+	$backup_nodes{ $port } = TestLib::tempdir;
+	$connstr_nodes{ $port } = "port=$port";
+	$archive_nodes{ $port } =  TestLib::tempdir;
+
+	# Extract the base backup wanted
+	my $current_dir = cwd();
+
+	# Temporary move to the place of extraction
+	chdir "$datadir_nodes{ $port }";
+	Archive::Tar->extract_archive($backup_file);
+	chdir "$current_dir";
+	configure_base_node($port);
+}
+
+# Create a backup on a node already running
+sub backup_node
+{
+	my ($port, $backup_name) = @_;
+
+	is_correct_port($port);
+
+	my $backup_path = "$backup_nodes{ $port }/$backup_name";
+
+	# Backup a node in tar format, it is more portable across platforms
+	system_or_bail("pg_basebackup -D $backup_path -p $port --format=t -x >> $log_path 2>&1");
+}
+
+# Run simple SQL query on a node
+sub psql_node
+{
+	my ($port, $query) = @_;
+
+	is_correct_port($port);
+
+	report_log("Running \"$query\" on node with port $port");
+	system_or_bail("psql -q --no-psqlrc -d $connstr_nodes{ $port } -c \"$query\" >> $log_path 2>&1");
+}
+
+# Run a query on a node and fetch back its result, This routine is
+# useful when needing server-side state data like data to define
+# a recovery target for example that depend on the environment
+# when this test suite is run.
+sub psql_get
+{
+	my ($port, $query) = @_;
+	my ($stdout, $stderr);
+
+	is_correct_port($port);
+
+	report_log("Running  \"$query\" on node with port $port to get back results");
+	my $result = run [ "psql", "-A", "-t", "-q",
+					   "-d $connstr_nodes{ $port }", "--no-psqlrc",
+					   "-c $query" ], '>', \$stdout, '2>', \$stderr;
+	chomp($stdout);
+	return $stdout;
+}
+
+# Perform an action with pg_ctl on a node
+sub pgctl_node
+{
+	my $port = shift;
+	my $action = shift;
+
+	is_correct_port($port);
+
+	report_log("Running \"$action\" on node with port $port");
+	system_or_bail("pg_ctl $action -w -D $datadir_nodes{ $port } >> $log_path 2>&1");
+}
+
+# Add a set of parameters to a configuration file
+sub append_to_file
+{
+	my($filename, $str) = @_;
+
+	open my $fh, ">>", $filename or die "could not open file $filename";
+	print $fh $str;
+	close $fh;
+}
+
+1;
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..5d6ebeb
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,59 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 8;
+
+use RecoveryTest;
+
+RecoveryTest::init_recovery_test('stream_rep');
+
+# Initialize master node
+my $port_master = get_free_port();
+my $backup_name = 'my_backup';
+init_node($port_master);
+
+# Start it
+pgctl_node($port_master, 'start');
+
+# Take backup
+backup_node($port_master, $backup_name);
+
+# Create streaming standby linking to master
+my $port_standby_1 = make_streaming_standby($port_master, $backup_name);
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+backup_node($port_standby_1, $backup_name);
+
+# Create second standby node linking to standby 1
+my $port_standby_2 = make_streaming_standby($port_standby_1, $backup_name);
+
+# Create some content on master and check its presence in standby 1 and 2
+psql_node($port_master, "CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a");
+
+# Sleep a bit to have time to replay things
+sleep 1;
+command_is(['psql', '-A', '-t',  '--no-psqlrc',
+		   "-d $connstr_nodes{ $port_standby_1 }", '-c', "SELECT count(*) FROM tab_int"],
+		   qq(1002
+),
+		   'check streamed content');
+command_is(['psql', '-A', '-t',  '--no-psqlrc',
+		   "-d $connstr_nodes{ $port_standby_2 }", '-c', "SELECT count(*) FROM tab_int"],
+		   qq(1002
+),
+		   'check streamed content');
+
+# Check that only READ-only queries can run on standbys
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+		   "-d $connstr_nodes{ $port_standby_1 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+		   'Read-only queries on standby 1');
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+		   "-d $connstr_nodes{ $port_standby_2 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+		   'Read-only queries on standby 2');
+
+# Stop nodes
+pgctl_node($port_master, 'stop', '-m', 'immediate');
+pgctl_node($port_standby_1, 'stop', '-m', 'immediate');
+pgctl_node($port_standby_2, 'stop', '-m', 'immediate');
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..a76ed65
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,47 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 3;
+use File::Copy;
+use RecoveryTest;
+
+RecoveryTest::init_recovery_test('archiving');
+
+# Initialize master node, doing archives
+my $port_master = get_free_port();
+my $backup_name = 'my_backup';
+init_node($port_master);
+enable_archiving($port_master);
+
+# Start it
+pgctl_node($port_master, 'start');
+
+# Take backup for slave
+backup_node($port_master, 'my_backup');
+
+# Initialize standby node from backup, fetching WAL from archives
+my $port_standby = make_archiving_standby($port_master, $backup_name);
+
+# Create some content on master
+psql_node($port_master,
+		  "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+
+# Force archiving of WAL file to make it present on master
+psql_node($port_master, "SELECT pg_switch_xlog()");
+
+# Add some more content, it should not be present on standby
+psql_node($port_master,
+		  "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+
+# Sleep a bit to have time to replay things
+sleep 2;
+command_is(['psql', '-A', '-t',  '--no-psqlrc',
+		   "-d $connstr_nodes{ $port_standby }", '-c', "SELECT count(*) FROM tab_int"],
+		   qq(1000
+),
+		   'check content from archives');
+
+# Stop nodes
+pgctl_node($port_master, 'stop', '-m', 'immediate');
+pgctl_node($port_standby, 'stop', '-m', 'immediate');
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..e725f35
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,134 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 21;
+
+use RecoveryTest;
+
+RecoveryTest::init_recovery_test('recovery_targets');
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $port_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+
+	my $port_standby = get_free_port();
+
+	init_node_from_backup($port_standby, $port_master, 'my_backup');
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+restore_command='cp -i $archive_nodes{ $port_master }/%f %p'
+));
+
+	foreach my $param_item (@$recovery_params)
+	{
+		append_to_file("$datadir_nodes{ $port_standby }/recovery.conf",
+					   qq($param_item
+));
+	}
+
+	pgctl_node($port_standby, 'start');
+
+	# Sleep a bit to have time to replay things
+	sleep 1;
+
+	# Create some content on master and check its presence in standby
+	command_is(['psql', '-A', '-t',  '--no-psqlrc',
+			   "-d $connstr_nodes{ $port_standby }", '-c',
+			   "SELECT count(*) FROM tab_int"],
+			   qq($num_rows
+),
+			   "check standby content for $test_name");
+
+	# Stop standby nodes
+	pgctl_node($port_standby, 'stop', '-m', 'immediate');
+}
+
+# Initialize master node
+my $port_master = get_free_port();
+init_node($port_master);
+enable_archiving($port_master);
+
+# Start it
+pgctl_node($port_master, 'start');
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+psql_node($port_master,
+		  "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+
+# Take backup from which all operations will be run
+backup_node($port_master, 'my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+psql_node($port_master,
+		  "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+my $recovery_txid = psql_get($port_master, "SELECT txid_current()");
+
+# More data, with recovery target timestamp
+psql_node($port_master,
+		  "INSERT INTO tab_int VALUES (generate_series(2001,3000))");
+my $recovery_time = psql_get($port_master, "SELECT now()");
+
+# Even more data, this time with a recovery target name
+psql_node($port_master,
+		  "INSERT INTO tab_int VALUES (generate_series(3001,4000))");
+my $recovery_name = "my_target";
+psql_node($port_master,
+		  "SELECT pg_create_restore_point('$recovery_name')");
+
+# Force archiving of WAL file
+psql_node($port_master, "SELECT pg_switch_xlog()");
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $port_master,
+					  \@recovery_params,
+					  "1000");
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $port_master,
+					  \@recovery_params,
+					  "2000");
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $port_master,
+					  \@recovery_params,
+					  "3000");
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $port_master,
+					  \@recovery_params,
+					  "4000");
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $port_master,
+					  \@recovery_params,
+					  "3000");
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $port_master,
+					  \@recovery_params,
+					  "2000");
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $port_master,
+					  \@recovery_params,
+					  "4000");
+
+pgctl_node($port_master, 'stop', '-m', 'immediate');
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..1c05e9e
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,65 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use TestLib;
+use Test::More tests => 3;
+
+use RecoveryTest;
+
+RecoveryTest::init_recovery_test('timeline_switch');
+
+# Initialize master node
+my $port_master = get_free_port();
+init_node($port_master);
+
+# Start it
+pgctl_node($port_master, 'start');
+
+# Take backup
+my $backup_name = 'my_backup';
+backup_node($port_master, $backup_name);
+
+# Create two standbys linking to it
+my $port_standby_1 = make_streaming_standby($port_master, $backup_name);
+diag("Started standby 1 with port $port_standby_1");
+my $port_standby_2 = make_streaming_standby($port_master, $backup_name);
+diag("Started standby 2 with port $port_standby_2");
+
+# Create some content on master
+psql_node($port_master, "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+
+# Sleep a bit to have time to replay things
+sleep 1;
+
+# Stop master, and promote standby 1, switching it to a new timeline
+pgctl_node($port_master, 'stop', '-m', 'immediate');
+pgctl_node($port_standby_1, 'promote');
+diag("Promoted standby 1");
+
+# Switch standby 2 to replay from standby 1
+remove_tree("$datadir_nodes{ $port_standby_2 }/recovery.conf");
+append_to_file("$datadir_nodes{ $port_standby_2 }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_standby_1 }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+pgctl_node($port_standby_2, "restart");
+sleep 1;
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done.
+psql_node($port_standby_1,
+		  "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+sleep 1;
+command_is(['psql', '-A', '-t',  '--no-psqlrc',
+			"-d $connstr_nodes{ $port_standby_2 }", '-c',
+			"SELECT count(*) FROM tab_int"], qq(2000
+),
+		   'check content of standby 2');
+
+# Stop nodes
+pgctl_node($port_standby_1, 'stop', '-m', 'immediate');
+pgctl_node($port_standby_2, 'stop', '-m', 'immediate');
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..56bb090
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,58 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 6;
+
+use RecoveryTest;
+
+RecoveryTest::init_recovery_test('replay_delay');
+
+# Initialize master node
+my $port_master = get_free_port();
+init_node($port_master);
+
+# Start it
+pgctl_node($port_master, 'start');
+
+# And some content
+psql_node($port_master, "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Take backup
+backup_node($port_master, 'my_backup');
+
+# Initialize node from backup
+my $port_standby = get_free_port();
+init_node_from_backup($port_standby, $port_master, 'my_backup');
+
+# Start second node, streaming from first one
+append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_master }'
+recovery_min_apply_delay = '2s'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+pgctl_node($port_standby, 'start');
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above
+psql_node($port_master, "INSERT INTO tab_int VALUES (generate_series(11,20))");
+# Here we should have only 10 rows
+sleep 1;
+command_is(['psql', '-A', '-t',  '--no-psqlrc',
+		   "-d $connstr_nodes{ $port_standby }", '-c', "SELECT count(*) FROM tab_int"],
+		   qq(10
+),
+		   'check content with delay of 1s');
+
+# Sleep a bit to have time to replay things and wait for delay to work
+sleep 1;
+command_is(['psql', '-A', '-t',  '--no-psqlrc',
+		   "-d $connstr_nodes{ $port_standby }", '-c', "SELECT count(*) FROM tab_int"],
+		   qq(20
+),
+		   'check content with delay 2s');
+
+# Stop nodes
+pgctl_node($port_master, 'stop', '-m', 'immediate');
+pgctl_node($port_standby, 'stop', '-m', 'immediate');
#17Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#16)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Mar 18, 2015 at 1:59 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

Feedback is of course welcome, but note that I am not seriously
expecting any until we get into 9.6 development cycle and I am adding
this patch to the next CF.

I have moved this patch to CF 2015-09, as I have enough patches to
take care of for now... Let's focus on Windows support and improvement
of logging for TAP in the first round. That will be already a good
step forward.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#18Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#17)
1 attachment(s)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Mon, Jun 29, 2015 at 10:11 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Mar 18, 2015 at 1:59 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

Feedback is of course welcome, but note that I am not seriously
expecting any until we get into 9.6 development cycle and I am adding
this patch to the next CF.

I have moved this patch to CF 2015-09, as I have enough patches to
take care of for now... Let's focus on Windows support and improvement
of logging for TAP in the first round. That will be already a good
step forward.

OK, attached is a new version of this patch, that I have largely
reworked to have more user-friendly routines for the tests. The number
of tests is still limited still it shows what this facility can do:
that's on purpose as it does not make much sense to code a complete
and complicated set of tests as long as the core routines are not
stable, hence let's focus on that first.
I have not done yet tests on Windows, I am expecting some tricks
needed for the archive and recovery commands generated for the tests.
Regards,
--
Michael

Attachments:

20150814_recovery_regressions.patchtext/x-diff; charset=US-ASCII; name=20150814_recovery_regressions.patchDownload
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index 22e5cae..8dd0fb7 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -125,38 +125,6 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
 sub append_to_file
 {
 	my ($filename, $str) = @_;
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..d6e51eb 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -17,7 +17,7 @@ SUBDIRS = regress isolation modules
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
 # because the SSL test suite is not secure to run on a multi-user system.
-ALWAYS_SUBDIRS = examples locale thread ssl
+ALWAYS_SUBDIRS = examples locale thread ssl recovery
 
 # We want to recurse to all subdirs for all standard targets, except that
 # installcheck and install should not recurse into the subdirectory "modules".
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 4927d45..cc1287f 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -12,6 +12,7 @@ our @EXPORT = qw(
   configure_hba_for_replication
   start_test_server
   restart_test_server
+  poll_query_until
   psql
   slurp_dir
   slurp_file
@@ -219,6 +220,37 @@ END
 	}
 }
 
+sub poll_query_until
+{
+	my ($query, $connstr) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
+		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
 sub psql
 {
 	my ($dbname, $sql) = @_;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/RecoveryTest.pm b/src/test/recovery/RecoveryTest.pm
new file mode 100644
index 0000000..4a98815
--- /dev/null
+++ b/src/test/recovery/RecoveryTest.pm
@@ -0,0 +1,309 @@
+package RecoveryTest;
+
+# Set of common routines for recovery regression tests for a PostgreSQL
+# cluster. This includes global variables and methods that can be used
+# by the various set of tests present to set up cluster nodes and
+# configure them according to the test scenario wanted.
+#
+# Cluster nodes can be freely created using initdb or using the existing
+# base backup of another node, with minimum configuration done when the
+# node is created for the first time like having a proper port number.
+# It is then up to the test to decide what to do with the newly-created
+# node.
+#
+# Environmenment configuration of each node is available through a set
+# of global variables provided by this package, hashed depending on the
+# port number of a node:
+# - connstr_nodes connection string to connect to this node
+# - datadir_nodes to get the data folder of a given node
+# - archive_nodes for the location of the WAL archives of a node
+# - backup_nodes for the location of base backups of a node
+#
+# Nodes are identified by their port number, which should be unique
+# for each node of the cluster as it is run locally.
+
+use Cwd;
+use TestLib;
+use Test::More;
+
+use Archive::Tar;
+use IPC::Run qw(run start);
+
+use Exporter 'import';
+
+our @EXPORT = qw(
+	%connstr_nodes
+	%datadir_nodes
+	%backup_nodes
+	%archive_nodes
+
+	append_to_file
+	backup_node
+	enable_archiving
+	enable_restoring
+	enable_streaming
+	get_free_port
+	init_node
+	init_node_from_backup
+	make_master
+	make_warm_standby
+	make_hot_standby
+	restart_node
+	start_node
+	stop_node
+);
+
+# Global variables for node data
+%datadir_nodes = {};	# PGDATA folders
+%backup_nodes = {};		# Backup base folder
+%archive_nodes = {};	# Archive base folder
+%connstr_nodes = {};	# Connection strings
+
+# Database used for each connection attempt via psql
+$ENV{PGDATABASE} = "postgres";
+
+# Tracker of active nodes
+my @active_nodes = ();
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_root }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+	my $copy_command = $windows_os ?
+		"copy \"$datadir_nodes{ $port_root }%f\" \"%p\"" :
+		"cp -i $archive_nodes{ $port_root }/%f %p";
+
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+restore_command='$copy_command'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $port = shift;
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$datadir_nodes{ $port }%f\"" :
+		"cp -i %p $archive_nodes{ $port }/%f";
+
+	# Enable archive_mode and archive_command on node
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization
+sub make_master
+{
+	my $port_master = get_free_port();
+	print "# Initializing master node wih port $port_master\n";
+	init_node($port_master);
+	start_node($port_master);
+	return $port_master;
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_hot_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing streaming mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_streaming($port_master, $port_standby);
+	start_node($port_standby);
+	return $port_standby;
+}
+
+# Node getting WAL only from archives
+sub make_warm_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing archive mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_restoring($port_master, $port_standby);
+	start_node($port_standby);
+	return $port_standby;
+}
+
+sub configure_base_node
+{
+	my $port = shift;
+
+	# Make configuration somewhat generic to test recovery
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+port = $port
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+wal_retrieve_retry_interval = '100ms'
+));
+
+	configure_hba_for_replication($datadir_nodes{ $port });
+}
+
+# Get a port number not in use currently for a new node
+# As port number retrieval is based on the nodes currently running,
+# be sure that the node that is consuming this port number has already
+# been started.
+sub get_free_port
+{
+	my $found = 0;
+	# XXX: Should this part use PG_VERSION_NUM?
+	my $port = 90600 % 16384 + 49152;
+
+	while ($found == 0)
+	{
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		my $ret = system("psql -X -p $port postgres < $devnull");
+		if ($ret != 0)
+		{
+			$found = 1;
+		}
+	}
+
+	print "# Found free port $port\n";
+	return $port;
+}
+
+# Low-level routines to initialize a node
+# Initialize a node from scratch
+sub init_node
+{
+	my $port = shift;
+
+	# Save configuration information
+	$datadir_nodes{ $port } = TestLib::tempdir;
+	$backup_nodes{ $port } = TestLib::tempdir;
+	$connstr_nodes{ $port } = "port=$port";
+	$archive_nodes{ $port } =  TestLib::tempdir;
+
+	standard_initdb($datadir_nodes{ $port });
+	configure_base_node($port);
+}
+
+# Initialize a node from an existing base backup
+sub init_node_from_backup
+{
+	my ($port, $root_port, $backup_name) = @_;
+
+	my $backup_path =  "$backup_nodes{ $root_port }/$backup_name";
+	my $backup_file = "$backup_path/base.tar";
+
+	print "Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	# Check existence of backup wanted
+	if ( ! -d $backup_path )
+	{
+		die "Backup $backup_path does not exist";
+	}
+
+	# Save configuration information
+	$datadir_nodes{ $port } = TestLib::tempdir;
+	$backup_nodes{ $port } = TestLib::tempdir;
+	$connstr_nodes{ $port } = "port=$port";
+	$archive_nodes{ $port } =  TestLib::tempdir;
+
+	# Extract the base backup wanted
+	my $current_dir = cwd();
+
+	# Temporary move to the place of extraction
+	chdir "$datadir_nodes{ $port }";
+	Archive::Tar->extract_archive($backup_file);
+	chdir "$current_dir";
+	configure_base_node($port);
+}
+
+# Start a node
+sub start_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port },
+				   '-l', "$log_path/node_$port.log",
+				   'start');
+	# Register this node as candidate for cleanup
+	push(@active_nodes, $port);
+}
+
+# Stop a node
+sub stop_node
+{
+	my $port = shift;
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   'immediate', 'stop');
+	# Unregister this node as candidate for cleanup
+	@active_nodes = grep { $_ ne $port } @active_nodes;
+}
+
+sub restart_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port }, '-m',
+				   'fast', 'restart');
+}
+
+# Create a backup on a node already running
+sub backup_node
+{
+	my ($port, $backup_name) = @_;
+
+	my $backup_path = "$backup_nodes{ $port }/$backup_name";
+
+	print "Taking backup from node $port\n";
+	# Backup a node in tar format, it is more portable across platforms
+	system_or_bail("pg_basebackup -D $backup_path -p $port --format=t -x");
+	print "# Backup finished\n"
+}
+
+# Add a set of parameters to a configuration file
+sub append_to_file
+{
+	my($filename, $str) = @_;
+
+	open my $fh, ">>", $filename or die "could not open file $filename";
+	print $fh $str;
+	close $fh;
+}
+
+END
+{
+	foreach my $port (@active_nodes)
+	{
+		stop_node($port);
+	}
+}
+
+1;
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..757a639
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,60 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+my $backup_name = 'my_backup';
+
+# Take backup
+backup_node($port_master, $backup_name);
+
+# Create streaming standby linking to master
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+backup_node($port_standby_1, $backup_name);
+
+# Create second standby node linking to standby 1
+my $port_standby_2 = make_hot_standby($port_standby_1, $backup_name);
+
+# Create some content on master and check its presence in standby 1 an
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a";
+
+# Wait for standbys to catch up
+my $current_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= replay_location FROM pg_stat_replication";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_master })
+	or die "Timed out while waiting for standby 1 to catch up";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT count(*) FROM tab_int";
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_1 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_2 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
+
+# Stop nodes
+stop_node($port_standby_2);
+stop_node($port_standby_1);
+stop_node($port_master);
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..756604d
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,48 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $port_master = get_free_port();
+my $backup_name = 'my_backup';
+init_node($port_master);
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Take backup for slave
+backup_node($port_master, 'my_backup');
+
+# Initialize standby node from backup, fetching WAL from archives
+my $port_standby = make_hot_standby($port_master, $backup_name);
+
+# Create some content on master
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $current_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Force archiving of WAL file to make it present on master
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Add some more content, it should not be present on standby
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(1000), 'check content from archives');
+
+# Stop nodes
+stop_node($port_standby);
+stop_node($port_master);
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..3246efc
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,139 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $port_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $port_standby = get_free_port();
+
+	init_node_from_backup($port_standby, $port_master, 'my_backup');
+	enable_restoring($port_master, $port_standby);
+
+	foreach my $param_item (@$recovery_params)
+	{
+		append_to_file("$datadir_nodes{ $port_standby }/recovery.conf",
+					   qq($param_item
+));
+	}
+
+	start_node($port_standby);
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = psql $connstr_nodes{ $port_standby },
+		"SELECT count(*) FROM tab_int";
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	stop_node($port_standby);
+}
+
+# Initialize master node
+my $port_master = get_free_port();
+init_node($port_master);
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $lsn1 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Take backup from which all operations will be run
+backup_node($port_master, 'my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+my $recovery_txid = psql $connstr_nodes{ $port_master },
+	"SELECT txid_current()";
+my $lsn2 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# More data, with recovery target timestamp
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(2001,3000))";
+my $recovery_time = psql $connstr_nodes{ $port_master }, "SELECT now()";
+my $lsn3 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Even more data, this time with a recovery target name
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))";
+my $recovery_name = "my_target";
+my $lsn4 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+psql $connstr_nodes{ $port_master },
+	"SELECT pg_create_restore_point('$recovery_name')";
+
+# Force archiving of WAL file
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $port_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+stop_node($port_master);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..15bcd05
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,65 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+
+# Take backup
+my $backup_name = 'my_backup';
+backup_node($port_master, $backup_name);
+
+# Create two standbys linking to it
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+my $port_standby_2 = make_hot_standby($port_master, $backup_name);
+
+# Create some content on master
+psql $connstr_nodes { $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop master, and promote standby 1, switching it to a new timeline
+stop_node($port_master);
+system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port_standby_1 },
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree("$datadir_nodes{ $port_standby_2 }/recovery.conf");
+append_to_file("$datadir_nodes{ $port_standby_2 }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_standby_1 }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+restart_node($port_standby_2);
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done.
+psql $connstr_nodes{ $port_standby_1 },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+$until_lsn = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT pg_current_xlog_location();";
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_2 })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(2000), 'check content of standby 2');
+
+# Stop nodes
+stop_node($port_standby_2);
+stop_node($port_standby_1);
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..419b835
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,53 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+
+# And some content
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a";
+
+# Take backup
+backup_node($port_master, 'my_backup');
+
+# Initialize node from backup
+my $port_standby = get_free_port();
+init_node_from_backup($port_standby, $port_master, 'my_backup');
+
+# Start second node, streaming from first one
+append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_master }'
+recovery_min_apply_delay = '2s'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+start_node($port_standby);
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(11,20))";
+sleep 1;
+# Here we should have only 10 rows
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+$result = psql $connstr_nodes{ $port_standby }, "SELECT count(*) FROM tab_int";
+is($result, qq(20), 'check content with delay of 2s');
+
+# Stop nodes
+stop_node($port_standby);
+stop_node($port_master);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index d3d736b..1355508 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -194,8 +194,9 @@ sub tapcheck
 	foreach my $test_path (@$tap_dirs)
 	{
 		# Like on Unix "make check-world", don't run the SSL test suite
-		# automatically.
+		# or the recovery test suite automatically.
 		next if ($test_path =~ /\/src\/test\/ssl\//);
+		next if ($test_path =~ /\/src\/test\/recovery\//);
 
 		my $dir = dirname($test_path);
 		chdir $dir;
#19Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#18)
1 attachment(s)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Aug 14, 2015 at 12:54 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Mon, Jun 29, 2015 at 10:11 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Mar 18, 2015 at 1:59 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

Feedback is of course welcome, but note that I am not seriously
expecting any until we get into 9.6 development cycle and I am adding
this patch to the next CF.

I have moved this patch to CF 2015-09, as I have enough patches to
take care of for now... Let's focus on Windows support and improvement
of logging for TAP in the first round. That will be already a good
step forward.

OK, attached is a new version of this patch, that I have largely
reworked to have more user-friendly routines for the tests. The number
of tests is still limited still it shows what this facility can do:
that's on purpose as it does not make much sense to code a complete
and complicated set of tests as long as the core routines are not
stable, hence let's focus on that first.
I have not done yet tests on Windows, I am expecting some tricks
needed for the archive and recovery commands generated for the tests.

Attached is v3. I have tested and fixed the tests such as they can run
on Windows. archive_command and restore_command are using Windows'
copy when needed. There was also a bug with the use of a hot standby
instead of a warm one, causing test 002 to fail.
I am rather happy with the shape of this patch now, so feel free to review it...
Regards,
--
Michael

Attachments:

20150814_recovery_regressions_v3.patchtext/x-diff; charset=US-ASCII; name=20150814_recovery_regressions_v3.patchDownload
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index 22e5cae..8dd0fb7 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -125,38 +125,6 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
 sub append_to_file
 {
 	my ($filename, $str) = @_;
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..d6e51eb 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -17,7 +17,7 @@ SUBDIRS = regress isolation modules
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
 # because the SSL test suite is not secure to run on a multi-user system.
-ALWAYS_SUBDIRS = examples locale thread ssl
+ALWAYS_SUBDIRS = examples locale thread ssl recovery
 
 # We want to recurse to all subdirs for all standard targets, except that
 # installcheck and install should not recurse into the subdirectory "modules".
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 4927d45..cc1287f 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -12,6 +12,7 @@ our @EXPORT = qw(
   configure_hba_for_replication
   start_test_server
   restart_test_server
+  poll_query_until
   psql
   slurp_dir
   slurp_file
@@ -219,6 +220,37 @@ END
 	}
 }
 
+sub poll_query_until
+{
+	my ($query, $connstr) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
+		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
 sub psql
 {
 	my ($dbname, $sql) = @_;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/RecoveryTest.pm b/src/test/recovery/RecoveryTest.pm
new file mode 100644
index 0000000..c015f3b
--- /dev/null
+++ b/src/test/recovery/RecoveryTest.pm
@@ -0,0 +1,316 @@
+package RecoveryTest;
+
+# Set of common routines for recovery regression tests for a PostgreSQL
+# cluster. This includes global variables and methods that can be used
+# by the various set of tests present to set up cluster nodes and
+# configure them according to the test scenario wanted.
+#
+# Cluster nodes can be freely created using initdb or using the existing
+# base backup of another node, with minimum configuration done when the
+# node is created for the first time like having a proper port number.
+# It is then up to the test to decide what to do with the newly-created
+# node.
+#
+# Environmenment configuration of each node is available through a set
+# of global variables provided by this package, hashed depending on the
+# port number of a node:
+# - connstr_nodes connection string to connect to this node
+# - datadir_nodes to get the data folder of a given node
+# - archive_nodes for the location of the WAL archives of a node
+# - backup_nodes for the location of base backups of a node
+#
+# Nodes are identified by their port number, which should be unique
+# for each node of the cluster as it is run locally.
+
+use Cwd;
+use TestLib;
+use Test::More;
+
+use Archive::Tar;
+use IPC::Run qw(run start);
+
+use Exporter 'import';
+
+our @EXPORT = qw(
+	%connstr_nodes
+	%datadir_nodes
+	%backup_nodes
+	%archive_nodes
+
+	append_to_file
+	backup_node
+	enable_archiving
+	enable_restoring
+	enable_streaming
+	get_free_port
+	init_node
+	init_node_from_backup
+	make_master
+	make_warm_standby
+	make_hot_standby
+	restart_node
+	start_node
+	stop_node
+);
+
+# Global variables for node data
+%datadir_nodes = {};	# PGDATA folders
+%backup_nodes = {};		# Backup base folder
+%archive_nodes = {};	# Archive base folder
+%connstr_nodes = {};	# Connection strings
+
+# Database used for each connection attempt via psql
+$ENV{PGDATABASE} = "postgres";
+
+# Tracker of active nodes
+my @active_nodes = ();
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_root }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+	my $path = $archive_nodes{ $port_root };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"$path\\\\%f\" \"%p\"" :
+		"cp -i $path/%f %p";
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+restore_command='$copy_command'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $port = shift;
+	my $path = $archive_nodes{ $port };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$path\\\\%f\"" :
+		"cp -i %p $path/%f";
+
+	# Enable archive_mode and archive_command on node
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization
+sub make_master
+{
+	my $port_master = get_free_port();
+	print "# Initializing master node wih port $port_master\n";
+	init_node($port_master);
+	start_node($port_master);
+	return $port_master;
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_hot_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing streaming mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_streaming($port_master, $port_standby);
+	start_node($port_standby);
+	return $port_standby;
+}
+
+# Node getting WAL only from archives
+sub make_warm_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing archive mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_restoring($port_master, $port_standby);
+	start_node($port_standby);
+	return $port_standby;
+}
+
+sub configure_base_node
+{
+	my $port = shift;
+
+	# Make configuration somewhat generic to test recovery
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+port = $port
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+wal_retrieve_retry_interval = '100ms'
+));
+
+	configure_hba_for_replication($datadir_nodes{ $port });
+}
+
+# Get a port number not in use currently for a new node
+# As port number retrieval is based on the nodes currently running,
+# be sure that the node that is consuming this port number has already
+# been started.
+sub get_free_port
+{
+	my $found = 0;
+	# XXX: Should this part use PG_VERSION_NUM?
+	my $port = 90600 % 16384 + 49152;
+
+	while ($found == 0)
+	{
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		my $ret = system("psql -X -p $port postgres < $devnull");
+		if ($ret != 0)
+		{
+			$found = 1;
+		}
+	}
+
+	print "# Found free port $port\n";
+	return $port;
+}
+
+# Low-level routines to initialize a node
+# Initialize a node from scratch
+sub init_node
+{
+	my $port = shift;
+
+	# Save configuration information
+	$datadir_nodes{ $port } = TestLib::tempdir;
+	$backup_nodes{ $port } = TestLib::tempdir;
+	$connstr_nodes{ $port } = "port=$port";
+	$archive_nodes{ $port } =  TestLib::tempdir;
+
+	standard_initdb($datadir_nodes{ $port });
+	configure_base_node($port);
+}
+
+# Initialize a node from an existing base backup
+sub init_node_from_backup
+{
+	my ($port, $root_port, $backup_name) = @_;
+
+	my $backup_path =  "$backup_nodes{ $root_port }/$backup_name";
+	my $backup_file = "$backup_path/base.tar";
+
+	print "Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	# Check existence of backup wanted
+	if ( ! -d $backup_path )
+	{
+		die "Backup $backup_path does not exist";
+	}
+
+	# Save configuration information
+	$datadir_nodes{ $port } = TestLib::tempdir;
+	$backup_nodes{ $port } = TestLib::tempdir;
+	$connstr_nodes{ $port } = "port=$port";
+	$archive_nodes{ $port } =  TestLib::tempdir;
+
+	# Extract the base backup wanted
+	my $current_dir = cwd();
+
+	# Temporary move to the place of extraction
+	chdir "$datadir_nodes{ $port }";
+	Archive::Tar->extract_archive($backup_file);
+	chdir "$current_dir";
+	configure_base_node($port);
+}
+
+# Start a node
+sub start_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port },
+				   '-l', "$log_path/node_$port.log",
+				   'start');
+	# Register this node as candidate for cleanup
+	push(@active_nodes, $port);
+}
+
+# Stop a node
+sub stop_node
+{
+	my $port = shift;
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   'immediate', 'stop');
+	# Unregister this node as candidate for cleanup
+	@active_nodes = grep { $_ ne $port } @active_nodes;
+}
+
+sub restart_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port }, '-m',
+				   'fast', 'restart');
+}
+
+# Create a backup on a node already running
+sub backup_node
+{
+	my ($port, $backup_name) = @_;
+
+	my $backup_path = "$backup_nodes{ $port }/$backup_name";
+
+	print "Taking backup from node $port\n";
+	# Backup a node in tar format, it is more portable across platforms
+	system_or_bail("pg_basebackup -D $backup_path -p $port --format=t -x");
+	print "# Backup finished\n"
+}
+
+# Add a set of parameters to a configuration file
+sub append_to_file
+{
+	my($filename, $str) = @_;
+
+	open my $fh, ">>", $filename or die "could not open file $filename";
+	print $fh $str;
+	close $fh;
+}
+
+END
+{
+	foreach my $port (@active_nodes)
+	{
+		stop_node($port);
+	}
+}
+
+1;
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..757a639
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,60 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+my $backup_name = 'my_backup';
+
+# Take backup
+backup_node($port_master, $backup_name);
+
+# Create streaming standby linking to master
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+backup_node($port_standby_1, $backup_name);
+
+# Create second standby node linking to standby 1
+my $port_standby_2 = make_hot_standby($port_standby_1, $backup_name);
+
+# Create some content on master and check its presence in standby 1 an
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a";
+
+# Wait for standbys to catch up
+my $current_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= replay_location FROM pg_stat_replication";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_master })
+	or die "Timed out while waiting for standby 1 to catch up";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT count(*) FROM tab_int";
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_1 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_2 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
+
+# Stop nodes
+stop_node($port_standby_2);
+stop_node($port_standby_1);
+stop_node($port_master);
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..2470654
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,48 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $port_master = get_free_port();
+my $backup_name = 'my_backup';
+init_node($port_master);
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Take backup for slave
+backup_node($port_master, 'my_backup');
+
+# Initialize standby node from backup, fetching WAL from archives
+my $port_standby = make_warm_standby($port_master, $backup_name);
+
+# Create some content on master
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $current_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Force archiving of WAL file to make it present on master
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Add some more content, it should not be present on standby
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(1000), 'check content from archives');
+
+# Stop nodes
+stop_node($port_standby);
+stop_node($port_master);
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..3246efc
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,139 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $port_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $port_standby = get_free_port();
+
+	init_node_from_backup($port_standby, $port_master, 'my_backup');
+	enable_restoring($port_master, $port_standby);
+
+	foreach my $param_item (@$recovery_params)
+	{
+		append_to_file("$datadir_nodes{ $port_standby }/recovery.conf",
+					   qq($param_item
+));
+	}
+
+	start_node($port_standby);
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = psql $connstr_nodes{ $port_standby },
+		"SELECT count(*) FROM tab_int";
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	stop_node($port_standby);
+}
+
+# Initialize master node
+my $port_master = get_free_port();
+init_node($port_master);
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $lsn1 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Take backup from which all operations will be run
+backup_node($port_master, 'my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+my $recovery_txid = psql $connstr_nodes{ $port_master },
+	"SELECT txid_current()";
+my $lsn2 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# More data, with recovery target timestamp
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(2001,3000))";
+my $recovery_time = psql $connstr_nodes{ $port_master }, "SELECT now()";
+my $lsn3 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Even more data, this time with a recovery target name
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))";
+my $recovery_name = "my_target";
+my $lsn4 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+psql $connstr_nodes{ $port_master },
+	"SELECT pg_create_restore_point('$recovery_name')";
+
+# Force archiving of WAL file
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $port_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+stop_node($port_master);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..15bcd05
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,65 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+
+# Take backup
+my $backup_name = 'my_backup';
+backup_node($port_master, $backup_name);
+
+# Create two standbys linking to it
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+my $port_standby_2 = make_hot_standby($port_master, $backup_name);
+
+# Create some content on master
+psql $connstr_nodes { $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop master, and promote standby 1, switching it to a new timeline
+stop_node($port_master);
+system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port_standby_1 },
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree("$datadir_nodes{ $port_standby_2 }/recovery.conf");
+append_to_file("$datadir_nodes{ $port_standby_2 }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_standby_1 }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+restart_node($port_standby_2);
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done.
+psql $connstr_nodes{ $port_standby_1 },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+$until_lsn = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT pg_current_xlog_location();";
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_2 })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(2000), 'check content of standby 2');
+
+# Stop nodes
+stop_node($port_standby_2);
+stop_node($port_standby_1);
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..419b835
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,53 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+
+# And some content
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a";
+
+# Take backup
+backup_node($port_master, 'my_backup');
+
+# Initialize node from backup
+my $port_standby = get_free_port();
+init_node_from_backup($port_standby, $port_master, 'my_backup');
+
+# Start second node, streaming from first one
+append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_master }'
+recovery_min_apply_delay = '2s'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+start_node($port_standby);
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(11,20))";
+sleep 1;
+# Here we should have only 10 rows
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+$result = psql $connstr_nodes{ $port_standby }, "SELECT count(*) FROM tab_int";
+is($result, qq(20), 'check content with delay of 2s');
+
+# Stop nodes
+stop_node($port_standby);
+stop_node($port_master);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index d3d736b..1355508 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -194,8 +194,9 @@ sub tapcheck
 	foreach my $test_path (@$tap_dirs)
 	{
 		# Like on Unix "make check-world", don't run the SSL test suite
-		# automatically.
+		# or the recovery test suite automatically.
 		next if ($test_path =~ /\/src\/test\/ssl\//);
+		next if ($test_path =~ /\/src\/test\/recovery\//);
 
 		my $dir = dirname($test_path);
 		chdir $dir;
#20Amir Rohan
amir.rohan@mail.com
In reply to: Michael Paquier (#19)
1 attachment(s)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 08/14/2015 06:32 AM, Michael Paquier wrote:

On Fri, Aug 14, 2015 at 12:54 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Mon, Jun 29, 2015 at 10:11 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Mar 18, 2015 at 1:59 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

Feedback is of course welcome, but note that I am not seriously
expecting any until we get into 9.6 development cycle and I am adding
this patch to the next CF.

I have moved this patch to CF 2015-09, as I have enough patches to
take care of for now... Let's focus on Windows support and improvement
of logging for TAP in the first round. That will be already a good
step forward.

OK, attached is a new version of this patch, that I have largely
reworked to have more user-friendly routines for the tests. The number
of tests is still limited still it shows what this facility can do:
that's on purpose as it does not make much sense to code a complete
and complicated set of tests as long as the core routines are not
stable, hence let's focus on that first.
I have not done yet tests on Windows, I am expecting some tricks
needed for the archive and recovery commands generated for the tests.

Attached is v3. I have tested and fixed the tests such as they can run
on Windows. archive_command and restore_command are using Windows'
copy when needed. There was also a bug with the use of a hot standby
instead of a warm one, causing test 002 to fail.
I am rather happy with the shape of this patch now, so feel free to review it...
Regards,

Michael, I've ran these and it worked fine for me.
See attached patch with a couple of minor fixes.

Amir

Attachments:

20150925_recovery_regressions_amir_fixes.patchtext/x-patch; name=20150925_recovery_regressions_amir_fixes.patchDownload
diff --git a/src/test/recovery/RecoveryTest.pm b/src/test/recovery/RecoveryTest.pm
index c015f3b..5cc977d 100644
--- a/src/test/recovery/RecoveryTest.pm
+++ b/src/test/recovery/RecoveryTest.pm
@@ -11,7 +11,7 @@ package RecoveryTest;
 # It is then up to the test to decide what to do with the newly-created
 # node.
 #
-# Environmenment configuration of each node is available through a set
+# Environment configuration of each node is available through a set
 # of global variables provided by this package, hashed depending on the
 # port number of a node:
 # - connstr_nodes connection string to connect to this node
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index 757a639..2c59b73 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -31,9 +31,9 @@ psql $connstr_nodes{ $port_master },
 my $current_lsn = psql $connstr_nodes{ $port_master },
 	"SELECT pg_current_xlog_location();";
 my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= replay_location FROM pg_stat_replication";
-poll_query_until($caughtup_query, $connstr_nodes{ $port_master })
-	or die "Timed out while waiting for standby 1 to catch up";
 poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby 1 to catch up";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_2 })
 	or die "Timed out while waiting for standby 2 to catch up";

 my $result = psql $connstr_nodes{ $port_standby_1 },
--
2.4.3

#21Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#20)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Sep 25, 2015 at 3:11 PM, Amir Rohan <amir.rohan@mail.com> wrote:

On 08/14/2015 06:32 AM, Michael Paquier wrote:

On Fri, Aug 14, 2015 at 12:54 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Mon, Jun 29, 2015 at 10:11 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Mar 18, 2015 at 1:59 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

Feedback is of course welcome, but note that I am not seriously
expecting any until we get into 9.6 development cycle and I am adding
this patch to the next CF.

I have moved this patch to CF 2015-09, as I have enough patches to
take care of for now... Let's focus on Windows support and improvement
of logging for TAP in the first round. That will be already a good
step forward.

OK, attached is a new version of this patch, that I have largely
reworked to have more user-friendly routines for the tests. The number
of tests is still limited still it shows what this facility can do:
that's on purpose as it does not make much sense to code a complete
and complicated set of tests as long as the core routines are not
stable, hence let's focus on that first.
I have not done yet tests on Windows, I am expecting some tricks
needed for the archive and recovery commands generated for the tests.

Attached is v3. I have tested and fixed the tests such as they can run
on Windows. archive_command and restore_command are using Windows'
copy when needed. There was also a bug with the use of a hot standby
instead of a warm one, causing test 002 to fail.
I am rather happy with the shape of this patch now, so feel free to

review it...

Regards,

Michael, I've ran these and it worked fine for me.
See attached patch with a couple of minor fixes.

Thanks! I still think that we could improve a bit more the way
parametrization is done in postgresql.conf when a node is initialized by
appending a list of parameters or have a set of hardcoded behaviors
including a set of default parameters and their values... But well feedback
is welcome regarding that. I also arrived at the conclusion that it would
be better to place the new package file in src/test/perl instead of
src/test/recovery to allow any users of the TAP tests to have it in their
PERL5LIB path and to be able to call the new routines to create and
manipulate nodes.
--
Michael

#22Amir Rohan
amir.rohan@mail.com
In reply to: Michael Paquier (#21)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 09/25/2015 09:29 AM, Michael Paquier wrote:

On Fri, Sep 25, 2015 at 3:11 PM, Amir Rohan <amir.rohan@mail.com
<mailto:amir.rohan@mail.com>> wrote:

On 08/14/2015 06:32 AM, Michael Paquier wrote:

I am rather happy with the shape of this patch now, so feel free

to review it...

Regards,

Michael, I've ran these and it worked fine for me.
See attached patch with a couple of minor fixes.

Thanks! I still think that we could improve a bit more the way
parametrization is done in postgresql.conf when a node is initialized by
appending a list of parameters or have a set of hardcoded behaviors
including a set of default parameters and their values... But well
feedback is welcome regarding that. I also arrived at the conclusion
that it would be better to place the new package file in src/test/perl
instead of src/test/recovery to allow any users of the TAP tests to have
it in their PERL5LIB path and to be able to call the new routines to
create and manipulate nodes.
--
Michael

Having a subcommand in Greg's PEG (http://github.com/gregs1104/peg),
that allows you to create one of several "canned" clusters would be
convenient as well, for manual testing and folling around with features.

Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#23Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#22)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Sep 25, 2015 at 5:57 PM, Amir Rohan <amir.rohan@mail.com> wrote:

Having a subcommand in Greg's PEG (http://github.com/gregs1104/peg),
that allows you to create one of several "canned" clusters would be
convenient as well, for manual testing and folling around with features.

That's the kind of thing that each serious developer on this mailing
list already has in a rather different shape but with the same final
result: offer a way to set up a cluster ready for hacking in a single
command, and being able to switch among them easily. I am not sure we
would really find a cross-platform script generic enough to satisfy
all the needs of folks here and integrate it directly in the tree :)
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#24Amir Rohan
amir.rohan@mail.com
In reply to: Michael Paquier (#23)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 09/25/2015 01:47 PM, Michael Paquier wrote:

On Fri, Sep 25, 2015 at 5:57 PM, Amir Rohan <amir.rohan@mail.com> wrote:

That's the kind of thing that each serious developer on this mailing
list already has in a rather different shape but with the same final
result:

Oh, I guess I'll have to write one then. :)

offer a way to set up a cluster ready for hacking in a single
command, and being able to switch among them easily. I am not sure we
would really find a cross-platform script generic enough to satisfy
all the needs of folks here and integrate it directly in the tree :)

Yes, perl/bash seems to be the standard and both have their shorcomings,
in either convenience or familiarity... a static binary +
scripts/configuration would be least resistance,and it doesn't actually
have to live in the tree, but it needs to be good enough out of the box
that someone new will prefer to use/improve it over rolling their own
from scratch, and people need to know it's there.

Coming back to the recovery testing package...
I was investigating some behaviour having to do with recovery
and tried your new library to write a repro case. This uncovers some
implicit assumptions in the package that can make things diffficult
when violated. I had to rewrite nearly every function I used.

Major pain points:

1) Lib stores metadata (ports,paths, etc') using ports as keys.
-> Assumes ports aren't reused.
-> Because assumes server keep running until the tear down.

And

2) Behaviour (paths in particular) is hardwired rather then overridable
defaults.

This is exactly what I needed to test, problems:
3) Can't stop server without clearing its testing data (the maps holding
paths and things). But that data might be specifically
needed, in particular the backup shouldn't disappear when the
server melts down or we have a very low-grade DBA on our hands.
4) Port assignment relies on liveness checks on running servers.
If a server is shut down and a new instantiated, the port will get
reused, data will get trashed, and various confusing things can happen.
5) Servers are shutdown with -m 'immediate', which can lead to races
in the script when archiving is turned on. That may be good for some
tests, but there's no control over it.

Other issues:
6. Directory structure, used one directory per thing but more logical
to place all things related to an instance under a single directory,
and name them according to role (57333_backup, and so on).
7. enable_restoring() uses "cp -i" 'archive_command', not a good fit
for an automated test.

Aside from running the tests, the convenience of writing them
needs to be considered. My perl is very weak, it's been at least
a decade, but It was difficult to make progress because everything
is geared toward a batch "pass/fail" run . Stdout is redirected,
and the log files can't be 'tail --retry -f' in another terminal,
because they're clobbered at every run. Also:
8. No canned way to output a pprinted overview of the running system
(paths, ports, for manual checking).
9. Finding things is difficult, See 6.
10. If a test passes/fails or dies due to a bug, everything is cleaned.
Great for testing, bad for postmortem.
11. a canned "server is responding to queries" helper would be convenient.

It might be a good idea to:
1) Never reuse ports during a test. Liveness checking is used
to avoid collisions, but should not determine order of assignment.
2) Decouple cleanup from server shutdown. Do the cleanup as the end of
test only, and allow the user to keep things around.
3) Adjust the directory structure to one top directory per server with
(PGDATA, backup, archive) subdirs.
4) Instead of passing ports around as keys, have _explicit functions
which can be called directly by the user (I'd like the backup *HERE*
please), with the current functions refactored to merely invoke them
by interpolating in the values associated with the port they were given.
4b) server shutdown should perheps be "smart" by default, or segmented
into calmly_bring_to_a_close(), pull_electric_plug() and
drop_down_the_stairs_into_swimming_pool().

Regards,
Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#25Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#24)
1 attachment(s)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Sep 26, 2015 at 10:25 AM, Amir Rohan wrote:

On 09/25/2015 01:47 PM, Michael Paquier wrote:

offer a way to set up a cluster ready for hacking in a single
command, and being able to switch among them easily. I am not sure we
would really find a cross-platform script generic enough to satisfy
all the needs of folks here and integrate it directly in the tree :)

Yes, perl/bash seems to be the standard and both have their shorcomings,
in either convenience or familiarity...

Note as well that bash does not run on Windows, so this should be in
perl for the cross-platform requirements.

Coming back to the recovery testing package...

Thanks for providing input on this patch!

I was investigating some behaviour having to do with recovery
and tried your new library to write a repro case. This uncovers some
implicit assumptions in the package that can make things diffficult
when violated. I had to rewrite nearly every function I used.

OK. Noted. Could you be more explicit here with for example an example
of script or a patch?

Major pain points:
1) Lib stores metadata (ports,paths, etc') using ports as keys.
-> Assumes ports aren't reused.

What would be the point of reusing the same port number in a test for
different nodes? Ports are assigned automatically depending on their
availability looking at if a server is listening to it so they are
never reused as long as the node is running so that's a non-issue IMO.
Any server instances created during the tests should never use a
user-defined port for portability. Hence using those ports as keys
just made sense. We could have for example custom names, that have
port values assigned to them, but that's actually an overkill and
complicates the whole facility.

-> Because assumes server keep running until the tear down.

Check. Locking down the port number is the problem here.

And
2) Behaviour (paths in particular) is hardwired rather then overridable
defaults.

This is the case of all the TAP tests. We could always use the same
base directory for all the nodes and then embed a sub-directory whose
name is decided using the port number. But I am not really sure if
that's a win.

This is exactly what I needed to test, problems:
3) Can't stop server without clearing its testing data (the maps holding
paths and things). But that data might be specifically
needed, in particular the backup shouldn't disappear when the
server melts down or we have a very low-grade DBA on our hands.

OK, you have a point here. You may indeed want routines for to enable
and disable a node completely decoupled from start and stop, with
something like enable_node and disable_node that basically registers
or unregisters it from the list of active nodes. I have updated the
patch this way.

4) Port assignment relies on liveness checks on running servers.
If a server is shut down and a new instantiated, the port will get
reused, data will get trashed, and various confusing things can happen.

Right. The safest way to do that is to check in get_free_port if a
port number is used by a registered node, and continue to loop in if
that's the case. So done.

5) Servers are shutdown with -m 'immediate', which can lead to races
in the script when archiving is turned on. That may be good for some
tests, but there's no control over it.

I hesitated with fast here actually. So changed this way. We would
want as wall a teardown command to stop the node with immediate and
unregister the node from the active list.

Other issues:
6. Directory structure, used one directory per thing but more logical
to place all things related to an instance under a single directory,
and name them according to role (57333_backup, and so on).

Er, well. The first version of the patch did so, and then I switched
to an approach closer to what the existing TAP facility is doing. But
well let's simplify things a bit.

7. enable_restoring() uses "cp -i" 'archive_command', not a good fit
for an automated test.

This seems like a good default to me, and actually that's portable on
Windows easily. One could always append a custom archive_command in a
test when for example testing conflicting archiving when archive_mode
= always.

Aside from running the tests, the convenience of writing them
needs to be considered. My perl is very weak, it's been at least
a decade, but It was difficult to make progress because everything
is geared toward a batch "pass/fail" run . Stdout is redirected,
and the log files can't be 'tail --retry -f' in another terminal,
because they're clobbered at every run.

This relies on the existing TAP infrastructure and this has been
considered as the most portable way of doing by including Windows.
This patch is not aiming at changing that, and will use as much as
possible the existing infrastructure.

8. No canned way to output a pprinted overview of the running system
(paths, ports, for manual checking).

Hm. Why not... Are you suggesting something like print_current_conf
that goes through all the registered nodes and outputs that? How would
you use it?

9. Finding things is difficult, See 6.

See my comment above.

10. If a test passes/fails or dies due to a bug, everything is cleaned.
Great for testing, bad for postmortem.

That's something directly related to TestLib.pm where
File:Temp:tempdir creates a temporary path with CLEANUP = 1. We had
discussions regarding that actually...

11. a canned "server is responding to queries" helper would be convenient.

Do you mean a wrapper on pg_isready? Do you have use cases in mind for it?

It might be a good idea to:
1) Never reuse ports during a test. Liveness checking is used
to avoid collisions, but should not determine order of assignment.

Agreed. As far as I can see the problem here is related to the fact
that the port of non-running server may be fetched by another one.
That's a bug of my patch.

2) Decouple cleanup from server shutdown. Do the cleanup as the end of
test only, and allow the user to keep things around.

Agreed here.

3) Adjust the directory structure to one top directory per server with
(PGDATA, backup, archive) subdirs.

Hm. OK. The first version of the patch actually did so.

4) Instead of passing ports around as keys, have _explicit functions
which can be called directly by the user (I'd like the backup *HERE*
please), with the current functions refactored to merely invoke them
by interpolating in the values associated with the port they were given.

I don't really see in what this would be a win. We definitely should
have all the data depending on temporary paths during the tests to
facilitate the cleanup wrapper work.

4b) server shutdown should perhaps be "smart" by default, or segmented
into calmly_bring_to_a_close(), pull_electric_plug() and
drop_down_the_stairs_into_swimming_pool().

Nope, not agreeing here. "immediate" is rather violent to stop a node,
hence I have switched it to use "fast" and there is now a
teardown_node routine that uses immediate, that's more aimed at
cleanup up existing things fiercely.

I have as well moved RecoveryTest.pm to src/test/perl so as all the
consumers of prove_check can use it by default, and decoupled
start_node from make_master and make_*_standby so as it is possible to
add for example new parameters to their postgresql.conf and
recovery.conf files before starting them.

Thanks a lot for the feedback! Attached is an updated patch with all
the things mentioned above done. Are included as well the typo fixes
you sent upthread.
Regards,
--
Michael

Attachments:

20151002_recovery_regressions_v4.patchtext/x-patch; charset=US-ASCII; name=20151002_recovery_regressions_v4.patchDownload
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..ea219d7 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -125,38 +125,6 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
 sub append_to_file
 {
 	my ($filename, $str) = @_;
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..d6e51eb 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -17,7 +17,7 @@ SUBDIRS = regress isolation modules
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
 # because the SSL test suite is not secure to run on a multi-user system.
-ALWAYS_SUBDIRS = examples locale thread ssl
+ALWAYS_SUBDIRS = examples locale thread ssl recovery
 
 # We want to recurse to all subdirs for all standard targets, except that
 # installcheck and install should not recurse into the subdirectory "modules".
diff --git a/src/test/perl/RecoveryTest.pm b/src/test/perl/RecoveryTest.pm
new file mode 100644
index 0000000..16bcf82
--- /dev/null
+++ b/src/test/perl/RecoveryTest.pm
@@ -0,0 +1,356 @@
+package RecoveryTest;
+
+# Set of common routines for recovery regression tests for a PostgreSQL
+# cluster. This includes global variables and methods that can be used
+# by the various set of tests present to set up cluster nodes and
+# configure them according to the test scenario wanted.
+#
+# Cluster nodes can be freely created using initdb or using the existing
+# base backup of another node, with minimum configuration done when the
+# node is created for the first time like having a proper port number.
+# It is then up to the test to decide what to do with the newly-created
+# node.
+#
+# Environment configuration of each node is available through a set
+# of global variables provided by this package, hashed depending on the
+# port number of a node:
+# - connstr_nodes connection string to connect to this node
+# - datadir_nodes to get the data folder of a given node
+# - archive_nodes for the location of the WAL archives of a node
+# - backup_nodes for the location of base backups of a node
+# - applname_nodes, application_name to use for a standby
+#
+# Nodes are identified by their port number, which should be unique
+# for each node of the cluster as it is run locally.
+
+use Cwd;
+use TestLib;
+use Test::More;
+
+use Archive::Tar;
+use IPC::Run qw(run start);
+
+use Exporter 'import';
+
+our @EXPORT = qw(
+	%connstr_nodes
+	%datadir_nodes
+	%backup_nodes
+	%archive_nodes
+	%applname_nodes
+
+	append_to_file
+	backup_node
+	disable_node
+	enable_archiving
+	enable_node
+	enable_restoring
+	enable_streaming
+	get_free_port
+	init_node
+	init_node_from_backup
+	make_master
+	make_warm_standby
+	make_hot_standby
+	restart_node
+	start_node
+	stop_node
+	teardown_node
+);
+
+# Global variables for node data
+%datadir_nodes = {};	# PGDATA folders
+%backup_nodes = {};		# Backup base folder
+%archive_nodes = {};	# Archive base folder
+%connstr_nodes = {};	# Connection strings
+%applname_nodes = {};	# application_name used for standbys
+
+# Database used for each connection attempt via psql
+$ENV{PGDATABASE} = "postgres";
+
+# Tracker of active nodes
+my @active_nodes = ();
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_root } application_name=$applname_nodes{ $port_standby }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+	my $path = $archive_nodes{ $port_root };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"$path\\\\%f\" \"%p\"" :
+		"cp -i $path/%f %p";
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+restore_command='$copy_command'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $port = shift;
+	my $path = $archive_nodes{ $port };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$path\\\\%f\"" :
+		"cp -i %p $path/%f";
+
+	# Enable archive_mode and archive_command on node
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization
+sub make_master
+{
+	my $port_master = get_free_port();
+	print "# Initializing master node wih port $port_master\n";
+	init_node($port_master);
+	return $port_master;
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_hot_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing streaming mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_streaming($port_master, $port_standby);
+	return $port_standby;
+}
+
+# Node getting WAL only from archives
+sub make_warm_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing archive mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_restoring($port_master, $port_standby);
+	return $port_standby;
+}
+
+sub configure_base_node
+{
+	my $port = shift;
+
+	# Make configuration somewhat generic to test recovery
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+port = $port
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+));
+
+	configure_hba_for_replication($datadir_nodes{ $port });
+}
+
+# Get a port number not in use currently for a new node
+# As port number retrieval is based on the nodes currently running and
+# their presence in the list of registered ports, be sure that the node
+# that is consuming this port number has already been started and that
+# it is not registered yet.
+sub get_free_port
+{
+	my $found = 0;
+	# XXX: Should this part use PG_VERSION_NUM?
+	my $port = 90600 % 16384 + 49152;
+
+	while ($found == 0)
+	{
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		my $ret = system("psql -X -p $port postgres < $devnull");
+		if ($ret != 0)
+		{
+			# Found a potential candidate, check first that it is
+			# not included in the list of registered nodes.
+			$found = 1 if ! grep (/$port/, @array);
+		}
+	}
+
+	print "# Found free port $port\n";
+	# Lock port number found
+	enable_node($port);
+	return $port;
+}
+
+# Low-level routines to initialize a node
+# Initialize a node from scratch
+sub init_node
+{
+	my $port = shift;
+
+	# Save configuration information
+	my $base_dir = TestLib::tempdir;
+	$datadir_nodes{ $port } = "$base_dir/pgdata";
+	print "Base directory $base_dir/pgdata, Port $port\n";
+	$backup_nodes{ $port } = "$base_dir/backup";
+	mkdir $backup_nodes{ $port };
+	$archive_nodes{ $port } =  "$base_dir/archives";
+	mkdir $archive_nodes{ $port };
+	$connstr_nodes{ $port } = "port=$port";
+	$applname_nodes{ $port } = "node_$port";
+
+	standard_initdb($datadir_nodes{ $port });
+	configure_base_node($port);
+}
+
+# Initialize a node from an existing base backup
+sub init_node_from_backup
+{
+	my ($port, $root_port, $backup_name) = @_;
+
+	my $backup_path =  "$backup_nodes{ $root_port }/$backup_name";
+	my $backup_file = "$backup_path/base.tar";
+
+	print "Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	# Check existence of backup wanted
+	if ( ! -d $backup_path )
+	{
+		die "Backup $backup_path does not exist";
+	}
+
+	# Save configuration information
+	my $base_dir = TestLib::tempdir;
+	$datadir_nodes{ $port } = "$base_dir/pgdata";
+	mkdir $datadir_nodes{ $port }, 0700;
+	$backup_nodes{ $port } = "$base_dir/backup";
+	mkdir $backup_nodes{ $port };
+	$archive_nodes{ $port } =  "$base_dir/archives";
+	mkdir $archive_nodes{ $port };
+	$connstr_nodes{ $port } = "port=$port";
+	$applname_nodes{ $port } = "node_$port";
+
+	# Extract the base backup wanted
+	my $current_dir = cwd();
+
+	# Temporary move to the place of extraction
+	chdir "$datadir_nodes{ $port }";
+	Archive::Tar->extract_archive($backup_file);
+	chdir "$current_dir";
+	configure_base_node($port);
+}
+
+# Start a node
+sub start_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port },
+				   '-l', "$log_path/node_$port.log",
+				   'start');
+}
+
+# Stop a node. This routine can be called during a test.
+sub stop_node
+{
+	my $port = shift;
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   'fast', 'stop');
+}
+
+sub restart_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port }, '-m',
+				   'fast', 'restart');
+}
+
+# Mark given node as active, making the port number used by this node
+# locked for this test.
+sub enable_node
+{
+	my $port = shift;
+	print "# Port $port has been locked for new node\n";
+	push(@active_nodes, $port);
+}
+
+# Disactivate given node, node should have been stopped before calling
+# this routine.
+sub disable_node
+{
+	my $port = shift;
+	print "# Port $port has been disabled\n";
+	@active_nodes = grep { $_ ne $port } @active_nodes;
+}
+
+# Remove any traces of given node.
+sub teardown_node
+{
+	my $port = shift;
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   'immediate', 'stop');
+	disable_node($port);
+}
+
+# Create a backup on a node already running
+sub backup_node
+{
+	my ($port, $backup_name) = @_;
+
+	my $backup_path = "$backup_nodes{ $port }/$backup_name";
+
+	print "Taking backup from node $port\n";
+	# Backup a node in tar format, it is more portable across platforms
+	system_or_bail("pg_basebackup -D $backup_path -p $port --format=t -x");
+	print "# Backup finished\n"
+}
+
+# Add a set of parameters to a configuration file
+sub append_to_file
+{
+	my($filename, $str) = @_;
+
+	open my $fh, ">>", $filename or die "could not open file $filename";
+	print $fh $str;
+	close $fh;
+}
+
+END
+{
+	foreach my $port (@active_nodes)
+	{
+		teardown_node($port);
+	}
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..0dc0d6d 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -12,6 +12,7 @@ our @EXPORT = qw(
   configure_hba_for_replication
   start_test_server
   restart_test_server
+  poll_query_until
   psql
   slurp_dir
   slurp_file
@@ -220,6 +221,37 @@ END
 	}
 }
 
+sub poll_query_until
+{
+	my ($query, $connstr) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
+		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
 sub psql
 {
 	my ($dbname, $sql) = @_;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..734793c
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,62 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+my $backup_name = 'my_backup';
+
+# Take backup
+backup_node($port_master, $backup_name);
+
+# Create streaming standby linking to master
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_1);
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+backup_node($port_standby_1, $backup_name);
+
+# Create second standby node linking to standby 1
+my $port_standby_2 = make_hot_standby($port_standby_1, $backup_name);
+start_node($port_standby_2);
+
+# Create some content on master and check its presence in standby 1 an
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a";
+
+# Wait for standbys to catch up
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_nodes{ $port_standby_1 }';";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_master })
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_nodes{ $port_standby_2 }';";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_master })
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT count(*) FROM tab_int";
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_1 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_2 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
+
+# Cleanup nodes
+teardown_node($port_standby_2);
+teardown_node($port_standby_1);
+teardown_node($port_master);
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..2356394
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,52 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $port_master = get_free_port();
+my $backup_name = 'my_backup';
+init_node($port_master);
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Take backup for slave
+backup_node($port_master, 'my_backup');
+
+# Initialize standby node from backup, fetching WAL from archives
+my $port_standby = make_warm_standby($port_master, $backup_name);
+append_to_file("$datadir_nodes{ $port_standby }/postgresql.conf", qq(
+wal_retrieve_retry_interval = '100ms'
+));
+start_node($port_standby);
+
+# Create some content on master
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $current_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Force archiving of WAL file to make it present on master
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Add some more content, it should not be present on standby
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(1000), 'check content from archives');
+
+# Cleanup nodes
+teardown_node($port_standby);
+teardown_node($port_master);
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..bfc894f
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,139 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $port_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $port_standby = get_free_port();
+
+	init_node_from_backup($port_standby, $port_master, 'my_backup');
+	enable_restoring($port_master, $port_standby);
+
+	foreach my $param_item (@$recovery_params)
+	{
+		append_to_file("$datadir_nodes{ $port_standby }/recovery.conf",
+					   qq($param_item
+));
+	}
+
+	start_node($port_standby);
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = psql $connstr_nodes{ $port_standby },
+		"SELECT count(*) FROM tab_int";
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	teardown_node($port_standby);
+}
+
+# Initialize master node
+my $port_master = get_free_port();
+init_node($port_master);
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $lsn1 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Take backup from which all operations will be run
+backup_node($port_master, 'my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+my $recovery_txid = psql $connstr_nodes{ $port_master },
+	"SELECT txid_current()";
+my $lsn2 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# More data, with recovery target timestamp
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(2001,3000))";
+my $recovery_time = psql $connstr_nodes{ $port_master }, "SELECT now()";
+my $lsn3 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Even more data, this time with a recovery target name
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))";
+my $recovery_name = "my_target";
+my $lsn4 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+psql $connstr_nodes{ $port_master },
+	"SELECT pg_create_restore_point('$recovery_name')";
+
+# Force archiving of WAL file
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $port_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+teardown_node($port_master);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..2455ca4
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,68 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+
+# Take backup
+my $backup_name = 'my_backup';
+backup_node($port_master, $backup_name);
+
+# Create two standbys linking to it
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_1);
+my $port_standby_2 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_2);
+
+# Create some content on master
+psql $connstr_nodes { $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+teardown_node($port_master);
+system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port_standby_1 },
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree("$datadir_nodes{ $port_standby_2 }/recovery.conf");
+append_to_file("$datadir_nodes{ $port_standby_2 }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_standby_1 }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+restart_node($port_standby_2);
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done.
+psql $connstr_nodes{ $port_standby_1 },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+$until_lsn = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT pg_current_xlog_location();";
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_2 })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(2000), 'check content of standby 2');
+
+# Stop nodes
+teardown_node($port_standby_2);
+teardown_node($port_standby_1);
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..76ea60a
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,54 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+
+# And some content
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a";
+
+# Take backup
+backup_node($port_master, 'my_backup');
+
+# Initialize node from backup
+my $port_standby = get_free_port();
+init_node_from_backup($port_standby, $port_master, 'my_backup');
+
+# Start second node, streaming from first one
+append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_master }'
+recovery_min_apply_delay = '2s'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+start_node($port_standby);
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(11,20))";
+sleep 1;
+# Here we should have only 10 rows
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+$result = psql $connstr_nodes{ $port_standby }, "SELECT count(*) FROM tab_int";
+is($result, qq(20), 'check content with delay of 2s');
+
+# Stop nodes
+teardown_node($port_standby);
+teardown_node($port_master);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index d3d736b..1355508 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -194,8 +194,9 @@ sub tapcheck
 	foreach my $test_path (@$tap_dirs)
 	{
 		# Like on Unix "make check-world", don't run the SSL test suite
-		# automatically.
+		# or the recovery test suite automatically.
 		next if ($test_path =~ /\/src\/test\/ssl\//);
+		next if ($test_path =~ /\/src\/test\/recovery\//);
 
 		my $dir = dirname($test_path);
 		chdir $dir;
#26Amir Rohan
amir.rohan@mail.com
In reply to: Michael Paquier (#25)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/02/2015 03:33 PM, Michael Paquier wrote:

Any server instances created during the tests should never use a
user-defined port for portability. Hence using those ports as keys
just made sense. We could have for example custom names, that have
port values assigned to them, but that's actually an overkill and
complicates the whole facility.

Something like:

global nPortsAssigned = 0;
AssignPort() -> return is_ok(nPortsAssigned++)

was what I used.

2) Behaviour (paths in particular) is hardwired rather then overridable
defaults.

This is the case of all the TAP tests. We could always use the same
base directory for all the nodes and then embed a sub-directory whose
name is decided using the port number. But I am not really sure if
that's a win.

I understand, but it eliminates the kind of scenarios this convenience
package lets you express... conveniently.

This is exactly what I needed to test, problems:
3) Can't stop server without clearing its testing data (the maps holding
paths and things). But that data might be specifically
needed, in particular the backup shouldn't disappear when the
server melts down or we have a very low-grade DBA on our hands.

OK, you have a point here. You may indeed want routines for to enable
and disable a node completely decoupled from start and stop, with
something like enable_node and disable_node that basically registers
or unregisters it from the list of active nodes. I have updated the
patch this way.

Excellent.

4) Port assignment relies on liveness checks on running servers.
If a server is shut down and a new instantiated, the port will get
reused, data will get trashed, and various confusing things can happen.

Right. The safest way to do that is to check in get_free_port if a
port number is used by a registered node, and continue to loop in if
that's the case. So done.

That eliminates the "sweet and gentle" variant of the scenario, but it's
susceptible to the "ABA problem":
https://en.wikipedia.org/wiki/ABA_problem
https://youtu.be/CmxkPChOcvw?t=786

Granted, you have to try fairly hard to shoot yourself in the leg,
but since the solution is so simple, why not? If we never reuse ports
within a single test, this goes away.

5) Servers are shutdown with -m 'immediate', which can lead to races
in the script when archiving is turned on. That may be good for some
tests, but there's no control over it.

I hesitated with fast here actually. So changed this way. We would
want as wall a teardown command to stop the node with immediate and
unregister the node from the active list.

In particular, I was shutting down an archiving node and the archiving
was truncated. I *think* smart doesn't do this. But again, it's really
that the test writer can't easily override, not that the default is wrong.

Other issues:
6. Directory structure, used one directory per thing but more logical
to place all things related to an instance under a single directory,
and name them according to role (57333_backup, and so on).

Er, well. The first version of the patch did so, and then I switched
to an approach closer to what the existing TAP facility is doing. But
well let's simplify things a bit.

I know, I know, but:
1) an instance is a "thing" in your script, so having its associated
paraphernalia in one place makes more sense (maybe only to me).
2) That's often how folks (well, how I) arrange things in deployment,
at least with archive/backup as symlinks to the nas.

Alternatively, naming the dirs with a prefix (srv_foo_HASH,
backup_foo_backupname_HASH, etc') would work as well.

7. enable_restoring() uses "cp -i" 'archive_command', not a good fit
for an automated test.

This seems like a good default to me, and actually that's portable on
Windows easily. One could always append a custom archive_command in a
test when for example testing conflicting archiving when archive_mode
= always.

Ok, I wasn't sure about this, but specifically activating a switch that
asks for input from the user during a test? hmm.

8. No canned way to output a pprinted overview of the running system
(paths, ports, for manual checking).

Hm. Why not... Are you suggesting something like print_current_conf
that goes through all the registered nodes and outputs that? How would
you use it?

For one thin, I could open a few terminals and `$(print_env_for_server
5437), so psql just worked.

I wish PEG had that as well.

10. If a test passes/fails or dies due to a bug, everything is cleaned.
Great for testing, bad for postmortem.

That's something directly related to TestLib.pm where
File:Temp:tempdir creates a temporary path with CLEANUP = 1. We had
discussions regarding that actually...

11. a canned "server is responding to queries" helper would be convenient.

Do you mean a wrapper on pg_isready? Do you have use cases in mind for it?

Block until recovery is finished, before testing. eliminate races, and
avoid the stupid sleep(3) I used.

It might be a good idea to:
1) Never reuse ports during a test. Liveness checking is used
to avoid collisions, but should not determine order of assignment.

Agreed. As far as I can see the problem here is related to the fact
that the port of non-running server may be fetched by another one.
That's a bug of my patch.

2) Decouple cleanup from server shutdown. Do the cleanup as the end of
test only, and allow the user to keep things around.

Agreed here.

3) Adjust the directory structure to one top directory per server with
(PGDATA, backup, archive) subdirs.

Hm. OK. The first version of the patch actually did so.

Well, why does "consistency with TAP test" trump the advantages I
mentioned? does TAP actually care?

4) Instead of passing ports around as keys, have _explicit functions
which can be called directly by the user (I'd like the backup *HERE*
please), with the current functions refactored to merely invoke them
by interpolating in the values associated with the port they were given.

I don't really see in what this would be a win. We definitely should
have all the data depending on temporary paths during the tests to
facilitate the cleanup wrapper work.

The trouble was that I reused paths between servers. If shutdown/cleanup
are decoupled, this is probably not needed.

4b) server shutdown should perhaps be "smart" by default, or segmented
into calmly_bring_to_a_close(), pull_electric_plug() and
drop_down_the_stairs_into_swimming_pool().

Nope, not agreeing here. "immediate" is rather violent to stop a node,
hence I have switched it to use "fast" and there is now a
teardown_node routine that uses immediate, that's more aimed at
cleanup up existing things fiercely.

Ok, not as the default, but possible to request a specific kind of
shutdown. I needed smart in my case. Plus, in a scenario, you might
expressly be testing behavior for a specific mode, it needs to be
controllable.

I have as well moved RecoveryTest.pm to src/test/perl so as all the
consumers of prove_check can use it by default, and decoupled
start_node from make_master and make_*_standby so as it is possible to
add for example new parameters to their postgresql.conf and
recovery.conf files before starting them.

Thanks a lot for the feedback! Attached is an updated patch with all
the things mentioned above done. Are included as well the typo fixes
you sent upthread.
Regards,

Great! I'll look at it and post again if there's more. If any of the
above extra explanations make it clearer why I suggested some changes
you didn't like...

Thanks,
Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#27Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#26)
1 attachment(s)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Oct 2, 2015 at 11:10 PM, Amir Rohan wrote:

On 10/02/2015 03:33 PM, Michael Paquier wrote:

Any server instances created during the tests should never use a
user-defined port for portability. Hence using those ports as keys
just made sense. We could have for example custom names, that have
port values assigned to them, but that's actually an overkill and
complicates the whole facility.

Something like:

global nPortsAssigned = 0;
AssignPort() -> return is_ok(nPortsAssigned++)

was what I used.

Why do you need that. Creating a node is in the most basic way a
matter of calling make_master, make_*_standby where a port number, or
identifier gets uniquely assigned to a node. The main point of the
approach taken by this patch is to make port assignment transparent
for the caller.

4) Port assignment relies on liveness checks on running servers.
If a server is shut down and a new instantiated, the port will get
reused, data will get trashed, and various confusing things can happen.

Right. The safest way to do that is to check in get_free_port if a
port number is used by a registered node, and continue to loop in if
that's the case. So done.

That eliminates the "sweet and gentle" variant of the scenario, but it's
susceptible to the "ABA problem":
https://en.wikipedia.org/wiki/ABA_problem
https://youtu.be/CmxkPChOcvw?t=786

I learnt a new thing here. That's basically an existing problem even
with the existing perl test infrastructure relying on TestLib.pm when
tests are run in parallel. What we would need here is a global mapping
file storing all the port numbers used by all the nodes currently in
the tests.

Granted, you have to try fairly hard to shoot yourself in the leg,
but since the solution is so simple, why not? If we never reuse ports
within a single test, this goes away.

Well, you can reuse the same port number in a test. Simply teardown
the existing node and then recreate a new one. I think that port
number assignment to a node should be transparent to the caller, in
our case the perl test script holding a scenario.

5) Servers are shutdown with -m 'immediate', which can lead to races
in the script when archiving is turned on. That may be good for some
tests, but there's no control over it.

I hesitated with fast here actually. So changed this way. We would
want as wall a teardown command to stop the node with immediate and
unregister the node from the active list.

In particular, I was shutting down an archiving node and the archiving
was truncated. I *think* smart doesn't do this. But again, it's really
that the test writer can't easily override, not that the default is wrong.

Ah, OK. Then fast is just fine. It shuts down the node correctly.
"smart" would wait for all the current connections to finish but I am
wondering if it currently matters here: I don't see yet a clear use
case yet where it would make sense to have multi-threaded script... If
somebody comes back with a clear idea here perhaps we could revisit
that but it does not seem worth it now.

Other issues:
6. Directory structure, used one directory per thing but more logical
to place all things related to an instance under a single directory,
and name them according to role (57333_backup, and so on).

Er, well. The first version of the patch did so, and then I switched
to an approach closer to what the existing TAP facility is doing. But
well let's simplify things a bit.

I know, I know, but:
1) an instance is a "thing" in your script, so having its associated
paraphernalia in one place makes more sense (maybe only to me).
2) That's often how folks (well, how I) arrange things in deployment,
at least with archive/backup as symlinks to the nas.

Alternatively, naming the dirs with a prefix (srv_foo_HASH,
backup_foo_backupname_HASH, etc') would work as well.

The useful portion about tempdir is that it cleans up itself
automatically should an error happen. It does not seem to me we want
use that.

7. enable_restoring() uses "cp -i" 'archive_command', not a good fit
for an automated test.

This seems like a good default to me, and actually that's portable on
Windows easily. One could always append a custom archive_command in a
test when for example testing conflicting archiving when archive_mode
= always.

Ok, I wasn't sure about this, but specifically activating a switch that
asks for input from the user during a test? hmm.

Er... The -i switch is a bad idea. I removed it. Honestly I don't
recall why it was here to begin with...

8. No canned way to output a pprinted overview of the running system
(paths, ports, for manual checking).

Hm. Why not... Are you suggesting something like print_current_conf
that goes through all the registered nodes and outputs that? How would
you use it?

For one thin, I could open a few terminals and `$(print_env_for_server
5437), so psql just worked.
I wish PEG had that as well.

Hm. Isn't that coupled with the case where a failure happens then but
tempdirs are cleaned up then? I would expect hackers to run those runs
until the end. If a failure happens, it would then be useful to get a
dump of what happens. However, it seems to me that we can get the same
information by logging all the information when creating a node in the
log file. I have added a routine in this sense, which is called each
time a node is initialized. It seems helpful either way.

11. a canned "server is responding to queries" helper would be convenient.

Do you mean a wrapper on pg_isready? Do you have use cases in mind for it?

Block until recovery is finished, before testing. eliminate races, and
avoid the stupid sleep(3) I used.

TODO

4b) server shutdown should perhaps be "smart" by default, or segmented
into calmly_bring_to_a_close(), pull_electric_plug() and
drop_down_the_stairs_into_swimming_pool().

Nope, not agreeing here. "immediate" is rather violent to stop a node,
hence I have switched it to use "fast" and there is now a
teardown_node routine that uses immediate, that's more aimed at
cleanup up existing things fiercely.

Ok, not as the default, but possible to request a specific kind of
shutdown. I needed smart in my case. Plus, in a scenario, you might
expressly be testing behavior for a specific mode, it needs to be
controllable.

If your test script is running with a single thread, "fast" or "smart"
would not really make a difference, no?

I have as well moved RecoveryTest.pm to src/test/perl so as all the
consumers of prove_check can use it by default, and decoupled
start_node from make_master and make_*_standby so as it is possible to
add for example new parameters to their postgresql.conf and
recovery.conf files before starting them.

Thanks a lot for the feedback! Attached is an updated patch with all
the things mentioned above done. Are included as well the typo fixes
you sent upthread.
Regards,

Great! I'll look at it and post again if there's more. If any of the
above extra explanations make it clearer why I suggested some changes
you didn't like...

I am attaching a new version. I found a small bug in test case 001
when checking if standby 2 has caught up. There is also this dump
function that is helpful. The -i switch in cp command has been removed
as well.
--
Michael

Attachments:

20151003_recovery_regressions_v5.patchtext/x-patch; charset=US-ASCII; name=20151003_recovery_regressions_v5.patchDownload
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..ea219d7 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -125,38 +125,6 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
 sub append_to_file
 {
 	my ($filename, $str) = @_;
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..d6e51eb 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -17,7 +17,7 @@ SUBDIRS = regress isolation modules
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
 # because the SSL test suite is not secure to run on a multi-user system.
-ALWAYS_SUBDIRS = examples locale thread ssl
+ALWAYS_SUBDIRS = examples locale thread ssl recovery
 
 # We want to recurse to all subdirs for all standard targets, except that
 # installcheck and install should not recurse into the subdirectory "modules".
diff --git a/src/test/perl/RecoveryTest.pm b/src/test/perl/RecoveryTest.pm
new file mode 100644
index 0000000..0aade6d
--- /dev/null
+++ b/src/test/perl/RecoveryTest.pm
@@ -0,0 +1,375 @@
+package RecoveryTest;
+
+# Set of common routines for recovery regression tests for a PostgreSQL
+# cluster. This includes global variables and methods that can be used
+# by the various set of tests present to set up cluster nodes and
+# configure them according to the test scenario wanted.
+#
+# Cluster nodes can be freely created using initdb or using the existing
+# base backup of another node, with minimum configuration done when the
+# node is created for the first time like having a proper port number.
+# It is then up to the test to decide what to do with the newly-created
+# node.
+#
+# Environment configuration of each node is available through a set
+# of global variables provided by this package, hashed depending on the
+# port number of a node:
+# - connstr_nodes connection string to connect to this node
+# - datadir_nodes to get the data folder of a given node
+# - archive_nodes for the location of the WAL archives of a node
+# - backup_nodes for the location of base backups of a node
+# - applname_nodes, application_name to use for a standby
+#
+# Nodes are identified by their port number, which should be unique
+# for each node of the cluster as it is run locally.
+
+use Cwd;
+use TestLib;
+use Test::More;
+
+use Archive::Tar;
+use IPC::Run qw(run start);
+
+use Exporter 'import';
+
+our @EXPORT = qw(
+	%connstr_nodes
+	%datadir_nodes
+	%backup_nodes
+	%archive_nodes
+	%applname_nodes
+
+	append_to_file
+	backup_node
+	disable_node
+	dump_node_info
+	enable_archiving
+	enable_node
+	enable_restoring
+	enable_streaming
+	get_free_port
+	init_node
+	init_node_from_backup
+	make_master
+	make_warm_standby
+	make_hot_standby
+	restart_node
+	start_node
+	stop_node
+	teardown_node
+);
+
+# Global variables for node data
+%datadir_nodes = {};	# PGDATA folders
+%backup_nodes = {};		# Backup base folder
+%archive_nodes = {};	# Archive base folder
+%connstr_nodes = {};	# Connection strings
+%applname_nodes = {};	# application_name used for standbys
+
+# Database used for each connection attempt via psql
+$ENV{PGDATABASE} = "postgres";
+
+# Tracker of active nodes
+my @active_nodes = ();
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_root } application_name=$applname_nodes{ $port_standby }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+	my $path = $archive_nodes{ $port_root };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"$path\\\\%f\" \"%p\"" :
+		"cp -i $path/%f %p";
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+restore_command='$copy_command'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $port = shift;
+	my $path = $archive_nodes{ $port };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$path\\\\%f\"" :
+		"cp %p $path/%f";
+
+	# Enable archive_mode and archive_command on node
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization
+sub make_master
+{
+	my $port_master = get_free_port();
+	print "# Initializing master node wih port $port_master\n";
+	init_node($port_master);
+	return $port_master;
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_hot_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing streaming mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_streaming($port_master, $port_standby);
+	return $port_standby;
+}
+
+# Node getting WAL only from archives
+sub make_warm_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing archive mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_restoring($port_master, $port_standby);
+	return $port_standby;
+}
+
+sub configure_base_node
+{
+	my $port = shift;
+
+	# Make configuration somewhat generic to test recovery
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+port = $port
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+));
+
+	configure_hba_for_replication($datadir_nodes{ $port });
+}
+
+# Get a port number not in use currently for a new node
+# As port number retrieval is based on the nodes currently running and
+# their presence in the list of registered ports, be sure that the node
+# that is consuming this port number has already been started and that
+# it is not registered yet.
+sub get_free_port
+{
+	my $found = 0;
+	# XXX: Should this part use PG_VERSION_NUM?
+	my $port = 90600 % 16384 + 49152;
+
+	while ($found == 0)
+	{
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		my $ret = system("psql -X -p $port postgres < $devnull");
+		if ($ret != 0)
+		{
+			# Found a potential candidate, check first that it is
+			# not included in the list of registered nodes.
+			$found = 1 if ! grep (/$port/, @array);
+		}
+	}
+
+	print "# Found free port $port\n";
+	# Lock port number found
+	enable_node($port);
+	return $port;
+}
+
+# Low-level routines to initialize a node
+# Initialize a node from scratch
+sub init_node
+{
+	my $port = shift;
+
+	# Save configuration information
+	my $base_dir = TestLib::tempdir;
+	$datadir_nodes{ $port } = "$base_dir/pgdata";
+	print "Base directory $base_dir/pgdata, Port $port\n";
+	$backup_nodes{ $port } = "$base_dir/backup";
+	mkdir $backup_nodes{ $port };
+	$archive_nodes{ $port } =  "$base_dir/archives";
+	mkdir $archive_nodes{ $port };
+	$connstr_nodes{ $port } = "port=$port";
+	$applname_nodes{ $port } = "node_$port";
+
+	# Log some information
+	dump_node_info($port);
+
+	standard_initdb($datadir_nodes{ $port });
+	configure_base_node($port);
+}
+
+# Initialize a node from an existing base backup
+sub init_node_from_backup
+{
+	my ($port, $root_port, $backup_name) = @_;
+
+	my $backup_path =  "$backup_nodes{ $root_port }/$backup_name";
+	my $backup_file = "$backup_path/base.tar";
+
+	print "Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	# Check existence of backup wanted
+	if ( ! -d $backup_path )
+	{
+		die "Backup $backup_path does not exist";
+	}
+
+	# Save configuration information
+	my $base_dir = TestLib::tempdir;
+	$datadir_nodes{ $port } = "$base_dir/pgdata";
+	mkdir $datadir_nodes{ $port }, 0700;
+	$backup_nodes{ $port } = "$base_dir/backup";
+	mkdir $backup_nodes{ $port };
+	$archive_nodes{ $port } =  "$base_dir/archives";
+	mkdir $archive_nodes{ $port };
+	$connstr_nodes{ $port } = "port=$port";
+	$applname_nodes{ $port } = "node_$port";
+
+	# Log some information
+	dump_node_info($port);
+
+	# Extract the base backup wanted
+	my $current_dir = cwd();
+
+	# Temporary move to the place of extraction
+	chdir "$datadir_nodes{ $port }";
+	Archive::Tar->extract_archive($backup_file);
+	chdir "$current_dir";
+	configure_base_node($port);
+}
+
+# Start a node
+sub start_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port },
+				   '-l', "$log_path/node_$port.log",
+				   'start');
+}
+
+# Stop a node. This routine can be called during a test.
+sub stop_node
+{
+	my $port = shift;
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   'fast', 'stop');
+}
+
+sub restart_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port }, '-m',
+				   'fast', 'restart');
+}
+
+# Mark given node as active, making the port number used by this node
+# locked for this test.
+sub enable_node
+{
+	my $port = shift;
+	print "# Port $port has been locked for new node\n";
+	push(@active_nodes, $port);
+}
+
+# Disactivate given node, node should have been stopped before calling
+# this routine.
+sub disable_node
+{
+	my $port = shift;
+	print "# Port $port has been disabled\n";
+	@active_nodes = grep { $_ ne $port } @active_nodes;
+}
+
+# Remove any traces of given node.
+sub teardown_node
+{
+	my $port = shift;
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   'immediate', 'stop');
+	disable_node($port);
+}
+
+# Create a backup on a node already running
+sub backup_node
+{
+	my ($port, $backup_name) = @_;
+
+	my $backup_path = "$backup_nodes{ $port }/$backup_name";
+
+	print "Taking backup from node $port\n";
+	# Backup a node in tar format, it is more portable across platforms
+	system_or_bail("pg_basebackup -D $backup_path -p $port --format=t -x");
+	print "# Backup finished\n"
+}
+
+# Dump information of a node
+sub dump_node_info
+{
+	my $port = shift;
+	print "New node initialized: ${port}\n";
+	print "Data directory: $datadir_nodes{ $port }\n";
+	print "Backup directory: $backup_nodes{ $port }\n";
+	print "Archive directory: $archive_nodes{ $port }\n";
+	print "Connection string: $connstr_nodes{ $port }\n";
+	print "Application name: applname_nodes{ $port }\n";
+}
+
+# Add a set of parameters to a configuration file
+sub append_to_file
+{
+	my($filename, $str) = @_;
+
+	open my $fh, ">>", $filename or die "could not open file $filename";
+	print $fh $str;
+	close $fh;
+}
+
+END
+{
+	foreach my $port (@active_nodes)
+	{
+		teardown_node($port);
+	}
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..0dc0d6d 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -12,6 +12,7 @@ our @EXPORT = qw(
   configure_hba_for_replication
   start_test_server
   restart_test_server
+  poll_query_until
   psql
   slurp_dir
   slurp_file
@@ -220,6 +221,37 @@ END
 	}
 }
 
+sub poll_query_until
+{
+	my ($query, $connstr) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
+		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
 sub psql
 {
 	my ($dbname, $sql) = @_;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..fd4bfbf
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,62 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+my $backup_name = 'my_backup';
+
+# Take backup
+backup_node($port_master, $backup_name);
+
+# Create streaming standby linking to master
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_1);
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+backup_node($port_standby_1, $backup_name);
+
+# Create second standby node linking to standby 1
+my $port_standby_2 = make_hot_standby($port_standby_1, $backup_name);
+start_node($port_standby_2);
+
+# Create some content on master and check its presence in standby 1 an
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a";
+
+# Wait for standbys to catch up
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_nodes{ $port_standby_1 }';";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_master })
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_nodes{ $port_standby_2 }';";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT count(*) FROM tab_int";
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_1 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_2 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
+
+# Cleanup nodes
+teardown_node($port_standby_2);
+teardown_node($port_standby_1);
+teardown_node($port_master);
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..2356394
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,52 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $port_master = get_free_port();
+my $backup_name = 'my_backup';
+init_node($port_master);
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Take backup for slave
+backup_node($port_master, 'my_backup');
+
+# Initialize standby node from backup, fetching WAL from archives
+my $port_standby = make_warm_standby($port_master, $backup_name);
+append_to_file("$datadir_nodes{ $port_standby }/postgresql.conf", qq(
+wal_retrieve_retry_interval = '100ms'
+));
+start_node($port_standby);
+
+# Create some content on master
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $current_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Force archiving of WAL file to make it present on master
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Add some more content, it should not be present on standby
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(1000), 'check content from archives');
+
+# Cleanup nodes
+teardown_node($port_standby);
+teardown_node($port_master);
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..bfc894f
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,139 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $port_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $port_standby = get_free_port();
+
+	init_node_from_backup($port_standby, $port_master, 'my_backup');
+	enable_restoring($port_master, $port_standby);
+
+	foreach my $param_item (@$recovery_params)
+	{
+		append_to_file("$datadir_nodes{ $port_standby }/recovery.conf",
+					   qq($param_item
+));
+	}
+
+	start_node($port_standby);
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = psql $connstr_nodes{ $port_standby },
+		"SELECT count(*) FROM tab_int";
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	teardown_node($port_standby);
+}
+
+# Initialize master node
+my $port_master = get_free_port();
+init_node($port_master);
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $lsn1 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Take backup from which all operations will be run
+backup_node($port_master, 'my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+my $recovery_txid = psql $connstr_nodes{ $port_master },
+	"SELECT txid_current()";
+my $lsn2 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# More data, with recovery target timestamp
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(2001,3000))";
+my $recovery_time = psql $connstr_nodes{ $port_master }, "SELECT now()";
+my $lsn3 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Even more data, this time with a recovery target name
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))";
+my $recovery_name = "my_target";
+my $lsn4 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+psql $connstr_nodes{ $port_master },
+	"SELECT pg_create_restore_point('$recovery_name')";
+
+# Force archiving of WAL file
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $port_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+teardown_node($port_master);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..2455ca4
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,68 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+
+# Take backup
+my $backup_name = 'my_backup';
+backup_node($port_master, $backup_name);
+
+# Create two standbys linking to it
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_1);
+my $port_standby_2 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_2);
+
+# Create some content on master
+psql $connstr_nodes { $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+teardown_node($port_master);
+system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port_standby_1 },
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree("$datadir_nodes{ $port_standby_2 }/recovery.conf");
+append_to_file("$datadir_nodes{ $port_standby_2 }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_standby_1 }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+restart_node($port_standby_2);
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done.
+psql $connstr_nodes{ $port_standby_1 },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+$until_lsn = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT pg_current_xlog_location();";
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_2 })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(2000), 'check content of standby 2');
+
+# Stop nodes
+teardown_node($port_standby_2);
+teardown_node($port_standby_1);
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..76ea60a
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,54 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+
+# And some content
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a";
+
+# Take backup
+backup_node($port_master, 'my_backup');
+
+# Initialize node from backup
+my $port_standby = get_free_port();
+init_node_from_backup($port_standby, $port_master, 'my_backup');
+
+# Start second node, streaming from first one
+append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_master }'
+recovery_min_apply_delay = '2s'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+start_node($port_standby);
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(11,20))";
+sleep 1;
+# Here we should have only 10 rows
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+$result = psql $connstr_nodes{ $port_standby }, "SELECT count(*) FROM tab_int";
+is($result, qq(20), 'check content with delay of 2s');
+
+# Stop nodes
+teardown_node($port_standby);
+teardown_node($port_master);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index d3d736b..1355508 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -194,8 +194,9 @@ sub tapcheck
 	foreach my $test_path (@$tap_dirs)
 	{
 		# Like on Unix "make check-world", don't run the SSL test suite
-		# automatically.
+		# or the recovery test suite automatically.
 		next if ($test_path =~ /\/src\/test\/ssl\//);
+		next if ($test_path =~ /\/src\/test\/recovery\//);
 
 		my $dir = dirname($test_path);
 		chdir $dir;
#28Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#27)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/03/2015 02:38 PM, Michael Paquier wrote:

On Fri, Oct 2, 2015 at 11:10 PM, Amir Rohan wrote:

On 10/02/2015 03:33 PM, Michael Paquier wrote:

Any server instances created during the tests should never use a
user-defined port for portability. Hence using those ports as keys
just made sense. We could have for example custom names, that have
port values assigned to them, but that's actually an overkill and
complicates the whole facility.

Something like:

global nPortsAssigned = 0;
AssignPort() -> return is_ok(nPortsAssigned++)

was what I used.

Why do you need that. Creating a node is in the most basic way a
matter of calling make_master, make_*_standby where a port number, or
identifier gets uniquely assigned to a node. The main point of the
approach taken by this patch is to make port assignment transparent
for the caller.

See next.

Granted, you have to try fairly hard to shoot yourself in the leg,
but since the solution is so simple, why not? If we never reuse ports
within a single test, this goes away.

Well, you can reuse the same port number in a test. Simply teardown
the existing node and then recreate a new one. I think that port
number assignment to a node should be transparent to the caller, in
our case the perl test script holding a scenario.

What part of "Never assign the same port twice during one test"
makes this "not transparent to the user"?

If you're thinking about parallel test, I don't think you
need to worry. Availability checks take care of one part,
and the portnum-as-map-key-is-test-local takes care of the
other.

But, see next.

4) Port assignment relies on liveness checks on running servers.
If a server is shut down and a new instantiated, the port will get
reused, data will get trashed, and various confusing things can happen.

Right. The safest way to do that is to check in get_free_port if a
port number is used by a registered node, and continue to loop in if
that's the case. So done.

That eliminates the "sweet and gentle" variant of the scenario, but it's
susceptible to the "ABA problem":
https://en.wikipedia.org/wiki/ABA_problem
https://youtu.be/CmxkPChOcvw?t=786

I learnt a new thing here. That's basically an existing problem even
with the existing perl test infrastructure relying on TestLib.pm when
tests are run in parallel. What we would need here is a global mapping
file storing all the port numbers used by all the nodes currently in
the tests.

Yeah, a poorman's way to ensure ports aren't reused (I wasn't very
clear at top of post ) is something like:

global nPortsAssigned = 0;

AssignPort():
basePort = BASE_PORT; # the lowest port we use
while(!available(basePort+nPortsAssigned)):
basePort++

nPortsAssigned++

return basePort;

It has its glaring faults, but would probably work ok.
In any case, I'm sure you can do better.

Granted, you have to try fairly hard to shoot yourself in the leg,
but since the solution is so simple, why not? If we never reuse ports
within a single test, this goes away.

Well, you can reuse the same port number in a test. Simply teardown
the existing node and then recreate a new one. I think that port
number assignment to a node should be transparent to the caller, in
our case the perl test script holding a scenario.

I was using you *never* want to reuse port numbers. That is, as long
as the lib ensures we never reuse ports within one test, all kinds
of corner cases are eliminated.

5) Servers are shutdown with -m 'immediate', which can lead to races
in the script when archiving is turned on. That may be good for some
tests, but there's no control over it.

I hesitated with fast here actually. So changed this way. We would
want as wall a teardown command to stop the node with immediate and
unregister the node from the active list.

In particular, I was shutting down an archiving node and the archiving
was truncated. I *think* smart doesn't do this. But again, it's really
that the test writer can't easily override, not that the default is wrong.

Ah, OK. Then fast is just fine. It shuts down the node correctly.
"smart" would wait for all the current connections to finish but I am
wondering if it currently matters here: I don't see yet a clear use
case yet where it would make sense to have multi-threaded script... If
somebody comes back with a clear idea here perhaps we could revisit
that but it does not seem worth it now.

My mistake. Perhaps what would be useful though is a way
to force "messy" shutdown. a kill -9, basically. It's a recovery
test suite, right?.

Other issues:
6. Directory structure, used one directory per thing but more logical
to place all things related to an instance under a single directory,
and name them according to role (57333_backup, and so on).

Er, well. The first version of the patch did so, and then I switched
to an approach closer to what the existing TAP facility is doing. But
well let's simplify things a bit.

I know, I know, but:
1) an instance is a "thing" in your script, so having its associated
paraphernalia in one place makes more sense (maybe only to me).
2) That's often how folks (well, how I) arrange things in deployment,
at least with archive/backup as symlinks to the nas.

Alternatively, naming the dirs with a prefix (srv_foo_HASH,
backup_foo_backupname_HASH, etc') would work as well.

The useful portion about tempdir is that it cleans up itself
automatically should an error happen. It does not seem to me we want
use that.

Ensuring cleanup and directory structure aren't inherently related.
Testlib makes cleanup easy if you're willing to accept its flat
structure. But writing something that does cleanup and lets yo
control directory structure is perfectly doable.

The question is only if you agree or not that having per-server
directories could be convenient. Tying into the next, if you
don't think anyone ever need to look into these directories
(which I disagree with), then dir structure indeed doesn't matter.

8. No canned way to output a pprinted overview of the running system
(paths, ports, for manual checking).

Hm. Why not... Are you suggesting something like print_current_conf
that goes through all the registered nodes and outputs that? How would
you use it?

For one thin, I could open a few terminals and `$(print_env_for_server
5437), so psql just worked.
I wish PEG had that as well.

Hm. Isn't that coupled with the case where a failure happens then but
tempdirs are cleaned up then?

But I've mentioned that's inconvenient as well. If you don't think
/that/ should be fixed, then yes, there's no point adding this.
Still, perhaps doing both (+previous) would make writing tests
easier.

At least for me, writing tests isn't simply a typing exercise.
I always need to inspect the system at stages during the test.
It's only after the test is ready that cleanup is useful, before
that it actually hampers work.

I would expect hackers to run those runs
until the end.

I agree -- when you're running them , but what about when you're
/writing/ them?

If a failure happens, it would then be useful to get a

dump of what happens. However, it seems to me that we can get the same
information by logging all the information when creating a node in the
log file. I have added a routine in this sense, which is called each
time a node is initialized. It seems helpful either way.

11. a canned "server is responding to queries" helper would be convenient.

Do you mean a wrapper on pg_isready? Do you have use cases in mind for it?

Block until recovery is finished, before testing. eliminate races, and
avoid the stupid sleep(3) I used.

TODO

4b) server shutdown should perhaps be "smart" by default, or segmented
into calmly_bring_to_a_close(), pull_electric_plug() and
drop_down_the_stairs_into_swimming_pool().

Nope, not agreeing here. "immediate" is rather violent to stop a node,
hence I have switched it to use "fast" and there is now a
teardown_node routine that uses immediate, that's more aimed at
cleanup up existing things fiercely.

Ok, not as the default, but possible to request a specific kind of
shutdown. I needed smart in my case. Plus, in a scenario, you might
expressly be testing behavior for a specific mode, it needs to be
controllable.

If your test script is running with a single thread, "fast" or "smart"
would not really make a difference, no?

It would If there's a bug in one of them and I'm trying to write
a regression test for it. Recall, this was part of broader view
of "provide defaults, allow override" I was suggesting.

I have as well moved RecoveryTest.pm to src/test/perl so as all the
consumers of prove_check can use it by default, and decoupled
start_node from make_master and make_*_standby so as it is possible to
add for example new parameters to their postgresql.conf and
recovery.conf files before starting them.

Thanks a lot for the feedback! Attached is an updated patch with all
the things mentioned above done. Are included as well the typo fixes
you sent upthread.
Regards,

Great! I'll look at it and post again if there's more. If any of the
above extra explanations make it clearer why I suggested some changes
you didn't like...

I am attaching a new version. I found a small bug in test case 001
when checking if standby 2 has caught up. There is also this dump
function that is helpful. The -i switch in cp command has been removed
as well.

I'm sorry I didn't review the code, but honestly my perl is so rusty I'm
afraid I'll embarrass myself :)

Thanks!
Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#29Amir Rohan
amir.rohan@zoho.com
In reply to: Amir Rohan (#28)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/03/2015 03:50 PM, Amir Rohan wrote:

On 10/03/2015 02:38 PM, Michael Paquier wrote:

On Fri, Oct 2, 2015 at 11:10 PM, Amir Rohan wrote:

On 10/02/2015 03:33 PM, Michael Paquier wrote:

Granted, you have to try fairly hard to shoot yourself in the leg,
but since the solution is so simple, why not? If we never reuse ports
within a single test, this goes away.

Well, you can reuse the same port number in a test. Simply teardown
the existing node and then recreate a new one. I think that port
number assignment to a node should be transparent to the caller, in
our case the perl test script holding a scenario.

What part of "Never assign the same port twice during one test"
makes this "not transparent to the user"?

If you're thinking about parallel tests, I don't think you
need to worry. Availability checks take care of one part,

Except now that I think of it, that's definitely a race:

Thread1: is_available(5432) -> True
Thread2: is_available(5432) -> True
Thread1: listen(5432) -> True
Thread2: listen(5432) -> #$@#$&@#$^&$#@&

I don't know if parallel tests are actually supported, though.
If theye are, you're right that this is a shared global
resource wrt concurrency.

Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#30Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#28)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Oct 3, 2015 at 9:50 PM, Amir Rohan <amir.rohan@zoho.com> wrote:

On 10/03/2015 02:38 PM, Michael Paquier wrote:

On Fri, Oct 2, 2015 at 11:10 PM, Amir Rohan wrote:

On 10/02/2015 03:33 PM, Michael Paquier wrote:

4) Port assignment relies on liveness checks on running servers.
If a server is shut down and a new instantiated, the port will get
reused, data will get trashed, and various confusing things can happen.

Right. The safest way to do that is to check in get_free_port if a
port number is used by a registered node, and continue to loop in if
that's the case. So done.

That eliminates the "sweet and gentle" variant of the scenario, but it's
susceptible to the "ABA problem":
https://en.wikipedia.org/wiki/ABA_problem
https://youtu.be/CmxkPChOcvw?t=786

I learnt a new thing here. That's basically an existing problem even
with the existing perl test infrastructure relying on TestLib.pm when
tests are run in parallel. What we would need here is a global mapping
file storing all the port numbers used by all the nodes currently in
the tests.

Yeah, a poorman's way to ensure ports aren't reused (I wasn't very
clear at top of post ) is something like:

global nPortsAssigned = 0;

AssignPort():
basePort = BASE_PORT; # the lowest port we use
while(!available(basePort+nPortsAssigned)):
basePort++

nPortsAssigned++

return basePort;

It has its glaring faults, but would probably work ok.
In any case, I'm sure you can do better.

Yeah, this would improve the exiting port lookup. I don't mind adding
a global variable in get_free_port for this purpose. This would
accelerate finding a free port in may cases for sure.

Granted, you have to try fairly hard to shoot yourself in the leg,
but since the solution is so simple, why not? If we never reuse ports
within a single test, this goes away.

Well, you can reuse the same port number in a test. Simply teardown
the existing node and then recreate a new one. I think that port
number assignment to a node should be transparent to the caller, in
our case the perl test script holding a scenario.

I was using you *never* want to reuse port numbers. That is, as long
as the lib ensures we never reuse ports within one test, all kinds
of corner cases are eliminated.

Hm, sure. Though I don't really why that would be mandatory to enforce
this condition as long as the list of ports occupied is in a single
place (as long as tests are not run in parallel...).

5) Servers are shutdown with -m 'immediate', which can lead to races
in the script when archiving is turned on. That may be good for some
tests, but there's no control over it.

I hesitated with fast here actually. So changed this way. We would
want as wall a teardown command to stop the node with immediate and
unregister the node from the active list.

In particular, I was shutting down an archiving node and the archiving
was truncated. I *think* smart doesn't do this. But again, it's really
that the test writer can't easily override, not that the default is wrong.

Ah, OK. Then fast is just fine. It shuts down the node correctly.
"smart" would wait for all the current connections to finish but I am
wondering if it currently matters here: I don't see yet a clear use
case yet where it would make sense to have multi-threaded script... If
somebody comes back with a clear idea here perhaps we could revisit
that but it does not seem worth it now.

My mistake. Perhaps what would be useful though is a way
to force "messy" shutdown. a kill -9, basically. It's a recovery
test suite, right?.

That's what the teardown is aimed at having, the immediate stop mode
would play that fairly good enough. There has been a patch from Tom
Lane around to stop a server should its postmaster.pid be missing as
well...

Other issues:
6. Directory structure, used one directory per thing but more logical
to place all things related to an instance under a single directory,
and name them according to role (57333_backup, and so on).

Er, well. The first version of the patch did so, and then I switched
to an approach closer to what the existing TAP facility is doing. But
well let's simplify things a bit.

I know, I know, but:
1) an instance is a "thing" in your script, so having its associated
paraphernalia in one place makes more sense (maybe only to me).
2) That's often how folks (well, how I) arrange things in deployment,
at least with archive/backup as symlinks to the nas.

Alternatively, naming the dirs with a prefix (srv_foo_HASH,
backup_foo_backupname_HASH, etc') would work as well.

The useful portion about tempdir is that it cleans up itself
automatically should an error happen. It does not seem to me we want
use that.

Ensuring cleanup and directory structure aren't inherently related.
Testlib makes cleanup easy if you're willing to accept its flat
structure. But writing something that does cleanup and lets yo
control directory structure is perfectly doable.

The question is only if you agree or not that having per-server
directories could be convenient. Tying into the next, if you
don't think anyone ever need to look into these directories
(which I disagree with), then dir structure indeed doesn't matter.

So your point is having one temp dir for the whole, right? I don't
disagree with that.

I would expect hackers to run those runs
until the end.

I agree -- when you're running them , but what about when you're
/writing/ them?

Well, I enforce CLEANUP=0 manually in TestLib.pm for now.

11. a canned "server is responding to queries" helper would be convenient.

Do you mean a wrapper on pg_isready? Do you have use cases in mind for it?

Block until recovery is finished, before testing. eliminate races, and
avoid the stupid sleep(3) I used.

TODO

Well. I just recalled this item in the list of things you mentioned. I
marked it but forgot to address it. It sounds right that we may want
something using pg_isready in a loop as a node in recovery would
reject connections.

4b) server shutdown should perhaps be "smart" by default, or segmented
into calmly_bring_to_a_close(), pull_electric_plug() and
drop_down_the_stairs_into_swimming_pool().

Nope, not agreeing here. "immediate" is rather violent to stop a node,
hence I have switched it to use "fast" and there is now a
teardown_node routine that uses immediate, that's more aimed at
cleanup up existing things fiercely.

Ok, not as the default, but possible to request a specific kind of
shutdown. I needed smart in my case. Plus, in a scenario, you might
expressly be testing behavior for a specific mode, it needs to be
controllable.

If your test script is running with a single thread, "fast" or "smart"
would not really make a difference, no?

It would If there's a bug in one of them and I'm trying to write
a regression test for it. Recall, this was part of broader view
of "provide defaults, allow override" I was suggesting.

We could then extend stop_node with an optional argument containing a
mode, with fast being the default. Sounds right?

I have as well moved RecoveryTest.pm to src/test/perl so as all the
consumers of prove_check can use it by default, and decoupled
start_node from make_master and make_*_standby so as it is possible to
add for example new parameters to their postgresql.conf and
recovery.conf files before starting them.

Thanks a lot for the feedback! Attached is an updated patch with all
the things mentioned above done. Are included as well the typo fixes
you sent upthread.
Regards,

Great! I'll look at it and post again if there's more. If any of the
above extra explanations make it clearer why I suggested some changes
you didn't like...

I am attaching a new version. I found a small bug in test case 001
when checking if standby 2 has caught up. There is also this dump
function that is helpful. The -i switch in cp command has been removed
as well.

I'm sorry I didn't review the code, but honestly my perl is so rusty I'm
afraid I'll embarrass myself :)

I don't pretend mine are good :) So we are two.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#31Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#30)
1 attachment(s)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Oct 3, 2015 at 10:47 PM, Michael Paquier wrote:

On Sat, Oct 3, 2015 at 9:50 PM, Amir Rohan wrote:

Block until recovery is finished, before testing. eliminate races, and
avoid the stupid sleep(3) I used.

TODO

Well. I just recalled this item in the list of things you mentioned. I
marked it but forgot to address it. It sounds right that we may want
something using pg_isready in a loop as a node in recovery would
reject connections.

I just hacked up an updated version with the following things:
- Optional argument for stop_node to define the stop mode of pg_ctl
- Addition of wait_for_node where pg_isready is used to wait until a
node is ready to accept queries
- Addition of a local lookup variable to track the last port assigned.
This accelerates get_free_port.
Regards,
--
Michael

Attachments:

20151004_recovery_regressions_v6.patchtext/x-patch; charset=US-ASCII; name=20151004_recovery_regressions_v6.patchDownload
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..ea219d7 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -125,38 +125,6 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
 sub append_to_file
 {
 	my ($filename, $str) = @_;
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..d6e51eb 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -17,7 +17,7 @@ SUBDIRS = regress isolation modules
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
 # because the SSL test suite is not secure to run on a multi-user system.
-ALWAYS_SUBDIRS = examples locale thread ssl
+ALWAYS_SUBDIRS = examples locale thread ssl recovery
 
 # We want to recurse to all subdirs for all standard targets, except that
 # installcheck and install should not recurse into the subdirectory "modules".
diff --git a/src/test/perl/RecoveryTest.pm b/src/test/perl/RecoveryTest.pm
new file mode 100644
index 0000000..aa8998c
--- /dev/null
+++ b/src/test/perl/RecoveryTest.pm
@@ -0,0 +1,401 @@
+package RecoveryTest;
+
+# Set of common routines for recovery regression tests for a PostgreSQL
+# cluster. This includes global variables and methods that can be used
+# by the various set of tests present to set up cluster nodes and
+# configure them according to the test scenario wanted.
+#
+# Cluster nodes can be freely created using initdb or using the existing
+# base backup of another node, with minimum configuration done when the
+# node is created for the first time like having a proper port number.
+# It is then up to the test to decide what to do with the newly-created
+# node.
+#
+# Environment configuration of each node is available through a set
+# of global variables provided by this package, hashed depending on the
+# port number of a node:
+# - connstr_nodes connection string to connect to this node
+# - datadir_nodes to get the data folder of a given node
+# - archive_nodes for the location of the WAL archives of a node
+# - backup_nodes for the location of base backups of a node
+# - applname_nodes, application_name to use for a standby
+#
+# Nodes are identified by their port number, which should be unique
+# for each node of the cluster as it is run locally.
+
+use Cwd;
+use TestLib;
+use Test::More;
+
+use Archive::Tar;
+use IPC::Run qw(run start);
+
+use Exporter 'import';
+
+our @EXPORT = qw(
+	%connstr_nodes
+	%datadir_nodes
+	%backup_nodes
+	%archive_nodes
+	%applname_nodes
+
+	append_to_file
+	backup_node
+	disable_node
+	dump_node_info
+	enable_archiving
+	enable_node
+	enable_restoring
+	enable_streaming
+	get_free_port
+	init_node
+	init_node_from_backup
+	make_master
+	make_warm_standby
+	make_hot_standby
+	restart_node
+	start_node
+	stop_node
+	teardown_node
+);
+
+# Global variables for node data
+%datadir_nodes = {};	# PGDATA folders
+%backup_nodes = {};		# Backup base folder
+%archive_nodes = {};	# Archive base folder
+%connstr_nodes = {};	# Connection strings
+%applname_nodes = {};	# application_name used for standbys
+
+# Tracking of last port value assigned to accelerate free port lookup.
+# XXX: Should this part use PG_VERSION_NUM?
+my $last_port_assigned =  90600 % 16384 + 49152;
+
+# Database used for each connection attempt via psql
+$ENV{PGDATABASE} = "postgres";
+
+# Tracker of active nodes
+my @active_nodes = ();
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_root } application_name=$applname_nodes{ $port_standby }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+	my $path = $archive_nodes{ $port_root };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"$path\\\\%f\" \"%p\"" :
+		"cp -i $path/%f %p";
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+restore_command='$copy_command'
+standby_mode=on
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $port = shift;
+	my $path = $archive_nodes{ $port };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$path\\\\%f\"" :
+		"cp %p $path/%f";
+
+	# Enable archive_mode and archive_command on node
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization
+sub make_master
+{
+	my $port_master = get_free_port();
+	print "# Initializing master node wih port $port_master\n";
+	init_node($port_master);
+	return $port_master;
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_hot_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing streaming mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_streaming($port_master, $port_standby);
+	return $port_standby;
+}
+
+# Node getting WAL only from archives
+sub make_warm_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing archive mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_restoring($port_master, $port_standby);
+	return $port_standby;
+}
+
+sub configure_base_node
+{
+	my $port = shift;
+
+	# Make configuration somewhat generic to test recovery
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+port = $port
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+));
+
+	configure_hba_for_replication($datadir_nodes{ $port });
+}
+
+# Get a port number not in use currently for a new node
+# As port number retrieval is based on the nodes currently running and
+# their presence in the list of registered ports, be sure that the node
+# that is consuming this port number has already been started and that
+# it is not registered yet.
+sub get_free_port
+{
+	my $found = 0;
+	my $port = $last_port_assigned;
+
+	while ($found == 0)
+	{
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		my $ret = system("psql -X -p $port postgres < $devnull");
+		if ($ret != 0)
+		{
+			# Found a potential candidate, check first that it is
+			# not included in the list of registered nodes.
+			$found = 1 if ! grep (/$port/, @array);
+		}
+	}
+
+	print "# Found free port $port\n";
+	# Lock port number found
+	enable_node($port);
+	$last_port_assigned = $port;
+	return $port;
+}
+
+# Low-level routines to initialize a node
+# Initialize a node from scratch
+sub init_node
+{
+	my $port = shift;
+
+	# Save configuration information
+	my $base_dir = TestLib::tempdir;
+	$datadir_nodes{ $port } = "$base_dir/pgdata";
+	print "Base directory $base_dir/pgdata, Port $port\n";
+	$backup_nodes{ $port } = "$base_dir/backup";
+	mkdir $backup_nodes{ $port };
+	$archive_nodes{ $port } =  "$base_dir/archives";
+	mkdir $archive_nodes{ $port };
+	$connstr_nodes{ $port } = "port=$port";
+	$applname_nodes{ $port } = "node_$port";
+
+	# Log some information
+	dump_node_info($port);
+
+	standard_initdb($datadir_nodes{ $port });
+	configure_base_node($port);
+}
+
+# Initialize a node from an existing base backup
+sub init_node_from_backup
+{
+	my ($port, $root_port, $backup_name) = @_;
+
+	my $backup_path =  "$backup_nodes{ $root_port }/$backup_name";
+	my $backup_file = "$backup_path/base.tar";
+
+	print "Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	# Check existence of backup wanted
+	if ( ! -d $backup_path )
+	{
+		die "Backup $backup_path does not exist";
+	}
+
+	# Save configuration information
+	my $base_dir = TestLib::tempdir;
+	$datadir_nodes{ $port } = "$base_dir/pgdata";
+	mkdir $datadir_nodes{ $port }, 0700;
+	$backup_nodes{ $port } = "$base_dir/backup";
+	mkdir $backup_nodes{ $port };
+	$archive_nodes{ $port } =  "$base_dir/archives";
+	mkdir $archive_nodes{ $port };
+	$connstr_nodes{ $port } = "port=$port";
+	$applname_nodes{ $port } = "node_$port";
+
+	# Log some information
+	dump_node_info($port);
+
+	# Extract the base backup wanted
+	my $current_dir = cwd();
+
+	# Temporary move to the place of extraction
+	chdir "$datadir_nodes{ $port }";
+	Archive::Tar->extract_archive($backup_file);
+	chdir "$current_dir";
+	configure_base_node($port);
+}
+
+# Start a node
+sub start_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port },
+				   '-l', "$log_path/node_$port.log",
+				   'start');
+}
+
+# Stop a node. This routine can be called during a test.
+sub stop_node
+{
+	my $port = shift;
+	my $mode = shift || "fast";
+
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   $mode, 'stop');
+}
+
+sub restart_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port }, '-m',
+				   'fast', 'restart');
+}
+
+# Wait until a node is able to accept queries. Useful have putting a node
+# in recovery.
+sub wait_for_node
+{
+	my $port         = shift;
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	while ($attempts < $max_attempts)
+	{
+		if (run_log('pg_isready', '-p', $port))
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+	return 0;
+}
+
+# Mark given node as active, making the port number used by this node
+# locked for this test.
+sub enable_node
+{
+	my $port = shift;
+	print "# Port $port has been locked for new node\n";
+	push(@active_nodes, $port);
+}
+
+# Disactivate given node, node should have been stopped before calling
+# this routine.
+sub disable_node
+{
+	my $port = shift;
+	print "# Port $port has been disabled\n";
+	@active_nodes = grep { $_ ne $port } @active_nodes;
+}
+
+# Remove any traces of given node.
+sub teardown_node
+{
+	my $port = shift;
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   'immediate', 'stop');
+	disable_node($port);
+}
+
+# Create a backup on a node already running
+sub backup_node
+{
+	my ($port, $backup_name) = @_;
+
+	my $backup_path = "$backup_nodes{ $port }/$backup_name";
+
+	print "Taking backup from node $port\n";
+	# Backup a node in tar format, it is more portable across platforms
+	system_or_bail("pg_basebackup -D $backup_path -p $port --format=t -x");
+	print "# Backup finished\n"
+}
+
+# Dump information of a node
+sub dump_node_info
+{
+	my $port = shift;
+	print "New node initialized: ${port}\n";
+	print "Data directory: $datadir_nodes{ $port }\n";
+	print "Backup directory: $backup_nodes{ $port }\n";
+	print "Archive directory: $archive_nodes{ $port }\n";
+	print "Connection string: $connstr_nodes{ $port }\n";
+	print "Application name: applname_nodes{ $port }\n";
+}
+
+# Add a set of parameters to a configuration file
+sub append_to_file
+{
+	my($filename, $str) = @_;
+
+	open my $fh, ">>", $filename or die "could not open file $filename";
+	print $fh $str;
+	close $fh;
+}
+
+END
+{
+	foreach my $port (@active_nodes)
+	{
+		teardown_node($port);
+	}
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..0dc0d6d 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -12,6 +12,7 @@ our @EXPORT = qw(
   configure_hba_for_replication
   start_test_server
   restart_test_server
+  poll_query_until
   psql
   slurp_dir
   slurp_file
@@ -220,6 +221,37 @@ END
 	}
 }
 
+sub poll_query_until
+{
+	my ($query, $connstr) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
+		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
 sub psql
 {
 	my ($dbname, $sql) = @_;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..fd4bfbf
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,62 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+my $backup_name = 'my_backup';
+
+# Take backup
+backup_node($port_master, $backup_name);
+
+# Create streaming standby linking to master
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_1);
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+backup_node($port_standby_1, $backup_name);
+
+# Create second standby node linking to standby 1
+my $port_standby_2 = make_hot_standby($port_standby_1, $backup_name);
+start_node($port_standby_2);
+
+# Create some content on master and check its presence in standby 1 an
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a";
+
+# Wait for standbys to catch up
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_nodes{ $port_standby_1 }';";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_master })
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_nodes{ $port_standby_2 }';";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT count(*) FROM tab_int";
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_1 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_2 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
+
+# Cleanup nodes
+teardown_node($port_standby_2);
+teardown_node($port_standby_1);
+teardown_node($port_master);
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..2356394
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,52 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $port_master = get_free_port();
+my $backup_name = 'my_backup';
+init_node($port_master);
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Take backup for slave
+backup_node($port_master, 'my_backup');
+
+# Initialize standby node from backup, fetching WAL from archives
+my $port_standby = make_warm_standby($port_master, $backup_name);
+append_to_file("$datadir_nodes{ $port_standby }/postgresql.conf", qq(
+wal_retrieve_retry_interval = '100ms'
+));
+start_node($port_standby);
+
+# Create some content on master
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $current_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Force archiving of WAL file to make it present on master
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Add some more content, it should not be present on standby
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(1000), 'check content from archives');
+
+# Cleanup nodes
+teardown_node($port_standby);
+teardown_node($port_master);
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..bfc894f
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,139 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $port_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $port_standby = get_free_port();
+
+	init_node_from_backup($port_standby, $port_master, 'my_backup');
+	enable_restoring($port_master, $port_standby);
+
+	foreach my $param_item (@$recovery_params)
+	{
+		append_to_file("$datadir_nodes{ $port_standby }/recovery.conf",
+					   qq($param_item
+));
+	}
+
+	start_node($port_standby);
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = psql $connstr_nodes{ $port_standby },
+		"SELECT count(*) FROM tab_int";
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	teardown_node($port_standby);
+}
+
+# Initialize master node
+my $port_master = get_free_port();
+init_node($port_master);
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $lsn1 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Take backup from which all operations will be run
+backup_node($port_master, 'my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+my $recovery_txid = psql $connstr_nodes{ $port_master },
+	"SELECT txid_current()";
+my $lsn2 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# More data, with recovery target timestamp
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(2001,3000))";
+my $recovery_time = psql $connstr_nodes{ $port_master }, "SELECT now()";
+my $lsn3 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Even more data, this time with a recovery target name
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))";
+my $recovery_name = "my_target";
+my $lsn4 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+psql $connstr_nodes{ $port_master },
+	"SELECT pg_create_restore_point('$recovery_name')";
+
+# Force archiving of WAL file
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $port_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+teardown_node($port_master);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..2455ca4
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,68 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+
+# Take backup
+my $backup_name = 'my_backup';
+backup_node($port_master, $backup_name);
+
+# Create two standbys linking to it
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_1);
+my $port_standby_2 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_2);
+
+# Create some content on master
+psql $connstr_nodes { $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+teardown_node($port_master);
+system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port_standby_1 },
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree("$datadir_nodes{ $port_standby_2 }/recovery.conf");
+append_to_file("$datadir_nodes{ $port_standby_2 }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_standby_1 }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+restart_node($port_standby_2);
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done.
+psql $connstr_nodes{ $port_standby_1 },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+$until_lsn = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT pg_current_xlog_location();";
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_2 })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(2000), 'check content of standby 2');
+
+# Stop nodes
+teardown_node($port_standby_2);
+teardown_node($port_standby_1);
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..76ea60a
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,54 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+
+# And some content
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a";
+
+# Take backup
+backup_node($port_master, 'my_backup');
+
+# Initialize node from backup
+my $port_standby = get_free_port();
+init_node_from_backup($port_standby, $port_master, 'my_backup');
+
+# Start second node, streaming from first one
+append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_master }'
+recovery_min_apply_delay = '2s'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+start_node($port_standby);
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(11,20))";
+sleep 1;
+# Here we should have only 10 rows
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+$result = psql $connstr_nodes{ $port_standby }, "SELECT count(*) FROM tab_int";
+is($result, qq(20), 'check content with delay of 2s');
+
+# Stop nodes
+teardown_node($port_standby);
+teardown_node($port_master);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index d3d736b..1355508 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -194,8 +194,9 @@ sub tapcheck
 	foreach my $test_path (@$tap_dirs)
 	{
 		# Like on Unix "make check-world", don't run the SSL test suite
-		# automatically.
+		# or the recovery test suite automatically.
 		next if ($test_path =~ /\/src\/test\/ssl\//);
+		next if ($test_path =~ /\/src\/test\/recovery\//);
 
 		my $dir = dirname($test_path);
 		chdir $dir;
#32Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#25)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/02/2015 03:33 PM, Michael Paquier wrote:

Michael, I'm afraid my email bungling has damaged your thread.

I didn't include an "In-reply-To" header when I posted:

trinity-b4a8035d-59af-4c42-a37e-258f0f28e44a-1443795007012@3capp-mailcom-lxa08.

And we subsequently had our discussion over there instead of here, where
the commitfest app is tracking it.

https://commitfest.postgresql.org/6/197/

Perhaps it would help a little if you posted the latest patch here as
well? So that at least the app picks it up again.

Apologies for my ML n00bness,
Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#33Andres Freund
andres@2ndquadrant.com
In reply to: Amir Rohan (#32)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On October 4, 2015 3:27:00 PM GMT+02:00, Amir Rohan <amir.rohan@zoho.com> wrote:

Perhaps it would help a little if you posted the latest patch here as
well? So that at least the app picks it up again.

You can as additional threads in the cf app.

--
Please excuse brevity and formatting - I am writing this on my mobile phone.

Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#34Amir Rohan
amir.rohan@zoho.com
In reply to: Andres Freund (#33)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/04/2015 04:29 PM, Andres Freund wrote:

On October 4, 2015 3:27:00 PM GMT+02:00, Amir Rohan <amir.rohan@zoho.com> wrote:

Perhaps it would help a little if you posted the latest patch here as
well? So that at least the app picks it up again.

You can as additional threads in the cf app.

Done, thank you.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#35Robert Haas
robertmhaas@gmail.com
In reply to: Michael Paquier (#27)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Oct 3, 2015 at 7:38 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

Granted, you have to try fairly hard to shoot yourself in the leg,
but since the solution is so simple, why not? If we never reuse ports
within a single test, this goes away.

Well, you can reuse the same port number in a test. Simply teardown
the existing node and then recreate a new one. I think that port
number assignment to a node should be transparent to the caller, in
our case the perl test script holding a scenario.

It seems that these days 'make check' creates a directory in /tmp
called /tmp/pg_regress-RANDOMSTUFF. Listening on TCP ports is
disabled, and the socket goes inside this directory with a name like
.s.PGSQL.PORT. You can connect with psql -h
/tmp/pg_regress-RANDOMSTUFF -p PORT, but not over TCP. This basically
removes the risk of TCP port number collisions, as well as the risk of
your temporary instance being hijacked by a malicious user on the same
machine. I'm not sure what we do on Windows, though.

In particular, I was shutting down an archiving node and the archiving
was truncated. I *think* smart doesn't do this. But again, it's really
that the test writer can't easily override, not that the default is wrong.

Ah, OK. Then fast is just fine. It shuts down the node correctly.
"smart" would wait for all the current connections to finish but I am
wondering if it currently matters here: I don't see yet a clear use
case yet where it would make sense to have multi-threaded script... If
somebody comes back with a clear idea here perhaps we could revisit
that but it does not seem worth it now.

I don't have anything brilliant to say about this point, but here's a
perhaps-not-brilliant comment:

If there's a bug in one of smart and fast shutdown and the other works
great, it would be nice to catch that.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#36Michael Paquier
michael.paquier@gmail.com
In reply to: Robert Haas (#35)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Oct 7, 2015 at 5:58 AM, Robert Haas wrote:

On Sat, Oct 3, 2015 at 7:38 AM, Michael Paquier wrote:
It seems that these days 'make check' creates a directory in /tmp
called /tmp/pg_regress-RANDOMSTUFF. Listening on TCP ports is
disabled, and the socket goes inside this directory with a name like
.s.PGSQL.PORT. You can connect with psql -h
/tmp/pg_regress-RANDOMSTUFF -p PORT, but not over TCP. This basically
removes the risk of TCP port number collisions, as well as the risk of
your temporary instance being hijacked by a malicious user on the same
machine.

Right, that's for example /var/folders/ on OSX, and this is defined
once per test run via $tempdir_short. PGHOST is set to that as well.

I'm not sure what we do on Windows, though.

sspi with include_realm through 127.0.0.1.

In particular, I was shutting down an archiving node and the archiving
was truncated. I *think* smart doesn't do this. But again, it's really
that the test writer can't easily override, not that the default is wrong.

Ah, OK. Then fast is just fine. It shuts down the node correctly.
"smart" would wait for all the current connections to finish but I am
wondering if it currently matters here: I don't see yet a clear use
case yet where it would make sense to have multi-threaded script... If
somebody comes back with a clear idea here perhaps we could revisit
that but it does not seem worth it now.

I don't have anything brilliant to say about this point, but here's a
perhaps-not-brilliant comment:

If there's a bug in one of smart and fast shutdown and the other works
great, it would be nice to catch that.

Yes, sure. I extended the patch to support other stop modes than fast,
the default being kept to fast if none is defined.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#37Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#36)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Oct 7, 2015 at 7:43 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Oct 7, 2015 at 5:58 AM, Robert Haas wrote:

On Sat, Oct 3, 2015 at 7:38 AM, Michael Paquier wrote:
It seems that these days 'make check' creates a directory in /tmp
called /tmp/pg_regress-RANDOMSTUFF. Listening on TCP ports is
disabled, and the socket goes inside this directory with a name like
.s.PGSQL.PORT. You can connect with psql -h
/tmp/pg_regress-RANDOMSTUFF -p PORT, but not over TCP. This basically
removes the risk of TCP port number collisions, as well as the risk of
your temporary instance being hijacked by a malicious user on the same
machine.

Right, that's for example /var/folders/ on OSX, and this is defined
once per test run via $tempdir_short. PGHOST is set to that as well.

Er, mistake here. That's actually once per standard_initdb, except
that all the tests I have included in my patch run it just once to
create a master node. It seems that it would be wiser to set one
socket dir per node then, remove the port assignment stuff, and use
tempdir_short as a key to define a node as well as in the connection
string to this node. I'll update the patch later today...
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#38Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#37)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Oct 7, 2015 at 7:51 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Oct 7, 2015 at 7:43 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Oct 7, 2015 at 5:58 AM, Robert Haas wrote:

On Sat, Oct 3, 2015 at 7:38 AM, Michael Paquier wrote:
It seems that these days 'make check' creates a directory in /tmp
called /tmp/pg_regress-RANDOMSTUFF. Listening on TCP ports is
disabled, and the socket goes inside this directory with a name like
.s.PGSQL.PORT. You can connect with psql -h
/tmp/pg_regress-RANDOMSTUFF -p PORT, but not over TCP. This basically
removes the risk of TCP port number collisions, as well as the risk of
your temporary instance being hijacked by a malicious user on the same
machine.

Right, that's for example /var/folders/ on OSX, and this is defined
once per test run via $tempdir_short. PGHOST is set to that as well.

Er, mistake here. That's actually once per standard_initdb, except
that all the tests I have included in my patch run it just once to
create a master node. It seems that it would be wiser to set one
socket dir per node then, remove the port assignment stuff, and use
tempdir_short as a key to define a node as well as in the connection
string to this node. I'll update the patch later today...

So, my conclusion regarding multiple calls of make_master is that we
should not allow to do it. On Unix/Linux we could have a separate unix
socket directory for each node, but not on Windows where
listen_addresses is set to look after 127.0.0.1. On Unix/Linux, PGHOST
is set by the master node to a tempdir once and for all. Hence, to
make the code more consistent, I think that we should keep the port
lookup machinery here. An updated patch is attached.
--
Michael

Attachments:

20151007_recovery_regressions_v7.patchtext/x-diff; charset=US-ASCII; name=20151007_recovery_regressions_v7.patchDownload
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..ea219d7 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -125,38 +125,6 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
 sub append_to_file
 {
 	my ($filename, $str) = @_;
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..d6e51eb 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -17,7 +17,7 @@ SUBDIRS = regress isolation modules
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
 # because the SSL test suite is not secure to run on a multi-user system.
-ALWAYS_SUBDIRS = examples locale thread ssl
+ALWAYS_SUBDIRS = examples locale thread ssl recovery
 
 # We want to recurse to all subdirs for all standard targets, except that
 # installcheck and install should not recurse into the subdirectory "modules".
diff --git a/src/test/perl/RecoveryTest.pm b/src/test/perl/RecoveryTest.pm
new file mode 100644
index 0000000..cbd2441
--- /dev/null
+++ b/src/test/perl/RecoveryTest.pm
@@ -0,0 +1,410 @@
+package RecoveryTest;
+
+# Set of common routines for recovery regression tests for a PostgreSQL
+# cluster. This includes global variables and methods that can be used
+# by the various set of tests present to set up cluster nodes and
+# configure them according to the test scenario wanted.
+#
+# Cluster nodes can be freely created using initdb or using the existing
+# base backup of another node, with minimum configuration done when the
+# node is created for the first time like having a proper port number.
+# It is then up to the test to decide what to do with the newly-created
+# node.
+#
+# Environment configuration of each node is available through a set
+# of global variables provided by this package, hashed depending on the
+# port number of a node:
+# - connstr_nodes connection string to connect to this node
+# - datadir_nodes to get the data folder of a given node
+# - archive_nodes for the location of the WAL archives of a node
+# - backup_nodes for the location of base backups of a node
+# - applname_nodes, application_name to use for a standby
+#
+# Nodes are identified by their port number, which should be unique
+# for each node of the cluster as it is run locally.
+
+use Cwd;
+use TestLib;
+use Test::More;
+
+use Archive::Tar;
+use IPC::Run qw(run start);
+
+use Exporter 'import';
+
+our @EXPORT = qw(
+	%connstr_nodes
+	%datadir_nodes
+	%backup_nodes
+	%archive_nodes
+	%applname_nodes
+
+	append_to_file
+	backup_node
+	disable_node
+	dump_node_info
+	enable_archiving
+	enable_node
+	enable_restoring
+	enable_streaming
+	get_free_port
+	init_node
+	init_node_from_backup
+	make_master
+	make_warm_standby
+	make_hot_standby
+	restart_node
+	start_node
+	stop_node
+	teardown_node
+);
+
+# Global variables for node data
+%datadir_nodes = {};	# PGDATA folders
+%backup_nodes = {};		# Backup base folder
+%archive_nodes = {};	# Archive base folder
+%connstr_nodes = {};	# Connection strings
+%applname_nodes = {};	# application_name used for standbys
+
+# Tracking of last port value assigned to accelerate free port lookup.
+# XXX: Should this part use PG_VERSION_NUM?
+my $last_port_assigned =  90600 % 16384 + 49152;
+
+# Flag to see if a master has already been initialized with its own PGHOST.
+my $master_initialized = 0;
+
+# Database used for each connection attempt via psql
+$ENV{PGDATABASE} = "postgres";
+
+# Tracker of active nodes
+my @active_nodes = ();
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_root } application_name=$applname_nodes{ $port_standby }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+	my $path = $archive_nodes{ $port_root };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"$path\\\\%f\" \"%p\"" :
+		"cp -i $path/%f %p";
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+restore_command='$copy_command'
+standby_mode=on
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $port = shift;
+	my $path = $archive_nodes{ $port };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$path\\\\%f\"" :
+		"cp %p $path/%f";
+
+	# Enable archive_mode and archive_command on node
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization. This should be called only once in a series
+# of tests because the rest of the logic relies on PGHOST being set only
+# once. This would not be an issue on Linux where one socket path per
+# node could be used by on Windows listen_addresses needs to be set
+# to look at 127.0.0.1.
+sub make_master
+{
+	my $port_master = get_free_port();
+	die "Master has already been initialized\n" if ($master_initialized);
+	print "# Initializing master node wih port $port_master\n";
+	init_node($port_master);
+	$master_initialized = 1;
+	return $port_master;
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_hot_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing streaming mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_streaming($port_master, $port_standby);
+	return $port_standby;
+}
+
+# Node getting WAL only from archives
+sub make_warm_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing archive mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_restoring($port_master, $port_standby);
+	return $port_standby;
+}
+
+sub configure_base_node
+{
+	my $port = shift;
+
+	# Make configuration somewhat generic to test recovery
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+port = $port
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+));
+
+	configure_hba_for_replication($datadir_nodes{ $port });
+}
+
+# Get a port number not in use currently for a new node
+# As port number retrieval is based on the nodes currently running and
+# their presence in the list of registered ports, be sure that the node
+# that is consuming this port number has already been started and that
+# it is not registered yet.
+sub get_free_port
+{
+	my $found = 0;
+	my $port = $last_port_assigned;
+
+	while ($found == 0)
+	{
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		my $ret = system("psql -X -p $port postgres < $devnull");
+		if ($ret != 0)
+		{
+			# Found a potential candidate, check first that it is
+			# not included in the list of registered nodes.
+			$found = 1 if ! grep (/$port/, @array);
+		}
+	}
+
+	print "# Found free port $port\n";
+	# Lock port number found
+	enable_node($port);
+	$last_port_assigned = $port;
+	return $port;
+}
+
+# Low-level routines to initialize a node
+# Initialize a node from scratch
+sub init_node
+{
+	my $port = shift;
+
+	# Save configuration information
+	my $base_dir = TestLib::tempdir;
+	$datadir_nodes{ $port } = "$base_dir/pgdata";
+	print "Base directory $base_dir/pgdata, Port $port\n";
+	$backup_nodes{ $port } = "$base_dir/backup";
+	mkdir $backup_nodes{ $port };
+	$archive_nodes{ $port } =  "$base_dir/archives";
+	mkdir $archive_nodes{ $port };
+	$connstr_nodes{ $port } = "port=$port";
+	$applname_nodes{ $port } = "node_$port";
+
+	# Log some information
+	dump_node_info($port);
+
+	standard_initdb($datadir_nodes{ $port });
+	configure_base_node($port);
+}
+
+# Initialize a node from an existing base backup
+sub init_node_from_backup
+{
+	my ($port, $root_port, $backup_name) = @_;
+
+	my $backup_path =  "$backup_nodes{ $root_port }/$backup_name";
+	my $backup_file = "$backup_path/base.tar";
+
+	print "Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	# Check existence of backup wanted
+	if ( ! -d $backup_path )
+	{
+		die "Backup $backup_path does not exist";
+	}
+
+	# Save configuration information
+	my $base_dir = TestLib::tempdir;
+	$datadir_nodes{ $port } = "$base_dir/pgdata";
+	mkdir $datadir_nodes{ $port }, 0700;
+	$backup_nodes{ $port } = "$base_dir/backup";
+	mkdir $backup_nodes{ $port };
+	$archive_nodes{ $port } =  "$base_dir/archives";
+	mkdir $archive_nodes{ $port };
+	$connstr_nodes{ $port } = "port=$port";
+	$applname_nodes{ $port } = "node_$port";
+
+	# Log some information
+	dump_node_info($port);
+
+	# Extract the base backup wanted
+	my $current_dir = cwd();
+
+	# Temporary move to the place of extraction
+	chdir "$datadir_nodes{ $port }";
+	Archive::Tar->extract_archive($backup_file);
+	chdir "$current_dir";
+	configure_base_node($port);
+}
+
+# Start a node
+sub start_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port },
+				   '-l', "$log_path/node_$port.log",
+				   'start');
+}
+
+# Stop a node. This routine can be called during a test.
+sub stop_node
+{
+	my $port = shift;
+	my $mode = shift || "fast";
+
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   $mode, 'stop');
+}
+
+sub restart_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port }, '-m',
+				   'fast', 'restart');
+}
+
+# Wait until a node is able to accept queries. Useful have putting a node
+# in recovery.
+sub wait_for_node
+{
+	my $port         = shift;
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	while ($attempts < $max_attempts)
+	{
+		if (run_log('pg_isready', '-p', $port))
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+	return 0;
+}
+
+# Mark given node as active, making the port number used by this node
+# locked for this test.
+sub enable_node
+{
+	my $port = shift;
+	print "# Port $port has been locked for new node\n";
+	push(@active_nodes, $port);
+}
+
+# Disactivate given node, node should have been stopped before calling
+# this routine.
+sub disable_node
+{
+	my $port = shift;
+	print "# Port $port has been disabled\n";
+	@active_nodes = grep { $_ ne $port } @active_nodes;
+}
+
+# Remove any traces of given node.
+sub teardown_node
+{
+	my $port = shift;
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   'immediate', 'stop');
+	disable_node($port);
+}
+
+# Create a backup on a node already running
+sub backup_node
+{
+	my ($port, $backup_name) = @_;
+
+	my $backup_path = "$backup_nodes{ $port }/$backup_name";
+
+	print "Taking backup from node $port\n";
+	# Backup a node in tar format, it is more portable across platforms
+	system_or_bail("pg_basebackup -D $backup_path -p $port --format=t -x");
+	print "# Backup finished\n"
+}
+
+# Dump information of a node
+sub dump_node_info
+{
+	my $port = shift;
+	print "New node initialized: ${port}\n";
+	print "Data directory: $datadir_nodes{ $port }\n";
+	print "Backup directory: $backup_nodes{ $port }\n";
+	print "Archive directory: $archive_nodes{ $port }\n";
+	print "Connection string: $connstr_nodes{ $port }\n";
+	print "Application name: applname_nodes{ $port }\n";
+}
+
+# Add a set of parameters to a configuration file
+sub append_to_file
+{
+	my($filename, $str) = @_;
+
+	open my $fh, ">>", $filename or die "could not open file $filename";
+	print $fh $str;
+	close $fh;
+}
+
+END
+{
+	foreach my $port (@active_nodes)
+	{
+		teardown_node($port);
+	}
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..0dc0d6d 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -12,6 +12,7 @@ our @EXPORT = qw(
   configure_hba_for_replication
   start_test_server
   restart_test_server
+  poll_query_until
   psql
   slurp_dir
   slurp_file
@@ -220,6 +221,37 @@ END
 	}
 }
 
+sub poll_query_until
+{
+	my ($query, $connstr) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
+		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
 sub psql
 {
 	my ($dbname, $sql) = @_;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..fd4bfbf
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,62 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+my $backup_name = 'my_backup';
+
+# Take backup
+backup_node($port_master, $backup_name);
+
+# Create streaming standby linking to master
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_1);
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+backup_node($port_standby_1, $backup_name);
+
+# Create second standby node linking to standby 1
+my $port_standby_2 = make_hot_standby($port_standby_1, $backup_name);
+start_node($port_standby_2);
+
+# Create some content on master and check its presence in standby 1 an
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a";
+
+# Wait for standbys to catch up
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_nodes{ $port_standby_1 }';";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_master })
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_nodes{ $port_standby_2 }';";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT count(*) FROM tab_int";
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_1 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_2 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
+
+# Cleanup nodes
+teardown_node($port_standby_2);
+teardown_node($port_standby_1);
+teardown_node($port_master);
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..c7977f0
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,51 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $port_master = make_master();
+my $backup_name = 'my_backup';
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Take backup for slave
+backup_node($port_master, $backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $port_standby = make_warm_standby($port_master, $backup_name);
+append_to_file("$datadir_nodes{ $port_standby }/postgresql.conf", qq(
+wal_retrieve_retry_interval = '100ms'
+));
+start_node($port_standby);
+
+# Create some content on master
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $current_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Force archiving of WAL file to make it present on master
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Add some more content, it should not be present on standby
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(1000), 'check content from archives');
+
+# Cleanup nodes
+teardown_node($port_standby);
+teardown_node($port_master);
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..c989f83
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,138 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $port_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $port_standby = get_free_port();
+
+	init_node_from_backup($port_standby, $port_master, 'my_backup');
+	enable_restoring($port_master, $port_standby);
+
+	foreach my $param_item (@$recovery_params)
+	{
+		append_to_file("$datadir_nodes{ $port_standby }/recovery.conf",
+					   qq($param_item
+));
+	}
+
+	start_node($port_standby);
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = psql $connstr_nodes{ $port_standby },
+		"SELECT count(*) FROM tab_int";
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	teardown_node($port_standby);
+}
+
+# Initialize master node
+my $port_master = make_master();
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $lsn1 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Take backup from which all operations will be run
+backup_node($port_master, 'my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+my $recovery_txid = psql $connstr_nodes{ $port_master },
+	"SELECT txid_current()";
+my $lsn2 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# More data, with recovery target timestamp
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(2001,3000))";
+my $recovery_time = psql $connstr_nodes{ $port_master }, "SELECT now()";
+my $lsn3 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Even more data, this time with a recovery target name
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))";
+my $recovery_name = "my_target";
+my $lsn4 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+psql $connstr_nodes{ $port_master },
+	"SELECT pg_create_restore_point('$recovery_name')";
+
+# Force archiving of WAL file
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $port_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+teardown_node($port_master);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..2455ca4
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,68 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+
+# Take backup
+my $backup_name = 'my_backup';
+backup_node($port_master, $backup_name);
+
+# Create two standbys linking to it
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_1);
+my $port_standby_2 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_2);
+
+# Create some content on master
+psql $connstr_nodes { $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+teardown_node($port_master);
+system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port_standby_1 },
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree("$datadir_nodes{ $port_standby_2 }/recovery.conf");
+append_to_file("$datadir_nodes{ $port_standby_2 }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_standby_1 }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+restart_node($port_standby_2);
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done.
+psql $connstr_nodes{ $port_standby_1 },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+$until_lsn = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT pg_current_xlog_location();";
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_2 })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(2000), 'check content of standby 2');
+
+# Stop nodes
+teardown_node($port_standby_2);
+teardown_node($port_standby_1);
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..76ea60a
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,54 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+
+# And some content
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a";
+
+# Take backup
+backup_node($port_master, 'my_backup');
+
+# Initialize node from backup
+my $port_standby = get_free_port();
+init_node_from_backup($port_standby, $port_master, 'my_backup');
+
+# Start second node, streaming from first one
+append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_master }'
+recovery_min_apply_delay = '2s'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+start_node($port_standby);
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(11,20))";
+sleep 1;
+# Here we should have only 10 rows
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+$result = psql $connstr_nodes{ $port_standby }, "SELECT count(*) FROM tab_int";
+is($result, qq(20), 'check content with delay of 2s');
+
+# Stop nodes
+teardown_node($port_standby);
+teardown_node($port_master);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index d3d736b..1355508 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -194,8 +194,9 @@ sub tapcheck
 	foreach my $test_path (@$tap_dirs)
 	{
 		# Like on Unix "make check-world", don't run the SSL test suite
-		# automatically.
+		# or the recovery test suite automatically.
 		next if ($test_path =~ /\/src\/test\/ssl\//);
+		next if ($test_path =~ /\/src\/test\/recovery\//);
 
 		my $dir = dirname($test_path);
 		chdir $dir;
#39Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#38)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/07/2015 09:27 AM, Michael Paquier wrote:

On Wed, Oct 7, 2015 at 7:51 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Oct 7, 2015 at 7:43 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Oct 7, 2015 at 5:58 AM, Robert Haas wrote:

On Sat, Oct 3, 2015 at 7:38 AM, Michael Paquier wrote:
It seems that these days 'make check' creates a directory in /tmp
called /tmp/pg_regress-RANDOMSTUFF. Listening on TCP ports is
disabled, and the socket goes inside this directory with a name like
.s.PGSQL.PORT. You can connect with psql -h
/tmp/pg_regress-RANDOMSTUFF -p PORT, but not over TCP. This basically
removes the risk of TCP port number collisions, as well as the risk of
your temporary instance being hijacked by a malicious user on the same
machine.

Right, that's for example /var/folders/ on OSX, and this is defined
once per test run via $tempdir_short. PGHOST is set to that as well.

Er, mistake here. That's actually once per standard_initdb, except
that all the tests I have included in my patch run it just once to
create a master node. It seems that it would be wiser to set one
socket dir per node then, remove the port assignment stuff, and use
tempdir_short as a key to define a node as well as in the connection
string to this node. I'll update the patch later today...

So, my conclusion regarding multiple calls of make_master is that we
should not allow to do it. On Unix/Linux we could have a separate unix
socket directory for each node, but not on Windows where
listen_addresses is set to look after 127.0.0.1. On Unix/Linux, PGHOST
is set by the master node to a tempdir once and for all. Hence, to
make the code more consistent, I think that we should keep the port
lookup machinery here. An updated patch is attached.

If parallel tests are supported, get_free_port is still racy even
with last_port_found because it's:
1) process-local.
2) even if it were shared, there's the race window between the
available-check and the listen() I mentioned upthread.

If parallel tests are explicitly disallowed, a comment to that
effect (and a note on things known to break) might help someone
down the road.

Also, the removal of poll_query_until from pg_rewind looks suspiciously
like a copy-paste gone bad. Pardon if I'm missing something.

Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#40Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#39)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Oct 7, 2015 at 4:16 PM, Amir Rohan <amir.rohan@zoho.com> wrote:

On 10/07/2015 09:27 AM, Michael Paquier wrote:

On Wed, Oct 7, 2015 at 7:51 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Oct 7, 2015 at 7:43 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Oct 7, 2015 at 5:58 AM, Robert Haas wrote:

On Sat, Oct 3, 2015 at 7:38 AM, Michael Paquier wrote:
It seems that these days 'make check' creates a directory in /tmp
called /tmp/pg_regress-RANDOMSTUFF. Listening on TCP ports is
disabled, and the socket goes inside this directory with a name like
.s.PGSQL.PORT. You can connect with psql -h
/tmp/pg_regress-RANDOMSTUFF -p PORT, but not over TCP. This basically
removes the risk of TCP port number collisions, as well as the risk of
your temporary instance being hijacked by a malicious user on the same
machine.

Right, that's for example /var/folders/ on OSX, and this is defined
once per test run via $tempdir_short. PGHOST is set to that as well.

Er, mistake here. That's actually once per standard_initdb, except
that all the tests I have included in my patch run it just once to
create a master node. It seems that it would be wiser to set one
socket dir per node then, remove the port assignment stuff, and use
tempdir_short as a key to define a node as well as in the connection
string to this node. I'll update the patch later today...

So, my conclusion regarding multiple calls of make_master is that we
should not allow to do it. On Unix/Linux we could have a separate unix
socket directory for each node, but not on Windows where
listen_addresses is set to look after 127.0.0.1. On Unix/Linux, PGHOST
is set by the master node to a tempdir once and for all. Hence, to
make the code more consistent, I think that we should keep the port
lookup machinery here. An updated patch is attached.

If parallel tests are supported, get_free_port is still racy even
with last_port_found because it's:
1) process-local.
2) even if it were shared, there's the race window between the
available-check and the listen() I mentioned upthread.

If parallel tests are explicitly disallowed, a comment to that
effect (and a note on things known to break) might help someone
down the road.

Actually, no, port lookup will not map and parallel tests would work
fine thinking more about it, each set of tests uses its own PGHOST to
a private unix socket directory so even if multiple tests use the same
port number they won't interact with each other because they connect
to different socket paths. MinGW is a problem though, and an existing
one in the perl test scripts, I recall that it can use make -j and
that's on Windows where PGHOST is mapping to 127.0.0.1 only.

Also, the removal of poll_query_until from pg_rewind looks suspiciously
like a copy-paste gone bad. Pardon if I'm missing something.

Perhaps. Do you have a suggestion regarding that? It seems to me that
this is more useful in TestLib.pm as-is.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#41Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#40)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/07/2015 10:29 AM, Michael Paquier wrote:

On Wed, Oct 7, 2015 at 4:16 PM, Amir Rohan <amir.rohan@zoho.com> wrote:

On 10/07/2015 09:27 AM, Michael Paquier wrote:

On Wed, Oct 7, 2015 at 7:51 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Oct 7, 2015 at 7:43 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Oct 7, 2015 at 5:58 AM, Robert Haas wrote:

On Sat, Oct 3, 2015 at 7:38 AM, Michael Paquier wrote:
It seems that these days 'make check' creates a directory in /tmp
called /tmp/pg_regress-RANDOMSTUFF. Listening on TCP ports is
disabled, and the socket goes inside this directory with a name like
.s.PGSQL.PORT. You can connect with psql -h
/tmp/pg_regress-RANDOMSTUFF -p PORT, but not over TCP. This basically
removes the risk of TCP port number collisions, as well as the risk of
your temporary instance being hijacked by a malicious user on the same
machine.

Right, that's for example /var/folders/ on OSX, and this is defined
once per test run via $tempdir_short. PGHOST is set to that as well.

Er, mistake here. That's actually once per standard_initdb, except
that all the tests I have included in my patch run it just once to
create a master node. It seems that it would be wiser to set one
socket dir per node then, remove the port assignment stuff, and use
tempdir_short as a key to define a node as well as in the connection
string to this node. I'll update the patch later today...

So, my conclusion regarding multiple calls of make_master is that we
should not allow to do it. On Unix/Linux we could have a separate unix
socket directory for each node, but not on Windows where
listen_addresses is set to look after 127.0.0.1. On Unix/Linux, PGHOST
is set by the master node to a tempdir once and for all. Hence, to
make the code more consistent, I think that we should keep the port
lookup machinery here. An updated patch is attached.

If parallel tests are supported, get_free_port is still racy even
with last_port_found because it's:
1) process-local.
2) even if it were shared, there's the race window between the
available-check and the listen() I mentioned upthread.

If parallel tests are explicitly disallowed, a comment to that
effect (and a note on things known to break) might help someone
down the road.

Actually, no, port lookup will not map and parallel tests would work
fine thinking more about it, each set of tests uses its own PGHOST to
a private unix socket directory so even if multiple tests use the same
port number they won't interact with each other because they connect
to different socket paths. MinGW is a problem though, and an existing
one in the perl test scripts, I recall that it can use make -j and
that's on Windows where PGHOST is mapping to 127.0.0.1 only.

ah, the portnum is actually a real tcp port only on windows, and
the race is limited to that case as you say. Note that in the
tcp case, using psql to check is wrong:
$ nc -l 8001 # listen on 8001
$ psql -X -h lo -p 8001 postgres < /dev/null psql: could not connect to
server: Connection refused
Is the server running on host "lo" (127.0.0.1) and accepting
TCP/IP connections on port 8001?

The port isn't free, but psql is really only checking if pg is there
and reports that the port is available. That's a fairly mild issue, though.

Also, the removal of poll_query_until from pg_rewind looks suspiciously
like a copy-paste gone bad. Pardon if I'm missing something.

Perhaps. Do you have a suggestion regarding that? It seems to me that
this is more useful in TestLib.pm as-is.

My mistake, the patch only shows some internal function being deleted
but RewindTest.pm (obviously) imports TestLib. You're right, TestLib is
a better place for it.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#42Craig Ringer
craig@2ndquadrant.com
In reply to: Michael Paquier (#21)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 25 September 2015 at 14:29, Michael Paquier
<michael.paquier@gmail.com> wrote:

I also arrived at the conclusion that it would be
better to place the new package file in src/test/perl instead of
src/test/recovery to allow any users of the TAP tests to have it in their
PERL5LIB path and to be able to call the new routines to create and
manipulate nodes.

While it's Python not Perl, you might find it interesting that support
for the replication protocol is being added to psycopg2, the Python
driver for PostgreSQL. I've been reviewing the patch at
https://github.com/psycopg/psycopg2/pull/322 .

I'm using it to write protocol validation for a logical decoding
plugin at the moment, so that the decoding plugin's output can be
validated in a consistent way for easily controlled inputs.

Perhaps it's worth teaching DBD::Pg to do this? Or adopting psycopg2
for some optional protocol tests...

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#43Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#41)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Oct 7, 2015 at 5:44 PM, Amir Rohan wrote:

On 10/07/2015 10:29 AM, Michael Paquier wrote:

On Wed, Oct 7, 2015 at 4:16 PM, Amir Rohan wrote:

Also, the removal of poll_query_until from pg_rewind looks suspiciously
like a copy-paste gone bad. Pardon if I'm missing something.

Perhaps. Do you have a suggestion regarding that? It seems to me that
this is more useful in TestLib.pm as-is.

My mistake, the patch only shows some internal function being deleted
but RewindTest.pm (obviously) imports TestLib. You're right, TestLib is
a better place for it.

OK. Here is a new patch version. I have removed the restriction
preventing to call make_master multiple times in the same script (one
may actually want to test some stuff related to logical decoding or
FDW for example, who knows...), forcing PGHOST to always use the same
value after it has been initialized. I have added a sanity check
though, it is not possible to create a node based on a base backup if
no master are defined. This looks like a cheap insurance... I also
refactored a bit the code, using the new init_node_info to fill in the
fields of a newly-initialized node, and I removed get_free_port,
init_node, init_node_from_backup, enable_restoring and
enable_streaming from the list of routines exposed to the users, those
can be used directly with make_master, make_warm_standby and
make_hot_standby. We could add them again if need be, somebody may
want to be able to get a free port, set up a node without those
generic routines, just that it does not seem necessary now.
Regards,
--
Michael

Attachments:

20151008_recovery_regressions_v8.patchtext/x-diff; charset=US-ASCII; name=20151008_recovery_regressions_v8.patchDownload
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..ea219d7 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -125,38 +125,6 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
 sub append_to_file
 {
 	my ($filename, $str) = @_;
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..d6e51eb 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -17,7 +17,7 @@ SUBDIRS = regress isolation modules
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
 # because the SSL test suite is not secure to run on a multi-user system.
-ALWAYS_SUBDIRS = examples locale thread ssl
+ALWAYS_SUBDIRS = examples locale thread ssl recovery
 
 # We want to recurse to all subdirs for all standard targets, except that
 # installcheck and install should not recurse into the subdirectory "modules".
diff --git a/src/test/perl/RecoveryTest.pm b/src/test/perl/RecoveryTest.pm
new file mode 100644
index 0000000..baa4d31
--- /dev/null
+++ b/src/test/perl/RecoveryTest.pm
@@ -0,0 +1,412 @@
+package RecoveryTest;
+
+# Set of common routines for recovery regression tests for a PostgreSQL
+# cluster. This includes global variables and methods that can be used
+# by the various set of tests present to set up cluster nodes and
+# configure them according to the test scenario wanted.
+#
+# Cluster nodes can be freely created using initdb or using the existing
+# base backup of another node, with minimum configuration done when the
+# node is created for the first time like having a proper port number.
+# It is then up to the test to decide what to do with the newly-created
+# node.
+#
+# Environment configuration of each node is available through a set
+# of global variables provided by this package, hashed depending on the
+# port number of a node:
+# - connstr_nodes connection string to connect to this node
+# - datadir_nodes to get the data folder of a given node
+# - archive_nodes for the location of the WAL archives of a node
+# - backup_nodes for the location of base backups of a node
+# - applname_nodes, application_name to use for a standby
+#
+# Nodes are identified by their port number, which should be unique
+# for each node of the cluster as it is run locally.
+
+use Cwd;
+use TestLib;
+use Test::More;
+
+use Archive::Tar;
+use IPC::Run qw(run start);
+
+use Exporter 'import';
+
+our @EXPORT = qw(
+	%connstr_nodes
+	%datadir_nodes
+	%backup_nodes
+	%archive_nodes
+	%applname_nodes
+
+	append_to_file
+	backup_node
+	disable_node
+	dump_node_info
+	enable_archiving
+	enable_node
+	enable_restoring
+	enable_streaming
+	get_free_port
+	init_node
+	init_node_from_backup
+	make_master
+	make_warm_standby
+	make_hot_standby
+	restart_node
+	start_node
+	stop_node
+	teardown_node
+);
+
+# Global variables for node data
+%datadir_nodes = {};	# PGDATA folders
+%backup_nodes = {};		# Backup base folder
+%archive_nodes = {};	# Archive base folder
+%connstr_nodes = {};	# Connection strings
+%applname_nodes = {};	# application_name used for standbys
+
+# Tracking of last port value assigned to accelerate free port lookup.
+# XXX: Should this part use PG_VERSION_NUM?
+my $last_port_assigned =  90600 % 16384 + 49152;
+
+# Value tracking the host value used for a single run.
+my $node_pghost = undef;
+
+# Database used for each connection attempt via psql
+$ENV{PGDATABASE} = "postgres";
+
+# Tracker of active nodes
+my @active_nodes = ();
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_root } application_name=$applname_nodes{ $port_standby }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+	my $path = $archive_nodes{ $port_root };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"$path\\\\%f\" \"%p\"" :
+		"cp -i $path/%f %p";
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+restore_command='$copy_command'
+standby_mode=on
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $port = shift;
+	my $path = $archive_nodes{ $port };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$path\\\\%f\"" :
+		"cp %p $path/%f";
+
+	# Enable archive_mode and archive_command on node
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization. This should be called only once in a series
+# of tests because the rest of the logic relies on PGHOST being set only
+# once. This would not be an issue on Linux where one socket path per
+# node could be used by on Windows listen_addresses needs to be set
+# to look at 127.0.0.1.
+sub make_master
+{
+	my $port_master = get_free_port();
+	print "# Initializing master node wih port $port_master\n";
+	init_node($port_master);
+	return $port_master;
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_hot_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing streaming mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_streaming($port_master, $port_standby);
+	return $port_standby;
+}
+
+# Node getting WAL only from archives
+sub make_warm_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing archive mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, restoring from first one
+	enable_restoring($port_master, $port_standby);
+	return $port_standby;
+}
+
+sub configure_base_node
+{
+	my $port = shift;
+
+	# Make configuration somewhat generic to test recovery
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+port = $port
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+));
+
+	configure_hba_for_replication($datadir_nodes{ $port });
+}
+
+# Get a port number not in use currently for a new node
+# As port number retrieval is based on the nodes currently running and
+# their presence in the list of registered ports, be sure that the node
+# that is consuming this port number has already been started and that
+# it is not registered yet.
+sub get_free_port
+{
+	my $found = 0;
+	my $port = $last_port_assigned;
+
+	while ($found == 0)
+	{
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		my $ret = system("psql -X -p $port postgres < $devnull");
+		if ($ret != 0)
+		{
+			# Found a potential candidate, check first that it is
+			# not included in the list of registered nodes.
+			$found = 1 if ! grep (/$port/, @array);
+		}
+	}
+
+	print "# Found free port $port\n";
+	# Lock port number found
+	enable_node($port);
+	$last_port_assigned = $port;
+	return $port;
+}
+
+# Low-level routines to initialize a node
+# Initialize a node from scratch
+sub init_node
+{
+	my $port = shift;
+
+	if (! defined($node_pghost))
+	{
+		$node_pghost = $windows_os ? "127.0.0.1" : tempdir_short;
+	}
+
+	init_node_info($port);
+	mkdir $backup_nodes{ $port };
+	mkdir $archive_nodes{ $port };
+
+	standard_initdb($datadir_nodes{ $port }, $node_pghost);
+	configure_base_node($port);
+}
+
+# Initialize all the information of a node freshly created in the
+# common set of variables for the whole set.
+sub init_node_info
+{
+	my $port = shift;
+	my $base_dir = TestLib::tempdir;
+
+	$datadir_nodes{ $port } = "$base_dir/pgdata";
+	$backup_nodes{ $port } = "$base_dir/backup";
+	$archive_nodes{ $port } =  "$base_dir/archives";
+	$connstr_nodes{ $port } = "port=$port host=$node_pghost";
+	$applname_nodes{ $port } = "node_$port";
+
+	# Log some information
+	dump_node_info($port);
+}
+
+# Initialize a node from an existing base backup
+sub init_node_from_backup
+{
+	my ($port, $root_port, $backup_name) = @_;
+
+	my $backup_path =  "$backup_nodes{ $root_port }/$backup_name";
+	my $backup_file = "$backup_path/base.tar";
+
+	die "Initializing node from backup but no master defined"
+		if (! defined($node_pghost));
+
+	print "Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	# Check existence of backup wanted
+	if ( ! -d $backup_path )
+	{
+		die "Backup $backup_path does not exist";
+	}
+
+	# Save configuration information
+	init_node_info($port);
+	mkdir $datadir_nodes{ $port }, 0700;
+	mkdir $backup_nodes{ $port };
+	mkdir $archive_nodes{ $port };
+
+	# Temporary move to the place of extraction
+	chdir "$datadir_nodes{ $port }";
+	Archive::Tar->extract_archive($backup_file);
+	chdir "$current_dir";
+	configure_base_node($port);
+}
+
+# Start a node
+sub start_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port },
+				   '-l', "$log_path/node_$port.log",
+				   'start');
+}
+
+# Stop a node. This routine can be called during a test.
+sub stop_node
+{
+	my $port = shift;
+	my $mode = shift || "fast";
+
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   $mode, 'stop');
+}
+
+sub restart_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port }, '-m',
+				   'fast', 'restart');
+}
+
+# Wait until a node is able to accept queries. Useful have putting a node
+# in recovery.
+sub wait_for_node
+{
+	my $port         = shift;
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	while ($attempts < $max_attempts)
+	{
+		if (run_log('pg_isready', '-p', $port))
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+	return 0;
+}
+
+# Mark given node as active, making the port number used by this node
+# locked for this test.
+sub enable_node
+{
+	my $port = shift;
+	print "# Port $port has been locked for new node\n";
+	push(@active_nodes, $port);
+}
+
+# Disactivate given node, node should have been stopped before calling
+# this routine.
+sub disable_node
+{
+	my $port = shift;
+	print "# Port $port has been disabled\n";
+	@active_nodes = grep { $_ ne $port } @active_nodes;
+}
+
+# Remove any traces of given node.
+sub teardown_node
+{
+	my $port = shift;
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   'immediate', 'stop');
+	disable_node($port);
+}
+
+# Create a backup on a node already running
+sub backup_node
+{
+	my ($port, $backup_name) = @_;
+
+	my $backup_path = "$backup_nodes{ $port }/$backup_name";
+
+	print "Taking backup from node $port\n";
+	# Backup a node in tar format, it is more portable across platforms
+	system_or_bail("pg_basebackup -D $backup_path -p $port --format=t -x");
+	print "# Backup finished\n"
+}
+
+# Dump information of a node
+sub dump_node_info
+{
+	my $port = shift;
+	print "New node initialized: ${port}\n";
+	print "Data directory: $datadir_nodes{ $port }\n";
+	print "Backup directory: $backup_nodes{ $port }\n";
+	print "Archive directory: $archive_nodes{ $port }\n";
+	print "Connection string: $connstr_nodes{ $port }\n";
+	print "Application name: applname_nodes{ $port }\n";
+}
+
+# Add a set of parameters to a configuration file
+sub append_to_file
+{
+	my($filename, $str) = @_;
+
+	open my $fh, ">>", $filename or die "could not open file $filename";
+	print $fh $str;
+	close $fh;
+}
+
+END
+{
+	foreach my $port (@active_nodes)
+	{
+		teardown_node($port);
+	}
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..34e4ce0 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -12,6 +12,7 @@ our @EXPORT = qw(
   configure_hba_for_replication
   start_test_server
   restart_test_server
+  poll_query_until
   psql
   slurp_dir
   slurp_file
@@ -136,11 +137,12 @@ sub tempdir_short
 sub standard_initdb
 {
 	my $pgdata = shift;
+	my $tempdir_short = shift;
+
+	$tempdir_short = tempdir_short if (! defined($tempdir_short));
 	system_or_bail('initdb', '-D', "$pgdata", '-A' , 'trust', '-N');
 	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
 
-	my $tempdir_short = tempdir_short;
-
 	open CONF, ">>$pgdata/postgresql.conf";
 	print CONF "\n# Added by TestLib.pm)\n";
 	print CONF "fsync = off\n";
@@ -220,6 +222,37 @@ END
 	}
 }
 
+sub poll_query_until
+{
+	my ($query, $connstr) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
+		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
 sub psql
 {
 	my ($dbname, $sql) = @_;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..fd4bfbf
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,62 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+my $backup_name = 'my_backup';
+
+# Take backup
+backup_node($port_master, $backup_name);
+
+# Create streaming standby linking to master
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_1);
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+backup_node($port_standby_1, $backup_name);
+
+# Create second standby node linking to standby 1
+my $port_standby_2 = make_hot_standby($port_standby_1, $backup_name);
+start_node($port_standby_2);
+
+# Create some content on master and check its presence in standby 1 an
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a";
+
+# Wait for standbys to catch up
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_nodes{ $port_standby_1 }';";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_master })
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_nodes{ $port_standby_2 }';";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT count(*) FROM tab_int";
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_1 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_2 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
+
+# Cleanup nodes
+teardown_node($port_standby_2);
+teardown_node($port_standby_1);
+teardown_node($port_master);
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..c7977f0
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,51 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $port_master = make_master();
+my $backup_name = 'my_backup';
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Take backup for slave
+backup_node($port_master, $backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $port_standby = make_warm_standby($port_master, $backup_name);
+append_to_file("$datadir_nodes{ $port_standby }/postgresql.conf", qq(
+wal_retrieve_retry_interval = '100ms'
+));
+start_node($port_standby);
+
+# Create some content on master
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $current_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Force archiving of WAL file to make it present on master
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Add some more content, it should not be present on standby
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(1000), 'check content from archives');
+
+# Cleanup nodes
+teardown_node($port_standby);
+teardown_node($port_master);
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..94b94a1
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,135 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $port_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $port_standby = make_warm_standby($port_master, 'my_backup');
+
+	foreach my $param_item (@$recovery_params)
+	{
+		append_to_file("$datadir_nodes{ $port_standby }/recovery.conf",
+					   qq($param_item
+));
+	}
+
+	start_node($port_standby);
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = psql $connstr_nodes{ $port_standby },
+		"SELECT count(*) FROM tab_int";
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	teardown_node($port_standby);
+}
+
+# Initialize master node
+my $port_master = make_master();
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $lsn1 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Take backup from which all operations will be run
+backup_node($port_master, 'my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+my $recovery_txid = psql $connstr_nodes{ $port_master },
+	"SELECT txid_current()";
+my $lsn2 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# More data, with recovery target timestamp
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(2001,3000))";
+my $recovery_time = psql $connstr_nodes{ $port_master }, "SELECT now()";
+my $lsn3 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Even more data, this time with a recovery target name
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))";
+my $recovery_name = "my_target";
+my $lsn4 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+psql $connstr_nodes{ $port_master },
+	"SELECT pg_create_restore_point('$recovery_name')";
+
+# Force archiving of WAL file
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $port_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+teardown_node($port_master);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..2455ca4
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,68 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+
+# Take backup
+my $backup_name = 'my_backup';
+backup_node($port_master, $backup_name);
+
+# Create two standbys linking to it
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_1);
+my $port_standby_2 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_2);
+
+# Create some content on master
+psql $connstr_nodes { $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+teardown_node($port_master);
+system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port_standby_1 },
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree("$datadir_nodes{ $port_standby_2 }/recovery.conf");
+append_to_file("$datadir_nodes{ $port_standby_2 }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_standby_1 }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+restart_node($port_standby_2);
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done.
+psql $connstr_nodes{ $port_standby_1 },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+$until_lsn = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT pg_current_xlog_location();";
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_2 })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(2000), 'check content of standby 2');
+
+# Stop nodes
+teardown_node($port_standby_2);
+teardown_node($port_standby_1);
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..4efd814
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,49 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+
+# And some content
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a";
+
+# Take backup
+my $backup_name = 'my_backup';
+backup_node($port_master, $backup_name);
+
+# Create streaming standby from backup
+my $port_standby = make_hot_standby($port_master, $backup_name);
+append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+recovery_min_apply_delay = '2s'
+));
+start_node($port_standby);
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(11,20))";
+sleep 1;
+# Here we should have only 10 rows
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+$result = psql $connstr_nodes{ $port_standby }, "SELECT count(*) FROM tab_int";
+is($result, qq(20), 'check content with delay of 2s');
+
+# Stop nodes
+teardown_node($port_standby);
+teardown_node($port_master);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index d3d736b..1355508 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -194,8 +194,9 @@ sub tapcheck
 	foreach my $test_path (@$tap_dirs)
 	{
 		# Like on Unix "make check-world", don't run the SSL test suite
-		# automatically.
+		# or the recovery test suite automatically.
 		next if ($test_path =~ /\/src\/test\/ssl\//);
+		next if ($test_path =~ /\/src\/test\/recovery\//);
 
 		my $dir = dirname($test_path);
 		chdir $dir;
#44Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#43)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/08/2015 08:19 AM, Michael Paquier wrote:

On Wed, Oct 7, 2015 at 5:44 PM, Amir Rohan wrote:

On 10/07/2015 10:29 AM, Michael Paquier wrote:

On Wed, Oct 7, 2015 at 4:16 PM, Amir Rohan wrote:

Also, the removal of poll_query_until from pg_rewind looks suspiciously
like a copy-paste gone bad. Pardon if I'm missing something.

Perhaps. Do you have a suggestion regarding that? It seems to me that
this is more useful in TestLib.pm as-is.

My mistake, the patch only shows some internal function being deleted
but RewindTest.pm (obviously) imports TestLib. You're right, TestLib is
a better place for it.

OK. Here is a new patch version. I have removed the restriction
preventing to call make_master multiple times in the same script (one
may actually want to test some stuff related to logical decoding or
FDW for example, who knows...), forcing PGHOST to always use the same
value after it has been initialized. I have added a sanity check
though, it is not possible to create a node based on a base backup if
no master are defined. This looks like a cheap insurance... I also
refactored a bit the code, using the new init_node_info to fill in the
fields of a newly-initialized node, and I removed get_free_port,
init_node, init_node_from_backup, enable_restoring and
enable_streaming from the list of routines exposed to the users, those
can be used directly with make_master, make_warm_standby and
make_hot_standby. We could add them again if need be, somebody may
want to be able to get a free port, set up a node without those
generic routines, just that it does not seem necessary now.
Regards,

If you'd like, I can write up some tests for cascading replication which
are currently missing.

Someone mentioned a daisy chain setup which sounds fun. Anything else in
particular? Also, it would be nice to have some canned way to measure
end-to-end replication latency for variable number of nodes.
What about going back through the commit log and writing some regression
tests for the real stinkers, if someone care to volunteer some candidate
bugs

Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#45Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#44)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Oct 8, 2015 at 3:59 PM, Amir Rohan <amir.rohan@zoho.com> wrote:

On 10/08/2015 08:19 AM, Michael Paquier wrote:

On Wed, Oct 7, 2015 at 5:44 PM, Amir Rohan wrote:

On 10/07/2015 10:29 AM, Michael Paquier wrote:

On Wed, Oct 7, 2015 at 4:16 PM, Amir Rohan wrote:

Also, the removal of poll_query_until from pg_rewind looks suspiciously
like a copy-paste gone bad. Pardon if I'm missing something.

Perhaps. Do you have a suggestion regarding that? It seems to me that
this is more useful in TestLib.pm as-is.

My mistake, the patch only shows some internal function being deleted
but RewindTest.pm (obviously) imports TestLib. You're right, TestLib is
a better place for it.

OK. Here is a new patch version. I have removed the restriction
preventing to call make_master multiple times in the same script (one
may actually want to test some stuff related to logical decoding or
FDW for example, who knows...), forcing PGHOST to always use the same
value after it has been initialized. I have added a sanity check
though, it is not possible to create a node based on a base backup if
no master are defined. This looks like a cheap insurance... I also
refactored a bit the code, using the new init_node_info to fill in the
fields of a newly-initialized node, and I removed get_free_port,
init_node, init_node_from_backup, enable_restoring and
enable_streaming from the list of routines exposed to the users, those
can be used directly with make_master, make_warm_standby and
make_hot_standby. We could add them again if need be, somebody may
want to be able to get a free port, set up a node without those
generic routines, just that it does not seem necessary now.
Regards,

If you'd like, I can write up some tests for cascading replication which
are currently missing.

001 is testing cascading, like that node1 -> node2 -> node3.

Someone mentioned a daisy chain setup which sounds fun. Anything else in
particular? Also, it would be nice to have some canned way to measure
end-to-end replication latency for variable number of nodes.

Hm. Do you mean comparing the LSN position between two nodes even if
both nodes are not connected to each other? What would you use it for?

What about going back through the commit log and writing some regression
tests for the real stinkers, if someone care to volunteer some candidate
bugs

I have drafted a list with a couple of items upthread:
/messages/by-id/CAB7nPqSgffSPhOcrhFoAsDAnipvn6WsH2nYkf1KayRm+9_MTGw@mail.gmail.com
So based on the existing patch the list becomes as follows:
- wal_retrieve_retry_interval with a high value, say setting to for
example 2/3s and loop until it is applied by checking it is it has
been received by the standby every second.
- recovery_target_action
- archive_cleanup_command
- recovery_end_command
- pg_xlog_replay_pause and pg_xlog_replay_resume
In the list of things that could have a test, I recall that we should
test as well 2PC with the recovery delay, look at a1105c3d. This could
be included in 005.
The advantage of implementing that now is that we could see if the
existing routines are solid enough or not. Still, looking at what the
patch has now I think that we had better get a committer look at it,
and if the core portion gets integrated we could already use it for
the patch implementing quorum synchronous replication and in doing
more advanced tests with pg_rewind regarding the timeline handling
(both patches of this CF). I don't mind adding more now, though I
think that the set of sample tests included in this version is enough
as a base implementation of the facility and shows what it can do.
Regards,
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#46Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#45)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/08/2015 10:39 AM, Michael Paquier wrote:

On Thu, Oct 8, 2015 at 3:59 PM, Amir Rohan <amir.rohan@zoho.com> wrote:

On 10/08/2015 08:19 AM, Michael Paquier wrote:

On Wed, Oct 7, 2015 at 5:44 PM, Amir Rohan wrote:

On 10/07/2015 10:29 AM, Michael Paquier wrote:

On Wed, Oct 7, 2015 at 4:16 PM, Amir Rohan wrote:

Also, the removal of poll_query_until from pg_rewind looks suspiciously
like a copy-paste gone bad. Pardon if I'm missing something.

Perhaps. Do you have a suggestion regarding that? It seems to me that
this is more useful in TestLib.pm as-is.

My mistake, the patch only shows some internal function being deleted
but RewindTest.pm (obviously) imports TestLib. You're right, TestLib is
a better place for it.

OK. Here is a new patch version. I have removed the restriction
preventing to call make_master multiple times in the same script (one
may actually want to test some stuff related to logical decoding or
FDW for example, who knows...), forcing PGHOST to always use the same
value after it has been initialized. I have added a sanity check
though, it is not possible to create a node based on a base backup if
no master are defined. This looks like a cheap insurance... I also
refactored a bit the code, using the new init_node_info to fill in the
fields of a newly-initialized node, and I removed get_free_port,
init_node, init_node_from_backup, enable_restoring and
enable_streaming from the list of routines exposed to the users, those
can be used directly with make_master, make_warm_standby and
make_hot_standby. We could add them again if need be, somebody may
want to be able to get a free port, set up a node without those
generic routines, just that it does not seem necessary now.
Regards,

If you'd like, I can write up some tests for cascading replication which
are currently missing.

001 is testing cascading, like that node1 -> node2 -> node3.

Someone mentioned a daisy chain setup which sounds fun. Anything else in
particular? Also, it would be nice to have some canned way to measure
end-to-end replication latency for variable number of nodes.

Hm. Do you mean comparing the LSN position between two nodes even if
both nodes are not connected to each other? What would you use it for?

In a cascading replication setup, the typical _time_ it takes for a
COMMIT on master to reach the slave (assuming constant WAL generation
rate) is an important operational metric.

It would be useful to catch future regressions for that metric,
which may happen even when a patch doesn't outright break cascading
replication. Just automating the measurement could be useful if
there's no pg facility that tracks performance over time in
a regimented fashion. I've seen multiple projects which consider
a "benchmark suite" to be part of its testing strategy.

As for the "daisy chain" thing, it was (IIRC) mentioned in a josh berkus
talk I caught on youtube. It's possible to setup cascading replication,
take down the master, and then reinsert it as replicating slave, so that
you end up with *all* servers replicating from the
ancestor in the chain, and no master. I think it was more
a fun hack then anything, but also an interesting corner case to
investigate.

What about going back through the commit log and writing some regression
tests for the real stinkers, if someone care to volunteer some candidate
bugs

I have drafted a list with a couple of items upthread:
/messages/by-id/CAB7nPqSgffSPhOcrhFoAsDAnipvn6WsH2nYkf1KayRm+9_MTGw@mail.gmail.com
So based on the existing patch the list becomes as follows:
- wal_retrieve_retry_interval with a high value, say setting to for
example 2/3s and loop until it is applied by checking it is it has
been received by the standby every second.
- recovery_target_action
- archive_cleanup_command
- recovery_end_command
- pg_xlog_replay_pause and pg_xlog_replay_resume
In the list of things that could have a test, I recall that we should
test as well 2PC with the recovery delay, look at a1105c3d. This could
be included in 005.

a1105c3 Mar 23 Fix copy & paste error in 4f1b890b137. Andres Freund
4f1b890 Mar 15 Merge the various forms of transaction commit & abort
records. Andres Freund

Is that the right commit?

The advantage of implementing that now is that we could see if the
existing routines are solid enough or not.

I can do this if you point me at a self-contained thread/#issue.

Still, looking at what the
patch has now I think that we had better get a committer look at it,
and if the core portion gets integrated we could already use it for
the patch implementing quorum synchronous replication and in doing
more advanced tests with pg_rewind regarding the timeline handling
(both patches of this CF).

I don't mind adding more now, though I
think that the set of sample tests included in this version is enough
as a base implementation of the facility and shows what it can do.

Sure.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#47Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#46)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Oct 8, 2015 at 6:03 PM, Amir Rohan wrote:

On 10/08/2015 10:39 AM, Michael Paquier wrote:

Someone mentioned a daisy chain setup which sounds fun. Anything else in
particular? Also, it would be nice to have some canned way to measure
end-to-end replication latency for variable number of nodes.

Hm. Do you mean comparing the LSN position between two nodes even if
both nodes are not connected to each other? What would you use it for?

In a cascading replication setup, the typical _time_ it takes for a
COMMIT on master to reach the slave (assuming constant WAL generation
rate) is an important operational metric.

Hm. You mean the exact amount of time it gets to be sure that a given
WAL position has been flushed on a cascading standby, be it across
multiple layers. Er, that's a bit tough without patching the backend
where I guess we would need to keep a track of when a LSN position has
been flushed. And calls of gettimeofday are expensive, so that does
not sound like a plausible alternative here to me...

It would be useful to catch future regressions for that metric,
which may happen even when a patch doesn't outright break cascading
replication. Just automating the measurement could be useful if
there's no pg facility that tracks performance over time in
a regimented fashion. I've seen multiple projects which consider
a "benchmark suite" to be part of its testing strategy.

Ah, OK. I see. That's a bit out of scope of this patch, and that's
really OS-dependent, but as long as the comparisons can be done on the
same OS it would make sense.

As for the "daisy chain" thing, it was (IIRC) mentioned in a josh berkus
talk I caught on youtube. It's possible to setup cascading replication,
take down the master, and then reinsert it as replicating slave, so that
you end up with *all* servers replicating from the
ancestor in the chain, and no master. I think it was more
a fun hack then anything, but also an interesting corner case to
investigate.

Ah, yes. I recall this one. I am sure it made the audience smile. All
the nodes link to each other in closed circle.

What about going back through the commit log and writing some regression
tests for the real stinkers, if someone care to volunteer some candidate
bugs

I have drafted a list with a couple of items upthread:
/messages/by-id/CAB7nPqSgffSPhOcrhFoAsDAnipvn6WsH2nYkf1KayRm+9_MTGw@mail.gmail.com
So based on the existing patch the list becomes as follows:
- wal_retrieve_retry_interval with a high value, say setting to for
example 2/3s and loop until it is applied by checking it is it has
been received by the standby every second.
- recovery_target_action
- archive_cleanup_command
- recovery_end_command
- pg_xlog_replay_pause and pg_xlog_replay_resume
In the list of things that could have a test, I recall that we should
test as well 2PC with the recovery delay, look at a1105c3d. This could
be included in 005.

a1105c3 Mar 23 Fix copy & paste error in 4f1b890b137. Andres Freund
4f1b890 Mar 15 Merge the various forms of transaction commit & abort
records. Andres Freund

Is that the right commit?

That's this one. a1105c3 was actually rather tricky... The idea is to
simply check the WAL replay delay with COMMIT PREPARED.

The advantage of implementing that now is that we could see if the
existing routines are solid enough or not.

I can do this if you point me at a self-contained thread/#issue.

Hm. This patch is already 900 lines, perhaps it would be wiser not to
make it more complicated for now..
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#48Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#47)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/08/2015 04:47 PM, Michael Paquier wrote:

On Thu, Oct 8, 2015 at 6:03 PM, Amir Rohan wrote:

On 10/08/2015 10:39 AM, Michael Paquier wrote:

Someone mentioned a daisy chain setup which sounds fun. Anything else in
particular? Also, it would be nice to have some canned way to measure
end-to-end replication latency for variable number of nodes.

Hm. Do you mean comparing the LSN position between two nodes even if
both nodes are not connected to each other? What would you use it for?

In a cascading replication setup, the typical _time_ it takes for a
COMMIT on master to reach the slave (assuming constant WAL generation
rate) is an important operational metric.

Hm. You mean the exact amount of time it gets to be sure that a given
WAL position has been flushed on a cascading standby, be it across
multiple layers. Er, that's a bit tough without patching the backend
where I guess we would need to keep a track of when a LSN position has
been flushed. And calls of gettimeofday are expensive, so that does
not sound like a plausible alternative here to me...

Wouldn't this work?

1) start timer
2) Grab pg_stat_replication.sent_location from master
3) pg_switch_xlog() # I /think/ we want this, could be wrong
4) Poll slave's pg_last_xlog_replay_location() until LSN shows up
5) stop timer

Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#49Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#48)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Oct 8, 2015 at 11:28 PM, Amir Rohan wrote:

Wouldn't this work?
1) start timer
2) Grab pg_stat_replication.sent_location from master
3) pg_switch_xlog() # I /think/ we want this, could be wrong

For a warm standby, you would want that, but this depends on two factors:
- The moment master completes archiving of this segment
- The moment standby restores it.
On slow machines, those two things become by far the bottleneck,
imagine a PI restricted on I/O with a low-class SD card in the worst
case (I maintain one, with a good card, still the I/O is a
bottleneck).

4) Poll slave's pg_last_xlog_replay_location() until LSN shows up
5) stop timer

That's not really solid, there is an interval of time between the
moment the LSN position is taken from the master and the standby. An
accurate method is to log/store on master when a given WAL position
has been flushed to disk, and do the same on slave at replay for this
LSN position. In any case this is doing to flood badly the logs of
both nodes, and as the backend cares about the performance of
operations in this code path we won't want to do that anyway.

To make it short, it seems to me that simply waiting until the LSN a
test is waiting for has been replayed is just but fine for this set of
tests to ensure their run consistency, let's not forget that this is
the goal here.
Regards,
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#50Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#49)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Oct 9, 2015 at 8:25 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Thu, Oct 8, 2015 at 11:28 PM, Amir Rohan wrote:

Wouldn't this work?
1) start timer
2) Grab pg_stat_replication.sent_location from master
3) pg_switch_xlog() # I /think/ we want this, could be wrong

For a warm standby, you would want that, but this depends on two factors:
- The moment master completes archiving of this segment
- The moment standby restores it.
On slow machines, those two things become by far the bottleneck,
imagine a PI restricted on I/O with a low-class SD card in the worst
case (I maintain one, with a good card, still the I/O is a
bottleneck).

4) Poll slave's pg_last_xlog_replay_location() until LSN shows up
5) stop timer

That's not really solid, there is an interval of time between the
moment the LSN position is taken from the master and the standby. An
accurate method is to log/store on master when a given WAL position
has been flushed to disk, and do the same on slave at replay for this
LSN position. In any case this is doing to flood badly the logs of
both nodes, and as the backend cares about the performance of
operations in this code path we won't want to do that anyway.

To make it short, it seems to me that simply waiting until the LSN a
test is waiting for has been replayed is just but fine for this set of
tests to ensure their run consistency, let's not forget that this is
the goal here.

In terms of features, it seems that this patch has everything it needs
to allow one to design tests to work on both Linux and Windows, and it
is careful regarding CVE-2014-0067. Thoughts about moving that as
"Ready for committer"?
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#51Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#50)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/09/2015 02:12 PM, Michael Paquier wrote:

On Fri, Oct 9, 2015 at 8:25 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Thu, Oct 8, 2015 at 11:28 PM, Amir Rohan wrote:

Wouldn't this work?
1) start timer
2) Grab pg_stat_replication.sent_location from master
3) pg_switch_xlog() # I /think/ we want this, could be wrong

For a warm standby, you would want that, but this depends on two factors:
- The moment master completes archiving of this segment
- The moment standby restores it.
On slow machines, those two things become by far the bottleneck,
imagine a PI restricted on I/O with a low-class SD card in the worst
case (I maintain one, with a good card, still the I/O is a
bottleneck).

4) Poll slave's pg_last_xlog_replay_location() until LSN shows up
5) stop timer

That's not really solid, there is an interval of time between the
moment the LSN position is taken from the master and the standby. An
accurate method is to log/store on master when a given WAL position
has been flushed to disk, and do the same on slave at replay for this
LSN position. In any case this is doing to flood badly the logs of
both nodes, and as the backend cares about the performance of
operations in this code path we won't want to do that anyway.

To make it short, it seems to me that simply waiting until the LSN a
test is waiting for has been replayed is just but fine for this set of
tests to ensure their run consistency, let's not forget that this is
the goal here.

In terms of features, it seems that this patch has everything it needs
to allow one to design tests to work on both Linux and Windows, and it
is careful regarding CVE-2014-0067. Thoughts about moving that as
"Ready for committer"?

Ok, I've put myself down as reviewer in cfapp. I don't think I can
provide any more useful feedback that would actually result in changes
at this point, but I'll read through the entire discussion once last
time and write down final comments/notes. After that I have no problem
marking this for a committer to look at.

Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#52Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#51)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Oct 9, 2015 at 8:47 PM, Amir Rohan wrote:

Ok, I've put myself down as reviewer in cfapp. I don't think I can
provide any more useful feedback that would actually result in changes
at this point, but I'll read through the entire discussion once last
time and write down final comments/notes. After that I have no problem
marking this for a committer to look at.

OK. If you have any comments or remarks, please do not hesitate at all!
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#53Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#52)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Oct 9, 2015 at 8:53 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Fri, Oct 9, 2015 at 8:47 PM, Amir Rohan wrote:

Ok, I've put myself down as reviewer in cfapp. I don't think I can
provide any more useful feedback that would actually result in changes
at this point, but I'll read through the entire discussion once last
time and write down final comments/notes. After that I have no problem
marking this for a committer to look at.

OK. If you have any comments or remarks, please do not hesitate at all!

So, to let everybody know the issue, Amir has reported me offlist a
bug in one of the tests that can be reproduced more easily on a slow
machine:

Amir wrote:
Before posting the summary, I ran the latest v8 patch on today's git
master (9c42727) and got some errors:
t/004_timeline_switch.pl ...
1..1
# ERROR: invalid input syntax for type pg_lsn: ""
# LINE 1: SELECT ''::pg_lsn <= pg_last_xlog_replay_location()
# ^
# No tests run!

And here is my reply:
This is a timing issue and can happen when standby1, the promoted
standby which standby2 reconnects to to check that recovery works with
a timeline jump, is still in recovery after being restarted. There is
a small windows where this is possible, and this gets easier to
reproduce on slow machines (did so on a VM). So the issue was in test
004. I have updated the script to check pg_is_in_recovery() to be sure
that the node exits recovery before querying it with
pg_current_xlog_location.

It is worth noticing that the following change has saved me a lot of pain:
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -259,6 +259,7 @@ sub psql
        my ($stdout, $stderr);
        print("# Running SQL command: $sql\n");
        run [ 'psql', '-X', '-A', '-t', '-q', '-d', $dbname, '-f',
'-'], '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
+       print "# Error output: $stderr\n" if $stderr ne "";
Perhaps we should consider backpatching it, it helped me find out the
issue I faced.

Attached is an updated patch fixing 004.
Regards,
--
Michael

Attachments:

20151010_recovery_regressions_v9.patchapplication/x-patch; name=20151010_recovery_regressions_v9.patchDownload
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..ea219d7 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -125,38 +125,6 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
 sub append_to_file
 {
 	my ($filename, $str) = @_;
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..d6e51eb 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -17,7 +17,7 @@ SUBDIRS = regress isolation modules
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
 # because the SSL test suite is not secure to run on a multi-user system.
-ALWAYS_SUBDIRS = examples locale thread ssl
+ALWAYS_SUBDIRS = examples locale thread ssl recovery
 
 # We want to recurse to all subdirs for all standard targets, except that
 # installcheck and install should not recurse into the subdirectory "modules".
diff --git a/src/test/perl/RecoveryTest.pm b/src/test/perl/RecoveryTest.pm
new file mode 100644
index 0000000..b60bf5c
--- /dev/null
+++ b/src/test/perl/RecoveryTest.pm
@@ -0,0 +1,412 @@
+package RecoveryTest;
+
+# Set of common routines for recovery regression tests for a PostgreSQL
+# cluster. This includes global variables and methods that can be used
+# by the various set of tests present to set up cluster nodes and
+# configure them according to the test scenario wanted.
+#
+# Cluster nodes can be freely created using initdb or using the existing
+# base backup of another node, with minimum configuration done when the
+# node is created for the first time like having a proper port number.
+# It is then up to the test to decide what to do with the newly-created
+# node.
+#
+# Environment configuration of each node is available through a set
+# of global variables provided by this package, hashed depending on the
+# port number of a node:
+# - connstr_nodes connection string to connect to this node
+# - datadir_nodes to get the data folder of a given node
+# - archive_nodes for the location of the WAL archives of a node
+# - backup_nodes for the location of base backups of a node
+# - applname_nodes, application_name to use for a standby
+#
+# Nodes are identified by their port number, which should be unique
+# for each node of the cluster as it is run locally.
+
+use Cwd;
+use TestLib;
+use Test::More;
+
+use Archive::Tar;
+use IPC::Run qw(run start);
+
+use Exporter 'import';
+
+our @EXPORT = qw(
+	%connstr_nodes
+	%datadir_nodes
+	%backup_nodes
+	%archive_nodes
+	%applname_nodes
+
+	append_to_file
+	backup_node
+	disable_node
+	dump_node_info
+	enable_archiving
+	enable_node
+	enable_restoring
+	enable_streaming
+	get_free_port
+	init_node
+	init_node_from_backup
+	make_master
+	make_warm_standby
+	make_hot_standby
+	restart_node
+	start_node
+	stop_node
+	teardown_node
+);
+
+# Global variables for node data
+%datadir_nodes = {};	# PGDATA folders
+%backup_nodes = {};		# Backup base folder
+%archive_nodes = {};	# Archive base folder
+%connstr_nodes = {};	# Connection strings
+%applname_nodes = {};	# application_name used for standbys
+
+# Tracking of last port value assigned to accelerate free port lookup.
+# XXX: Should this part use PG_VERSION_NUM?
+my $last_port_assigned =  90600 % 16384 + 49152;
+
+# Value tracking the host value used for a single run.
+my $node_pghost = undef;
+
+# Database used for each connection attempt via psql
+$ENV{PGDATABASE} = "postgres";
+
+# Tracker of active nodes
+my @active_nodes = ();
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_root } application_name=$applname_nodes{ $port_standby }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $port_root = shift; # Instance to link to
+	my $port_standby = shift;
+	my $path = $archive_nodes{ $port_root };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"$path\\\\%f\" \"%p\"" :
+		"cp -i $path/%f %p";
+	append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+restore_command='$copy_command'
+standby_mode=on
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $port = shift;
+	my $path = $archive_nodes{ $port };
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$path\\\\%f\"" :
+		"cp %p $path/%f";
+
+	# Enable archive_mode and archive_command on node
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization. This should be called only once in a series
+# of tests because the rest of the logic relies on PGHOST being set only
+# once. This would not be an issue on Linux where one socket path per
+# node could be used by on Windows listen_addresses needs to be set
+# to look at 127.0.0.1.
+sub make_master
+{
+	my $port_master = get_free_port();
+	print "# Initializing master node wih port $port_master\n";
+	init_node($port_master);
+	return $port_master;
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_hot_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing streaming mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, streaming from first one
+	enable_streaming($port_master, $port_standby);
+	return $port_standby;
+}
+
+# Node getting WAL only from archives
+sub make_warm_standby
+{
+	my $port_master = shift;
+	my $backup_name = shift;
+	my $port_standby = get_free_port();
+
+	print "# Initializing archive mode for node $port_standby from node $port_master\n";
+	init_node_from_backup($port_standby, $port_master, $backup_name);
+
+	# Start second node, restoring from first one
+	enable_restoring($port_master, $port_standby);
+	return $port_standby;
+}
+
+sub configure_base_node
+{
+	my $port = shift;
+
+	# Make configuration somewhat generic to test recovery
+	append_to_file("$datadir_nodes{ $port }/postgresql.conf", qq(
+port = $port
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+));
+
+	configure_hba_for_replication($datadir_nodes{ $port });
+}
+
+# Get a port number not in use currently for a new node
+# As port number retrieval is based on the nodes currently running and
+# their presence in the list of registered ports, be sure that the node
+# that is consuming this port number has already been started and that
+# it is not registered yet.
+sub get_free_port
+{
+	my $found = 0;
+	my $port = $last_port_assigned;
+
+	while ($found == 0)
+	{
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		my $ret = system("psql -X -p $port postgres < $devnull");
+		if ($ret != 0)
+		{
+			# Found a potential candidate, check first that it is
+			# not included in the list of registered nodes.
+			$found = 1 if ! grep (/$port/, @array);
+		}
+	}
+
+	print "# Found free port $port\n";
+	# Lock port number found
+	enable_node($port);
+	$last_port_assigned = $port;
+	return $port;
+}
+
+# Low-level routines to initialize a node
+# Initialize a node from scratch
+sub init_node
+{
+	my $port = shift;
+
+	if (! defined($node_pghost))
+	{
+		$node_pghost = $windows_os ? "127.0.0.1" : tempdir_short;
+	}
+
+	init_node_info($port);
+	mkdir $backup_nodes{ $port };
+	mkdir $archive_nodes{ $port };
+
+	standard_initdb($datadir_nodes{ $port }, $node_pghost);
+	configure_base_node($port);
+}
+
+# Initialize all the information of a node freshly created in the
+# common set of variables for the whole set.
+sub init_node_info
+{
+	my $port = shift;
+	my $base_dir = TestLib::tempdir;
+
+	$datadir_nodes{ $port } = "$base_dir/pgdata";
+	$backup_nodes{ $port } = "$base_dir/backup";
+	$archive_nodes{ $port } =  "$base_dir/archives";
+	$connstr_nodes{ $port } = "port=$port host=$node_pghost";
+	$applname_nodes{ $port } = "node_$port";
+
+	# Log some information
+	dump_node_info($port);
+}
+
+# Initialize a node from an existing base backup
+sub init_node_from_backup
+{
+	my ($port, $root_port, $backup_name) = @_;
+
+	my $backup_path =  "$backup_nodes{ $root_port }/$backup_name";
+	my $backup_file = "$backup_path/base.tar";
+
+	die "Initializing node from backup but no master defined"
+		if (! defined($node_pghost));
+
+	print "Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	# Check existence of backup wanted
+	if ( ! -d $backup_path )
+	{
+		die "Backup $backup_path does not exist";
+	}
+
+	# Save configuration information
+	init_node_info($port);
+	mkdir $datadir_nodes{ $port }, 0700;
+	mkdir $backup_nodes{ $port };
+	mkdir $archive_nodes{ $port };
+
+	# Temporary move to the place of extraction
+	chdir "$datadir_nodes{ $port }";
+	Archive::Tar->extract_archive($backup_file);
+	chdir "$current_dir";
+	configure_base_node($port);
+}
+
+# Start a node
+sub start_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port },
+				   '-l', "$log_path/node_$port.log",
+				   'start');
+}
+
+# Stop a node. This routine can be called during a test.
+sub stop_node
+{
+	my $port = shift;
+	my $mode = shift || "fast";
+
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   $mode, 'stop');
+}
+
+sub restart_node
+{
+	my $port = shift;
+	system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port }, '-m',
+				   'fast', 'restart');
+}
+
+# Wait until a node is able to accept queries. Useful have putting a node
+# in recovery.
+sub wait_for_node
+{
+	my $port         = shift;
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	while ($attempts < $max_attempts)
+	{
+		if (run_log('pg_isready', '-p', $port))
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+	return 0;
+}
+
+# Mark given node as active, making the port number used by this node
+# locked for this test.
+sub enable_node
+{
+	my $port = shift;
+	print "# Port $port has been locked for new node\n";
+	push(@active_nodes, $port);
+}
+
+# Disactivate given node, node should have been stopped before calling
+# this routine.
+sub disable_node
+{
+	my $port = shift;
+	print "# Port $port has been disabled\n";
+	@active_nodes = grep { $_ ne $port } @active_nodes;
+}
+
+# Remove any traces of given node.
+sub teardown_node
+{
+	my $port = shift;
+	system('pg_ctl', '-D', $datadir_nodes{ $port }, '-m',
+		   'immediate', 'stop');
+	disable_node($port);
+}
+
+# Create a backup on a node already running
+sub backup_node
+{
+	my ($port, $backup_name) = @_;
+
+	my $backup_path = "$backup_nodes{ $port }/$backup_name";
+
+	print "Taking backup from node $port\n";
+	# Backup a node in tar format, it is more portable across platforms
+	system_or_bail("pg_basebackup -D $backup_path -p $port --format=t -x");
+	print "# Backup finished\n"
+}
+
+# Dump information of a node
+sub dump_node_info
+{
+	my $port = shift;
+	print "New node initialized: $port\n";
+	print "Data directory: $datadir_nodes{ $port }\n";
+	print "Backup directory: $backup_nodes{ $port }\n";
+	print "Archive directory: $archive_nodes{ $port }\n";
+	print "Connection string: $connstr_nodes{ $port }\n";
+	print "Application name: $applname_nodes{ $port }\n";
+}
+
+# Add a set of parameters to a configuration file
+sub append_to_file
+{
+	my($filename, $str) = @_;
+
+	open my $fh, ">>", $filename or die "could not open file $filename";
+	print $fh $str;
+	close $fh;
+}
+
+END
+{
+	foreach my $port (@active_nodes)
+	{
+		teardown_node($port);
+	}
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..66fc750 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -12,6 +12,7 @@ our @EXPORT = qw(
   configure_hba_for_replication
   start_test_server
   restart_test_server
+  poll_query_until
   psql
   slurp_dir
   slurp_file
@@ -136,11 +137,12 @@ sub tempdir_short
 sub standard_initdb
 {
 	my $pgdata = shift;
+	my $tempdir_short = shift;
+
+	$tempdir_short = tempdir_short if (! defined($tempdir_short));
 	system_or_bail('initdb', '-D', "$pgdata", '-A' , 'trust', '-N');
 	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
 
-	my $tempdir_short = tempdir_short;
-
 	open CONF, ">>$pgdata/postgresql.conf";
 	print CONF "\n# Added by TestLib.pm)\n";
 	print CONF "fsync = off\n";
@@ -220,12 +222,44 @@ END
 	}
 }
 
+sub poll_query_until
+{
+	my ($query, $connstr) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
+		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
 sub psql
 {
 	my ($dbname, $sql) = @_;
 	my ($stdout, $stderr);
 	print("# Running SQL command: $sql\n");
 	run [ 'psql', '-X', '-A', '-t', '-q', '-d', $dbname, '-f', '-' ], '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
+	print "# Error output: $stderr\n" if $stderr ne "";
 	chomp $stdout;
 	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
 	return $stdout;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..fd4bfbf
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,62 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+my $backup_name = 'my_backup';
+
+# Take backup
+backup_node($port_master, $backup_name);
+
+# Create streaming standby linking to master
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_1);
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+backup_node($port_standby_1, $backup_name);
+
+# Create second standby node linking to standby 1
+my $port_standby_2 = make_hot_standby($port_standby_1, $backup_name);
+start_node($port_standby_2);
+
+# Create some content on master and check its presence in standby 1 an
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a";
+
+# Wait for standbys to catch up
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_nodes{ $port_standby_1 }';";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_master })
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_nodes{ $port_standby_2 }';";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT count(*) FROM tab_int";
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_1 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	"-d $connstr_nodes{ $port_standby_2 }", '-c', "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
+
+# Cleanup nodes
+teardown_node($port_standby_2);
+teardown_node($port_standby_1);
+teardown_node($port_master);
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..c7977f0
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,51 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $port_master = make_master();
+my $backup_name = 'my_backup';
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Take backup for slave
+backup_node($port_master, $backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $port_standby = make_warm_standby($port_master, $backup_name);
+append_to_file("$datadir_nodes{ $port_standby }/postgresql.conf", qq(
+wal_retrieve_retry_interval = '100ms'
+));
+start_node($port_standby);
+
+# Create some content on master
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $current_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Force archiving of WAL file to make it present on master
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Add some more content, it should not be present on standby
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(1000), 'check content from archives');
+
+# Cleanup nodes
+teardown_node($port_standby);
+teardown_node($port_master);
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..94b94a1
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,135 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $port_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $port_standby = make_warm_standby($port_master, 'my_backup');
+
+	foreach my $param_item (@$recovery_params)
+	{
+		append_to_file("$datadir_nodes{ $port_standby }/recovery.conf",
+					   qq($param_item
+));
+	}
+
+	start_node($port_standby);
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = psql $connstr_nodes{ $port_standby },
+		"SELECT count(*) FROM tab_int";
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	teardown_node($port_standby);
+}
+
+# Initialize master node
+my $port_master = make_master();
+enable_archiving($port_master);
+
+# Start it
+start_node($port_master);
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $lsn1 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Take backup from which all operations will be run
+backup_node($port_master, 'my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+my $recovery_txid = psql $connstr_nodes{ $port_master },
+	"SELECT txid_current()";
+my $lsn2 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# More data, with recovery target timestamp
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(2001,3000))";
+my $recovery_time = psql $connstr_nodes{ $port_master }, "SELECT now()";
+my $lsn3 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Even more data, this time with a recovery target name
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))";
+my $recovery_name = "my_target";
+my $lsn4 = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+psql $connstr_nodes{ $port_master },
+	"SELECT pg_create_restore_point('$recovery_name')";
+
+# Force archiving of WAL file
+psql $connstr_nodes{ $port_master }, "SELECT pg_switch_xlog()";
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $port_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $port_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $port_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $port_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+teardown_node($port_master);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..77ebd22
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,71 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+
+# Take backup
+my $backup_name = 'my_backup';
+backup_node($port_master, $backup_name);
+
+# Create two standbys linking to it
+my $port_standby_1 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_1);
+my $port_standby_2 = make_hot_standby($port_master, $backup_name);
+start_node($port_standby_2);
+
+# Create some content on master
+psql $connstr_nodes { $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_1 })
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+teardown_node($port_master);
+system_or_bail('pg_ctl', '-w', '-D', $datadir_nodes{ $port_standby_1 },
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree("$datadir_nodes{ $port_standby_2 }/recovery.conf");
+append_to_file("$datadir_nodes{ $port_standby_2 }/recovery.conf", qq(
+primary_conninfo='$connstr_nodes{ $port_standby_1 }'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+restart_node($port_standby_2);
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done. Standby 1 needs
+# to exit recovery first before moving on with the test.
+poll_query_until("SELECT pg_is_in_recovery() <> true",
+				 $connstr_nodes{ $port_standby_1 });
+psql $connstr_nodes{ $port_standby_1 },
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+$until_lsn = psql $connstr_nodes{ $port_standby_1 },
+	"SELECT pg_current_xlog_location();";
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby_2 })
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $connstr_nodes{ $port_standby_2 },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(2000), 'check content of standby 2');
+
+# Stop nodes
+teardown_node($port_standby_2);
+teardown_node($port_standby_1);
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..4efd814
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,49 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $port_master = make_master();
+start_node($port_master);
+
+# And some content
+psql $connstr_nodes{ $port_master },
+	"CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a";
+
+# Take backup
+my $backup_name = 'my_backup';
+backup_node($port_master, $backup_name);
+
+# Create streaming standby from backup
+my $port_standby = make_hot_standby($port_master, $backup_name);
+append_to_file("$datadir_nodes{ $port_standby }/recovery.conf", qq(
+recovery_min_apply_delay = '2s'
+));
+start_node($port_standby);
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+psql $connstr_nodes{ $port_master },
+	"INSERT INTO tab_int VALUES (generate_series(11,20))";
+sleep 1;
+# Here we should have only 10 rows
+my $result = psql $connstr_nodes{ $port_standby },
+	"SELECT count(*) FROM tab_int";
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = psql $connstr_nodes{ $port_master },
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $connstr_nodes{ $port_standby })
+	or die "Timed out while waiting for standby to catch up";
+$result = psql $connstr_nodes{ $port_standby }, "SELECT count(*) FROM tab_int";
+is($result, qq(20), 'check content with delay of 2s');
+
+# Stop nodes
+teardown_node($port_standby);
+teardown_node($port_master);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index d3d736b..1355508 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -194,8 +194,9 @@ sub tapcheck
 	foreach my $test_path (@$tap_dirs)
 	{
 		# Like on Unix "make check-world", don't run the SSL test suite
-		# automatically.
+		# or the recovery test suite automatically.
 		next if ($test_path =~ /\/src\/test\/ssl\//);
+		next if ($test_path =~ /\/src\/test\/recovery\//);
 
 		my $dir = dirname($test_path);
 		chdir $dir;
#54Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#53)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/10/2015 02:43 PM, Michael Paquier wrote:

On Fri, Oct 9, 2015 at 8:53 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Fri, Oct 9, 2015 at 8:47 PM, Amir Rohan wrote:

Ok, I've put myself down as reviewer in cfapp. I don't think I can
provide any more useful feedback that would actually result in changes
at this point, but I'll read through the entire discussion once last
time and write down final comments/notes. After that I have no problem
marking this for a committer to look at.

OK. If you have any comments or remarks, please do not hesitate at all!

So, to let everybody know the issue, Amir has reported me offlist a
bug in one of the tests that can be reproduced more easily on a slow
machine:

Yeah, I usually stick to the list for discussion, but I ran an earlier
version without issues and thought this might be a problem with my
system as I've changed things a bit this week.

Now that v9 fixes the probkem, here's a summary from going over the
entire thread one last time:

# Windows and TAP sets
Noah (2015-03) mentioned TAP doesn't work on windows, and hoped
this would include some work on that.

IIUC, the facilities and tests do run on windows, but focus was there
and not the preexisting TAP suite.

# Test coverage (in the future)
Andres wanted a test for xid/multixid wraparound which also raises
the question of the tests that will need to be written in the future.

The patch focuses on providing facilities, while providing new coverage
for several features. There should be a TODO list on the wiki (bug
tracker, actually), where the list of tests to be written can be managed.

Some were mentioned in the thread (multi/xid wraparound
hot_standby_feedback, max_standby_archive_delay and
max_standby_streaming_delay? recovery_target_action? some in your
original list?), but threads
are precisely where these things get lost in the cracks.

# Interactive use vs. TAP tests

Early on the goal was also to provide something for interactive use
in order to test scenarios. The shift has focused to the TAP tests
and some of the choices in the API reflect that. Interactive use
is possible, but wasn't a central requirement.

# Directory structure

I suggested keeping backup/log/PGDATA per instance, rejected.

# Parallel tests and port collisions

Lots about this. Final result is no port races are possible because
dedicated dirs are used per test, per instance. And because tcp
isn't used for connections on any platform (can you confirm that's
true on windows as well? I'm not familiar with sspi and what OSI
layer it lives on)

# Allow test to specify shutdown mode

Added

# decouple cleanup from node shutdown

Added (in latest patches?)

# Conveniences for test writing vs. running

My suggestions weren't picked up, but for one thing setting CLEANUP=0
in the lib (which means editing it...) can be useful for writers.

# blocking until server ready

pg_isready wrapper added.

# Multiple masters

back and forth, but supported in latest version.

That's it. I've ran the latest (v9) tests works and passed on my system
(fedora 64bit) and also under docker with --cpu-quota=10000, which
simulates a slow machine.

Michael, is there anything else to do here or shall I mark this for
committer review?

Regards,
Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#55Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#54)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Oct 10, 2015 at 9:04 PM, Amir Rohan wrote:

Now that v9 fixes the probkem, here's a summary from going over the
entire thread one last time:

Thanks a lot for the summary of the events.

# Windows and TAP sets
Noah (2015-03) mentioned TAP doesn't work on windows, and hoped
this would include some work on that.
IIUC, the facilities and tests do run on windows, but focus was there
and not the preexisting TAP suite.

They do work on Windows, see 13d856e.

# Test coverage (in the future)
Andres wanted a test for xid/multixid wraparound which also raises
the question of the tests that will need to be written in the future.

I recall that this would have needed extra functions on the backend...

The patch focuses on providing facilities, while providing new coverage
for several features. There should be a TODO list on the wiki (bug
tracker, actually), where the list of tests to be written can be managed.
Some were mentioned in the thread (multi/xid wraparound
hot_standby_feedback, max_standby_archive_delay and
max_standby_streaming_delay? recovery_target_action? some in your
original list?), but threads
are precisely where these things get lost in the cracks.

Sure, that's an on-going task.

# Directory structure
I suggested keeping backup/log/PGDATA per instance, rejected.

I guess that I am still flexible on this one, the node information
(own PGDATA, connection string, port, etc.) is logged as well so this
is not a big deal to me...

# Parallel tests and port collisions
Lots about this. Final result is no port races are possible because
dedicated dirs are used per test, per instance. And because tcp
isn't used for connections on any platform (can you confirm that's
true on windows as well? I'm not familiar with sspi and what OSI
layer it lives on)

On Windows you remain with the problem that all nodes initialized
using TestLib.pm will listen to 127.0.0.1, sspi being used to ensure
that the connection at user level is secure (additional entries in
pg_hba.conf are added).

# decouple cleanup from node shutdown
Added (in latest patches?)

Yes this was added.

Michael, is there anything else to do here or shall I mark this for
committer review?

I have nothing else. Thanks a lot!
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#56Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#55)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/10/2015 04:32 PM, Michael Paquier wrote:

On Sat, Oct 10, 2015 at 9:04 PM, Amir Rohan wrote:

Now that v9 fixes the problem, here's a summary from going over the
entire thread one last time:

Thanks a lot for the summary of the events.

# Windows and TAP sets
Noah (2015-03) mentioned TAP doesn't work on windows, and hoped
this would include some work on that.
IIUC, the facilities and tests do run on windows, but focus was there
and not the preexisting TAP suite.

They do work on Windows, see 13d856e.

Thanks, I did not know that.

# Test coverage (in the future)
Andres wanted a test for xid/multixid wraparound which also raises
the question of the tests that will need to be written in the future.

I recall that this would have needed extra functions on the backend...

The patch focuses on providing facilities, while providing new coverage
for several features. There should be a TODO list on the wiki (bug
tracker, actually), where the list of tests to be written can be managed.
Some were mentioned in the thread (multi/xid wraparound
hot_standby_feedback, max_standby_archive_delay and
max_standby_streaming_delay? recovery_target_action? some in your
original list?), but threads
are precisely where these things get lost in the cracks.

Sure, that's an on-going task.

# Directory structure
I suggested keeping backup/log/PGDATA per instance, rejected.

I guess that I am still flexible on this one, the node information
(own PGDATA, connection string, port, etc.) is logged as well so this
is not a big deal to me...

# Parallel tests and port collisions
Lots about this. Final result is no port races are possible because
dedicated dirs are used per test, per instance. And because tcp
isn't used for connections on any platform (can you confirm that's
true on windows as well? I'm not familiar with sspi and what OSI
layer it lives on)

On Windows you remain with the problem that all nodes initialized
using TestLib.pm will listen to 127.0.0.1, sspi being used to ensure
that the connection at user level is secure (additional entries in
pg_hba.conf are added).

# decouple cleanup from node shutdown
Added (in latest patches?)

Yes this was added.

Michael, is there anything else to do here or shall I mark this for
committer review?

I have nothing else. Thanks a lot!

Ok, marked for committer, I hope I'm following "correct" cf procedure.

Regards,
Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#57Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#55)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/10/2015 04:32 PM, Michael Paquier wrote:

On Sat, Oct 10, 2015 at 9:04 PM, Amir Rohan wrote:

The patch focuses on providing facilities, while providing new coverage
for several features. There should be a TODO list on the wiki (bug
tracker, actually), where the list of tests to be written can be managed.
Some were mentioned in the thread (multi/xid wraparound
hot_standby_feedback, max_standby_archive_delay and
max_standby_streaming_delay? recovery_target_action? some in your
original list?), but threads
are precisely where these things get lost in the cracks.

Sure, that's an on-going task.

I was arguing that it's an on-going task that would do
better if it had a TODO list, instead of "ideas for tests"
being scattered across 50-100 messages spanning a year or
more in one thread or another. You may disagree.

Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#58Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#57)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Oct 10, 2015 at 10:52 PM, Amir Rohan <amir.rohan@zoho.com> wrote:

On 10/10/2015 04:32 PM, Michael Paquier wrote:
I was arguing that it's an on-going task that would do
better if it had a TODO list, instead of "ideas for tests"
being scattered across 50-100 messages spanning a year or
more in one thread or another. You may disagree.

Let's be clear. I am fully in line with your point.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#59Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#58)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/11/2015 02:47 AM, Michael Paquier wrote:

On Sat, Oct 10, 2015 at 10:52 PM, Amir Rohan <amir.rohan@zoho.com> wrote:

On 10/10/2015 04:32 PM, Michael Paquier wrote:
I was arguing that it's an on-going task that would do
better if it had a TODO list, instead of "ideas for tests"
being scattered across 50-100 messages spanning a year or
more in one thread or another. You may disagree.

Let's be clear. I am fully in line with your point.

I apologize -- that didn't came out right.
What I meant to suggest was "open an issue" to track
any works that needs to be done. But I guess that's
not the PG way.

Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#60Michael Paquier
michael.paquier@gmail.com
In reply to: Amir Rohan (#59)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sun, Oct 11, 2015 at 4:44 PM, Amir Rohan wrote:

On 10/11/2015 02:47 AM, Michael Paquier wrote:
I apologize -- that didn't came out right.
What I meant to suggest was "open an issue" to track
any works that needs to be done. But I guess that's
not the PG way.

No problem. I was not clear either. We could create a new item in the
TODO list (https://wiki.postgresql.org/wiki/Todo) and link it to
dedicated page on the wiki where all the potential tests would be
listed.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#61Amir Rohan
amir.rohan@zoho.com
In reply to: Michael Paquier (#60)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 10/11/2015 01:19 PM, Michael Paquier wrote:

On Sun, Oct 11, 2015 at 4:44 PM, Amir Rohan wrote:

On 10/11/2015 02:47 AM, Michael Paquier wrote:
I apologize -- that didn't came out right.
What I meant to suggest was "open an issue" to track
any works that needs to be done. But I guess that's
not the PG way.

No problem. I was not clear either. We could create a new item in the
TODO list (https://wiki.postgresql.org/wiki/Todo) and link it to
dedicated page on the wiki where all the potential tests would be
listed.

It couldn't hurt but also may be just a waste of your time.
I'm just realizing how central an issue tracker is to how I work and
how much not having one irritates me. Tough luck for me I guess.

Regards,
Amir

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#62Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#53)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Hi, I just started looking this over a bit. The first thing I noticed
is that it adds a dependency on Archive::Tar which isn't already used
anywhere else. Did anybody check whether this exists back in 5.8
installations?

Why is "recovery" added to ALWAYS_SUBDIRS in src/test/Makefile instead
of to SUBDIRS? Seems a strange choice.

Instead of adding
print "# Error output: $stderr\n" if $stderr ne "";
to sub psql, I think it would be better to add line separators, which
would be clearer if the error output ever turns into a multiline error
messages. It would still show as empty if no stderr is produced; so I
think something like
if ($stderr ne '')
{
print "#### Begin standard error\n"
print $stderr;
print "#### End standard error\n";
}
or something like that.

In my days of Perl, it was starting to become frowned upon to call
subroutines without parenthesizing arguments. Is that no longer the
case? Because I notice there are many places in this patch and pre-
existing that call psql with an argument list without parens. And it's
a bit odd because I couldn't find any other subroutine that we're using
in that way.

In 005_replay_delay there's a 2s delay configured; then we test whether
something is replayed in 1s. I hate tests that run for a long time, but
is 2s good enough considering that some of our test animals in buildfarm
are really slow?

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#63Erik Rijkers
er@xs4all.nl
In reply to: Alvaro Herrera (#62)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 2015-11-18 16:21, Alvaro Herrera wrote:

Hi, I just started looking this over a bit. The first thing I noticed
is that it adds a dependency on Archive::Tar which isn't already used
anywhere else. Did anybody check whether this exists back in 5.8
installations?

Apparently it did not yet exist in core then, Module::CoreList says
5.9.3:

$ perl -MModule::CoreList -e ' print
Module::CoreList->first_release('Archive::Tar'), "\n";'
5.009003

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#64Noah Misch
noah@leadboat.com
In reply to: Alvaro Herrera (#62)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Nov 18, 2015 at 01:21:45PM -0200, Alvaro Herrera wrote:

In my days of Perl, it was starting to become frowned upon to call
subroutines without parenthesizing arguments. Is that no longer the
case?

I've not witnessed those frowns.

Because I notice there are many places in this patch and pre-
existing that call psql with an argument list without parens. And it's
a bit odd because I couldn't find any other subroutine that we're using
in that way.

TestLib.pm has unparenthesized calls to "standard_initdb", "start" and "run".
070_dropuser.pl has such calls to "start_test_server" and "psql".

In 005_replay_delay there's a 2s delay configured; then we test whether
something is replayed in 1s. I hate tests that run for a long time, but
is 2s good enough considering that some of our test animals in buildfarm
are really slow?

That test will be unreliable, agreed.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#65Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#62)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Nov 19, 2015 at 12:21 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Hi, I just started looking this over a bit. The first thing I noticed
is that it adds a dependency on Archive::Tar which isn't already used
anywhere else. Did anybody check whether this exists back in 5.8
installations?

Actually I didn't and that's a good point, we have decided to support
TAP down to 5.8.9. The only reason why I introduced this dependency is
that there is no easy native way to copy an entire folder in perl, and
that's for handling base backups. There are things like File::NCopy of
File::Copy::Recursive however it does not seem like a good idea to
depend on other modules that IPC::Run. Would it be better to have an
in-core module dedicated to that similar to SimpleTee.pm? Or are you
guys fine to accept a dependency with another module?

Why is "recovery" added to ALWAYS_SUBDIRS in src/test/Makefile instead
of to SUBDIRS? Seems a strange choice.

Because I thought that it should not be part of the main regression
suite, like ssl/. Feel free to correct me if my feeling is wrong.

Instead of adding
print "# Error output: $stderr\n" if $stderr ne "";
to sub psql, I think it would be better to add line separators, which
would be clearer if the error output ever turns into a multiline error
messages. It would still show as empty if no stderr is produced; so I
think something like
if ($stderr ne '')
{
print "#### Begin standard error\n"
print $stderr;
print "#### End standard error\n";
}
or something like that.

Yes, that would be better.

In my days of Perl, it was starting to become frowned upon to call
subroutines without parenthesizing arguments. Is that no longer the
case? Because I notice there are many places in this patch and pre-
existing that call psql with an argument list without parens. And it's
a bit odd because I couldn't find any other subroutine that we're using
in that way.

Hm, yeah. If we decide about a perl coding policy I would be happy to
follow it. Personally I prefer usually using parenthesis however if we
decide to make the calls consistent we had better address that as a
separate patch.

In 005_replay_delay there's a 2s delay configured; then we test whether
something is replayed in 1s. I hate tests that run for a long time, but
is 2s good enough considering that some of our test animals in buildfarm
are really slow?

A call to poll_query_until ensures that we wait for the standby to
replay once the minimum replay threshold is reached. Even with a slow
machine the first query would still see only 10 rows at the first try,
and then wait for the standby to replay before checking if 20 rows are
visible. Or I am not following your point.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#66Robert Haas
robertmhaas@gmail.com
In reply to: Alvaro Herrera (#62)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Nov 18, 2015 at 10:21 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

In my days of Perl, it was starting to become frowned upon to call
subroutines without parenthesizing arguments. Is that no longer the
case? Because I notice there are many places in this patch and pre-
existing that call psql with an argument list without parens. And it's
a bit odd because I couldn't find any other subroutine that we're using
in that way.

I've been coding in Perl for more than 20 years and have never heard
of such a rule.

Maybe I am not part of the "in" crowd.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#67Mike Blackwell
mike.blackwell@rrd.com
In reply to: Robert Haas (#66)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Nov 19, 2015 at 10:05 AM, Robert Haas <robertmhaas@gmail.com> wrote:

On Wed, Nov 18, 2015 at 10:21 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

In my days of Perl, it was starting to become frowned upon to call
subroutines without parenthesizing arguments. Is that no longer the
case?

​As I understand it, there are several reasons not to make function calls
in Perl without parenthesis. Whether they are good reasons is a question
for the user. Modern Perl <http://onyxneon.com/books/modern_perl/&gt; chapter
5 covers most of them.

__________________________________________________________________________________
*Mike Blackwell | Technical Analyst, Distribution Services/Rollout
Management | RR Donnelley*
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com

<http://www.rrdonnelley.com/&gt;
* <Mike.Blackwell@rrd.com>*

#68David Steele
david@pgmasters.net
In reply to: Robert Haas (#66)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 11/19/15 11:05 AM, Robert Haas wrote:

On Wed, Nov 18, 2015 at 10:21 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

In my days of Perl, it was starting to become frowned upon to call
subroutines without parenthesizing arguments. Is that no longer the
case? Because I notice there are many places in this patch and pre-
existing that call psql with an argument list without parens. And it's
a bit odd because I couldn't find any other subroutine that we're using
in that way.

I've been coding in Perl for more than 20 years and have never heard
of such a rule.

I follow the convention of using parentheses for all function calls in
Perl, though this stems more from my greater familiarity with languages
that require them than any adherence to vague Perl conventions.

I do think it makes the code [more] readable.

--
-David
david@pgmasters.net

#69Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#53)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

I just noticed that RecoveryTest.pm is lacking "use strict; use
warnings;". With those added, there's a number of problems reported:

Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 66.
Global symbol "%backup_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 67.
Global symbol "%archive_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 68.
Global symbol "%connstr_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 69.
Global symbol "%applname_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 70.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 92.
Global symbol "%connstr_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 93.
Global symbol "%applname_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 93.
Global symbol "%archive_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 104.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 111.
Global symbol "%archive_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 121.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 130.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 185.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 197.
Global symbol "@array" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 220.
Global symbol "%backup_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 243.
Global symbol "%archive_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 244.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 246.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 257.
Global symbol "%backup_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 258.
Global symbol "%archive_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 259.
Global symbol "%connstr_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 260.
Global symbol "%applname_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 261.
Global symbol "%backup_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 272.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 287.
Global symbol "%backup_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 288.
Global symbol "%archive_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 289.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 292.
Global symbol "$current_dir" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 294.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 302.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 313.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 320.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 367.
Global symbol "%backup_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 377.
Global symbol "%datadir_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 390.
Global symbol "%backup_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 391.
Global symbol "%archive_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 392.
Global symbol "%connstr_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 393.
Global symbol "%applname_nodes" requires explicit package name at /pgsql/source/master/src/test/perl/RecoveryTest.pm line 394.

Most of them are easily fixable by adding the correct "my" lines; but at
least @array and $current_dir require more code to be written.

TBH all that business with arrays that are kept in sync looks too
contrived to me. Could we have a Perl object representing each node
instead? That would require a "PostgresNode" package (or similar). The
RecoveryTest.pm would have a single %nodes hash. Also, you don't need
@active_nodes, just a flag in PostgresNode, and have the stop routine do
nothing if node is not marked active. Also: if you pass the "root node"
when creating a node that will become a standby, you don't need to pass
it when calling, say, enable_streaming; the root node becomes an
instance variable. (Hmm, actually, if we do that, I wonder what if in
the future we want to test node promotion and a standby is repointed to
a new master. Maybe we don't want to have this knowledge in the Perl
code at all.)

In get_free_port, isn't it easier to use pg_isready rather than psql?

I've been messing with 003 because I think it's a bit too repetitive.
Will finish it after you post a fixed version of RecoveryTest.pm.

Thanks!

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#70Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#65)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

On Thu, Nov 19, 2015 at 12:21 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Hi, I just started looking this over a bit. The first thing I noticed
is that it adds a dependency on Archive::Tar which isn't already used
anywhere else. Did anybody check whether this exists back in 5.8
installations?

Actually I didn't and that's a good point, we have decided to support
TAP down to 5.8.9. The only reason why I introduced this dependency is
that there is no easy native way to copy an entire folder in perl, and
that's for handling base backups. There are things like File::NCopy of
File::Copy::Recursive however it does not seem like a good idea to
depend on other modules that IPC::Run. Would it be better to have an
in-core module dedicated to that similar to SimpleTee.pm? Or are you
guys fine to accept a dependency with another module?

It would be a lot better to not have to rely on another module existing
everywhere. I'd rather have another simple module, following
SimpleTee's example. Since this doesn't have to be terribly generic, it
should be reasonably short, I hope.

Why is "recovery" added to ALWAYS_SUBDIRS in src/test/Makefile instead
of to SUBDIRS? Seems a strange choice.

Because I thought that it should not be part of the main regression
suite, like ssl/. Feel free to correct me if my feeling is wrong.

As I understand, the problem with "ssl" is that it messes with
system-wide settings, which is not the case here. I'm inclined to move
it to SUBDIRS. As an example, "modules" is not part of the main
regression suite either.

In my days of Perl, it was starting to become frowned upon to call
subroutines without parenthesizing arguments. Is that no longer the
case? Because I notice there are many places in this patch and pre-
existing that call psql with an argument list without parens. And it's
a bit odd because I couldn't find any other subroutine that we're using
in that way.

Hm, yeah. If we decide about a perl coding policy I would be happy to
follow it. Personally I prefer usually using parenthesis however if we
decide to make the calls consistent we had better address that as a
separate patch.

Some votes against, some votes for. Ultimately, it seems that this
depends on the committer. I don't really care all that much about this
TBH.

In 005_replay_delay there's a 2s delay configured; then we test whether
something is replayed in 1s. I hate tests that run for a long time, but
is 2s good enough considering that some of our test animals in buildfarm
are really slow?

A call to poll_query_until ensures that we wait for the standby to
replay once the minimum replay threshold is reached. Even with a slow
machine the first query would still see only 10 rows at the first try,
and then wait for the standby to replay before checking if 20 rows are
visible. Or I am not following your point.

Ah, I see. Maybe it's fine then, or else I'm not following your point
;-)

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#71Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#70)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Tue, Nov 24, 2015 at 6:27 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Michael Paquier wrote:

On Thu, Nov 19, 2015 at 12:21 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Hi, I just started looking this over a bit. The first thing I noticed
is that it adds a dependency on Archive::Tar which isn't already used
anywhere else. Did anybody check whether this exists back in 5.8
installations?

Actually I didn't and that's a good point, we have decided to support
TAP down to 5.8.9. The only reason why I introduced this dependency is
that there is no easy native way to copy an entire folder in perl, and
that's for handling base backups. There are things like File::NCopy of
File::Copy::Recursive however it does not seem like a good idea to
depend on other modules that IPC::Run. Would it be better to have an
in-core module dedicated to that similar to SimpleTee.pm? Or are you
guys fine to accept a dependency with another module?

It would be a lot better to not have to rely on another module existing
everywhere. I'd rather have another simple module, following
SimpleTee's example. Since this doesn't have to be terribly generic, it
should be reasonably short, I hope.

Sure, that would be a simple function that does directory lookup and
recursive calls. I'll move ahead with that then and reuse it in the
recovery logic.

Why is "recovery" added to ALWAYS_SUBDIRS in src/test/Makefile instead
of to SUBDIRS? Seems a strange choice.

Because I thought that it should not be part of the main regression
suite, like ssl/. Feel free to correct me if my feeling is wrong.

As I understand, the problem with "ssl" is that it messes with
system-wide settings, which is not the case here. I'm inclined to move
it to SUBDIRS. As an example, "modules" is not part of the main
regression suite either.

OK, I'll move it back to it then.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#72Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#69)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Thanks for the review.

On Tue, Nov 24, 2015 at 6:15 AM, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

I just noticed that RecoveryTest.pm is lacking "use strict; use
warnings;". With those added, there's a number of problems reported:

Most of them are easily fixable by adding the correct "my" lines; but at

least @array and $current_dir require more code to be written.

Oops.

TBH all that business with arrays that are kept in sync looks too
contrived to me. Could we have a Perl object representing each node
instead?

Not really to be honest.

That would require a "PostgresNode" package (or similar). The
RecoveryTest.pm would have a single %nodes hash. Also, you don't need
@active_nodes, just a flag in PostgresNode, and have the stop routine do
nothing if node is not marked active. Also: if you pass the "root node"
when creating a node that will become a standby, you don't need to pass
it when calling, say, enable_streaming; the root node becomes an
instance variable. (Hmm, actually, if we do that, I wonder what if in
the future we want to test node promotion and a standby is repointed to
a new master. Maybe we don't want to have this knowledge in the Perl
code at all.)

I think I'll get the idea. In short all the parametrization will just
happen at object level, as well as basic actions on the nodes like start,
stop, restart etc.

In get_free_port, isn't it easier to use pg_isready rather than psql?

Will switch.

I've been messing with 003 because I think it's a bit too repetitive.
Will finish it after you post a fixed version of RecoveryTest.pm.

Sure, thanks.

I'll rework this patch and will update a new version soon.

Thanks again for the review.
--
Michael

#73Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#72)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Tue, Nov 24, 2015 at 2:14 PM, Michael Paquier
<michael.paquier@gmail.com>wrote:

I'll rework this patch and will update a new version soon.

So, attached is a new patch addressing all the comments received. The new
version has the following changes:
- Print more verbosely stderr output in case of error in psql
- Add recovery test suite to SUBDIRS in src/test/Makefile
- Add strict and warnings to what is used in the new modules of this patch
- Manage node information using package/class PostgresNode.pm and have
RecoveryTest use it. I have actually made PostgresNode bare-bone and simple
on purpose: one can initialize the node, append configuration parameters to
it and manage it through start/stop/restart (we may want to add reload and
promote actually if needed). However, more complex configuration is left to
RecoveryTest.pm, which is in charge of appending the configuration
dedicated to streaming, archiving, etc though a set of routines working on
PostgresNode objects. I have also arrived at the conclusion that it is not
really worth adding a node status flag in PostgresNode because the port
number saved there is sufficient when doing free port lookup, and the list
of nodes used in a recovery test are saved in an array.
- Add new module RecursiveCopy to be used for base backups. This removes
the dependency with Archive::Tar. PostgresNode makes use of that when
initializing a node from a backup.
- Tests have been updated to use the PostgresNode objects instead of the
port number as identifier. That's more portable.

Hopefully I have missed nothing.
Regards,
--
Michael

Attachments:

20151124_recovery_regression_v9.patchapplication/x-patch; name=20151124_recovery_regression_v9.patchDownload
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..ea219d7 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -125,38 +125,6 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
 sub append_to_file
 {
 	my ($filename, $str) = @_;
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..7f7754f 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -12,7 +12,7 @@ subdir = src/test
 top_builddir = ../..
 include $(top_builddir)/src/Makefile.global
 
-SUBDIRS = regress isolation modules
+SUBDIRS = regress isolation modules recovery
 
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
new file mode 100644
index 0000000..90ad3bd
--- /dev/null
+++ b/src/test/perl/PostgresNode.pm
@@ -0,0 +1,162 @@
+# PostgresNode, simple node representation for regression tests
+#
+# Regression tests should use this basic class infrastructure to define
+# nodes that need to be used in the complex scenarios. This object is wanted
+# simple with only a basic set of routines able to configure, initialize
+# and manage a node.
+
+package PostgresNode;
+
+use strict;
+use warnings;
+
+use RecursiveCopy;
+use TestLib;
+
+sub new {
+	my $class = shift;
+	my $pghost = shift;
+	my $pgport = shift;
+	my $self = {
+		_port => undef,
+		_host => undef,
+		_basedir => undef,
+		_connstr => undef,
+		_applname => undef
+			};
+
+	# Set up each field
+	$self->{_port} = $pgport;
+	$self->{_host} = $pghost;
+	$self->{_basedir} = TestLib::tempdir;
+	$self->{_connstr} = "port=$pgport host=$pghost";
+	$self->{_applname} = "node_$pgport";
+	bless $self, $class;
+	return $self;
+}
+
+# Get routines for various variables
+sub getPort {
+	my( $self ) = @_;
+	return $self->{_port};
+}
+sub getHost {
+	my( $self ) = @_;
+	return $self->{_host};
+}
+sub getConnStr {
+	my( $self ) = @_;
+	return $self->{_connstr};
+}
+sub getDataDir {
+	my ( $self ) = @_;
+	return $self->{_basedir} . '/pgdata';
+}
+sub getApplName {
+	my ( $self ) = @_;
+	return $self->{_applname};
+}
+sub getArchiveDir {
+	my ( $self ) = @_;
+	return $self->{_basedir} . '/archives';
+}
+sub getBackupDir {
+	my ( $self ) = @_;
+	return $self->{_basedir} . '/backup';
+}
+
+# Dump node information
+sub dumpNodeInfo {
+	my ( $self ) = @_;
+	print 'Data directory: ' . $self->getDataDir() . "\n";
+	print 'Backup directory: ' . $self->getBackupDir() . "\n";
+	print 'Archive directory: ' . $self->getArchiveDir() . "\n";
+	print 'Connection string: ' . $self->getConnStr() . "\n";
+	print 'Application name: ' . $self->getApplName() . "\n";
+}
+
+# Actions on node
+sub initNode
+{
+	my ( $self ) = @_;
+	my $port = $self->getPort();
+
+	mkdir $self->getBackupDir();
+	mkdir $self->getArchiveDir();
+
+	standard_initdb($self->getDataDir(), $self->getHost());
+	$self->appendConf('postgresql.conf', qq(
+port = $port
+));
+	configure_hba_for_replication($self->getDataDir());
+}
+sub appendConf
+{
+	my ($self, $filename, $str) = @_;
+
+	my $conffile = $self->getDataDir() . '/' . $filename;
+
+	open my $fh, ">>", $conffile or die "could not open file $filename";
+	print $fh $str;
+	close $fh;
+}
+sub backupNode
+{
+	my ($self, $backup_name) = @_;
+    my $backup_path = $self->getBackupDir() . '/' . $backup_name;
+	my $port = $self->getPort();
+
+	print "# Taking backup $backup_name from node with port $port\n";
+	system_or_bail("pg_basebackup -D $backup_path -p $port -x");
+	print "# Backup finished\n";
+}
+sub initNodeFromBackup
+{
+	my ( $self, $root_node, $backup_name ) = @_;
+	my $backup_path = $root_node->getBackupDir() . '/' . $backup_name;
+	my $port = $self->getPort();
+	my $root_port = $root_node->getPort();
+
+	print "Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	die "Backup $backup_path does not exist" if (! -d $backup_path);
+
+	mkdir $self->getBackupDir();
+	mkdir $self->getArchiveDir();
+
+	my $data_path = $self->getDataDir();
+    rmdir($data_path);
+    RecursiveCopy::copypath($backup_path, $data_path);
+    chmod(0700, $data_path);
+
+	# Base configuration for this node
+	$self->appendConf('postgresql.conf', qq(
+port = $port
+));
+	configure_hba_for_replication($self->getDataDir());
+}
+sub startNode
+{
+	my ( $self ) = @_;
+	my $port = $self->getPort();
+	print "Starting node with port $port\n";
+    system_or_bail('pg_ctl', '-w', '-D', $self->getDataDir(),
+				   '-l', "$log_path/node_$port.log",
+				   'start');
+}
+sub stopNode
+{
+	my ( $self, $mode ) = @_;
+	my $port = $self->getPort();
+	$mode = 'fast' if (!defined($mode));
+	print "Stopping node with port $port with $mode mode\n";
+	system('pg_ctl', '-D', $self->getDataDir(), '-m',
+		   $mode, 'stop');
+}
+sub restartNode
+{
+	my ( $self ) = @_;
+	$self->stopNode();
+	$self->startNode();
+}
+
+1;
diff --git a/src/test/perl/RecoveryTest.pm b/src/test/perl/RecoveryTest.pm
new file mode 100644
index 0000000..1056a1c
--- /dev/null
+++ b/src/test/perl/RecoveryTest.pm
@@ -0,0 +1,251 @@
+# Set of common routines for recovery regression tests for a PostgreSQL
+# cluster. This includes methods that can be used by the various set of
+# tests present to set up cluster nodes and configure them according to
+# the test scenario wanted.
+#
+# This module makes use of PostgresNode for node manipulation, performing
+# higher-level operations to create standby nodes or setting them up
+# for archiving and replication.
+#
+# Nodes are identified by their port number and have one allocated when
+# created, hence it is unique for each node of the cluster as it is run
+# locally. PGHOST is equally set to a unique value for the duration of
+# each test.
+
+package RecoveryTest;
+
+use strict;
+use warnings;
+
+use Cwd;
+use PostgresNode;
+use RecursiveCopy;
+use TestLib;
+use Test::More;
+
+use IPC::Run qw(run start);
+
+use Exporter 'import';
+
+our @EXPORT = qw(
+	%active_nodes
+
+	enable_archiving
+	enable_restoring
+	enable_streaming
+	get_free_port
+	make_master
+	make_archive_standby
+	make_stream_standby
+	teardown_node
+);
+
+# Tracking of last port value assigned to accelerate free port lookup.
+# XXX: Should this part use PG_VERSION_NUM?
+my $last_port_assigned =  90600 % 16384 + 49152;
+
+# Value tracking the host value used for a single run.
+my $node_pghost = $windows_os ? "127.0.0.1" : tempdir_short;
+
+
+# Database used for each connection attempt via psql
+$ENV{PGDATABASE} = "postgres";
+
+# Tracker of active nodes
+my @active_nodes = ();
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $node_root = shift; # Instance to link to
+	my $node_standby = shift;
+	my $root_connstr = $node_root->getConnStr();
+	my $applname = $node_standby->getApplName();
+
+	$node_standby->appendConf('recovery.conf', qq(
+primary_conninfo='$root_connstr application_name=$applname'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $node_root = shift; # Instance to link to
+	my $node_standby = shift;
+	my $path = $node_root->getArchiveDir();
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"$path\\\\%f\" \"%p\"" :
+		"cp -i $path/%f %p";
+	$node_standby->appendConf('recovery.conf', qq(
+restore_command='$copy_command'
+standby_mode=on
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $node = shift;
+	my $path = $node->getArchiveDir();
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$path\\\\%f\"" :
+		"cp %p $path/%f";
+
+	# Enable archive_mode and archive_command on node
+	$node->appendConf('postgresql.conf', qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization.
+sub make_master
+{
+	my $node_master = get_free_port();
+	my $port_master = $node_master->getPort();
+	print "# Initializing master node wih port $port_master\n";
+	$node_master->initNode();
+	configure_base_node($node_master);
+	return $node_master;
+}
+
+sub configure_base_node
+{
+	my $node = shift;
+
+	$node->appendConf('postgresql.conf', qq(
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+));
+	configure_hba_for_replication($node->getDataDir());
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_stream_standby
+{
+	my $node_master = shift;
+	my $backup_name = shift;
+	my $node_standby = get_free_port();
+	my $master_port = $node_master->getPort();
+	my $standby_port = $node_standby->getPort();
+
+	print "# Initializing streaming mode for node $standby_port from node $master_port\n";
+	$node_standby->initNodeFromBackup($node_master, $backup_name);
+	configure_base_node($node_standby);
+
+	# Start second node, streaming from first one
+	enable_streaming($node_master, $node_standby);
+	return $node_standby;
+}
+
+# Node getting WAL only from archives
+sub make_archive_standby
+{
+	my $node_master = shift;
+	my $backup_name = shift;
+	my $node_standby = get_free_port();
+	my $master_port = $node_master->getPort();
+	my $standby_port = $node_standby->getPort();
+
+	print "# Initializing archive mode for node $standby_port from node $master_port\n";
+	$node_standby->initNodeFromBackup($node_master, $backup_name);
+	configure_base_node($node_standby);
+
+	# Start second node, restoring from first one
+	enable_restoring($node_master, $node_standby);
+	return $node_standby;
+}
+
+# Get a port number not in use currently for a new node
+# As port number retrieval is based on the nodes currently running and
+# their presence in the list of registered ports, be sure that the node
+# that is consuming this port number has already been started and that
+# it is not registered yet.
+sub get_free_port
+{
+	my $found = 0;
+	my $port = $last_port_assigned;
+
+	while ($found == 0)
+	{
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		if (!run_log(['pg_isready', '-p', $port]))
+		{
+			$found = 1;
+			# Found a potential candidate, check first that it is
+			# not included in the list of registered nodes.
+			foreach my $node (@active_nodes)
+			{
+				$found = 0 if ($node->getPort() == $port);
+			}
+		}
+	}
+
+	print "# Found free port $port\n";
+	# Lock port number found by creating a new node
+	my $node = new PostgresNode($node_pghost, $port);
+
+	# Add node to list of nodes currently in use
+	push(@active_nodes, $node);
+	$last_port_assigned = $port;
+	return $node;
+}
+
+# Wait until a node is able to accept queries. Useful when putting a node
+# in recovery and wait for it to be able to work particularly on slow
+# machines.
+sub wait_for_node
+{
+	my $node         = shift;
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	while ($attempts < $max_attempts)
+	{
+		if (run_log(['pg_isready', '-p', $node->getPort()]))
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+	return 0;
+}
+
+# Remove any traces of given node.
+sub teardown_node
+{
+	my $node = shift;
+
+	$node->stopNode('immediate');
+	@active_nodes = grep { $_ ne $node } @active_nodes;
+}
+
+END
+{
+	foreach my $node (@active_nodes)
+	{
+		teardown_node($node);
+	}
+}
+
+1;
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
new file mode 100644
index 0000000..f06f975
--- /dev/null
+++ b/src/test/perl/RecursiveCopy.pm
@@ -0,0 +1,42 @@
+# RecursiveCopy, a simple recursive copy implementation
+#
+# Having this implementation has the advantage to not have the regression
+# test code rely on any external module for this simple operation.
+
+package RecursiveCopy;
+use strict;
+use warnings;
+
+use File::Basename;
+use File::Copy;
+
+sub copypath {
+	my $srcpath = shift;
+	my $destpath = shift;
+
+	die "Cannot operate on symlinks" if ( -l $srcpath || -l $destpath );
+
+	# This source path is a file, simply copy it to destination with the
+	# same name.
+	die "Destination path $destpath exists as file" if ( -f $destpath );
+	if ( -f $srcpath )
+	{
+		my $filename = basename($destpath);
+		copy($srcpath, "$destpath");
+		return 1;
+	}
+
+	die "Destination needs to be a directory" if (! -d $srcpath);
+	mkdir($destpath);
+
+	# Scan existing source directory and recursively copy everything.
+	opendir(my $directory, $srcpath);
+	while (my $entry = readdir($directory)) {
+		next if ($entry eq '.' || $entry eq '..');
+		RecursiveCopy::copypath("$srcpath/$entry", "$destpath/$entry");
+	}
+	closedir($directory);
+	return 1;
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..359ef8a 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -12,6 +12,7 @@ our @EXPORT = qw(
   configure_hba_for_replication
   start_test_server
   restart_test_server
+  poll_query_until
   psql
   slurp_dir
   slurp_file
@@ -136,11 +137,12 @@ sub tempdir_short
 sub standard_initdb
 {
 	my $pgdata = shift;
+	my $tempdir_short = shift;
+
+	$tempdir_short = tempdir_short if (! defined($tempdir_short));
 	system_or_bail('initdb', '-D', "$pgdata", '-A' , 'trust', '-N');
 	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
 
-	my $tempdir_short = tempdir_short;
-
 	open CONF, ">>$pgdata/postgresql.conf";
 	print CONF "\n# Added by TestLib.pm)\n";
 	print CONF "fsync = off\n";
@@ -220,12 +222,49 @@ END
 	}
 }
 
+sub poll_query_until
+{
+	my ($query, $connstr) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
+		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
 sub psql
 {
 	my ($dbname, $sql) = @_;
 	my ($stdout, $stderr);
 	print("# Running SQL command: $sql\n");
 	run [ 'psql', '-X', '-A', '-t', '-q', '-d', $dbname, '-f', '-' ], '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
+	if ($stderr ne "")
+	{
+		print "#### Begin standard error\n";
+		print $stderr;
+		print "#### End standard error\n";
+	}
 	chomp $stdout;
 	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
 	return $stdout;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..aae1026
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,67 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+
+use RecoveryTest;
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->startNode();
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_master->backupNode($backup_name);
+
+# Create streaming standby linking to master
+my $node_standby_1 = make_stream_standby($node_master, $backup_name);
+$node_standby_1->startNode();
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+$node_standby_1->backupNode($backup_name);
+
+# Create second standby node linking to standby 1
+my $node_standby_2 = make_stream_standby($node_standby_1, $backup_name);
+$node_standby_2->startNode();
+$node_standby_2->backupNode($backup_name);
+
+# Create some content on master and check its presence in standby 1 an
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a";
+
+# Wait for standbys to catch up
+my $applname_1 = $node_standby_1->getApplName();
+my $applname_2 = $node_standby_2->getApplName();
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_1';";
+poll_query_until($caughtup_query, $node_master->getConnStr())
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_2';";
+poll_query_until($caughtup_query, $node_standby_1->getConnStr())
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = psql $node_standby_1->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result = psql $node_standby_2->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->getConnStr(), '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->getConnStr(), '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
+
+# Cleanup nodes
+teardown_node($node_standby_2);
+teardown_node($node_standby_1);
+teardown_node($node_master);
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..c3e8465
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,51 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $node_master = make_master();
+my $backup_name = 'my_backup';
+enable_archiving($node_master);
+
+# Start it
+$node_master->startNode();
+
+# Take backup for slave
+$node_master->backupNode($backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $node_standby = make_archive_standby($node_master, $backup_name);
+$node_standby->appendConf('postgresql.conf', qq(
+wal_retrieve_retry_interval = '100ms'
+));
+$node_standby->startNode();
+
+# Create some content on master
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $current_lsn = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# Force archiving of WAL file to make it present on master
+psql $node_master->getConnStr(), "SELECT pg_switch_xlog()";
+
+# Add some more content, it should not be present on standby
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $node_standby->getConnStr())
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $node_standby->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+is($result, qq(1000), 'check content from archives');
+
+# Cleanup nodes
+teardown_node($node_standby);
+teardown_node($node_master);
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..995e8e4
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,135 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $node_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $node_standby = make_archive_standby($node_master, 'my_backup');
+
+	foreach my $param_item (@$recovery_params)
+	{
+		$node_standby->appendConf('recovery.conf',
+					   qq($param_item
+));
+	}
+
+	$node_standby->startNode();
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	poll_query_until($caughtup_query, $node_standby->getConnStr())
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = psql $node_standby->getConnStr(),
+		"SELECT count(*) FROM tab_int";
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	teardown_node($node_standby);
+}
+
+# Initialize master node
+my $node_master = make_master();
+enable_archiving($node_master);
+
+# Start it
+$node_master->startNode();
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $lsn1 = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# Take backup from which all operations will be run
+$node_master->backupNode('my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+my $recovery_txid = psql $node_master->getConnStr(),
+	"SELECT txid_current()";
+my $lsn2 = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# More data, with recovery target timestamp
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(2001,3000))";
+my $recovery_time = psql $node_master->getConnStr(), "SELECT now()";
+my $lsn3 = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# Even more data, this time with a recovery target name
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))";
+my $recovery_name = "my_target";
+my $lsn4 = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+psql $node_master->getConnStr(),
+	"SELECT pg_create_restore_point('$recovery_name')";
+
+# Force archiving of WAL file
+psql $node_master->getConnStr(), "SELECT pg_switch_xlog()";
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $node_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+teardown_node($node_master);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..509571c
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,72 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->startNode();
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backupNode($backup_name);
+
+# Create two standbys linking to it
+my $node_standby_1 = make_stream_standby($node_master, $backup_name);
+$node_standby_1->startNode();
+my $node_standby_2 = make_stream_standby($node_master, $backup_name);
+$node_standby_2->startNode();
+
+# Create some content on master
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $until_lsn = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $node_standby_1->getConnStr())
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+teardown_node($node_master);
+system_or_bail('pg_ctl', '-w', '-D', $node_standby_1->getDataDir(),
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree($node_standby_2->getDataDir() . '/recovery.conf');
+my $connstr_1 = $node_standby_1->getConnStr();
+$node_standby_2->appendConf('recovery.conf', qq(
+primary_conninfo='$connstr_1'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+$node_standby_2->restartNode();
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done. Standby 1 needs
+# to exit recovery first before moving on with the test.
+poll_query_until("SELECT pg_is_in_recovery() <> true",
+				 $node_standby_1->getConnStr());
+psql $node_standby_1->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+$until_lsn = psql $node_standby_1->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $node_standby_2->getConnStr())
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $node_standby_2->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+is($result, qq(2000), 'check content of standby 2');
+
+# Stop nodes
+teardown_node($node_standby_2);
+teardown_node($node_standby_1);
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..c209b55
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,49 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->startNode();
+
+# And some content
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a";
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backupNode($backup_name);
+
+# Create streaming standby from backup
+my $node_standby = make_stream_standby($node_master, $backup_name);
+$node_standby->appendConf('recovery.conf', qq(
+recovery_min_apply_delay = '2s'
+));
+$node_standby->startNode();
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(11,20))";
+sleep 1;
+# Here we should have only 10 rows
+my $result = psql $node_standby->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $node_standby->getConnStr())
+	or die "Timed out while waiting for standby to catch up";
+$result = psql $node_standby->getConnStr(), "SELECT count(*) FROM tab_int";
+is($result, qq(20), 'check content with delay of 2s');
+
+# Stop nodes
+teardown_node($node_standby);
+teardown_node($node_master);
#74Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#73)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

- Manage node information using package/class PostgresNode.pm and have
RecoveryTest use it. I have actually made PostgresNode bare-bone and simple
on purpose: one can initialize the node, append configuration parameters to
it and manage it through start/stop/restart (we may want to add reload and
promote actually if needed).

This looks great as a starting point. I think we should make TestLib
depend on PostgresNode instead of the other way around. I will have a
look at that (I realize this means messing with the existing tests).

I have also arrived at the conclusion that it is not really worth
adding a node status flag in PostgresNode because the port number
saved there is sufficient when doing free port lookup, and the list of
nodes used in a recovery test are saved in an array.

I don't disagree with this in principle, but I think the design that you
get a new PostgresNode object by calling get_free_port is strange. I
think the port lookup code should be part of either TestLib or
PostgresNode, not RecoveryTest.

- Add new module RecursiveCopy to be used for base backups. This removes
the dependency with Archive::Tar. PostgresNode makes use of that when
initializing a node from a backup.

Great.

- Tests have been updated to use the PostgresNode objects instead of the
port number as identifier. That's more portable.

Makes sense.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#75Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#74)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Nov 25, 2015 at 6:22 AM, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

Michael Paquier wrote:

- Manage node information using package/class PostgresNode.pm and have
RecoveryTest use it. I have actually made PostgresNode bare-bone and

simple

on purpose: one can initialize the node, append configuration parameters

to

it and manage it through start/stop/restart (we may want to add reload

and

promote actually if needed).

This looks great as a starting point. I think we should make TestLib
depend on PostgresNode instead of the other way around. I will have a
look at that (I realize this means messing with the existing tests).

Makes sense. My thoughts following that is that we should keep a track of
the nodes started as an array which is part of TestLib, with PGHOST set
once at startup using tempdir_short. That's surely an refactoring patch
somewhat independent of the recovery test suite. I would not mind writing
something among those lines if needed.

I have also arrived at the conclusion that it is not really worth
adding a node status flag in PostgresNode because the port number
saved there is sufficient when doing free port lookup, and the list of
nodes used in a recovery test are saved in an array.

I don't disagree with this in principle, but I think the design that you
get a new PostgresNode object by calling get_free_port is strange. I
think the port lookup code should be part of either TestLib or
PostgresNode, not RecoveryTest.

I'd vote for TestLib. I have written PostgresNode this way to allow users
to set up arbitrary port numbers if they'd like to do so. That's more
flexible.
--
Michael

#76Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#75)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

On Wed, Nov 25, 2015 at 6:22 AM, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

Michael Paquier wrote:

This looks great as a starting point. I think we should make TestLib
depend on PostgresNode instead of the other way around. I will have a
look at that (I realize this means messing with the existing tests).

Makes sense. My thoughts following that is that we should keep a track of
the nodes started as an array which is part of TestLib, with PGHOST set
once at startup using tempdir_short. That's surely an refactoring patch
somewhat independent of the recovery test suite. I would not mind writing
something among those lines if needed.

OK, please do.

We can split this up in two patches: one introducing PostgresNode
(+ RecursiveCopy) together with the refactoring of existing test code,
and a subsequent one introducing RecoveryTest and the corresponding
subdir. Sounds good?

I have also arrived at the conclusion that it is not really worth
adding a node status flag in PostgresNode because the port number
saved there is sufficient when doing free port lookup, and the list of
nodes used in a recovery test are saved in an array.

I don't disagree with this in principle, but I think the design that you
get a new PostgresNode object by calling get_free_port is strange. I
think the port lookup code should be part of either TestLib or
PostgresNode, not RecoveryTest.

I'd vote for TestLib. I have written PostgresNode this way to allow users
to set up arbitrary port numbers if they'd like to do so. That's more
flexible.

That works for me.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#77Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#76)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Nov 25, 2015 at 10:55 AM, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

Michael Paquier wrote:

On Wed, Nov 25, 2015 at 6:22 AM, Alvaro Herrera <

alvherre@2ndquadrant.com>

wrote:

Michael Paquier wrote:

This looks great as a starting point. I think we should make TestLib
depend on PostgresNode instead of the other way around. I will have a
look at that (I realize this means messing with the existing tests).

Makes sense. My thoughts following that is that we should keep a track of
the nodes started as an array which is part of TestLib, with PGHOST set
once at startup using tempdir_short. That's surely an refactoring patch
somewhat independent of the recovery test suite. I would not mind writing
something among those lines if needed.

OK, please do.

We can split this up in two patches: one introducing PostgresNode
(+ RecursiveCopy) together with the refactoring of existing test code,
and a subsequent one introducing RecoveryTest and the corresponding
subdir. Sounds good?

Yeah, that matches my line of thoughts. Will do so.
--
Michael

#78Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#77)
2 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Nov 25, 2015 at 11:00 AM, Michael Paquier <michael.paquier@gmail.com

wrote:

On Wed, Nov 25, 2015 at 10:55 AM, Alvaro Herrera <alvherre@2ndquadrant.com

wrote:

Michael Paquier wrote:

On Wed, Nov 25, 2015 at 6:22 AM, Alvaro Herrera <

alvherre@2ndquadrant.com>

wrote:

Michael Paquier wrote:

This looks great as a starting point. I think we should make TestLib
depend on PostgresNode instead of the other way around. I will have a
look at that (I realize this means messing with the existing tests).

Makes sense. My thoughts following that is that we should keep a track

of

the nodes started as an array which is part of TestLib, with PGHOST set
once at startup using tempdir_short. That's surely an refactoring patch
somewhat independent of the recovery test suite. I would not mind

writing

something among those lines if needed.

OK, please do.

We can split this up in two patches: one introducing PostgresNode
(+ RecursiveCopy) together with the refactoring of existing test code,
and a subsequent one introducing RecoveryTest and the corresponding
subdir. Sounds good?

Yeah, that matches my line of thoughts. Will do so.

The result of a couple of hours of hacking is attached:
- 0001 is the refactoring adding PostgresNode and RecursiveCopy. I have
also found that it is quite advantageous to move some of the routines that
are synonyms of system() and the stuff used for logging into another
low-level library that PostgresNode depends on, that I called TestBase in
this patch. This way, all the infrastructure depends on the same logging
management. Existing tests have been refactored to fit into the new code,
and this leads to a couple of simplifications particularly in pg_rewind
tests because there is no more need to have there routines for environment
cleanup and logging. I have done tests on OSX and Windows using it and
tests are passing. I have as well tested that ssl tests were working.
- 0002 adds the recovery tests with RecoveryTest.pm now located in
src/test/recovery.
Regards,
--
Michael

Attachments:

0001-Refactor-TAP-tests-to-use-common-node-management-sys.patchapplication/x-patch; name=0001-Refactor-TAP-tests-to-use-common-node-management-sys.patchDownload
From 686a58ae7edabc4d48767115283d2390c24c305a Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Wed, 25 Nov 2015 21:39:31 +0900
Subject: [PATCH 1/2] Refactor TAP tests to use common node management system

All the existing TAP tests now use a new module called PostgresNode that
centralizes logging, backup and definitions of a Postgres node used in
the tests. Some low-level content of TestLib is split into a second
module called TestBase which contains routines used by PostgresNode for
logging and running commands.

This is in preparation for a more advanced facility dedicated at testing
recovery scenarios directly in core.
---
 src/bin/initdb/t/001_initdb.pl                 |   3 +-
 src/bin/pg_basebackup/t/010_pg_basebackup.pl   |  29 ++-
 src/bin/pg_controldata/t/001_pg_controldata.pl |  13 +-
 src/bin/pg_ctl/t/001_start_stop.pl             |   5 +-
 src/bin/pg_ctl/t/002_status.pl                 |  17 +-
 src/bin/pg_rewind/RewindTest.pm                | 122 +++++-------
 src/bin/pg_rewind/t/003_extrafiles.pl          |   4 +-
 src/bin/pg_rewind/t/004_pg_xlog_symlink.pl     |   3 +
 src/bin/scripts/t/010_clusterdb.pl             |  23 ++-
 src/bin/scripts/t/011_clusterdb_all.pl         |  13 +-
 src/bin/scripts/t/020_createdb.pl              |  12 +-
 src/bin/scripts/t/030_createlang.pl            |  19 +-
 src/bin/scripts/t/040_createuser.pl            |  20 +-
 src/bin/scripts/t/050_dropdb.pl                |  12 +-
 src/bin/scripts/t/060_droplang.pl              |  10 +-
 src/bin/scripts/t/070_dropuser.pl              |  10 +-
 src/bin/scripts/t/080_pg_isready.pl            |   8 +-
 src/bin/scripts/t/090_reindexdb.pl             |  19 +-
 src/bin/scripts/t/091_reindexdb_all.pl         |   9 +-
 src/bin/scripts/t/100_vacuumdb.pl              |  18 +-
 src/bin/scripts/t/101_vacuumdb_all.pl          |  10 +-
 src/bin/scripts/t/102_vacuumdb_stages.pl       |  12 +-
 src/test/perl/PostgresNode.pm                  | 233 +++++++++++++++++++++++
 src/test/perl/RecursiveCopy.pm                 |  42 ++++
 src/test/perl/TestBase.pm                      | 109 +++++++++++
 src/test/perl/TestLib.pm                       | 254 +++++++------------------
 src/test/ssl/ServerSetup.pm                    |  24 +--
 src/test/ssl/t/001_ssltests.pl                 |  32 ++--
 28 files changed, 702 insertions(+), 383 deletions(-)
 create mode 100644 src/test/perl/PostgresNode.pm
 create mode 100644 src/test/perl/RecursiveCopy.pm
 create mode 100644 src/test/perl/TestBase.pm

diff --git a/src/bin/initdb/t/001_initdb.pl b/src/bin/initdb/t/001_initdb.pl
index 299dcf5..3b5d7af 100644
--- a/src/bin/initdb/t/001_initdb.pl
+++ b/src/bin/initdb/t/001_initdb.pl
@@ -4,10 +4,11 @@
 
 use strict;
 use warnings;
+use TestBase;
 use TestLib;
 use Test::More tests => 14;
 
-my $tempdir = TestLib::tempdir;
+my $tempdir = TestBase::tempdir;
 my $xlogdir = "$tempdir/pgxlog";
 my $datadir = "$tempdir/data";
 
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index dc96bbf..65ed4de 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -2,6 +2,8 @@ use strict;
 use warnings;
 use Cwd;
 use Config;
+use PostgresNode;
+use TestBase;
 use TestLib;
 use Test::More tests => 51;
 
@@ -9,8 +11,15 @@ program_help_ok('pg_basebackup');
 program_version_ok('pg_basebackup');
 program_options_handling_ok('pg_basebackup');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $tempdir = TestBase::tempdir;
+
+my $node = get_new_node();
+# Initialize node without replication settings
+$node->initNode(0);
+$node->startNode();
+my $pgdata = $node->getDataDir();
+
+$ENV{PGPORT} = $node->getPort();
 
 command_fails(['pg_basebackup'],
 	'pg_basebackup needs target directory specified');
@@ -26,19 +35,19 @@ if (open BADCHARS, ">>$tempdir/pgdata/FOO\xe0\xe0\xe0BAR")
 	close BADCHARS;
 }
 
-configure_hba_for_replication "$tempdir/pgdata";
-system_or_bail 'pg_ctl', '-D', "$tempdir/pgdata", 'reload';
+$node->setReplicationConf();
+system_or_bail 'pg_ctl', '-D', $pgdata, 'reload';
 
 command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup fails because of WAL configuration');
 
-open CONF, ">>$tempdir/pgdata/postgresql.conf";
+open CONF, ">>$pgdata/postgresql.conf";
 print CONF "max_replication_slots = 10\n";
 print CONF "max_wal_senders = 10\n";
 print CONF "wal_level = archive\n";
 close CONF;
-restart_test_server;
+$node->restartNode();
 
 command_ok([ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup runs');
@@ -81,13 +90,13 @@ command_fails(
 
 # Tar format doesn't support filenames longer than 100 bytes.
 my $superlongname = "superlongname_" . ("x" x 100);
-my $superlongpath = "$tempdir/pgdata/$superlongname";
+my $superlongpath = "$pgdata/$superlongname";
 
 open FILE, ">$superlongpath" or die "unable to create file $superlongpath";
 close FILE;
 command_fails([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l1", '-Ft' ],
 	'pg_basebackup tar with long name fails');
-unlink "$tempdir/pgdata/$superlongname";
+unlink "$pgdata/$superlongname";
 
 # The following tests test symlinks. Windows doesn't have symlinks, so
 # skip on Windows.
@@ -98,7 +107,7 @@ SKIP: {
 	# to our physical temp location.  That way we can use shorter names
 	# for the tablespace directories, which hopefully won't run afoul of
 	# the 99 character length limit.
-	my $shorter_tempdir = tempdir_short . "/tempdir";
+	my $shorter_tempdir = TestBase::tempdir_short . "/tempdir";
 	symlink "$tempdir", $shorter_tempdir;
 
 	mkdir "$tempdir/tblspc1";
@@ -120,7 +129,7 @@ SKIP: {
 			"-T$shorter_tempdir/tblspc1=$tempdir/tbackup/tblspc1" ],
 		'plain format with tablespaces succeeds with tablespace mapping');
 	ok(-d "$tempdir/tbackup/tblspc1", 'tablespace was relocated');
-	opendir(my $dh, "$tempdir/pgdata/pg_tblspc") or die;
+	opendir(my $dh, "$pgdata/pg_tblspc") or die;
 	ok( (   grep {
 		-l "$tempdir/backup1/pg_tblspc/$_"
 			and readlink "$tempdir/backup1/pg_tblspc/$_" eq
diff --git a/src/bin/pg_controldata/t/001_pg_controldata.pl b/src/bin/pg_controldata/t/001_pg_controldata.pl
index e2b0d42..343223b 100644
--- a/src/bin/pg_controldata/t/001_pg_controldata.pl
+++ b/src/bin/pg_controldata/t/001_pg_controldata.pl
@@ -1,16 +1,19 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
-my $tempdir = TestLib::tempdir;
-
 program_help_ok('pg_controldata');
 program_version_ok('pg_controldata');
 program_options_handling_ok('pg_controldata');
 command_fails(['pg_controldata'], 'pg_controldata without arguments fails');
 command_fails([ 'pg_controldata', 'nonexistent' ],
-	'pg_controldata with nonexistent directory fails');
-standard_initdb "$tempdir/data";
-command_like([ 'pg_controldata', "$tempdir/data" ],
+			  'pg_controldata with nonexistent directory fails');
+
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
+
+command_like([ 'pg_controldata', $node->getDataDir() ],
 	qr/checkpoint/, 'pg_controldata produces output');
diff --git a/src/bin/pg_ctl/t/001_start_stop.pl b/src/bin/pg_ctl/t/001_start_stop.pl
index f57abce..d76fe80 100644
--- a/src/bin/pg_ctl/t/001_start_stop.pl
+++ b/src/bin/pg_ctl/t/001_start_stop.pl
@@ -1,11 +1,12 @@
 use strict;
 use warnings;
 use Config;
+use TestBase;
 use TestLib;
 use Test::More tests => 17;
 
-my $tempdir       = TestLib::tempdir;
-my $tempdir_short = TestLib::tempdir_short;
+my $tempdir       = TestBase::tempdir;
+my $tempdir_short = TestBase::tempdir_short;
 
 program_help_ok('pg_ctl');
 program_version_ok('pg_ctl');
diff --git a/src/bin/pg_ctl/t/002_status.pl b/src/bin/pg_ctl/t/002_status.pl
index 31f7c72..74cb68a 100644
--- a/src/bin/pg_ctl/t/002_status.pl
+++ b/src/bin/pg_ctl/t/002_status.pl
@@ -1,22 +1,25 @@
 use strict;
 use warnings;
+use PostgresNode;
+use TestBase;
 use TestLib;
 use Test::More tests => 3;
 
-my $tempdir       = TestLib::tempdir;
-my $tempdir_short = TestLib::tempdir_short;
+my $tempdir       = TestBase::tempdir;
+my $tempdir_short = TestBase::tempdir_short;
 
 command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/nonexistent" ],
 	4, 'pg_ctl status with nonexistent directory');
 
-standard_initdb "$tempdir/data";
+my $node = get_new_node();
+$node->initNode();
 
-command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/data" ],
+command_exit_is([ 'pg_ctl', 'status', '-D', $node->getDataDir() ],
 	3, 'pg_ctl status with server not running');
 
 system_or_bail 'pg_ctl', '-l', "$tempdir/logfile", '-D',
-  "$tempdir/data", '-w', 'start';
-command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/data" ],
+  $node->getDataDir(), '-w', 'start';
+command_exit_is([ 'pg_ctl', 'status', '-D', $node->getDataDir() ],
 	0, 'pg_ctl status with server running');
 
-system_or_bail 'pg_ctl', 'stop', '-D', "$tempdir/data", '-m', 'fast';
+system_or_bail 'pg_ctl', 'stop', '-D', $node->getDataDir(), '-m', 'fast';
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..5124df8 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -37,6 +37,8 @@ package RewindTest;
 use strict;
 use warnings;
 
+use PostgresNode;
+use TestBase;
 use TestLib;
 use Test::More;
 
@@ -47,10 +49,8 @@ use IPC::Run qw(run start);
 
 use Exporter 'import';
 our @EXPORT = qw(
-  $connstr_master
-  $connstr_standby
-  $test_master_datadir
-  $test_standby_datadir
+  $node_master
+  $node_standby
 
   append_to_file
   master_psql
@@ -66,15 +66,9 @@ our @EXPORT = qw(
   clean_rewind_test
 );
 
-our $test_master_datadir  = "$tmp_check/data_master";
-our $test_standby_datadir = "$tmp_check/data_standby";
-
-# Define non-conflicting ports for both nodes.
-my $port_master  = $ENV{PGPORT};
-my $port_standby = $port_master + 1;
-
-my $connstr_master  = "port=$port_master";
-my $connstr_standby = "port=$port_standby";
+# Ports of both nodes.
+our $node_master = undef;
+our $node_standby = undef;
 
 $ENV{PGDATABASE} = "postgres";
 
@@ -82,16 +76,16 @@ sub master_psql
 {
 	my $cmd = shift;
 
-	system_or_bail 'psql', '-q', '--no-psqlrc', '-d', $connstr_master,
-	  '-c', "$cmd";
+	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
+	  $node_master->getConnStr(), '-c', "$cmd";
 }
 
 sub standby_psql
 {
 	my $cmd = shift;
 
-	system_or_bail 'psql', '-q', '--no-psqlrc', '-d', $connstr_standby,
-	  '-c', "$cmd";
+	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
+      $node_standby->getConnStr(), '-c', "$cmd";
 }
 
 # Run a query against the master, and check that the output matches what's
@@ -104,7 +98,7 @@ sub check_query
 	# we want just the output, no formatting
 	my $result = run [
 		'psql',          '-q', '-A', '-t', '--no-psqlrc', '-d',
-		$connstr_master, '-c', $query ],
+		$node_master->getConnStr(), '-c', $query ],
 	  '>', \$stdout, '2>', \$stderr;
 
 	# We don't use ok() for the exit code and stderr, because we want this
@@ -169,12 +163,11 @@ sub append_to_file
 sub setup_cluster
 {
 	# Initialize master, data checksums are mandatory
-	rmtree($test_master_datadir);
-	standard_initdb($test_master_datadir);
+	$node_master = get_new_node();
+	$node_master->initNode();
 
 	# Custom parameters for master's postgresql.conf
-	append_to_file(
-		"$test_master_datadir/postgresql.conf", qq(
+	$node_master->appendConf("postgresql.conf", qq(
 wal_level = hot_standby
 max_wal_senders = 2
 wal_keep_segments = 20
@@ -185,17 +178,11 @@ hot_standby = on
 autovacuum = off
 max_connections = 10
 ));
-
-	# Accept replication connections on master
-	configure_hba_for_replication $test_master_datadir;
 }
 
 sub start_master
 {
-	system_or_bail('pg_ctl' , '-w',
-				   '-D' , $test_master_datadir,
-				   '-l',  "$log_path/master.log",
-				   "-o", "-p $port_master", 'start');
+	$node_master->startNode();
 
 	#### Now run the test-specific parts to initialize the master before setting
 	# up standby
@@ -203,24 +190,19 @@ sub start_master
 
 sub create_standby
 {
+	$node_standby = get_new_node();
+	$node_master->backupNode('my_backup');
+	$node_standby->initNodeFromBackup($node_master, 'my_backup');
+	my $connstr_master = $node_master->getConnStr();
 
-	# Set up standby with necessary parameter
-	rmtree $test_standby_datadir;
-
-	# Base backup is taken with xlog files included
-	system_or_bail('pg_basebackup', '-D', $test_standby_datadir,
-				   '-p', $port_master, '-x');
-	append_to_file(
-		"$test_standby_datadir/recovery.conf", qq(
+	$node_standby->appendConf("recovery.conf", qq(
 primary_conninfo='$connstr_master application_name=rewind_standby'
 standby_mode=on
 recovery_target_timeline='latest'
 ));
 
 	# Start standby
-	system_or_bail('pg_ctl', '-w', '-D', $test_standby_datadir,
-				   '-l', "$log_path/standby.log",
-				   '-o', "-p $port_standby", 'start');
+	$node_standby->startNode();
 
 	# The standby may have WAL to apply before it matches the primary.  That
 	# is fine, because no test examines the standby before promotion.
@@ -234,14 +216,14 @@ sub promote_standby
 	# Wait for the standby to receive and write all WAL.
 	my $wal_received_query =
 "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = 'rewind_standby';";
-	poll_query_until($wal_received_query, $connstr_master)
+	poll_query_until($wal_received_query, $node_master->getConnStr())
 	  or die "Timed out while waiting for standby to receive and write WAL";
 
 	# Now promote slave and insert some new data on master, this will put
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
-	system_or_bail('pg_ctl', '-w', '-D', $test_standby_datadir, 'promote');
-	poll_query_until("SELECT NOT pg_is_in_recovery()", $connstr_standby)
+	system_or_bail('pg_ctl', '-w', '-D', $node_standby->getDataDir(), 'promote');
+	poll_query_until("SELECT NOT pg_is_in_recovery()", $node_standby->getConnStr())
 	  or die "Timed out while waiting for promotion of standby";
 
 	# Force a checkpoint after the promotion. pg_rewind looks at the control
@@ -256,9 +238,13 @@ sub promote_standby
 sub run_pg_rewind
 {
 	my $test_mode = shift;
+	my $master_pgdata = $node_master->getDataDir();
+	my $standby_pgdata = $node_standby->getDataDir();
+	my $standby_connstr = $node_standby->getConnStr('postgres');
+	my $tmp_folder = TestBase::tempdir;
 
 	# Stop the master and be ready to perform the rewind
-	system_or_bail('pg_ctl', '-D', $test_master_datadir, '-m', 'fast', 'stop');
+	$node_master->stopNode();
 
 	# At this point, the rewind processing is ready to run.
 	# We now have a very simple scenario with a few diverged WAL record.
@@ -267,20 +253,19 @@ sub run_pg_rewind
 
 	# Keep a temporary postgresql.conf for master node or it would be
 	# overwritten during the rewind.
-	copy("$test_master_datadir/postgresql.conf",
-		 "$tmp_check/master-postgresql.conf.tmp");
+	copy("$master_pgdata/postgresql.conf",
+		 "$tmp_folder/master-postgresql.conf.tmp");
 
 	# Now run pg_rewind
 	if ($test_mode eq "local")
 	{
 		# Do rewind using a local pgdata as source
 		# Stop the master and be ready to perform the rewind
-		system_or_bail('pg_ctl', '-D', $test_standby_datadir,
-					   '-m', 'fast', 'stop');
+		$node_standby->stopNode();
 		command_ok(['pg_rewind',
 					"--debug",
-					"--source-pgdata=$test_standby_datadir",
-					"--target-pgdata=$test_master_datadir"],
+					"--source-pgdata=$standby_pgdata",
+					"--target-pgdata=$master_pgdata"],
 				   'pg_rewind local');
 	}
 	elsif ($test_mode eq "remote")
@@ -289,33 +274,30 @@ sub run_pg_rewind
 		command_ok(['pg_rewind',
 					"--debug",
 					"--source-server",
-					"port=$port_standby dbname=postgres",
-					"--target-pgdata=$test_master_datadir"],
+					$standby_connstr,
+					"--target-pgdata=$master_pgdata"],
 				   'pg_rewind remote');
 	}
 	else
 	{
-
 		# Cannot come here normally
 		die("Incorrect test mode specified");
 	}
 
 	# Now move back postgresql.conf with old settings
-	move("$tmp_check/master-postgresql.conf.tmp",
-		 "$test_master_datadir/postgresql.conf");
+	move("$tmp_folder/master-postgresql.conf.tmp",
+		 "$master_pgdata/postgresql.conf");
 
 	# Plug-in rewound node to the now-promoted standby node
-	append_to_file(
-		"$test_master_datadir/recovery.conf", qq(
+	my $port_standby = $node_standby->getPort();
+	$node_master->appendConf('recovery.conf', qq(
 primary_conninfo='port=$port_standby'
 standby_mode=on
 recovery_target_timeline='latest'
 ));
 
 	# Restart the master to check that rewind went correctly
-	system_or_bail('pg_ctl', '-w', '-D', $test_master_datadir,
-				   '-l', "$log_path/master.log",
-				   '-o', "-p $port_master", 'start');
+	$node_master->restartNode();
 
 	#### Now run the test-specific parts to check the result
 }
@@ -323,22 +305,6 @@ recovery_target_timeline='latest'
 # Clean up after the test. Stop both servers, if they're still running.
 sub clean_rewind_test
 {
-	if ($test_master_datadir)
-	{
-		system
-		  'pg_ctl', '-D', $test_master_datadir, '-m', 'immediate', 'stop';
-	}
-	if ($test_standby_datadir)
-	{
-		system
-		  'pg_ctl', '-D', $test_standby_datadir, '-m', 'immediate', 'stop';
-	}
-}
-
-# Stop the test servers, just in case they're still running.
-END
-{
-	my $save_rc = $?;
-	clean_rewind_test();
-	$? = $save_rc;
+	teardown_node($node_master) if (defined($node_master));
+	teardown_node($node_standby) if (defined($node_standby));
 }
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index d317f53..8494121 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -2,6 +2,7 @@
 
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 4;
 
@@ -17,7 +18,7 @@ sub run_test
 	RewindTest::setup_cluster();
 	RewindTest::start_master();
 
-	my $test_master_datadir = $RewindTest::test_master_datadir;
+	my $test_master_datadir = $node_master->getDataDir();
 
 	# Create a subdir and files that will be present in both
 	mkdir "$test_master_datadir/tst_both_dir";
@@ -30,6 +31,7 @@ sub run_test
 	RewindTest::create_standby();
 
 	# Create different subdirs and files in master and standby
+	my $test_standby_datadir = $node_standby->getDataDir();
 
 	mkdir "$test_standby_datadir/tst_standby_dir";
 	append_to_file "$test_standby_datadir/tst_standby_dir/standby_file1",
diff --git a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
index c5f72e2..4870f6a 100644
--- a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
+++ b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
@@ -5,6 +5,7 @@ use strict;
 use warnings;
 use File::Copy;
 use File::Path qw(rmtree);
+use TestBase;
 use TestLib;
 use Test::More;
 if ($windows_os)
@@ -28,6 +29,8 @@ sub run_test
 	rmtree($master_xlogdir);
 	RewindTest::setup_cluster();
 
+	my $test_master_datadir = $node_master->getDataDir();
+
 	# turn pg_xlog into a symlink
 	print("moving $test_master_datadir/pg_xlog to $master_xlogdir\n");
 	move("$test_master_datadir/pg_xlog", $master_xlogdir) or die;
diff --git a/src/bin/scripts/t/010_clusterdb.pl b/src/bin/scripts/t/010_clusterdb.pl
index dc0d78a..80b3cd9 100644
--- a/src/bin/scripts/t/010_clusterdb.pl
+++ b/src/bin/scripts/t/010_clusterdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
@@ -7,20 +8,24 @@ program_help_ok('clusterdb');
 program_version_ok('clusterdb');
 program_options_handling_ok('clusterdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
-	[ 'clusterdb', 'postgres' ],
+$ENV{PGPORT} = $node->getPort();
+$ENV{PGDATABASE} = 'postgres';
+
+issues_sql_like($node,
+	[ 'clusterdb' ],
 	qr/statement: CLUSTER;/,
 	'SQL CLUSTER run');
 
-command_fails([ 'clusterdb', '-t', 'nonexistent', 'postgres' ],
-	'fails with nonexistent table');
+command_fails([ 'clusterdb', '-t', 'nonexistent' ],
+			  'fails with nonexistent table');
 
 psql 'postgres',
-'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';
-issues_sql_like(
-	[ 'clusterdb', '-t', 'test1', 'postgres' ],
+	'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';
+issues_sql_like($node,
+	[ 'clusterdb', '-t', 'test1' ],
 	qr/statement: CLUSTER test1;/,
 	'cluster specific table');
diff --git a/src/bin/scripts/t/011_clusterdb_all.pl b/src/bin/scripts/t/011_clusterdb_all.pl
index 7769f70..b66ade3 100644
--- a/src/bin/scripts/t/011_clusterdb_all.pl
+++ b/src/bin/scripts/t/011_clusterdb_all.pl
@@ -1,12 +1,19 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+# cluster -a is not compatible with -d, hence enforce environment variables
+# correctly.
+$ENV{PGDATABASE} = 'postgres';
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'clusterdb', '-a' ],
 	qr/statement: CLUSTER.*statement: CLUSTER/s,
 	'cluster all databases');
diff --git a/src/bin/scripts/t/020_createdb.pl b/src/bin/scripts/t/020_createdb.pl
index a44283c..5d502b1 100644
--- a/src/bin/scripts/t/020_createdb.pl
+++ b/src/bin/scripts/t/020_createdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
@@ -7,14 +8,17 @@ program_help_ok('createdb');
 program_version_ok('createdb');
 program_options_handling_ok('createdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'createdb', 'foobar1' ],
 	qr/statement: CREATE DATABASE foobar1/,
 	'SQL CREATE DATABASE run');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'createdb', '-l', 'C', '-E', 'LATIN1', '-T', 'template0', 'foobar2' ],
 	qr/statement: CREATE DATABASE foobar2 ENCODING 'LATIN1'/,
 	'create database with encoding');
diff --git a/src/bin/scripts/t/030_createlang.pl b/src/bin/scripts/t/030_createlang.pl
index 7ff0a3e..2bfc540 100644
--- a/src/bin/scripts/t/030_createlang.pl
+++ b/src/bin/scripts/t/030_createlang.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 14;
 
@@ -7,18 +8,22 @@ program_help_ok('createlang');
 program_version_ok('createlang');
 program_options_handling_ok('createlang');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
+
+$ENV{PGPORT} = $node->getPort();
+$ENV{PGDATABASE} = 'postgres';
 
 command_fails(
-	[ 'createlang', 'plpgsql', 'postgres' ],
+	[ 'createlang', 'plpgsql' ],
 	'fails if language already exists');
 
-psql 'postgres', 'DROP EXTENSION plpgsql';
-issues_sql_like(
-	[ 'createlang', 'plpgsql', 'postgres' ],
+psql $node->getConnStr('postgres'), 'DROP EXTENSION plpgsql';
+issues_sql_like($node,
+	[ 'createlang', 'plpgsql' ],
 	qr/statement: CREATE EXTENSION "plpgsql"/,
 	'SQL CREATE EXTENSION run');
 
-command_like([ 'createlang', '--list', 'postgres' ],
+command_like([ 'createlang', '--list' ],
 	qr/plpgsql/, 'list output');
diff --git a/src/bin/scripts/t/040_createuser.pl b/src/bin/scripts/t/040_createuser.pl
index 4d44e14..0238b2f 100644
--- a/src/bin/scripts/t/040_createuser.pl
+++ b/src/bin/scripts/t/040_createuser.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 17;
 
@@ -7,24 +8,29 @@ program_help_ok('createuser');
 program_version_ok('createuser');
 program_options_handling_ok('createuser');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+$ENV{PGDATABASE} = 'postgres';
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'createuser', 'user1' ],
 qr/statement: CREATE ROLE user1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN;/,
 	'SQL CREATE USER run');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'createuser', '-L', 'role1' ],
 qr/statement: CREATE ROLE role1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT NOLOGIN;/,
 	'create a non-login role');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'createuser', '-r', 'user2' ],
 qr/statement: CREATE ROLE user2 NOSUPERUSER NOCREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a CREATEROLE user');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'createuser', '-s', 'user3' ],
 qr/statement: CREATE ROLE user3 SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a superuser');
 
-command_fails([ 'createuser', 'user1' ], 'fails if role already exists');
+command_fails([ 'createuser', 'user1' ],
+			  'fails if role already exists');
diff --git a/src/bin/scripts/t/050_dropdb.pl b/src/bin/scripts/t/050_dropdb.pl
index 3065e50..fc11e46 100644
--- a/src/bin/scripts/t/050_dropdb.pl
+++ b/src/bin/scripts/t/050_dropdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,11 +8,14 @@ program_help_ok('dropdb');
 program_version_ok('dropdb');
 program_options_handling_ok('dropdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-psql 'postgres', 'CREATE DATABASE foobar1';
-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+psql $node->getConnStr('postgres'), 'CREATE DATABASE foobar1';
+issues_sql_like($node,
 	[ 'dropdb', 'foobar1' ],
 	qr/statement: DROP DATABASE foobar1/,
 	'SQL DROP DATABASE run');
diff --git a/src/bin/scripts/t/060_droplang.pl b/src/bin/scripts/t/060_droplang.pl
index 6a21d7e..bccd9ce 100644
--- a/src/bin/scripts/t/060_droplang.pl
+++ b/src/bin/scripts/t/060_droplang.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,10 +8,13 @@ program_help_ok('droplang');
 program_version_ok('droplang');
 program_options_handling_ok('droplang');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'droplang', 'plpgsql', 'postgres' ],
 	qr/statement: DROP EXTENSION "plpgsql"/,
 	'SQL DROP EXTENSION run');
diff --git a/src/bin/scripts/t/070_dropuser.pl b/src/bin/scripts/t/070_dropuser.pl
index bbb3b79..9d5bef7 100644
--- a/src/bin/scripts/t/070_dropuser.pl
+++ b/src/bin/scripts/t/070_dropuser.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,11 +8,14 @@ program_help_ok('dropuser');
 program_version_ok('dropuser');
 program_options_handling_ok('dropuser');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
+
+$ENV{PGPORT} = $node->getPort();
 
 psql 'postgres', 'CREATE ROLE foobar1';
-issues_sql_like(
+issues_sql_like($node,
 	[ 'dropuser', 'foobar1' ],
 	qr/statement: DROP ROLE foobar1/,
 	'SQL DROP ROLE run');
diff --git a/src/bin/scripts/t/080_pg_isready.pl b/src/bin/scripts/t/080_pg_isready.pl
index f432505..47aef9b 100644
--- a/src/bin/scripts/t/080_pg_isready.pl
+++ b/src/bin/scripts/t/080_pg_isready.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 10;
 
@@ -9,7 +10,10 @@ program_options_handling_ok('pg_isready');
 
 command_fails(['pg_isready'], 'fails with no server running');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
+
+$ENV{PGPORT} = $node->getPort();
 
 command_ok(['pg_isready'], 'succeeds with server running');
diff --git a/src/bin/scripts/t/090_reindexdb.pl b/src/bin/scripts/t/090_reindexdb.pl
index 42628c2..9bcf7ec 100644
--- a/src/bin/scripts/t/090_reindexdb.pl
+++ b/src/bin/scripts/t/090_reindexdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 20;
 
@@ -7,35 +8,37 @@ program_help_ok('reindexdb');
 program_version_ok('reindexdb');
 program_options_handling_ok('reindexdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
+$ENV{PGPORT} = $node->getPort();
 $ENV{PGOPTIONS} = '--client-min-messages=WARNING';
 
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', 'postgres' ],
 	qr/statement: REINDEX DATABASE postgres;/,
 	'SQL REINDEX run');
 
 psql 'postgres',
   'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a);';
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX TABLE test1;/,
 	'reindex specific table');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', '-i', 'test1x', 'postgres' ],
 	qr/statement: REINDEX INDEX test1x;/,
 	'reindex specific index');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', '-S', 'pg_catalog', 'postgres' ],
 	qr/statement: REINDEX SCHEMA pg_catalog;/,
 	'reindex specific schema');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', '-s', 'postgres' ],
 	qr/statement: REINDEX SYSTEM postgres;/,
 	'reindex system tables');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', '-v', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX \(VERBOSE\) TABLE test1;/,
 	'reindex with verbose output');
diff --git a/src/bin/scripts/t/091_reindexdb_all.pl b/src/bin/scripts/t/091_reindexdb_all.pl
index ffadf29..732ed1d 100644
--- a/src/bin/scripts/t/091_reindexdb_all.pl
+++ b/src/bin/scripts/t/091_reindexdb_all.pl
@@ -1,14 +1,17 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
+$ENV{PGPORT} = $node->getPort();
 $ENV{PGOPTIONS} = '--client-min-messages=WARNING';
 
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', '-a' ],
 	qr/statement: REINDEX.*statement: REINDEX/s,
 	'reindex all databases');
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index ac160ba..5b825f1 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 18;
 
@@ -7,26 +8,29 @@ program_help_ok('vacuumdb');
 program_version_ok('vacuumdb');
 program_options_handling_ok('vacuumdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'vacuumdb', 'postgres' ],
 	qr/statement: VACUUM;/,
 	'SQL VACUUM run');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'vacuumdb', '-f', 'postgres' ],
 	qr/statement: VACUUM \(FULL\);/,
 	'vacuumdb -f');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'vacuumdb', '-F', 'postgres' ],
 	qr/statement: VACUUM \(FREEZE\);/,
 	'vacuumdb -F');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'vacuumdb', '-z', 'postgres' ],
 	qr/statement: VACUUM \(ANALYZE\);/,
 	'vacuumdb -z');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'vacuumdb', '-Z', 'postgres' ],
 	qr/statement: ANALYZE;/,
 	'vacuumdb -Z');
diff --git a/src/bin/scripts/t/101_vacuumdb_all.pl b/src/bin/scripts/t/101_vacuumdb_all.pl
index e90f321..829d7c9 100644
--- a/src/bin/scripts/t/101_vacuumdb_all.pl
+++ b/src/bin/scripts/t/101_vacuumdb_all.pl
@@ -1,12 +1,16 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'vacuumdb', '-a' ],
 	qr/statement: VACUUM.*statement: VACUUM/s,
 	'vacuum all databases');
diff --git a/src/bin/scripts/t/102_vacuumdb_stages.pl b/src/bin/scripts/t/102_vacuumdb_stages.pl
index 57b980e..db3288d 100644
--- a/src/bin/scripts/t/102_vacuumdb_stages.pl
+++ b/src/bin/scripts/t/102_vacuumdb_stages.pl
@@ -1,12 +1,16 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 4;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'vacuumdb', '--analyze-in-stages', 'postgres' ],
 qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE.*
@@ -17,7 +21,7 @@ qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
 	'analyze three times');
 
 
-issues_sql_like(
+issues_sql_like($node,
 	[ 'vacuumdb', '--analyze-in-stages', '--all' ],
 qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE.*
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
new file mode 100644
index 0000000..ff0e90d
--- /dev/null
+++ b/src/test/perl/PostgresNode.pm
@@ -0,0 +1,233 @@
+# PostgresNode, simple node representation for regression tests
+#
+# Regression tests should use this basic class infrastructure to define
+# nodes that need used in the complex scenarios. This object is wanted
+# simple with only a basic set of routines able to configure, initialize
+# and manage a node.
+
+package PostgresNode;
+
+use strict;
+use warnings;
+
+use RecursiveCopy;
+use TestBase;
+
+sub new {
+	my $class = shift;
+	my $pghost = shift;
+	my $pgport = shift;
+	my $self = {
+		_port => undef,
+		_host => undef,
+		_basedir => undef,
+		_applname => undef,
+		_logfile => undef
+			};
+
+	# Set up each field
+	$self->{_port} = $pgport;
+	$self->{_host} = $pghost;
+	$self->{_basedir} = TestBase::tempdir;
+	$self->{_applname} = "node_$pgport";
+	$self->{_logfile} = "$log_path/node_$pgport.log";
+	bless $self, $class;
+	$self->dumpNodeInfo();
+	return $self;
+}
+
+# Get routines for various variables
+sub getPort {
+	my ( $self ) = @_;
+	return $self->{_port};
+}
+sub getHost {
+	my ( $self ) = @_;
+	return $self->{_host};
+}
+sub getConnStr {
+	my ( $self, $dbname ) = @_;
+	my $pgport = $self->getPort();
+	my $pghost = $self->getHost();
+	if (!defined($dbname))
+	{
+		return "port=$pgport host=$pghost";
+	}
+	return "port=$pgport host=$pghost dbname=$dbname";
+}
+sub getDataDir {
+	my ( $self ) = @_;
+	my $basedir = $self->{_basedir};
+	return "$basedir/pgdata";
+}
+sub getApplName {
+	my ( $self ) = @_;
+	return $self->{_applname};
+}
+sub getLogFile {
+	my ( $self ) = @_;
+	return $self->{_logfile};
+}
+sub getArchiveDir {
+	my ( $self ) = @_;
+	my $basedir = $self->{_basedir};
+	return "$basedir/archives";
+}
+sub getBackupDir {
+	my ( $self ) = @_;
+	my $basedir = $self->{_basedir};
+	return "$basedir/backup";
+}
+
+# Dump node information
+sub dumpNodeInfo {
+	my ( $self ) = @_;
+	print 'Data directory: ' . $self->getDataDir() . "\n";
+	print 'Backup directory: ' . $self->getBackupDir() . "\n";
+	print 'Archive directory: ' . $self->getArchiveDir() . "\n";
+	print 'Connection string: ' . $self->getConnStr() . "\n";
+	print 'Application name: ' . $self->getApplName() . "\n";
+	print 'Log file: ' . $self->getLogFile() . "\n";
+}
+
+sub setReplicationConf
+{
+	my ( $self ) = @_;
+	my $pgdata = $self->getDataDir();
+
+	open my $hba, ">>$pgdata/pg_hba.conf";
+	print $hba "\n# Allow replication (set up by PostgresNode.pm)\n";
+	if (! $windows_os)
+	{
+		print $hba "local replication all trust\n";
+	}
+	else
+    {
+		print $hba "host replication all 127.0.0.1/32 sspi include_realm=1 map=regress\n";
+	}
+	close $hba;
+
+}
+
+# Initialize a new cluster for testing.
+#
+# Authentication is set up so that only the current OS user can access the
+# cluster. On Unix, we use Unix domain socket connections, with the socket in
+# a directory that's only accessible to the current user to ensure that.
+# On Windows, we use SSPI authentication to ensure the same (by pg_regress
+# --config-auth).
+sub initNode
+{
+	my ( $self, $repconf ) = @_;
+	my $port = $self->getPort();
+	my $pgdata = $self->getDataDir();
+	my $host = $self->getHost();
+
+	$repconf = 1 if (!defined($repconf));
+
+	mkdir $self->getBackupDir();
+	mkdir $self->getArchiveDir();
+
+	system_or_bail('initdb', '-D', $pgdata, '-A' , 'trust', '-N');
+	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
+
+	open my $conf, ">>$pgdata/postgresql.conf";
+	print $conf "\n# Added by TestLib.pm)\n";
+	print $conf "fsync = off\n";
+	print $conf "log_statement = all\n";
+	print $conf "port = $port\n";
+	if ($windows_os)
+    {
+		print $conf "listen_addresses = '$host'\n";
+	}
+	else
+	{
+		print $conf "unix_socket_directories = '$host'\n";
+		print $conf "listen_addresses = ''\n";
+	}
+	close $conf;
+
+	$self->setReplicationConf() if ($repconf);
+}
+sub appendConf
+{
+	my ($self, $filename, $str) = @_;
+
+	my $conffile = $self->getDataDir() . '/' . $filename;
+
+	open my $fh, ">>", $conffile or die "could not open file $filename";
+	print $fh $str;
+	close $fh;
+}
+sub backupNode
+{
+	my ($self, $backup_name) = @_;
+    my $backup_path = $self->getBackupDir() . '/' . $backup_name;
+	my $port = $self->getPort();
+
+	print "# Taking backup $backup_name from node with port $port\n";
+	system_or_bail("pg_basebackup -D $backup_path -p $port -x");
+	print "# Backup finished\n";
+}
+sub initNodeFromBackup
+{
+	my ( $self, $root_node, $backup_name ) = @_;
+	my $backup_path = $root_node->getBackupDir() . '/' . $backup_name;
+	my $port = $self->getPort();
+	my $root_port = $root_node->getPort();
+
+	print "Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	die "Backup $backup_path does not exist" if (! -d $backup_path);
+
+	mkdir $self->getBackupDir();
+	mkdir $self->getArchiveDir();
+
+	my $data_path = $self->getDataDir();
+    rmdir($data_path);
+    RecursiveCopy::copypath($backup_path, $data_path);
+    chmod(0700, $data_path);
+
+	# Base configuration for this node
+	$self->appendConf('postgresql.conf', qq(
+port = $port
+));
+	$self->setReplicationConf();
+}
+sub startNode
+{
+	my ( $self ) = @_;
+	my $port = $self->getPort();
+	my $pgdata = $self->getDataDir();
+	print("### Starting test server in $pgdata\n");
+	my $ret = system_log('pg_ctl', '-w', '-D', $self->getDataDir(),
+						 '-l', $self->getLogFile(),
+						 'start');
+
+	if ($ret != 0)
+	{
+		print "# pg_ctl failed; logfile:\n";
+		system('cat', $self->getLogFile());
+		BAIL_OUT("pg_ctl failed");
+	}
+}
+sub stopNode
+{
+	my ( $self, $mode ) = @_;
+	my $port = $self->getPort();
+	my $pgdata = $self->getDataDir();
+	$mode = 'fast' if (!defined($mode));
+	print "### Stopping node in $pgdata with port $port using mode $mode\n";
+	system_log('pg_ctl', '-D', $pgdata, '-m',
+			   $mode, 'stop');
+}
+sub restartNode
+{
+	my ( $self ) = @_;
+	my $port = $self->getPort();
+	my $pgdata = $self->getDataDir();
+	my $logfile = $self->getLogFile();
+	system_log('pg_ctl', '-D', $pgdata, '-w', '-l',
+			   $logfile, 'restart');
+}
+
+1;
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
new file mode 100644
index 0000000..f06f975
--- /dev/null
+++ b/src/test/perl/RecursiveCopy.pm
@@ -0,0 +1,42 @@
+# RecursiveCopy, a simple recursive copy implementation
+#
+# Having this implementation has the advantage to not have the regression
+# test code rely on any external module for this simple operation.
+
+package RecursiveCopy;
+use strict;
+use warnings;
+
+use File::Basename;
+use File::Copy;
+
+sub copypath {
+	my $srcpath = shift;
+	my $destpath = shift;
+
+	die "Cannot operate on symlinks" if ( -l $srcpath || -l $destpath );
+
+	# This source path is a file, simply copy it to destination with the
+	# same name.
+	die "Destination path $destpath exists as file" if ( -f $destpath );
+	if ( -f $srcpath )
+	{
+		my $filename = basename($destpath);
+		copy($srcpath, "$destpath");
+		return 1;
+	}
+
+	die "Destination needs to be a directory" if (! -d $srcpath);
+	mkdir($destpath);
+
+	# Scan existing source directory and recursively copy everything.
+	opendir(my $directory, $srcpath);
+	while (my $entry = readdir($directory)) {
+		next if ($entry eq '.' || $entry eq '..');
+		RecursiveCopy::copypath("$srcpath/$entry", "$destpath/$entry");
+	}
+	closedir($directory);
+	return 1;
+}
+
+1;
diff --git a/src/test/perl/TestBase.pm b/src/test/perl/TestBase.pm
new file mode 100644
index 0000000..3a61939
--- /dev/null
+++ b/src/test/perl/TestBase.pm
@@ -0,0 +1,109 @@
+# Set of low-level routines dedicated to base tasks for regression
+# tests, like command execution and logging. This module should not
+# have dependencies with other modules dedicated to regression tests
+# of Postgres.
+
+package TestBase;
+
+use strict;
+use warnings;
+
+use Config;
+use Exporter 'import';
+use File::Basename;
+use File::Spec;
+use File::Temp ();
+use IPC::Run qw(run start);
+
+use Test::More;
+
+our @EXPORT = qw(
+  system_or_bail
+  system_log
+  run_log
+
+  $tmp_check
+  $log_path
+  $windows_os
+);
+
+# Internal modules
+use SimpleTee;
+
+our $windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
+
+# Open log file. For each test, the log file name uses the name of the
+# file launching this module, without the .pl suffix.
+our ($tmp_check, $log_path);
+$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
+$log_path = "$tmp_check/log";
+mkdir $tmp_check;
+mkdir $log_path;
+my $test_logfile = basename($0);
+$test_logfile =~ s/\.[^.]+$//;
+$test_logfile = "$log_path/regress_log_$test_logfile";
+open TESTLOG, '>', $test_logfile or die "Cannot open STDOUT to logfile: $!";
+
+# Hijack STDOUT and STDERR to the log file
+open(ORIG_STDOUT, ">&STDOUT");
+open(ORIG_STDERR, ">&STDERR");
+open(STDOUT, ">&TESTLOG");
+open(STDERR, ">&TESTLOG");
+
+# The test output (ok ...) needs to be printed to the original STDOUT so
+# that the 'prove' program can parse it, and display it to the user in
+# real time. But also copy it to the log file, to provide more context
+# in the log.
+my $builder = Test::More->builder;
+my $fh = $builder->output;
+tie *$fh, "SimpleTee", *ORIG_STDOUT, *TESTLOG;
+$fh = $builder->failure_output;
+tie *$fh, "SimpleTee", *ORIG_STDERR, *TESTLOG;
+
+# Enable auto-flushing for all the file handles. Stderr and stdout are
+# redirected to the same file, and buffering causes the lines to appear
+# in the log in confusing order.
+autoflush STDOUT 1;
+autoflush STDERR 1;
+autoflush TESTLOG 1;
+
+#
+# Helper functions
+#
+sub tempdir
+{
+	return File::Temp::tempdir(
+		'tmp_testXXXX',
+		DIR => $ENV{TESTDIR} || cwd(),
+		CLEANUP => 1);
+}
+
+sub tempdir_short
+{
+
+	# Use a separate temp dir outside the build tree for the
+	# Unix-domain socket, to avoid file name length issues.
+	return File::Temp::tempdir(CLEANUP => 1);
+}
+
+sub system_or_bail
+{
+	if (system_log(@_) != 0)
+	{
+		BAIL_OUT("system $_[0] failed: $?");
+	}
+}
+
+sub system_log
+{
+	print("# Running: " . join(" ", @_) ."\n");
+	return system(@_);
+}
+
+sub run_log
+{
+	print("# Running: " . join(" ", @{$_[0]}) ."\n");
+	return run (@_);
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..a1473c6 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -6,18 +6,11 @@ use warnings;
 use Config;
 use Exporter 'import';
 our @EXPORT = qw(
-  tempdir
-  tempdir_short
-  standard_initdb
-  configure_hba_for_replication
-  start_test_server
-  restart_test_server
+  get_new_node
+  teardown_node
   psql
   slurp_dir
   slurp_file
-  system_or_bail
-  system_log
-  run_log
 
   command_ok
   command_fails
@@ -27,10 +20,6 @@ our @EXPORT = qw(
   program_options_handling_ok
   command_like
   issues_sql_like
-
-  $tmp_check
-  $log_path
-  $windows_os
 );
 
 use Cwd;
@@ -39,46 +28,10 @@ use File::Spec;
 use File::Temp ();
 use IPC::Run qw(run start);
 
-use SimpleTee;
-
 use Test::More;
 
-our $windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
-
-# Open log file. For each test, the log file name uses the name of the
-# file launching this module, without the .pl suffix.
-our ($tmp_check, $log_path);
-$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
-$log_path = "$tmp_check/log";
-mkdir $tmp_check;
-mkdir $log_path;
-my $test_logfile = basename($0);
-$test_logfile =~ s/\.[^.]+$//;
-$test_logfile = "$log_path/regress_log_$test_logfile";
-open TESTLOG, '>', $test_logfile or die "Cannot open STDOUT to logfile: $!";
-
-# Hijack STDOUT and STDERR to the log file
-open(ORIG_STDOUT, ">&STDOUT");
-open(ORIG_STDERR, ">&STDERR");
-open(STDOUT, ">&TESTLOG");
-open(STDERR, ">&TESTLOG");
-
-# The test output (ok ...) needs to be printed to the original STDOUT so
-# that the 'prove' program can parse it, and display it to the user in
-# real time. But also copy it to the log file, to provide more context
-# in the log.
-my $builder = Test::More->builder;
-my $fh = $builder->output;
-tie *$fh, "SimpleTee", *ORIG_STDOUT, *TESTLOG;
-$fh = $builder->failure_output;
-tie *$fh, "SimpleTee", *ORIG_STDERR, *TESTLOG;
-
-# Enable auto-flushing for all the file handles. Stderr and stdout are
-# redirected to the same file, and buffering causes the lines to appear
-# in the log in confusing order.
-autoflush STDOUT 1;
-autoflush STDERR 1;
-autoflush TESTLOG 1;
+# Internal modules
+use TestBase;
 
 # Set to untranslated messages, to be able to compare program output
 # with expected strings.
@@ -94,129 +47,74 @@ delete $ENV{PGREQUIRESSL};
 delete $ENV{PGSERVICE};
 delete $ENV{PGSSLMODE};
 delete $ENV{PGUSER};
-
-if (!$ENV{PGPORT})
-{
-	$ENV{PGPORT} = 65432;
-}
-
-$ENV{PGPORT} = int($ENV{PGPORT}) % 65536;
-
-
-#
-# Helper functions
-#
-
-
-sub tempdir
-{
-	return File::Temp::tempdir(
-		'tmp_testXXXX',
-		DIR => $ENV{TESTDIR} || cwd(),
-		CLEANUP => 1);
-}
-
-sub tempdir_short
-{
-
-	# Use a separate temp dir outside the build tree for the
-	# Unix-domain socket, to avoid file name length issues.
-	return File::Temp::tempdir(CLEANUP => 1);
-}
-
-# Initialize a new cluster for testing.
-#
-# The PGHOST environment variable is set to connect to the new cluster.
-#
-# Authentication is set up so that only the current OS user can access the
-# cluster. On Unix, we use Unix domain socket connections, with the socket in
-# a directory that's only accessible to the current user to ensure that.
-# On Windows, we use SSPI authentication to ensure the same (by pg_regress
-# --config-auth).
-sub standard_initdb
-{
-	my $pgdata = shift;
-	system_or_bail('initdb', '-D', "$pgdata", '-A' , 'trust', '-N');
-	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
-
-	my $tempdir_short = tempdir_short;
-
-	open CONF, ">>$pgdata/postgresql.conf";
-	print CONF "\n# Added by TestLib.pm)\n";
-	print CONF "fsync = off\n";
-	if ($windows_os)
+delete $ENV{PGPORT};
+delete $ENV{PGHOST};
+
+# PGHOST is set once and for all through a single series of tests
+# when this module is loaded.
+my $test_pghost = $windows_os ? "127.0.0.1" : TestBase::tempdir_short;
+$ENV{PGHOST} = $test_pghost;
+$ENV{PGDATABASE} = 'postgres';
+
+# Tracking of last port value assigned to accelerate free port lookup.
+# XXX: Should this part use PG_VERSION_NUM?
+my $last_port_assigned =  90600 % 16384 + 49152;
+# Tracker of active nodes
+my @active_nodes = ();
+
+# get free port
+# register nodes in array
+# Get a port number not in use currently for a new node
+# As port number retrieval is based on the nodes currently running and
+# their presence in the list of registered ports, be sure that the node
+# that is consuming this port number has already been started and that
+# it is not registered yet.
+sub get_new_node
+{
+	my $found = 0;
+	my $port = $last_port_assigned;
+
+	while ($found == 0)
 	{
-		print CONF "listen_addresses = '127.0.0.1'\n";
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		if (!run_log(['pg_isready', '-p', $port]))
+		{
+			$found = 1;
+			# Found a potential candidate, check first that it is
+			# not included in the list of registered nodes.
+			foreach my $node (@active_nodes)
+			{
+				$found = 0 if ($node->getPort() == $port);
+			}
+		}
 	}
-	else
-	{
-		print CONF "unix_socket_directories = '$tempdir_short'\n";
-		print CONF "listen_addresses = ''\n";
-	}
-	close CONF;
-
-	$ENV{PGHOST}         = $windows_os ? "127.0.0.1" : $tempdir_short;
-}
 
-# Set up the cluster to allow replication connections, in the same way that
-# standard_initdb does for normal connections.
-sub configure_hba_for_replication
-{
-	my $pgdata = shift;
+	print "# Found free port $port\n";
+	# Lock port number found by creating a new node
+	my $node = new PostgresNode($test_pghost, $port);
 
-	open HBA, ">>$pgdata/pg_hba.conf";
-	print HBA "\n# Allow replication (set up by TestLib.pm)\n";
-	if (! $windows_os)
-	{
-		print HBA "local replication all trust\n";
-	}
-	else
-	{
-		print HBA "host replication all 127.0.0.1/32 sspi include_realm=1 map=regress\n";
-	}
-	close HBA;
+	# Add node to list of nodes currently in use
+	push(@active_nodes, $node);
+	$last_port_assigned = $port;
+	return $node;
 }
 
-my ($test_server_datadir, $test_server_logfile);
-
-
-# Initialize a new cluster for testing in given directory, and start it.
-sub start_test_server
+# Remove any traces of given node.
+sub teardown_node
 {
-	my ($tempdir) = @_;
-	my $ret;
+	my $node = shift;
 
-	print("### Starting test server in $tempdir\n");
-	standard_initdb "$tempdir/pgdata";
-
-	$ret = system_log('pg_ctl', '-D', "$tempdir/pgdata", '-w', '-l',
-	  "$log_path/postmaster.log", '-o', "--log-statement=all",
-	  'start');
-
-	if ($ret != 0)
-	{
-		print "# pg_ctl failed; logfile:\n";
-		system('cat', "$log_path/postmaster.log");
-		BAIL_OUT("pg_ctl failed");
-	}
-
-	$test_server_datadir = "$tempdir/pgdata";
-	$test_server_logfile = "$log_path/postmaster.log";
-}
-
-sub restart_test_server
-{
-	print("### Restarting test server\n");
-	system_log('pg_ctl', '-D', $test_server_datadir, '-w', '-l',
-	  $test_server_logfile, 'restart');
+	$node->stopNode('immediate');
+	@active_nodes = grep { $_ ne $node } @active_nodes;
 }
 
 END
 {
-	if ($test_server_datadir)
+	foreach my $node (@active_nodes)
 	{
-		system_log('pg_ctl', '-D', $test_server_datadir, '-m',
-		  'immediate', 'stop');
+		teardown_node($node);
 	}
 }
 
@@ -226,6 +124,12 @@ sub psql
 	my ($stdout, $stderr);
 	print("# Running SQL command: $sql\n");
 	run [ 'psql', '-X', '-A', '-t', '-q', '-d', $dbname, '-f', '-' ], '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
+	if ($stderr ne "")
+	{
+		print "#### Begin standard error\n";
+		print $stderr;
+		print "#### End standard error\n";
+	}
 	chomp $stdout;
 	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
 	return $stdout;
@@ -249,32 +153,10 @@ sub slurp_file
 	return $contents;
 }
 
-sub system_or_bail
-{
-	if (system_log(@_) != 0)
-	{
-		BAIL_OUT("system $_[0] failed: $?");
-	}
-}
-
-sub system_log
-{
-	print("# Running: " . join(" ", @_) ."\n");
-	return system(@_);
-}
-
-sub run_log
-{
-	print("# Running: " . join(" ", @{$_[0]}) ."\n");
-	return run (@_);
-}
-
 
 #
 # Test functions
 #
-
-
 sub command_ok
 {
 	my ($cmd, $test_name) = @_;
@@ -354,11 +236,11 @@ sub command_like
 
 sub issues_sql_like
 {
-	my ($cmd, $expected_sql, $test_name) = @_;
-	truncate $test_server_logfile, 0;
+	my ($node, $cmd, $expected_sql, $test_name) = @_;
+	truncate $node->getLogFile(), 0;
 	my $result = run_log($cmd);
 	ok($result, "@$cmd exit code 0");
-	my $log = slurp_file($test_server_logfile);
+	my $log = slurp_file($node->getLogFile());
 	like($log, $expected_sql, "$test_name: SQL found in server log");
 }
 
diff --git a/src/test/ssl/ServerSetup.pm b/src/test/ssl/ServerSetup.pm
index a6c77b5..ccd4754 100644
--- a/src/test/ssl/ServerSetup.pm
+++ b/src/test/ssl/ServerSetup.pm
@@ -18,6 +18,7 @@ package ServerSetup;
 
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use File::Basename;
 use File::Copy;
@@ -45,7 +46,7 @@ sub copy_files
 
 sub configure_test_server_for_ssl
 {
-	my $tempdir    = $_[0];
+	my $pgdata     = $_[0];
 	my $serverhost = $_[1];
 
 	# Create test users and databases
@@ -55,7 +56,7 @@ sub configure_test_server_for_ssl
 	psql 'postgres', "CREATE DATABASE certdb";
 
 	# enable logging etc.
-	open CONF, ">>$tempdir/pgdata/postgresql.conf";
+	open CONF, ">>$pgdata/postgresql.conf";
 	print CONF "fsync=off\n";
 	print CONF "log_connections=on\n";
 	print CONF "log_hostname=on\n";
@@ -68,17 +69,17 @@ sub configure_test_server_for_ssl
 	close CONF;
 
 # Copy all server certificates and keys, and client root cert, to the data dir
-	copy_files("ssl/server-*.crt", "$tempdir/pgdata");
-	copy_files("ssl/server-*.key", "$tempdir/pgdata");
-	chmod(0600, glob "$tempdir/pgdata/server-*.key") or die $!;
-	copy_files("ssl/root+client_ca.crt", "$tempdir/pgdata");
-	copy_files("ssl/root+client.crl",    "$tempdir/pgdata");
+	copy_files("ssl/server-*.crt", $pgdata);
+	copy_files("ssl/server-*.key", $pgdata);
+	chmod(0600, glob "$pgdata/server-*.key") or die $!;
+	copy_files("ssl/root+client_ca.crt", $pgdata);
+	copy_files("ssl/root+client.crl",    $pgdata);
 
   # Only accept SSL connections from localhost. Our tests don't depend on this
   # but seems best to keep it as narrow as possible for security reasons.
   #
   # When connecting to certdb, also check the client certificate.
-	open HBA, ">$tempdir/pgdata/pg_hba.conf";
+	open HBA, ">$pgdata/pg_hba.conf";
 	print HBA
 "# TYPE  DATABASE        USER            ADDRESS                 METHOD\n";
 	print HBA
@@ -96,12 +97,13 @@ sub configure_test_server_for_ssl
 # the server so that the configuration takes effect.
 sub switch_server_cert
 {
-	my $tempdir  = $_[0];
+	my $node     = $_[0];
 	my $certfile = $_[1];
+	my $pgdata   = $node->getDataDir();
 
 	diag "Restarting server with certfile \"$certfile\"...";
 
-	open SSLCONF, ">$tempdir/pgdata/sslconfig.conf";
+	open SSLCONF, ">$pgdata/sslconfig.conf";
 	print SSLCONF "ssl=on\n";
 	print SSLCONF "ssl_ca_file='root+client_ca.crt'\n";
 	print SSLCONF "ssl_cert_file='$certfile.crt'\n";
@@ -110,5 +112,5 @@ sub switch_server_cert
 	close SSLCONF;
 
 	# Stop and restart server to reload the new config.
-	restart_test_server();
+	$node->restartNode();
 }
diff --git a/src/test/ssl/t/001_ssltests.pl b/src/test/ssl/t/001_ssltests.pl
index 0d6f339..6a32af1 100644
--- a/src/test/ssl/t/001_ssltests.pl
+++ b/src/test/ssl/t/001_ssltests.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+use PostgresNode;
+use TestBase;
 use TestLib;
 use Test::More tests => 38;
 use ServerSetup;
@@ -25,8 +27,6 @@ BEGIN
 # postgresql-ssl-regression.test.
 my $SERVERHOSTADDR = '127.0.0.1';
 
-my $tempdir = TestLib::tempdir;
-
 # Define a couple of helper functions to test connecting to the server.
 
 my $common_connstr;
@@ -74,10 +74,16 @@ chmod 0600, "ssl/client.key";
 
 #### Part 0. Set up the server.
 
-diag "setting up data directory in \"$tempdir\"...";
-start_test_server($tempdir);
-configure_test_server_for_ssl($tempdir, $SERVERHOSTADDR);
-switch_server_cert($tempdir, 'server-cn-only');
+diag "setting up data directory...";
+my $node = get_new_node();
+$node->initNode();
+# PGHOST is enforced here to set up the node, subsequent connections
+# will use a dedicated connection string.
+$ENV{PGHOST} = $node->getHost();
+$ENV{PGPORT} = $node->getPort();
+$node->startNode();
+configure_test_server_for_ssl($node->getDataDir(), $SERVERHOSTADDR);
+switch_server_cert($node, 'server-cn-only');
 
 ### Part 1. Run client-side tests.
 ###
@@ -150,7 +156,7 @@ test_connect_ok("sslmode=verify-ca host=wronghost.test");
 test_connect_fails("sslmode=verify-full host=wronghost.test");
 
 # Test Subject Alternative Names.
-switch_server_cert($tempdir, 'server-multiple-alt-names');
+switch_server_cert($node, 'server-multiple-alt-names');
 
 diag "test hostname matching with X509 Subject Alternative Names";
 $common_connstr =
@@ -165,7 +171,7 @@ test_connect_fails("host=deep.subdomain.wildcard.pg-ssltest.test");
 
 # Test certificate with a single Subject Alternative Name. (this gives a
 # slightly different error message, that's all)
-switch_server_cert($tempdir, 'server-single-alt-name');
+switch_server_cert($node, 'server-single-alt-name');
 
 diag "test hostname matching with a single X509 Subject Alternative Name";
 $common_connstr =
@@ -178,7 +184,7 @@ test_connect_fails("host=deep.subdomain.wildcard.pg-ssltest.test");
 
 # Test server certificate with a CN and SANs. Per RFCs 2818 and 6125, the CN
 # should be ignored when the certificate has both.
-switch_server_cert($tempdir, 'server-cn-and-alt-names');
+switch_server_cert($node, 'server-cn-and-alt-names');
 
 diag "test certificate with both a CN and SANs";
 $common_connstr =
@@ -190,7 +196,7 @@ test_connect_fails("host=common-name.pg-ssltest.test");
 
 # Finally, test a server certificate that has no CN or SANs. Of course, that's
 # not a very sensible certificate, but libpq should handle it gracefully.
-switch_server_cert($tempdir, 'server-no-names');
+switch_server_cert($node, 'server-no-names');
 $common_connstr =
 "user=ssltestuser dbname=trustdb sslcert=invalid sslrootcert=ssl/root+server_ca.crt hostaddr=$SERVERHOSTADDR";
 
@@ -199,7 +205,7 @@ test_connect_fails("sslmode=verify-full host=common-name.pg-ssltest.test");
 
 # Test that the CRL works
 diag "Testing client-side CRL";
-switch_server_cert($tempdir, 'server-revoked');
+switch_server_cert($node, 'server-revoked');
 
 $common_connstr =
 "user=ssltestuser dbname=trustdb sslcert=invalid hostaddr=$SERVERHOSTADDR host=common-name.pg-ssltest.test";
@@ -233,7 +239,3 @@ test_connect_fails(
 test_connect_fails(
 "user=ssltestuser sslcert=ssl/client-revoked.crt sslkey=ssl/client-revoked.key"
 );
-
-
-# All done! Save the log, before the temporary installation is deleted
-copy("$tempdir/client-log", "./client-log");
-- 
2.6.3

0002-Add-recovery-test-suite.patchapplication/x-patch; name=0002-Add-recovery-test-suite.patchDownload
From 9a8495d413a3fff8edc76ff84ee905a49262a1b5 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Wed, 25 Nov 2015 22:06:17 +0900
Subject: [PATCH 2/2] Add recovery test suite

Using the infrastructure put in place by last commit, this commit adds a
new set of tests dedicated to nodes in recovery, standbys, backup and
more advanced cluster operations. A couple of tests showing how to use
those routines is given as well.
---
 src/bin/pg_rewind/RewindTest.pm             |  32 -----
 src/test/Makefile                           |   2 +-
 src/test/perl/TestLib.pm                    |  32 +++++
 src/test/recovery/.gitignore                |   3 +
 src/test/recovery/Makefile                  |  17 +++
 src/test/recovery/README                    |  19 +++
 src/test/recovery/RecoveryTest.pm           | 179 ++++++++++++++++++++++++++++
 src/test/recovery/t/001_stream_rep.pl       |  67 +++++++++++
 src/test/recovery/t/002_archiving.pl        |  51 ++++++++
 src/test/recovery/t/003_recovery_targets.pl | 135 +++++++++++++++++++++
 src/test/recovery/t/004_timeline_switch.pl  |  76 ++++++++++++
 src/test/recovery/t/005_replay_delay.pl     |  49 ++++++++
 12 files changed, 629 insertions(+), 33 deletions(-)
 create mode 100644 src/test/recovery/.gitignore
 create mode 100644 src/test/recovery/Makefile
 create mode 100644 src/test/recovery/README
 create mode 100644 src/test/recovery/RecoveryTest.pm
 create mode 100644 src/test/recovery/t/001_stream_rep.pl
 create mode 100644 src/test/recovery/t/002_archiving.pl
 create mode 100644 src/test/recovery/t/003_recovery_targets.pl
 create mode 100644 src/test/recovery/t/004_timeline_switch.pl
 create mode 100644 src/test/recovery/t/005_replay_delay.pl

diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index 5124df8..660b03f 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -119,38 +119,6 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
 sub append_to_file
 {
 	my ($filename, $str) = @_;
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..7f7754f 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -12,7 +12,7 @@ subdir = src/test
 top_builddir = ../..
 include $(top_builddir)/src/Makefile.global
 
-SUBDIRS = regress isolation modules
+SUBDIRS = regress isolation modules recovery
 
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index a1473c6..091043e 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -8,6 +8,7 @@ use Exporter 'import';
 our @EXPORT = qw(
   get_new_node
   teardown_node
+  poll_query_until
   psql
   slurp_dir
   slurp_file
@@ -110,6 +111,37 @@ sub teardown_node
 	@active_nodes = grep { $_ ne $node } @active_nodes;
 }
 
+sub poll_query_until
+{
+	my ($query, $connstr) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
+		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
 END
 {
 	foreach my $node (@active_nodes)
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/RecoveryTest.pm b/src/test/recovery/RecoveryTest.pm
new file mode 100644
index 0000000..2b03233
--- /dev/null
+++ b/src/test/recovery/RecoveryTest.pm
@@ -0,0 +1,179 @@
+# Set of common routines for recovery regression tests for a PostgreSQL
+# cluster. This includes methods that can be used by the various set of
+# tests present to set up cluster nodes and configure them according to
+# the test scenario wanted.
+#
+# This module makes use of PostgresNode for node manipulation, performing
+# higher-level operations to create standby nodes or setting them up
+# for archiving and replication.
+#
+# Nodes are identified by their port number and have one allocated when
+# created, hence it is unique for each node of the cluster as it is run
+# locally. PGHOST is equally set to a unique value for the duration of
+# each test.
+
+package RecoveryTest;
+
+use strict;
+use warnings;
+
+use Cwd;
+use PostgresNode;
+use RecursiveCopy;
+use TestBase;
+use TestLib;
+use Test::More;
+
+use IPC::Run qw(run start);
+
+use Exporter 'import';
+
+our @EXPORT = qw(
+	enable_archiving
+	enable_restoring
+	enable_streaming
+	make_master
+	make_archive_standby
+	make_stream_standby
+);
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $node_root = shift; # Instance to link to
+	my $node_standby = shift;
+	my $root_connstr = $node_root->getConnStr();
+	my $applname = $node_standby->getApplName();
+
+	$node_standby->appendConf('recovery.conf', qq(
+primary_conninfo='$root_connstr application_name=$applname'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $node_root = shift; # Instance to link to
+	my $node_standby = shift;
+	my $path = $node_root->getArchiveDir();
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"$path\\\\%f\" \"%p\"" :
+		"cp -i $path/%f %p";
+	$node_standby->appendConf('recovery.conf', qq(
+restore_command='$copy_command'
+standby_mode=on
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $node = shift;
+	my $path = $node->getArchiveDir();
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$path\\\\%f\"" :
+		"cp %p $path/%f";
+
+	# Enable archive_mode and archive_command on node
+	$node->appendConf('postgresql.conf', qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization.
+sub make_master
+{
+	my $node_master = get_new_node();
+	my $port_master = $node_master->getPort();
+	print "# Initializing master node wih port $port_master\n";
+	$node_master->initNode();
+	configure_base_node($node_master);
+	return $node_master;
+}
+
+sub configure_base_node
+{
+	my $node = shift;
+
+	$node->appendConf('postgresql.conf', qq(
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+));
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_stream_standby
+{
+	my $node_master = shift;
+	my $backup_name = shift;
+	my $node_standby = get_new_node();
+	my $master_port = $node_master->getPort();
+	my $standby_port = $node_standby->getPort();
+
+	print "# Initializing streaming mode for node $standby_port from node $master_port\n";
+	$node_standby->initNodeFromBackup($node_master, $backup_name);
+	configure_base_node($node_standby);
+
+	# Start second node, streaming from first one
+	enable_streaming($node_master, $node_standby);
+	return $node_standby;
+}
+
+# Node getting WAL only from archives
+sub make_archive_standby
+{
+	my $node_master = shift;
+	my $backup_name = shift;
+	my $node_standby = get_new_node();
+	my $master_port = $node_master->getPort();
+	my $standby_port = $node_standby->getPort();
+
+	print "# Initializing archive mode for node $standby_port from node $master_port\n";
+	$node_standby->initNodeFromBackup($node_master, $backup_name);
+	configure_base_node($node_standby);
+
+	# Start second node, restoring from first one
+	enable_restoring($node_master, $node_standby);
+	return $node_standby;
+}
+
+# Wait until a node is able to accept queries. Useful when putting a node
+# in recovery and wait for it to be able to work particularly on slow
+# machines.
+sub wait_for_node
+{
+	my $node         = shift;
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	while ($attempts < $max_attempts)
+	{
+		if (run_log(['pg_isready', '-p', $node->getPort()]))
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+	return 0;
+}
+
+1;
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..aae1026
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,67 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+
+use RecoveryTest;
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->startNode();
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_master->backupNode($backup_name);
+
+# Create streaming standby linking to master
+my $node_standby_1 = make_stream_standby($node_master, $backup_name);
+$node_standby_1->startNode();
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+$node_standby_1->backupNode($backup_name);
+
+# Create second standby node linking to standby 1
+my $node_standby_2 = make_stream_standby($node_standby_1, $backup_name);
+$node_standby_2->startNode();
+$node_standby_2->backupNode($backup_name);
+
+# Create some content on master and check its presence in standby 1 an
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a";
+
+# Wait for standbys to catch up
+my $applname_1 = $node_standby_1->getApplName();
+my $applname_2 = $node_standby_2->getApplName();
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_1';";
+poll_query_until($caughtup_query, $node_master->getConnStr())
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_2';";
+poll_query_until($caughtup_query, $node_standby_1->getConnStr())
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = psql $node_standby_1->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result = psql $node_standby_2->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->getConnStr(), '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->getConnStr(), '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
+
+# Cleanup nodes
+teardown_node($node_standby_2);
+teardown_node($node_standby_1);
+teardown_node($node_master);
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..c3e8465
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,51 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $node_master = make_master();
+my $backup_name = 'my_backup';
+enable_archiving($node_master);
+
+# Start it
+$node_master->startNode();
+
+# Take backup for slave
+$node_master->backupNode($backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $node_standby = make_archive_standby($node_master, $backup_name);
+$node_standby->appendConf('postgresql.conf', qq(
+wal_retrieve_retry_interval = '100ms'
+));
+$node_standby->startNode();
+
+# Create some content on master
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $current_lsn = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# Force archiving of WAL file to make it present on master
+psql $node_master->getConnStr(), "SELECT pg_switch_xlog()";
+
+# Add some more content, it should not be present on standby
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $node_standby->getConnStr())
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $node_standby->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+is($result, qq(1000), 'check content from archives');
+
+# Cleanup nodes
+teardown_node($node_standby);
+teardown_node($node_master);
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..995e8e4
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,135 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $node_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $node_standby = make_archive_standby($node_master, 'my_backup');
+
+	foreach my $param_item (@$recovery_params)
+	{
+		$node_standby->appendConf('recovery.conf',
+					   qq($param_item
+));
+	}
+
+	$node_standby->startNode();
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	poll_query_until($caughtup_query, $node_standby->getConnStr())
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = psql $node_standby->getConnStr(),
+		"SELECT count(*) FROM tab_int";
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	teardown_node($node_standby);
+}
+
+# Initialize master node
+my $node_master = make_master();
+enable_archiving($node_master);
+
+# Start it
+$node_master->startNode();
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $lsn1 = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# Take backup from which all operations will be run
+$node_master->backupNode('my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+my $recovery_txid = psql $node_master->getConnStr(),
+	"SELECT txid_current()";
+my $lsn2 = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# More data, with recovery target timestamp
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(2001,3000))";
+my $recovery_time = psql $node_master->getConnStr(), "SELECT now()";
+my $lsn3 = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# Even more data, this time with a recovery target name
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))";
+my $recovery_name = "my_target";
+my $lsn4 = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+psql $node_master->getConnStr(),
+	"SELECT pg_create_restore_point('$recovery_name')";
+
+# Force archiving of WAL file
+psql $node_master->getConnStr(), "SELECT pg_switch_xlog()";
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $node_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+teardown_node($node_master);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..c78c92a
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,76 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use PostgresNode;
+use TestBase;
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->startNode();
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backupNode($backup_name);
+
+# Create two standbys linking to it
+my $node_standby_1 = make_stream_standby($node_master, $backup_name);
+$node_standby_1->startNode();
+my $node_standby_2 = make_stream_standby($node_master, $backup_name);
+$node_standby_2->startNode();
+
+# Create some content on master
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $until_lsn = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $node_standby_1->getConnStr())
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+teardown_node($node_master);
+system_or_bail('pg_ctl', '-w', '-D', $node_standby_1->getDataDir(),
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree($node_standby_2->getDataDir() . '/recovery.conf');
+my $connstr_1 = $node_standby_1->getConnStr();
+$node_standby_2->appendConf('recovery.conf', qq(
+primary_conninfo='$connstr_1'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+$node_standby_2->restartNode();
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done. Standby 1 needs
+# to exit recovery first before moving on with the test.
+poll_query_until("SELECT pg_is_in_recovery() <> true",
+				 $node_standby_1->getConnStr());
+psql $node_standby_1->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+$until_lsn = psql $node_standby_1->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $node_standby_2->getConnStr())
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $node_standby_2->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+is($result, qq(2000), 'check content of standby 2');
+
+# Stop nodes
+teardown_node($node_standby_2);
+teardown_node($node_standby_1);
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..c209b55
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,49 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->startNode();
+
+# And some content
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a";
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backupNode($backup_name);
+
+# Create streaming standby from backup
+my $node_standby = make_stream_standby($node_master, $backup_name);
+$node_standby->appendConf('recovery.conf', qq(
+recovery_min_apply_delay = '2s'
+));
+$node_standby->startNode();
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(11,20))";
+sleep 1;
+# Here we should have only 10 rows
+my $result = psql $node_standby->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($caughtup_query, $node_standby->getConnStr())
+	or die "Timed out while waiting for standby to catch up";
+$result = psql $node_standby->getConnStr(), "SELECT count(*) FROM tab_int";
+is($result, qq(20), 'check content with delay of 2s');
+
+# Stop nodes
+teardown_node($node_standby);
+teardown_node($node_master);
-- 
2.6.3

#79Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#78)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

The result of a couple of hours of hacking is attached:
- 0001 is the refactoring adding PostgresNode and RecursiveCopy. I have
also found that it is quite advantageous to move some of the routines that
are synonyms of system() and the stuff used for logging into another
low-level library that PostgresNode depends on, that I called TestBase in
this patch. This way, all the infrastructure depends on the same logging
management. Existing tests have been refactored to fit into the new code,
and this leads to a couple of simplifications particularly in pg_rewind
tests because there is no more need to have there routines for environment
cleanup and logging. I have done tests on OSX and Windows using it and
tests are passing. I have as well tested that ssl tests were working.

Here's another version of this. I changed the packages a bit more. For
starters, I moved the routines around a bit; some of your choices seemed
more about keeping stuff where it was originally rather than moving it
to where it made sense. These are the routines in each module:

TestBase: system_or_bail system_log run_log slurp_dir slurp_file
append_to_file

TestLib: get_new_node teardown_node psql poll_query_until command_ok
command_fails command_exit_is program_help_ok program_version_ok
program_options_handling_ok command_like issues_sql_like

I tried to get rid of teardown_node by having a DESTROY method for
PostgresNode; that method would call "pg_ctl stop -m immediate". That
would have been much cleaner. However, when a test fails this doesn't
work sanely because the END block for File::Temp runs earlier than that
DESTROY block, which means the datadir is already gone by the time
pg_ctl stop runs, so the node stop doesn't work at all. (Perhaps we
could fix this by noting postmaster's PID at start time, and then
sending a signal directly instead of relying on pg_ctl).

I moved all the initialization code (deleting stuff from environment,
detecting Windows, opening SimpleTie filedescs etc) into BEGIN blocks,
which run earlier than any other code.

I perltidy'ed PostgresNode (and all the other files actually), to have
the style match the rest of our code. I also updated some code to be
more Perlish.

I added a lot of error checking in RecursiveCopy.

You had a "cat" call somewhere, which I replaced with slurp_file.

I considered updating RewindTest so that it didn't have to export the
node global variables, but decided not to, not because of the huge code
churn for the t/*.pl files but because of the problem with the DESTROY
method above: it didn't actually buy anything.

Hmm. I just noticed RewindTest sets $ENV{PGDATABASE} outside BEGIN. Not
sure what to think of that. Could instead pass the database name in
$node->getConnStr() calls, like run_pg_rewind() is already doing.

I tried all the t/ tests we have and all of them pass for me. If I'm
able, I will push this on my Sunday late evening, so that I can fix
whatever gets red on Monday first thing ...

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

taptest.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/bin/initdb/t/001_initdb.pl b/src/bin/initdb/t/001_initdb.pl
index 299dcf5..3b5d7af 100644
--- a/src/bin/initdb/t/001_initdb.pl
+++ b/src/bin/initdb/t/001_initdb.pl
@@ -4,10 +4,11 @@
 
 use strict;
 use warnings;
+use TestBase;
 use TestLib;
 use Test::More tests => 14;
 
-my $tempdir = TestLib::tempdir;
+my $tempdir = TestBase::tempdir;
 my $xlogdir = "$tempdir/pgxlog";
 my $datadir = "$tempdir/data";
 
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index dc96bbf..65ed4de 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -2,6 +2,8 @@ use strict;
 use warnings;
 use Cwd;
 use Config;
+use PostgresNode;
+use TestBase;
 use TestLib;
 use Test::More tests => 51;
 
@@ -9,8 +11,15 @@ program_help_ok('pg_basebackup');
 program_version_ok('pg_basebackup');
 program_options_handling_ok('pg_basebackup');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $tempdir = TestBase::tempdir;
+
+my $node = get_new_node();
+# Initialize node without replication settings
+$node->initNode(0);
+$node->startNode();
+my $pgdata = $node->getDataDir();
+
+$ENV{PGPORT} = $node->getPort();
 
 command_fails(['pg_basebackup'],
 	'pg_basebackup needs target directory specified');
@@ -26,19 +35,19 @@ if (open BADCHARS, ">>$tempdir/pgdata/FOO\xe0\xe0\xe0BAR")
 	close BADCHARS;
 }
 
-configure_hba_for_replication "$tempdir/pgdata";
-system_or_bail 'pg_ctl', '-D', "$tempdir/pgdata", 'reload';
+$node->setReplicationConf();
+system_or_bail 'pg_ctl', '-D', $pgdata, 'reload';
 
 command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup fails because of WAL configuration');
 
-open CONF, ">>$tempdir/pgdata/postgresql.conf";
+open CONF, ">>$pgdata/postgresql.conf";
 print CONF "max_replication_slots = 10\n";
 print CONF "max_wal_senders = 10\n";
 print CONF "wal_level = archive\n";
 close CONF;
-restart_test_server;
+$node->restartNode();
 
 command_ok([ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup runs');
@@ -81,13 +90,13 @@ command_fails(
 
 # Tar format doesn't support filenames longer than 100 bytes.
 my $superlongname = "superlongname_" . ("x" x 100);
-my $superlongpath = "$tempdir/pgdata/$superlongname";
+my $superlongpath = "$pgdata/$superlongname";
 
 open FILE, ">$superlongpath" or die "unable to create file $superlongpath";
 close FILE;
 command_fails([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l1", '-Ft' ],
 	'pg_basebackup tar with long name fails');
-unlink "$tempdir/pgdata/$superlongname";
+unlink "$pgdata/$superlongname";
 
 # The following tests test symlinks. Windows doesn't have symlinks, so
 # skip on Windows.
@@ -98,7 +107,7 @@ SKIP: {
 	# to our physical temp location.  That way we can use shorter names
 	# for the tablespace directories, which hopefully won't run afoul of
 	# the 99 character length limit.
-	my $shorter_tempdir = tempdir_short . "/tempdir";
+	my $shorter_tempdir = TestBase::tempdir_short . "/tempdir";
 	symlink "$tempdir", $shorter_tempdir;
 
 	mkdir "$tempdir/tblspc1";
@@ -120,7 +129,7 @@ SKIP: {
 			"-T$shorter_tempdir/tblspc1=$tempdir/tbackup/tblspc1" ],
 		'plain format with tablespaces succeeds with tablespace mapping');
 	ok(-d "$tempdir/tbackup/tblspc1", 'tablespace was relocated');
-	opendir(my $dh, "$tempdir/pgdata/pg_tblspc") or die;
+	opendir(my $dh, "$pgdata/pg_tblspc") or die;
 	ok( (   grep {
 		-l "$tempdir/backup1/pg_tblspc/$_"
 			and readlink "$tempdir/backup1/pg_tblspc/$_" eq
diff --git a/src/bin/pg_controldata/t/001_pg_controldata.pl b/src/bin/pg_controldata/t/001_pg_controldata.pl
index e2b0d42..343223b 100644
--- a/src/bin/pg_controldata/t/001_pg_controldata.pl
+++ b/src/bin/pg_controldata/t/001_pg_controldata.pl
@@ -1,16 +1,19 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
-my $tempdir = TestLib::tempdir;
-
 program_help_ok('pg_controldata');
 program_version_ok('pg_controldata');
 program_options_handling_ok('pg_controldata');
 command_fails(['pg_controldata'], 'pg_controldata without arguments fails');
 command_fails([ 'pg_controldata', 'nonexistent' ],
-	'pg_controldata with nonexistent directory fails');
-standard_initdb "$tempdir/data";
-command_like([ 'pg_controldata', "$tempdir/data" ],
+			  'pg_controldata with nonexistent directory fails');
+
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
+
+command_like([ 'pg_controldata', $node->getDataDir() ],
 	qr/checkpoint/, 'pg_controldata produces output');
diff --git a/src/bin/pg_ctl/t/001_start_stop.pl b/src/bin/pg_ctl/t/001_start_stop.pl
index f57abce..d76fe80 100644
--- a/src/bin/pg_ctl/t/001_start_stop.pl
+++ b/src/bin/pg_ctl/t/001_start_stop.pl
@@ -1,11 +1,12 @@
 use strict;
 use warnings;
 use Config;
+use TestBase;
 use TestLib;
 use Test::More tests => 17;
 
-my $tempdir       = TestLib::tempdir;
-my $tempdir_short = TestLib::tempdir_short;
+my $tempdir       = TestBase::tempdir;
+my $tempdir_short = TestBase::tempdir_short;
 
 program_help_ok('pg_ctl');
 program_version_ok('pg_ctl');
diff --git a/src/bin/pg_ctl/t/002_status.pl b/src/bin/pg_ctl/t/002_status.pl
index 31f7c72..74cb68a 100644
--- a/src/bin/pg_ctl/t/002_status.pl
+++ b/src/bin/pg_ctl/t/002_status.pl
@@ -1,22 +1,25 @@
 use strict;
 use warnings;
+use PostgresNode;
+use TestBase;
 use TestLib;
 use Test::More tests => 3;
 
-my $tempdir       = TestLib::tempdir;
-my $tempdir_short = TestLib::tempdir_short;
+my $tempdir       = TestBase::tempdir;
+my $tempdir_short = TestBase::tempdir_short;
 
 command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/nonexistent" ],
 	4, 'pg_ctl status with nonexistent directory');
 
-standard_initdb "$tempdir/data";
+my $node = get_new_node();
+$node->initNode();
 
-command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/data" ],
+command_exit_is([ 'pg_ctl', 'status', '-D', $node->getDataDir() ],
 	3, 'pg_ctl status with server not running');
 
 system_or_bail 'pg_ctl', '-l', "$tempdir/logfile", '-D',
-  "$tempdir/data", '-w', 'start';
-command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/data" ],
+  $node->getDataDir(), '-w', 'start';
+command_exit_is([ 'pg_ctl', 'status', '-D', $node->getDataDir() ],
 	0, 'pg_ctl status with server running');
 
-system_or_bail 'pg_ctl', 'stop', '-D', "$tempdir/data", '-m', 'fast';
+system_or_bail 'pg_ctl', 'stop', '-D', $node->getDataDir(), '-m', 'fast';
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..2943b5d 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -9,22 +9,20 @@ package RewindTest;
 # To run a test, the test script (in t/ subdirectory) calls the functions
 # in this module. These functions should be called in this sequence:
 #
-# 1. init_rewind_test - sets up log file etc.
+# 1. setup_cluster - creates a PostgreSQL cluster that runs as the master
 #
-# 2. setup_cluster - creates a PostgreSQL cluster that runs as the master
+# 2. start_master - starts the master server
 #
-# 3. start_master - starts the master server
-#
-# 4. create_standby - runs pg_basebackup to initialize a standby server, and
+# 3. create_standby - runs pg_basebackup to initialize a standby server, and
 #    sets it up to follow the master.
 #
-# 5. promote_standby - runs "pg_ctl promote" to promote the standby server.
+# 4. promote_standby - runs "pg_ctl promote" to promote the standby server.
 # The old master keeps running.
 #
-# 6. run_pg_rewind - stops the old master (if it's still running) and runs
+# 5. run_pg_rewind - stops the old master (if it's still running) and runs
 # pg_rewind to synchronize it with the now-promoted standby server.
 #
-# 7. clean_rewind_test - stops both servers used in the test, if they're
+# 6. clean_rewind_test - stops both servers used in the test, if they're
 # still running.
 #
 # The test script can use the helper functions master_psql and standby_psql
@@ -37,27 +35,23 @@ package RewindTest;
 use strict;
 use warnings;
 
-use TestLib;
-use Test::More;
-
 use Config;
+use Exporter 'import';
 use File::Copy;
 use File::Path qw(rmtree);
 use IPC::Run qw(run start);
+use TestBase;
+use TestLib;
+use Test::More;
 
-use Exporter 'import';
 our @EXPORT = qw(
-  $connstr_master
-  $connstr_standby
-  $test_master_datadir
-  $test_standby_datadir
+  $node_master
+  $node_standby
 
-  append_to_file
   master_psql
   standby_psql
   check_query
 
-  init_rewind_test
   setup_cluster
   start_master
   create_standby
@@ -66,15 +60,9 @@ our @EXPORT = qw(
   clean_rewind_test
 );
 
-our $test_master_datadir  = "$tmp_check/data_master";
-our $test_standby_datadir = "$tmp_check/data_standby";
-
-# Define non-conflicting ports for both nodes.
-my $port_master  = $ENV{PGPORT};
-my $port_standby = $port_master + 1;
-
-my $connstr_master  = "port=$port_master";
-my $connstr_standby = "port=$port_standby";
+# Our nodes.
+our $node_master;
+our $node_standby;
 
 $ENV{PGDATABASE} = "postgres";
 
@@ -82,16 +70,16 @@ sub master_psql
 {
 	my $cmd = shift;
 
-	system_or_bail 'psql', '-q', '--no-psqlrc', '-d', $connstr_master,
-	  '-c', "$cmd";
+	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
+	  $node_master->getConnStr(), '-c', "$cmd";
 }
 
 sub standby_psql
 {
 	my $cmd = shift;
 
-	system_or_bail 'psql', '-q', '--no-psqlrc', '-d', $connstr_standby,
-	  '-c', "$cmd";
+	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
+      $node_standby->getConnStr(), '-c', "$cmd";
 }
 
 # Run a query against the master, and check that the output matches what's
@@ -104,7 +92,7 @@ sub check_query
 	# we want just the output, no formatting
 	my $result = run [
 		'psql',          '-q', '-A', '-t', '--no-psqlrc', '-d',
-		$connstr_master, '-c', $query ],
+		$node_master->getConnStr(), '-c', $query ],
 	  '>', \$stdout, '2>', \$stderr;
 
 	# We don't use ok() for the exit code and stderr, because we want this
@@ -125,56 +113,14 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
-sub append_to_file
-{
-	my ($filename, $str) = @_;
-
-	open my $fh, ">>", $filename or die "could not open file $filename";
-	print $fh $str;
-	close $fh;
-}
-
 sub setup_cluster
 {
 	# Initialize master, data checksums are mandatory
-	rmtree($test_master_datadir);
-	standard_initdb($test_master_datadir);
+	$node_master = get_new_node();
+	$node_master->initNode();
 
 	# Custom parameters for master's postgresql.conf
-	append_to_file(
-		"$test_master_datadir/postgresql.conf", qq(
+	$node_master->appendConf("postgresql.conf", qq(
 wal_level = hot_standby
 max_wal_senders = 2
 wal_keep_segments = 20
@@ -185,17 +131,11 @@ hot_standby = on
 autovacuum = off
 max_connections = 10
 ));
-
-	# Accept replication connections on master
-	configure_hba_for_replication $test_master_datadir;
 }
 
 sub start_master
 {
-	system_or_bail('pg_ctl' , '-w',
-				   '-D' , $test_master_datadir,
-				   '-l',  "$log_path/master.log",
-				   "-o", "-p $port_master", 'start');
+	$node_master->startNode();
 
 	#### Now run the test-specific parts to initialize the master before setting
 	# up standby
@@ -203,24 +143,19 @@ sub start_master
 
 sub create_standby
 {
+	$node_standby = get_new_node();
+	$node_master->backupNode('my_backup');
+	$node_standby->initNodeFromBackup($node_master, 'my_backup');
+	my $connstr_master = $node_master->getConnStr();
 
-	# Set up standby with necessary parameter
-	rmtree $test_standby_datadir;
-
-	# Base backup is taken with xlog files included
-	system_or_bail('pg_basebackup', '-D', $test_standby_datadir,
-				   '-p', $port_master, '-x');
-	append_to_file(
-		"$test_standby_datadir/recovery.conf", qq(
+	$node_standby->appendConf("recovery.conf", qq(
 primary_conninfo='$connstr_master application_name=rewind_standby'
 standby_mode=on
 recovery_target_timeline='latest'
 ));
 
 	# Start standby
-	system_or_bail('pg_ctl', '-w', '-D', $test_standby_datadir,
-				   '-l', "$log_path/standby.log",
-				   '-o', "-p $port_standby", 'start');
+	$node_standby->startNode();
 
 	# The standby may have WAL to apply before it matches the primary.  That
 	# is fine, because no test examines the standby before promotion.
@@ -234,14 +169,14 @@ sub promote_standby
 	# Wait for the standby to receive and write all WAL.
 	my $wal_received_query =
 "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = 'rewind_standby';";
-	poll_query_until($wal_received_query, $connstr_master)
+	poll_query_until($node_master, $wal_received_query)
 	  or die "Timed out while waiting for standby to receive and write WAL";
 
 	# Now promote slave and insert some new data on master, this will put
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
-	system_or_bail('pg_ctl', '-w', '-D', $test_standby_datadir, 'promote');
-	poll_query_until("SELECT NOT pg_is_in_recovery()", $connstr_standby)
+	system_or_bail('pg_ctl', '-w', '-D', $node_standby->getDataDir(), 'promote');
+	poll_query_until($node_standby, "SELECT NOT pg_is_in_recovery()")
 	  or die "Timed out while waiting for promotion of standby";
 
 	# Force a checkpoint after the promotion. pg_rewind looks at the control
@@ -256,9 +191,13 @@ sub promote_standby
 sub run_pg_rewind
 {
 	my $test_mode = shift;
+	my $master_pgdata = $node_master->getDataDir();
+	my $standby_pgdata = $node_standby->getDataDir();
+	my $standby_connstr = $node_standby->getConnStr('postgres');
+	my $tmp_folder = TestBase::tempdir;
 
 	# Stop the master and be ready to perform the rewind
-	system_or_bail('pg_ctl', '-D', $test_master_datadir, '-m', 'fast', 'stop');
+	$node_master->stopNode();
 
 	# At this point, the rewind processing is ready to run.
 	# We now have a very simple scenario with a few diverged WAL record.
@@ -267,20 +206,19 @@ sub run_pg_rewind
 
 	# Keep a temporary postgresql.conf for master node or it would be
 	# overwritten during the rewind.
-	copy("$test_master_datadir/postgresql.conf",
-		 "$tmp_check/master-postgresql.conf.tmp");
+	copy("$master_pgdata/postgresql.conf",
+		 "$tmp_folder/master-postgresql.conf.tmp");
 
 	# Now run pg_rewind
 	if ($test_mode eq "local")
 	{
 		# Do rewind using a local pgdata as source
 		# Stop the master and be ready to perform the rewind
-		system_or_bail('pg_ctl', '-D', $test_standby_datadir,
-					   '-m', 'fast', 'stop');
+		$node_standby->stopNode();
 		command_ok(['pg_rewind',
 					"--debug",
-					"--source-pgdata=$test_standby_datadir",
-					"--target-pgdata=$test_master_datadir"],
+					"--source-pgdata=$standby_pgdata",
+					"--target-pgdata=$master_pgdata"],
 				   'pg_rewind local');
 	}
 	elsif ($test_mode eq "remote")
@@ -289,33 +227,30 @@ sub run_pg_rewind
 		command_ok(['pg_rewind',
 					"--debug",
 					"--source-server",
-					"port=$port_standby dbname=postgres",
-					"--target-pgdata=$test_master_datadir"],
+					$standby_connstr,
+					"--target-pgdata=$master_pgdata"],
 				   'pg_rewind remote');
 	}
 	else
 	{
-
 		# Cannot come here normally
 		die("Incorrect test mode specified");
 	}
 
 	# Now move back postgresql.conf with old settings
-	move("$tmp_check/master-postgresql.conf.tmp",
-		 "$test_master_datadir/postgresql.conf");
+	move("$tmp_folder/master-postgresql.conf.tmp",
+		 "$master_pgdata/postgresql.conf");
 
 	# Plug-in rewound node to the now-promoted standby node
-	append_to_file(
-		"$test_master_datadir/recovery.conf", qq(
+	my $port_standby = $node_standby->getPort();
+	$node_master->appendConf('recovery.conf', qq(
 primary_conninfo='port=$port_standby'
 standby_mode=on
 recovery_target_timeline='latest'
 ));
 
 	# Restart the master to check that rewind went correctly
-	system_or_bail('pg_ctl', '-w', '-D', $test_master_datadir,
-				   '-l', "$log_path/master.log",
-				   '-o', "-p $port_master", 'start');
+	$node_master->restartNode();
 
 	#### Now run the test-specific parts to check the result
 }
@@ -323,22 +258,6 @@ recovery_target_timeline='latest'
 # Clean up after the test. Stop both servers, if they're still running.
 sub clean_rewind_test
 {
-	if ($test_master_datadir)
-	{
-		system
-		  'pg_ctl', '-D', $test_master_datadir, '-m', 'immediate', 'stop';
-	}
-	if ($test_standby_datadir)
-	{
-		system
-		  'pg_ctl', '-D', $test_standby_datadir, '-m', 'immediate', 'stop';
-	}
-}
-
-# Stop the test servers, just in case they're still running.
-END
-{
-	my $save_rc = $?;
-	clean_rewind_test();
-	$? = $save_rc;
+	teardown_node($node_master) if (defined($node_master));
+	teardown_node($node_standby) if (defined($node_standby));
 }
diff --git a/src/bin/pg_rewind/t/001_basic.pl b/src/bin/pg_rewind/t/001_basic.pl
index 1764b17..be7b929 100644
--- a/src/bin/pg_rewind/t/001_basic.pl
+++ b/src/bin/pg_rewind/t/001_basic.pl
@@ -1,9 +1,11 @@
+# Basic pg_rewind test.
+
 use strict;
 use warnings;
-use TestLib;
-use Test::More tests => 8;
 
 use RewindTest;
+use TestLib;
+use Test::More tests => 8;
 
 sub run_test
 {
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index d317f53..16d9366 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -2,13 +2,12 @@
 
 use strict;
 use warnings;
-use TestLib;
-use Test::More tests => 4;
 
 use File::Find;
-
 use RewindTest;
-
+use TestLib;
+use TestBase;
+use Test::More tests => 4;
 
 sub run_test
 {
@@ -17,7 +16,7 @@ sub run_test
 	RewindTest::setup_cluster();
 	RewindTest::start_master();
 
-	my $test_master_datadir = $RewindTest::test_master_datadir;
+	my $test_master_datadir = $node_master->getDataDir();
 
 	# Create a subdir and files that will be present in both
 	mkdir "$test_master_datadir/tst_both_dir";
@@ -30,6 +29,7 @@ sub run_test
 	RewindTest::create_standby();
 
 	# Create different subdirs and files in master and standby
+	my $test_standby_datadir = $node_standby->getDataDir();
 
 	mkdir "$test_standby_datadir/tst_standby_dir";
 	append_to_file "$test_standby_datadir/tst_standby_dir/standby_file1",
diff --git a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
index c5f72e2..1bc4bba 100644
--- a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
+++ b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
@@ -3,10 +3,13 @@
 #
 use strict;
 use warnings;
+
 use File::Copy;
 use File::Path qw(rmtree);
-use TestLib;
+use RewindTest;
 use Test::More;
+use TestBase;
+
 if ($windows_os)
 {
 	plan skip_all => 'symlinks not supported on Windows';
@@ -17,17 +20,17 @@ else
 	plan tests => 4;
 }
 
-use RewindTest;
-
 sub run_test
 {
 	my $test_mode = shift;
 
-	my $master_xlogdir = "$tmp_check/xlog_master";
+	my $master_xlogdir = "${TestBase::tmp_check}/xlog_master";
 
 	rmtree($master_xlogdir);
 	RewindTest::setup_cluster();
 
+	my $test_master_datadir = $node_master->getDataDir();
+
 	# turn pg_xlog into a symlink
 	print("moving $test_master_datadir/pg_xlog to $master_xlogdir\n");
 	move("$test_master_datadir/pg_xlog", $master_xlogdir) or die;
diff --git a/src/bin/scripts/t/010_clusterdb.pl b/src/bin/scripts/t/010_clusterdb.pl
index dc0d78a..5dee16c 100644
--- a/src/bin/scripts/t/010_clusterdb.pl
+++ b/src/bin/scripts/t/010_clusterdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 13;
 
@@ -7,20 +8,24 @@ program_help_ok('clusterdb');
 program_version_ok('clusterdb');
 program_options_handling_ok('clusterdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
-	[ 'clusterdb', 'postgres' ],
+$ENV{PGPORT} = $node->getPort();
+$ENV{PGDATABASE} = 'postgres';
+
+issues_sql_like($node,
+	[ 'clusterdb' ],
 	qr/statement: CLUSTER;/,
 	'SQL CLUSTER run');
 
-command_fails([ 'clusterdb', '-t', 'nonexistent', 'postgres' ],
-	'fails with nonexistent table');
+command_fails([ 'clusterdb', '-t', 'nonexistent' ],
+			  'fails with nonexistent table');
 
 psql 'postgres',
-'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';
-issues_sql_like(
-	[ 'clusterdb', '-t', 'test1', 'postgres' ],
+	'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';
+issues_sql_like($node,
+	[ 'clusterdb', '-t', 'test1' ],
 	qr/statement: CLUSTER test1;/,
 	'cluster specific table');
diff --git a/src/bin/scripts/t/011_clusterdb_all.pl b/src/bin/scripts/t/011_clusterdb_all.pl
index 7769f70..9c1c8f8 100644
--- a/src/bin/scripts/t/011_clusterdb_all.pl
+++ b/src/bin/scripts/t/011_clusterdb_all.pl
@@ -1,12 +1,19 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+# cluster -a is not compatible with -d, hence enforce environment variables
+# correctly.
+$ENV{PGDATABASE} = 'postgres';
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'clusterdb', '-a' ],
 	qr/statement: CLUSTER.*statement: CLUSTER/s,
 	'cluster all databases');
diff --git a/src/bin/scripts/t/020_createdb.pl b/src/bin/scripts/t/020_createdb.pl
index a44283c..784164c 100644
--- a/src/bin/scripts/t/020_createdb.pl
+++ b/src/bin/scripts/t/020_createdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 13;
 
@@ -7,14 +8,17 @@ program_help_ok('createdb');
 program_version_ok('createdb');
 program_options_handling_ok('createdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'createdb', 'foobar1' ],
 	qr/statement: CREATE DATABASE foobar1/,
 	'SQL CREATE DATABASE run');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'createdb', '-l', 'C', '-E', 'LATIN1', '-T', 'template0', 'foobar2' ],
 	qr/statement: CREATE DATABASE foobar2 ENCODING 'LATIN1'/,
 	'create database with encoding');
diff --git a/src/bin/scripts/t/030_createlang.pl b/src/bin/scripts/t/030_createlang.pl
index 7ff0a3e..9413153 100644
--- a/src/bin/scripts/t/030_createlang.pl
+++ b/src/bin/scripts/t/030_createlang.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 14;
 
@@ -7,18 +8,22 @@ program_help_ok('createlang');
 program_version_ok('createlang');
 program_options_handling_ok('createlang');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
+
+$ENV{PGPORT} = $node->getPort();
+$ENV{PGDATABASE} = 'postgres';
 
 command_fails(
-	[ 'createlang', 'plpgsql', 'postgres' ],
+	[ 'createlang', 'plpgsql' ],
 	'fails if language already exists');
 
-psql 'postgres', 'DROP EXTENSION plpgsql';
-issues_sql_like(
-	[ 'createlang', 'plpgsql', 'postgres' ],
+psql $node->getConnStr('postgres'), 'DROP EXTENSION plpgsql';
+issues_sql_like($node,
+	[ 'createlang', 'plpgsql' ],
 	qr/statement: CREATE EXTENSION "plpgsql"/,
 	'SQL CREATE EXTENSION run');
 
-command_like([ 'createlang', '--list', 'postgres' ],
+command_like([ 'createlang', '--list' ],
 	qr/plpgsql/, 'list output');
diff --git a/src/bin/scripts/t/040_createuser.pl b/src/bin/scripts/t/040_createuser.pl
index 4d44e14..c322c17 100644
--- a/src/bin/scripts/t/040_createuser.pl
+++ b/src/bin/scripts/t/040_createuser.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 17;
 
@@ -7,24 +8,29 @@ program_help_ok('createuser');
 program_version_ok('createuser');
 program_options_handling_ok('createuser');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+$ENV{PGDATABASE} = 'postgres';
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'createuser', 'user1' ],
 qr/statement: CREATE ROLE user1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN;/,
 	'SQL CREATE USER run');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'createuser', '-L', 'role1' ],
 qr/statement: CREATE ROLE role1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT NOLOGIN;/,
 	'create a non-login role');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'createuser', '-r', 'user2' ],
 qr/statement: CREATE ROLE user2 NOSUPERUSER NOCREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a CREATEROLE user');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'createuser', '-s', 'user3' ],
 qr/statement: CREATE ROLE user3 SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a superuser');
 
-command_fails([ 'createuser', 'user1' ], 'fails if role already exists');
+command_fails([ 'createuser', 'user1' ],
+			  'fails if role already exists');
diff --git a/src/bin/scripts/t/050_dropdb.pl b/src/bin/scripts/t/050_dropdb.pl
index 3065e50..5e0d984 100644
--- a/src/bin/scripts/t/050_dropdb.pl
+++ b/src/bin/scripts/t/050_dropdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,11 +8,14 @@ program_help_ok('dropdb');
 program_version_ok('dropdb');
 program_options_handling_ok('dropdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-psql 'postgres', 'CREATE DATABASE foobar1';
-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+psql $node->getConnStr('postgres'), 'CREATE DATABASE foobar1';
+issues_sql_like($node,
 	[ 'dropdb', 'foobar1' ],
 	qr/statement: DROP DATABASE foobar1/,
 	'SQL DROP DATABASE run');
diff --git a/src/bin/scripts/t/060_droplang.pl b/src/bin/scripts/t/060_droplang.pl
index 6a21d7e..7c5bc23 100644
--- a/src/bin/scripts/t/060_droplang.pl
+++ b/src/bin/scripts/t/060_droplang.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,10 +8,13 @@ program_help_ok('droplang');
 program_version_ok('droplang');
 program_options_handling_ok('droplang');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'droplang', 'plpgsql', 'postgres' ],
 	qr/statement: DROP EXTENSION "plpgsql"/,
 	'SQL DROP EXTENSION run');
diff --git a/src/bin/scripts/t/070_dropuser.pl b/src/bin/scripts/t/070_dropuser.pl
index bbb3b79..b8608cd 100644
--- a/src/bin/scripts/t/070_dropuser.pl
+++ b/src/bin/scripts/t/070_dropuser.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,11 +8,14 @@ program_help_ok('dropuser');
 program_version_ok('dropuser');
 program_options_handling_ok('dropuser');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
+
+$ENV{PGPORT} = $node->getPort();
 
 psql 'postgres', 'CREATE ROLE foobar1';
-issues_sql_like(
+issues_sql_like($node,
 	[ 'dropuser', 'foobar1' ],
 	qr/statement: DROP ROLE foobar1/,
 	'SQL DROP ROLE run');
diff --git a/src/bin/scripts/t/080_pg_isready.pl b/src/bin/scripts/t/080_pg_isready.pl
index f432505..b2014dd 100644
--- a/src/bin/scripts/t/080_pg_isready.pl
+++ b/src/bin/scripts/t/080_pg_isready.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 10;
 
@@ -9,7 +10,10 @@ program_options_handling_ok('pg_isready');
 
 command_fails(['pg_isready'], 'fails with no server running');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
+
+$ENV{PGPORT} = $node->getPort();
 
 command_ok(['pg_isready'], 'succeeds with server running');
diff --git a/src/bin/scripts/t/090_reindexdb.pl b/src/bin/scripts/t/090_reindexdb.pl
index 42628c2..3873316 100644
--- a/src/bin/scripts/t/090_reindexdb.pl
+++ b/src/bin/scripts/t/090_reindexdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 20;
 
@@ -7,35 +8,37 @@ program_help_ok('reindexdb');
 program_version_ok('reindexdb');
 program_options_handling_ok('reindexdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
+$ENV{PGPORT} = $node->getPort();
 $ENV{PGOPTIONS} = '--client-min-messages=WARNING';
 
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', 'postgres' ],
 	qr/statement: REINDEX DATABASE postgres;/,
 	'SQL REINDEX run');
 
 psql 'postgres',
   'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a);';
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX TABLE test1;/,
 	'reindex specific table');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', '-i', 'test1x', 'postgres' ],
 	qr/statement: REINDEX INDEX test1x;/,
 	'reindex specific index');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', '-S', 'pg_catalog', 'postgres' ],
 	qr/statement: REINDEX SCHEMA pg_catalog;/,
 	'reindex specific schema');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', '-s', 'postgres' ],
 	qr/statement: REINDEX SYSTEM postgres;/,
 	'reindex system tables');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', '-v', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX \(VERBOSE\) TABLE test1;/,
 	'reindex with verbose output');
diff --git a/src/bin/scripts/t/091_reindexdb_all.pl b/src/bin/scripts/t/091_reindexdb_all.pl
index ffadf29..7d83533 100644
--- a/src/bin/scripts/t/091_reindexdb_all.pl
+++ b/src/bin/scripts/t/091_reindexdb_all.pl
@@ -1,14 +1,17 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
+$ENV{PGPORT} = $node->getPort();
 $ENV{PGOPTIONS} = '--client-min-messages=WARNING';
 
-issues_sql_like(
+issues_sql_like($node,
 	[ 'reindexdb', '-a' ],
 	qr/statement: REINDEX.*statement: REINDEX/s,
 	'reindex all databases');
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index ac160ba..7103aae 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 18;
 
@@ -7,26 +8,29 @@ program_help_ok('vacuumdb');
 program_version_ok('vacuumdb');
 program_options_handling_ok('vacuumdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'vacuumdb', 'postgres' ],
 	qr/statement: VACUUM;/,
 	'SQL VACUUM run');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'vacuumdb', '-f', 'postgres' ],
 	qr/statement: VACUUM \(FULL\);/,
 	'vacuumdb -f');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'vacuumdb', '-F', 'postgres' ],
 	qr/statement: VACUUM \(FREEZE\);/,
 	'vacuumdb -F');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'vacuumdb', '-z', 'postgres' ],
 	qr/statement: VACUUM \(ANALYZE\);/,
 	'vacuumdb -z');
-issues_sql_like(
+issues_sql_like($node,
 	[ 'vacuumdb', '-Z', 'postgres' ],
 	qr/statement: ANALYZE;/,
 	'vacuumdb -Z');
diff --git a/src/bin/scripts/t/101_vacuumdb_all.pl b/src/bin/scripts/t/101_vacuumdb_all.pl
index e90f321..728cdf6 100644
--- a/src/bin/scripts/t/101_vacuumdb_all.pl
+++ b/src/bin/scripts/t/101_vacuumdb_all.pl
@@ -1,12 +1,16 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'vacuumdb', '-a' ],
 	qr/statement: VACUUM.*statement: VACUUM/s,
 	'vacuum all databases');
diff --git a/src/bin/scripts/t/102_vacuumdb_stages.pl b/src/bin/scripts/t/102_vacuumdb_stages.pl
index 57b980e..ddda9f5 100644
--- a/src/bin/scripts/t/102_vacuumdb_stages.pl
+++ b/src/bin/scripts/t/102_vacuumdb_stages.pl
@@ -1,12 +1,16 @@
 use strict;
 use warnings;
+
 use TestLib;
 use Test::More tests => 4;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->initNode();
+$node->startNode();
 
-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,
 	[ 'vacuumdb', '--analyze-in-stages', 'postgres' ],
 qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE.*
@@ -17,7 +21,7 @@ qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
 	'analyze three times');
 
 
-issues_sql_like(
+issues_sql_like($node,
 	[ 'vacuumdb', '--analyze-in-stages', '--all' ],
 qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE.*
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
new file mode 100644
index 0000000..9bf9853
--- /dev/null
+++ b/src/test/perl/PostgresNode.pm
@@ -0,0 +1,252 @@
+# PostgresNode, simple node representation for regression tests.
+#
+# Regression tests should use this basic class infrastructure to define nodes
+# that need used in the test modules/scripts.
+package PostgresNode;
+
+use strict;
+use warnings;
+
+use RecursiveCopy;
+use TestBase;
+use Test::More;
+
+sub new
+{
+	my $class  = shift;
+	my $pghost = shift;
+	my $pgport = shift;
+	my $self   = {
+		_port     => undef,
+		_host     => undef,
+		_basedir  => undef,
+		_applname => undef,
+		_logfile  => undef };
+
+	# Set up each field
+	$self->{_port}     = $pgport;
+	$self->{_host}     = $pghost;
+	$self->{_basedir}  = TestBase::tempdir;
+	$self->{_applname} = "node_$pgport";
+	$self->{_logfile}  = "$TestBase::log_path/node_$pgport.log";
+
+	bless $self, $class;
+	$self->dumpNodeInfo();
+
+	return $self;
+}
+
+# Get routines for various variables
+sub getPort
+{
+	my ($self) = @_;
+	return $self->{_port};
+}
+
+sub getHost
+{
+	my ($self) = @_;
+	return $self->{_host};
+}
+
+sub getConnStr
+{
+	my ($self, $dbname) = @_;
+	my $pgport = $self->getPort();
+	my $pghost = $self->getHost();
+	if (!defined($dbname))
+	{
+		return "port=$pgport host=$pghost";
+	}
+	return "port=$pgport host=$pghost dbname=$dbname";
+}
+
+sub getDataDir
+{
+	my ($self) = @_;
+	my $basedir = $self->{_basedir};
+	return "$basedir/pgdata";
+}
+
+sub getApplName
+{
+	my ($self) = @_;
+	return $self->{_applname};
+}
+
+sub getLogFile
+{
+	my ($self) = @_;
+	return $self->{_logfile};
+}
+
+sub getArchiveDir
+{
+	my ($self) = @_;
+	my $basedir = $self->{_basedir};
+	return "$basedir/archives";
+}
+
+sub getBackupDir
+{
+	my ($self) = @_;
+	my $basedir = $self->{_basedir};
+	return "$basedir/backup";
+}
+
+# Dump node information
+sub dumpNodeInfo
+{
+	my ($self) = @_;
+	print 'Data directory: ' . $self->getDataDir() . "\n";
+	print 'Backup directory: ' . $self->getBackupDir() . "\n";
+	print 'Archive directory: ' . $self->getArchiveDir() . "\n";
+	print 'Connection string: ' . $self->getConnStr() . "\n";
+	print 'Application name: ' . $self->getApplName() . "\n";
+	print 'Log file: ' . $self->getLogFile() . "\n";
+}
+
+sub setReplicationConf
+{
+	my ($self) = @_;
+	my $pgdata = $self->getDataDir();
+
+	open my $hba, ">>$pgdata/pg_hba.conf";
+	print $hba "\n# Allow replication (set up by PostgresNode.pm)\n";
+	if (!$windows_os)
+	{
+		print $hba "local replication all trust\n";
+	}
+	else
+	{
+		print $hba
+"host replication all 127.0.0.1/32 sspi include_realm=1 map=regress\n";
+	}
+	close $hba;
+}
+
+# Initialize a new cluster for testing.
+#
+# Authentication is set up so that only the current OS user can access the
+# cluster. On Unix, we use Unix domain socket connections, with the socket in
+# a directory that's only accessible to the current user to ensure that.
+# On Windows, we use SSPI authentication to ensure the same (by pg_regress
+# --config-auth).
+sub initNode
+{
+	my ($self, $repconf) = @_;
+	my $port   = $self->getPort();
+	my $pgdata = $self->getDataDir();
+	my $host   = $self->getHost();
+
+	$repconf = 1 if (!defined($repconf));
+
+	mkdir $self->getBackupDir();
+	mkdir $self->getArchiveDir();
+
+	system_or_bail('initdb', '-D', $pgdata, '-A', 'trust', '-N');
+	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
+
+	open my $conf, ">>$pgdata/postgresql.conf";
+	print $conf "\n# Added by TestLib.pm)\n";
+	print $conf "fsync = off\n";
+	print $conf "log_statement = all\n";
+	print $conf "port = $port\n";
+	if ($windows_os)
+	{
+		print $conf "listen_addresses = '$host'\n";
+	}
+	else
+	{
+		print $conf "unix_socket_directories = '$host'\n";
+		print $conf "listen_addresses = ''\n";
+	}
+	close $conf;
+
+	$self->setReplicationConf() if ($repconf);
+}
+
+sub appendConf
+{
+	my ($self, $filename, $str) = @_;
+
+	my $conffile = $self->getDataDir() . '/' . $filename;
+
+	append_to_file($conffile, $str);
+}
+
+sub backupNode
+{
+	my ($self, $backup_name) = @_;
+	my $backup_path = $self->getBackupDir() . '/' . $backup_name;
+	my $port        = $self->getPort();
+
+	print "# Taking backup $backup_name from node with port $port\n";
+	system_or_bail("pg_basebackup -D $backup_path -p $port -x");
+	print "# Backup finished\n";
+}
+
+sub initNodeFromBackup
+{
+	my ($self, $root_node, $backup_name) = @_;
+	my $backup_path = $root_node->getBackupDir() . '/' . $backup_name;
+	my $port        = $self->getPort();
+	my $root_port   = $root_node->getPort();
+
+	print
+"Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	die "Backup $backup_path does not exist" unless -d $backup_path;
+
+	mkdir $self->getBackupDir();
+	mkdir $self->getArchiveDir();
+
+	my $data_path = $self->getDataDir();
+	rmdir($data_path);
+	RecursiveCopy::copypath($backup_path, $data_path);
+	chmod(0700, $data_path);
+
+	# Base configuration for this node
+	$self->appendConf('postgresql.conf',
+		qq(
+port = $port
+));
+	$self->setReplicationConf();
+}
+
+sub startNode
+{
+	my ($self) = @_;
+	my $port   = $self->getPort();
+	my $pgdata = $self->getDataDir();
+	print("### Starting test server in $pgdata\n");
+	my $ret = system_log('pg_ctl', '-w', '-D', $self->getDataDir(),
+		'-l', $self->getLogFile(), 'start');
+
+	if ($ret != 0)
+	{
+		print "# pg_ctl failed; logfile:\n";
+		print slurp_file($self->getLogFile());
+		BAIL_OUT("pg_ctl failed");
+	}
+}
+
+sub stopNode
+{
+	my ($self, $mode) = @_;
+	my $port   = $self->getPort();
+	my $pgdata = $self->getDataDir();
+	$mode = 'fast' if (!defined($mode));
+	print "### Stopping node in $pgdata with port $port using mode $mode\n";
+	system_log('pg_ctl', '-D', $pgdata, '-m', $mode, 'stop');
+}
+
+sub restartNode
+{
+	my ($self)  = @_;
+	my $port    = $self->getPort();
+	my $pgdata  = $self->getDataDir();
+	my $logfile = $self->getLogFile();
+	system_log('pg_ctl', '-D', $pgdata, '-w', '-l', $logfile, 'restart');
+}
+
+1;
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
new file mode 100644
index 0000000..4e58ad3
--- /dev/null
+++ b/src/test/perl/RecursiveCopy.pm
@@ -0,0 +1,42 @@
+# RecursiveCopy, a simple recursive copy implementation
+package RecursiveCopy;
+
+use strict;
+use warnings;
+
+use File::Basename;
+use File::Copy;
+
+sub copypath
+{
+	my $srcpath  = shift;
+	my $destpath = shift;
+
+	die "Cannot operate on symlinks" if -l $srcpath or -l $destpath;
+
+	# This source path is a file, simply copy it to destination with the
+	# same name.
+	die "Destination path $destpath exists as file" if -f $destpath;
+	if (-f $srcpath)
+	{
+		copy($srcpath, $destpath)
+			or die "copy $srcpath -> $destpath failed: $!";
+		return 1;
+	}
+
+	die "Destination needs to be a directory" unless -d $srcpath;
+	mkdir($destpath) or die "mkdir($destpath) failed: $!";
+
+	# Scan existing source directory and recursively copy everything.
+	opendir(my $directory, $srcpath) or die "could not opendir($srcpath): $!";
+	while (my $entry = readdir($directory))
+	{
+		next if ($entry eq '.' || $entry eq '..');
+		RecursiveCopy::copypath("$srcpath/$entry", "$destpath/$entry")
+			or die "copypath $srcpath/$entry -> $destpath/$entry failed";
+	}
+	closedir($directory);
+	return 1;
+}
+
+1;
diff --git a/src/test/perl/TestBase.pm b/src/test/perl/TestBase.pm
new file mode 100644
index 0000000..72c8481
--- /dev/null
+++ b/src/test/perl/TestBase.pm
@@ -0,0 +1,143 @@
+# Set of low-level routines dedicated to base tasks for regression tests, like
+# command execution and logging.
+#
+# This module should not depend on any other PostgreSQL regression test
+# modules.
+package TestBase;
+
+use strict;
+use warnings;
+
+use Config;
+use Exporter 'import';
+use File::Basename;
+use File::Spec;
+use File::Temp ();
+use IPC::Run qw(run);
+use SimpleTee;
+use Test::More;
+
+our @EXPORT = qw(
+  system_or_bail
+  system_log
+  run_log
+  slurp_dir
+  slurp_file
+  append_to_file
+
+  $windows_os
+);
+
+our ($windows_os, $tmp_check, $log_path, $test_logfile);
+
+BEGIN
+{
+	$windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
+
+	# Determine output directories, and create them.  The base path is the
+	# TESTDIR environment variable, which is normally set by the invoking
+	# Makefile.
+	$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
+	$log_path = "$tmp_check/log";
+
+	mkdir $tmp_check;
+	mkdir $log_path;
+
+	# Open the test log file, whose name depends on the test name.
+	$test_logfile = basename($0);
+	$test_logfile =~ s/\.[^.]+$//;
+	$test_logfile = "$log_path/regress_log_$test_logfile";
+	open TESTLOG, '>', $test_logfile
+	  or die "could not open STDOUT to logfile \"$test_logfile\": $!";
+
+	# Hijack STDOUT and STDERR to the log file
+	open(ORIG_STDOUT, ">&STDOUT");
+	open(ORIG_STDERR, ">&STDERR");
+	open(STDOUT,      ">&TESTLOG");
+	open(STDERR,      ">&TESTLOG");
+
+	# The test output (ok ...) needs to be printed to the original STDOUT so
+	# that the 'prove' program can parse it, and display it to the user in
+	# real time. But also copy it to the log file, to provide more context
+	# in the log.
+	my $builder = Test::More->builder;
+	my $fh      = $builder->output;
+	tie *$fh, "SimpleTee", *ORIG_STDOUT, *TESTLOG;
+	$fh = $builder->failure_output;
+	tie *$fh, "SimpleTee", *ORIG_STDERR, *TESTLOG;
+
+	# Enable auto-flushing for all the file handles. Stderr and stdout are
+	# redirected to the same file, and buffering causes the lines to appear
+	# in the log in confusing order.
+	autoflush STDOUT 1;
+	autoflush STDERR 1;
+	autoflush TESTLOG 1;
+}
+
+#
+# Helper functions
+#
+sub tempdir
+{
+	return File::Temp::tempdir(
+		'tmp_testXXXX',
+		DIR => $ENV{TESTDIR} || cwd(),
+		CLEANUP => 1);
+}
+
+sub tempdir_short
+{
+	# Use a separate temp dir outside the build tree for the
+	# Unix-domain socket, to avoid file name length issues.
+	return File::Temp::tempdir(CLEANUP => 1);
+}
+
+sub system_log
+{
+	print("# Running: " . join(" ", @_) . "\n");
+	return system(@_);
+}
+
+sub system_or_bail
+{
+	if (system_log(@_) != 0)
+	{
+		BAIL_OUT("system $_[0] failed");
+	}
+}
+
+sub run_log
+{
+	print("# Running: " . join(" ", @{ $_[0] }) . "\n");
+	return run(@_);
+}
+
+sub slurp_dir
+{
+	my ($dir) = @_;
+	opendir(my $dh, $dir)
+	  or die "could not opendir \"$dir\": $!";
+	my @direntries = readdir $dh;
+	closedir $dh;
+	return @direntries;
+}
+
+sub slurp_file
+{
+	local $/;
+	local @ARGV = @_;
+	my $contents = <>;
+	$contents =~ s/\r//g if $Config{osname} eq 'msys';
+	return $contents;
+}
+
+sub append_to_file
+{
+	my ($filename, $str) = @_;
+
+	open my $fh, ">>", $filename or die "could not open \"$filename\": $!";
+	print $fh $str;
+	close $fh;
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..83202f6 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -4,20 +4,21 @@ use strict;
 use warnings;
 
 use Config;
+use Cwd;
 use Exporter 'import';
+use File::Basename;
+use File::Spec;
+use File::Temp ();
+use IPC::Run qw(run start);
+use PostgresNode;
+use Test::More;
+use TestBase;
+
 our @EXPORT = qw(
-  tempdir
-  tempdir_short
-  standard_initdb
-  configure_hba_for_replication
-  start_test_server
-  restart_test_server
+  get_new_node
+  teardown_node
   psql
-  slurp_dir
-  slurp_file
-  system_or_bail
-  system_log
-  run_log
+  poll_query_until
 
   command_ok
   command_fails
@@ -27,197 +28,91 @@ our @EXPORT = qw(
   program_options_handling_ok
   command_like
   issues_sql_like
-
-  $tmp_check
-  $log_path
-  $windows_os
 );
 
-use Cwd;
-use File::Basename;
-use File::Spec;
-use File::Temp ();
-use IPC::Run qw(run start);
+our ($test_pghost, $last_port_assigned);
+our (@all_nodes,   @active_nodes);
 
-use SimpleTee;
-
-use Test::More;
-
-our $windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
-
-# Open log file. For each test, the log file name uses the name of the
-# file launching this module, without the .pl suffix.
-our ($tmp_check, $log_path);
-$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
-$log_path = "$tmp_check/log";
-mkdir $tmp_check;
-mkdir $log_path;
-my $test_logfile = basename($0);
-$test_logfile =~ s/\.[^.]+$//;
-$test_logfile = "$log_path/regress_log_$test_logfile";
-open TESTLOG, '>', $test_logfile or die "Cannot open STDOUT to logfile: $!";
-
-# Hijack STDOUT and STDERR to the log file
-open(ORIG_STDOUT, ">&STDOUT");
-open(ORIG_STDERR, ">&STDERR");
-open(STDOUT, ">&TESTLOG");
-open(STDERR, ">&TESTLOG");
-
-# The test output (ok ...) needs to be printed to the original STDOUT so
-# that the 'prove' program can parse it, and display it to the user in
-# real time. But also copy it to the log file, to provide more context
-# in the log.
-my $builder = Test::More->builder;
-my $fh = $builder->output;
-tie *$fh, "SimpleTee", *ORIG_STDOUT, *TESTLOG;
-$fh = $builder->failure_output;
-tie *$fh, "SimpleTee", *ORIG_STDERR, *TESTLOG;
-
-# Enable auto-flushing for all the file handles. Stderr and stdout are
-# redirected to the same file, and buffering causes the lines to appear
-# in the log in confusing order.
-autoflush STDOUT 1;
-autoflush STDERR 1;
-autoflush TESTLOG 1;
-
-# Set to untranslated messages, to be able to compare program output
-# with expected strings.
-delete $ENV{LANGUAGE};
-delete $ENV{LC_ALL};
-$ENV{LC_MESSAGES} = 'C';
-
-delete $ENV{PGCONNECT_TIMEOUT};
-delete $ENV{PGDATA};
-delete $ENV{PGDATABASE};
-delete $ENV{PGHOSTADDR};
-delete $ENV{PGREQUIRESSL};
-delete $ENV{PGSERVICE};
-delete $ENV{PGSSLMODE};
-delete $ENV{PGUSER};
-
-if (!$ENV{PGPORT})
+BEGIN
 {
-	$ENV{PGPORT} = 65432;
+	# Set to untranslated messages, to be able to compare program output
+	# with expected strings.
+	delete $ENV{LANGUAGE};
+	delete $ENV{LC_ALL};
+	$ENV{LC_MESSAGES} = 'C';
+
+	delete $ENV{PGCONNECT_TIMEOUT};
+	delete $ENV{PGDATA};
+	delete $ENV{PGDATABASE};
+	delete $ENV{PGHOSTADDR};
+	delete $ENV{PGREQUIRESSL};
+	delete $ENV{PGSERVICE};
+	delete $ENV{PGSSLMODE};
+	delete $ENV{PGUSER};
+	delete $ENV{PGPORT};
+	delete $ENV{PGHOST};
+
+	# PGHOST is set once and for all through a single series of tests when
+	# this module is loaded.
+	$test_pghost = $windows_os ? "127.0.0.1" : TestBase::tempdir_short();
+	$ENV{PGHOST} = $test_pghost;
+	$ENV{PGDATABASE} = 'postgres';
+
+	# Tracking of last port value assigned to accelerate free port lookup.
+	# XXX: Should this use PG_VERSION_NUM?
+	$last_port_assigned = 90600 % 16384 + 49152;
+
+	# Tracker of active nodes
+	@all_nodes    = ();
+	@active_nodes = ();
 }
 
-$ENV{PGPORT} = int($ENV{PGPORT}) % 65536;
-
-
+# Build a new PostgresNode object, assigning a free port number.
 #
-# Helper functions
-#
-
-
-sub tempdir
+# We also register the node, to avoid the port number from being reused
+# for another node even when this one is not active.
+sub get_new_node
 {
-	return File::Temp::tempdir(
-		'tmp_testXXXX',
-		DIR => $ENV{TESTDIR} || cwd(),
-		CLEANUP => 1);
-}
+	my $found = 0;
+	my $port  = $last_port_assigned;
 
-sub tempdir_short
-{
-
-	# Use a separate temp dir outside the build tree for the
-	# Unix-domain socket, to avoid file name length issues.
-	return File::Temp::tempdir(CLEANUP => 1);
-}
-
-# Initialize a new cluster for testing.
-#
-# The PGHOST environment variable is set to connect to the new cluster.
-#
-# Authentication is set up so that only the current OS user can access the
-# cluster. On Unix, we use Unix domain socket connections, with the socket in
-# a directory that's only accessible to the current user to ensure that.
-# On Windows, we use SSPI authentication to ensure the same (by pg_regress
-# --config-auth).
-sub standard_initdb
-{
-	my $pgdata = shift;
-	system_or_bail('initdb', '-D', "$pgdata", '-A' , 'trust', '-N');
-	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
-
-	my $tempdir_short = tempdir_short;
-
-	open CONF, ">>$pgdata/postgresql.conf";
-	print CONF "\n# Added by TestLib.pm)\n";
-	print CONF "fsync = off\n";
-	if ($windows_os)
+	while ($found == 0)
 	{
-		print CONF "listen_addresses = '127.0.0.1'\n";
-	}
-	else
-	{
-		print CONF "unix_socket_directories = '$tempdir_short'\n";
-		print CONF "listen_addresses = ''\n";
-	}
-	close CONF;
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		if (!run_log([ 'pg_isready', '-p', $port ]))
+		{
+			$found = 1;
 
-	$ENV{PGHOST}         = $windows_os ? "127.0.0.1" : $tempdir_short;
-}
-
-# Set up the cluster to allow replication connections, in the same way that
-# standard_initdb does for normal connections.
-sub configure_hba_for_replication
-{
-	my $pgdata = shift;
-
-	open HBA, ">>$pgdata/pg_hba.conf";
-	print HBA "\n# Allow replication (set up by TestLib.pm)\n";
-	if (! $windows_os)
-	{
-		print HBA "local replication all trust\n";
-	}
-	else
-	{
-		print HBA "host replication all 127.0.0.1/32 sspi include_realm=1 map=regress\n";
-	}
-	close HBA;
-}
-
-my ($test_server_datadir, $test_server_logfile);
-
-
-# Initialize a new cluster for testing in given directory, and start it.
-sub start_test_server
-{
-	my ($tempdir) = @_;
-	my $ret;
-
-	print("### Starting test server in $tempdir\n");
-	standard_initdb "$tempdir/pgdata";
-
-	$ret = system_log('pg_ctl', '-D', "$tempdir/pgdata", '-w', '-l',
-	  "$log_path/postmaster.log", '-o', "--log-statement=all",
-	  'start');
-
-	if ($ret != 0)
-	{
-		print "# pg_ctl failed; logfile:\n";
-		system('cat', "$log_path/postmaster.log");
-		BAIL_OUT("pg_ctl failed");
+			# Found a potential candidate port number.  Check first that it is
+			# not included in the list of registered nodes.
+			foreach my $node (@all_nodes)
+			{
+				$found = 0 if ($node->getPort() == $port);
+			}
+		}
 	}
 
-	$test_server_datadir = "$tempdir/pgdata";
-	$test_server_logfile = "$log_path/postmaster.log";
+	print "# Found free port $port\n";
+
+	# Lock port number found by creating a new node
+	my $node = new PostgresNode($test_pghost, $port);
+
+	# Add node to list of nodes currently in use
+	push(@all_nodes,    $node);
+	push(@active_nodes, $node);
+	$last_port_assigned = $port;
+
+	return $node;
 }
 
-sub restart_test_server
+sub teardown_node
 {
-	print("### Restarting test server\n");
-	system_log('pg_ctl', '-D', $test_server_datadir, '-w', '-l',
-	  $test_server_logfile, 'restart');
-}
+	my $node = shift;
 
-END
-{
-	if ($test_server_datadir)
-	{
-		system_log('pg_ctl', '-D', $test_server_datadir, '-m',
-		  'immediate', 'stop');
-	}
+	$node->stopNode('immediate');
+	@active_nodes = grep { $_ ne $node } @active_nodes;
 }
 
 sub psql
@@ -225,56 +120,55 @@ sub psql
 	my ($dbname, $sql) = @_;
 	my ($stdout, $stderr);
 	print("# Running SQL command: $sql\n");
-	run [ 'psql', '-X', '-A', '-t', '-q', '-d', $dbname, '-f', '-' ], '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
+	run [ 'psql', '-X', '-A', '-t', '-q', '-d', $dbname, '-f', '-' ],
+	  '<', \$sql, '>', \$stdout, '2>', \$stderr
+	  or die;
+	if ($stderr ne "")
+	{
+		print "#### Begin standard error\n";
+		print $stderr;
+		print "#### End standard error\n";
+	}
 	chomp $stdout;
 	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
 	return $stdout;
 }
 
-sub slurp_dir
+# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
+sub poll_query_until
 {
-	my ($dir) = @_;
-	opendir(my $dh, $dir) or die;
-	my @direntries = readdir $dh;
-	closedir $dh;
-	return @direntries;
-}
+	my ($node, $query) = @_;
 
-sub slurp_file
-{
-	local $/;
-	local @ARGV = @_;
-	my $contents = <>;
-	$contents =~ s/\r//g if $Config{osname} eq 'msys';
-	return $contents;
-}
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
 
-sub system_or_bail
-{
-	if (system_log(@_) != 0)
+	while ($attempts < $max_attempts)
 	{
-		BAIL_OUT("system $_[0] failed: $?");
+		my $cmd = [ 'psql', '-At', '-c', $query, '-d', $node->getConnStr() ];
+		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
 	}
-}
 
-sub system_log
-{
-	print("# Running: " . join(" ", @_) ."\n");
-	return system(@_);
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
 }
 
-sub run_log
-{
-	print("# Running: " . join(" ", @{$_[0]}) ."\n");
-	return run (@_);
-}
-
-
 #
 # Test functions
 #
-
-
 sub command_ok
 {
 	my ($cmd, $test_name) = @_;
@@ -292,7 +186,7 @@ sub command_fails
 sub command_exit_is
 {
 	my ($cmd, $expected, $test_name) = @_;
-	print("# Running: " . join(" ", @{$cmd}) ."\n");
+	print("# Running: " . join(" ", @{$cmd}) . "\n");
 	my $h = start $cmd;
 	$h->finish();
 
@@ -303,8 +197,10 @@ sub command_exit_is
 	# assuming the Unix convention, which will always return 0 on Windows as
 	# long as the process was not terminated by an exception. To work around
 	# that, use $h->full_result on Windows instead.
-	my $result = ($Config{osname} eq "MSWin32") ?
-		($h->full_results)[0] : $h->result(0);
+	my $result =
+	    ($Config{osname} eq "MSWin32")
+	  ? ($h->full_results)[0]
+	  : $h->result(0);
 	is($result, $expected, $test_name);
 }
 
@@ -335,8 +231,8 @@ sub program_options_handling_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --not-a-valid-option\n");
-	my $result = run [ $cmd, '--not-a-valid-option' ], '>', \$stdout, '2>',
-	  \$stderr;
+	my $result = run [ $cmd, '--not-a-valid-option' ], '>', \$stdout,
+	  '2>', \$stderr;
 	ok(!$result, "$cmd with invalid option nonzero exit code");
 	isnt($stderr, '', "$cmd with invalid option prints error message");
 }
@@ -354,11 +250,11 @@ sub command_like
 
 sub issues_sql_like
 {
-	my ($cmd, $expected_sql, $test_name) = @_;
-	truncate $test_server_logfile, 0;
+	my ($node, $cmd, $expected_sql, $test_name) = @_;
+	truncate $node->getLogFile(), 0;
 	my $result = run_log($cmd);
 	ok($result, "@$cmd exit code 0");
-	my $log = slurp_file($test_server_logfile);
+	my $log = slurp_file($node->getLogFile());
 	like($log, $expected_sql, "$test_name: SQL found in server log");
 }
 
diff --git a/src/test/ssl/ServerSetup.pm b/src/test/ssl/ServerSetup.pm
index a6c77b5..ccd4754 100644
--- a/src/test/ssl/ServerSetup.pm
+++ b/src/test/ssl/ServerSetup.pm
@@ -18,6 +18,7 @@ package ServerSetup;
 
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use File::Basename;
 use File::Copy;
@@ -45,7 +46,7 @@ sub copy_files
 
 sub configure_test_server_for_ssl
 {
-	my $tempdir    = $_[0];
+	my $pgdata     = $_[0];
 	my $serverhost = $_[1];
 
 	# Create test users and databases
@@ -55,7 +56,7 @@ sub configure_test_server_for_ssl
 	psql 'postgres', "CREATE DATABASE certdb";
 
 	# enable logging etc.
-	open CONF, ">>$tempdir/pgdata/postgresql.conf";
+	open CONF, ">>$pgdata/postgresql.conf";
 	print CONF "fsync=off\n";
 	print CONF "log_connections=on\n";
 	print CONF "log_hostname=on\n";
@@ -68,17 +69,17 @@ sub configure_test_server_for_ssl
 	close CONF;
 
 # Copy all server certificates and keys, and client root cert, to the data dir
-	copy_files("ssl/server-*.crt", "$tempdir/pgdata");
-	copy_files("ssl/server-*.key", "$tempdir/pgdata");
-	chmod(0600, glob "$tempdir/pgdata/server-*.key") or die $!;
-	copy_files("ssl/root+client_ca.crt", "$tempdir/pgdata");
-	copy_files("ssl/root+client.crl",    "$tempdir/pgdata");
+	copy_files("ssl/server-*.crt", $pgdata);
+	copy_files("ssl/server-*.key", $pgdata);
+	chmod(0600, glob "$pgdata/server-*.key") or die $!;
+	copy_files("ssl/root+client_ca.crt", $pgdata);
+	copy_files("ssl/root+client.crl",    $pgdata);
 
   # Only accept SSL connections from localhost. Our tests don't depend on this
   # but seems best to keep it as narrow as possible for security reasons.
   #
   # When connecting to certdb, also check the client certificate.
-	open HBA, ">$tempdir/pgdata/pg_hba.conf";
+	open HBA, ">$pgdata/pg_hba.conf";
 	print HBA
 "# TYPE  DATABASE        USER            ADDRESS                 METHOD\n";
 	print HBA
@@ -96,12 +97,13 @@ sub configure_test_server_for_ssl
 # the server so that the configuration takes effect.
 sub switch_server_cert
 {
-	my $tempdir  = $_[0];
+	my $node     = $_[0];
 	my $certfile = $_[1];
+	my $pgdata   = $node->getDataDir();
 
 	diag "Restarting server with certfile \"$certfile\"...";
 
-	open SSLCONF, ">$tempdir/pgdata/sslconfig.conf";
+	open SSLCONF, ">$pgdata/sslconfig.conf";
 	print SSLCONF "ssl=on\n";
 	print SSLCONF "ssl_ca_file='root+client_ca.crt'\n";
 	print SSLCONF "ssl_cert_file='$certfile.crt'\n";
@@ -110,5 +112,5 @@ sub switch_server_cert
 	close SSLCONF;
 
 	# Stop and restart server to reload the new config.
-	restart_test_server();
+	$node->restartNode();
 }
diff --git a/src/test/ssl/t/001_ssltests.pl b/src/test/ssl/t/001_ssltests.pl
index 0d6f339..6a32af1 100644
--- a/src/test/ssl/t/001_ssltests.pl
+++ b/src/test/ssl/t/001_ssltests.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+use PostgresNode;
+use TestBase;
 use TestLib;
 use Test::More tests => 38;
 use ServerSetup;
@@ -25,8 +27,6 @@ BEGIN
 # postgresql-ssl-regression.test.
 my $SERVERHOSTADDR = '127.0.0.1';
 
-my $tempdir = TestLib::tempdir;
-
 # Define a couple of helper functions to test connecting to the server.
 
 my $common_connstr;
@@ -74,10 +74,16 @@ chmod 0600, "ssl/client.key";
 
 #### Part 0. Set up the server.
 
-diag "setting up data directory in \"$tempdir\"...";
-start_test_server($tempdir);
-configure_test_server_for_ssl($tempdir, $SERVERHOSTADDR);
-switch_server_cert($tempdir, 'server-cn-only');
+diag "setting up data directory...";
+my $node = get_new_node();
+$node->initNode();
+# PGHOST is enforced here to set up the node, subsequent connections
+# will use a dedicated connection string.
+$ENV{PGHOST} = $node->getHost();
+$ENV{PGPORT} = $node->getPort();
+$node->startNode();
+configure_test_server_for_ssl($node->getDataDir(), $SERVERHOSTADDR);
+switch_server_cert($node, 'server-cn-only');
 
 ### Part 1. Run client-side tests.
 ###
@@ -150,7 +156,7 @@ test_connect_ok("sslmode=verify-ca host=wronghost.test");
 test_connect_fails("sslmode=verify-full host=wronghost.test");
 
 # Test Subject Alternative Names.
-switch_server_cert($tempdir, 'server-multiple-alt-names');
+switch_server_cert($node, 'server-multiple-alt-names');
 
 diag "test hostname matching with X509 Subject Alternative Names";
 $common_connstr =
@@ -165,7 +171,7 @@ test_connect_fails("host=deep.subdomain.wildcard.pg-ssltest.test");
 
 # Test certificate with a single Subject Alternative Name. (this gives a
 # slightly different error message, that's all)
-switch_server_cert($tempdir, 'server-single-alt-name');
+switch_server_cert($node, 'server-single-alt-name');
 
 diag "test hostname matching with a single X509 Subject Alternative Name";
 $common_connstr =
@@ -178,7 +184,7 @@ test_connect_fails("host=deep.subdomain.wildcard.pg-ssltest.test");
 
 # Test server certificate with a CN and SANs. Per RFCs 2818 and 6125, the CN
 # should be ignored when the certificate has both.
-switch_server_cert($tempdir, 'server-cn-and-alt-names');
+switch_server_cert($node, 'server-cn-and-alt-names');
 
 diag "test certificate with both a CN and SANs";
 $common_connstr =
@@ -190,7 +196,7 @@ test_connect_fails("host=common-name.pg-ssltest.test");
 
 # Finally, test a server certificate that has no CN or SANs. Of course, that's
 # not a very sensible certificate, but libpq should handle it gracefully.
-switch_server_cert($tempdir, 'server-no-names');
+switch_server_cert($node, 'server-no-names');
 $common_connstr =
 "user=ssltestuser dbname=trustdb sslcert=invalid sslrootcert=ssl/root+server_ca.crt hostaddr=$SERVERHOSTADDR";
 
@@ -199,7 +205,7 @@ test_connect_fails("sslmode=verify-full host=common-name.pg-ssltest.test");
 
 # Test that the CRL works
 diag "Testing client-side CRL";
-switch_server_cert($tempdir, 'server-revoked');
+switch_server_cert($node, 'server-revoked');
 
 $common_connstr =
 "user=ssltestuser dbname=trustdb sslcert=invalid hostaddr=$SERVERHOSTADDR host=common-name.pg-ssltest.test";
@@ -233,7 +239,3 @@ test_connect_fails(
 test_connect_fails(
 "user=ssltestuser sslcert=ssl/client-revoked.crt sslkey=ssl/client-revoked.key"
 );
-
-
-# All done! Save the log, before the temporary installation is deleted
-copy("$tempdir/client-log", "./client-log");
#80Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#79)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Nov 28, 2015 at 7:53 AM, Alvaro Herrera wrote:

I moved all the initialization code (deleting stuff from environment,
detecting Windows, opening SimpleTie filedescs etc) into BEGIN blocks,
which run earlier than any other code.

Ah, OK. Thanks. That makes visibly the whole set of modules more consistent.

Hmm. I just noticed RewindTest sets $ENV{PGDATABASE} outside BEGIN. Not
sure what to think of that. Could instead pass the database name in
$node->getConnStr() calls, like run_pg_rewind() is already doing.

Yes, let's remove that and pass the database name to getConnStr().

I tried all the t/ tests we have and all of them pass for me. If I'm
able, I will push this on my Sunday late evening, so that I can fix
whatever gets red on Monday first thing ...

I have done as well additional tests on Windows and this patch is
showing a green status.

A separate issue, but as long as we are working on this set of tests:
I have noticed that config_default.pl is missing the flag tap_tests in
its list. See the patch attached. Could you apply that as well and
backpatch?

I have as well noticed that RewindTest.pm is missing "1;" on its last
line. When this is loaded this would lead to compilation errors.
--
Michael

Attachments:

20151128_fix_config_msvc.patchtext/x-patch; charset=US-ASCII; name=20151128_fix_config_msvc.patchDownload
diff --git a/src/tools/msvc/config_default.pl b/src/tools/msvc/config_default.pl
index b9f2ff4..e50be7e 100644
--- a/src/tools/msvc/config_default.pl
+++ b/src/tools/msvc/config_default.pl
@@ -3,7 +3,7 @@ use strict;
 use warnings;
 
 our $config = {
-	asserts => 0,    # --enable-cassert
+	asserts  => 0,    # --enable-cassert
 	  # integer_datetimes=>1,   # --enable-integer-datetimes - on is now default
 	  # float4byval=>1,         # --disable-float4-byval, on by default
 
@@ -13,18 +13,19 @@ our $config = {
 	# blocksize => 8,         # --with-blocksize, 8kB by default
 	# wal_blocksize => 8,     # --with-wal-blocksize, 8kB by default
 	# wal_segsize => 16,      # --with-wal-segsize, 16MB by default
-	ldap     => 1,        # --with-ldap
-	extraver => undef,    # --with-extra-version=<string>
-	nls      => undef,    # --enable-nls=<path>
-	tcl      => undef,    # --with-tls=<path>
-	perl     => undef,    # --with-perl
-	python   => undef,    # --with-python=<path>
-	openssl  => undef,    # --with-openssl=<path>
-	uuid     => undef,    # --with-ossp-uuid
-	xml      => undef,    # --with-libxml=<path>
-	xslt     => undef,    # --with-libxslt=<path>
-	iconv    => undef,    # (not in configure, path to iconv)
-	zlib     => undef     # --with-zlib=<path>
+	ldap      => 1,        # --with-ldap
+	extraver  => undef,    # --with-extra-version=<string>
+	nls       => undef,    # --enable-nls=<path>
+	tap_tests => undef,    # --enable-tap-tests
+	tcl       => undef,    # --with-tls=<path>
+	perl      => undef,    # --with-perl
+	python    => undef,    # --with-python=<path>
+	openssl   => undef,    # --with-openssl=<path>
+	uuid      => undef,    # --with-ossp-uuid
+	xml       => undef,    # --with-libxml=<path>
+	xslt      => undef,    # --with-libxslt=<path>
+	iconv     => undef,    # (not in configure, path to iconv)
+	zlib      => undef     # --with-zlib=<path>
 };
 
 1;
#81Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#80)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

On Sat, Nov 28, 2015 at 7:53 AM, Alvaro Herrera wrote:

Hmm. I just noticed RewindTest sets $ENV{PGDATABASE} outside BEGIN. Not
sure what to think of that. Could instead pass the database name in
$node->getConnStr() calls, like run_pg_rewind() is already doing.

Yes, let's remove that and pass the database name to getConnStr().

Ok.

A separate issue, but as long as we are working on this set of tests:
I have noticed that config_default.pl is missing the flag tap_tests in
its list. See the patch attached. Could you apply that as well and
backpatch?

I have as well noticed that RewindTest.pm is missing "1;" on its last
line. When this is loaded this would lead to compilation errors.

Sure.

I tried all the t/ tests we have and all of them pass for me. If I'm
able, I will push this on my Sunday late evening, so that I can fix
whatever gets red on Monday first thing ...

I have done as well additional tests on Windows and this patch is
showing a green status.

Great.

Here's your recovery test patch rebased, for your (and others'!)
perusal. It passes for me. (Test 003 is unchanged.)

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

0002.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..7f7754f 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -12,7 +12,7 @@ subdir = src/test
 top_builddir = ../..
 include $(top_builddir)/src/Makefile.global
 
-SUBDIRS = regress isolation modules
+SUBDIRS = regress isolation modules recovery
 
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/RecoveryTest.pm b/src/test/recovery/RecoveryTest.pm
new file mode 100644
index 0000000..3706847
--- /dev/null
+++ b/src/test/recovery/RecoveryTest.pm
@@ -0,0 +1,177 @@
+# Set of common routines for recovery regression tests for a PostgreSQL
+# cluster. This includes methods that can be used by the various set of
+# tests present to set up cluster nodes and configure them according to
+# the test scenario wanted.
+#
+# This module makes use of PostgresNode for node manipulation, performing
+# higher-level operations to create standby nodes or setting them up
+# for archiving and replication.
+#
+# Nodes are identified by their port number and have one allocated when
+# created, hence it is unique for each node of the cluster as it is run
+# locally. PGHOST is equally set to a unique value for the duration of
+# each test.
+
+package RecoveryTest;
+
+use strict;
+use warnings;
+
+use Cwd;
+use Exporter 'import';
+use IPC::Run qw(run start);
+use PostgresNode;
+use RecursiveCopy;
+use TestBase;
+use TestLib;
+use Test::More;
+
+our @EXPORT = qw(
+	enable_archiving
+	enable_restoring
+	enable_streaming
+	make_master
+	make_archive_standby
+	make_stream_standby
+);
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $node_root = shift; # Instance to link to
+	my $node_standby = shift;
+	my $root_connstr = $node_root->getConnStr();
+	my $applname = $node_standby->getApplName();
+
+	$node_standby->appendConf('recovery.conf', qq(
+primary_conninfo='$root_connstr application_name=$applname'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $node_root = shift; # Instance to link to
+	my $node_standby = shift;
+	my $path = $node_root->getArchiveDir();
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"$path\\\\%f\" \"%p\"" :
+		"cp -i $path/%f %p";
+	$node_standby->appendConf('recovery.conf', qq(
+restore_command='$copy_command'
+standby_mode=on
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $node = shift;
+	my $path = $node->getArchiveDir();
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$path\\\\%f\"" :
+		"cp %p $path/%f";
+
+	# Enable archive_mode and archive_command on node
+	$node->appendConf('postgresql.conf', qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization.
+sub make_master
+{
+	my $node_master = get_new_node();
+	my $port_master = $node_master->getPort();
+	print "# Initializing master node wih port $port_master\n";
+	$node_master->initNode();
+	configure_base_node($node_master);
+	return $node_master;
+}
+
+sub configure_base_node
+{
+	my $node = shift;
+
+	$node->appendConf('postgresql.conf', qq(
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+));
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_stream_standby
+{
+	my $node_master = shift;
+	my $backup_name = shift;
+	my $node_standby = get_new_node();
+	my $master_port = $node_master->getPort();
+	my $standby_port = $node_standby->getPort();
+
+	print "# Initializing streaming mode for node $standby_port from node $master_port\n";
+	$node_standby->initNodeFromBackup($node_master, $backup_name);
+	configure_base_node($node_standby);
+
+	# Start second node, streaming from first one
+	enable_streaming($node_master, $node_standby);
+	return $node_standby;
+}
+
+# Node getting WAL only from archives
+sub make_archive_standby
+{
+	my $node_master = shift;
+	my $backup_name = shift;
+	my $node_standby = get_new_node();
+	my $master_port = $node_master->getPort();
+	my $standby_port = $node_standby->getPort();
+
+	print "# Initializing archive mode for node $standby_port from node $master_port\n";
+	$node_standby->initNodeFromBackup($node_master, $backup_name);
+	configure_base_node($node_standby);
+
+	# Start second node, restoring from first one
+	enable_restoring($node_master, $node_standby);
+	return $node_standby;
+}
+
+# Wait until a node is able to accept queries. Useful when putting a node
+# in recovery and wait for it to be able to work particularly on slow
+# machines.
+sub wait_for_node
+{
+	my $node         = shift;
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	while ($attempts < $max_attempts)
+	{
+		if (run_log(['pg_isready', '-p', $node->getPort()]))
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+	return 0;
+}
+
+1;
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..e902d42
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,67 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+
+use RecoveryTest;
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->startNode();
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_master->backupNode($backup_name);
+
+# Create streaming standby linking to master
+my $node_standby_1 = make_stream_standby($node_master, $backup_name);
+$node_standby_1->startNode();
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+$node_standby_1->backupNode($backup_name);
+
+# Create second standby node linking to standby 1
+my $node_standby_2 = make_stream_standby($node_standby_1, $backup_name);
+$node_standby_2->startNode();
+$node_standby_2->backupNode($backup_name);
+
+# Create some content on master and check its presence in standby 1 an
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a";
+
+# Wait for standbys to catch up
+my $applname_1 = $node_standby_1->getApplName();
+my $applname_2 = $node_standby_2->getApplName();
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_1';";
+poll_query_until($node_master, $caughtup_query)
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_2';";
+poll_query_until($node_standby_1, $caughtup_query)
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = psql $node_standby_1->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result = psql $node_standby_2->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->getConnStr(), '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->getConnStr(), '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
+
+# Cleanup nodes
+teardown_node($node_standby_2);
+teardown_node($node_standby_1);
+teardown_node($node_master);
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..16dbbc1
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,51 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $node_master = make_master();
+my $backup_name = 'my_backup';
+enable_archiving($node_master);
+
+# Start it
+$node_master->startNode();
+
+# Take backup for slave
+$node_master->backupNode($backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $node_standby = make_archive_standby($node_master, $backup_name);
+$node_standby->appendConf('postgresql.conf', qq(
+wal_retrieve_retry_interval = '100ms'
+));
+$node_standby->startNode();
+
+# Create some content on master
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $current_lsn = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# Force archiving of WAL file to make it present on master
+psql $node_master->getConnStr(), "SELECT pg_switch_xlog()";
+
+# Add some more content, it should not be present on standby
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($node_standby, $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $node_standby->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+is($result, qq(1000), 'check content from archives');
+
+# Cleanup nodes
+teardown_node($node_standby);
+teardown_node($node_master);
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..6c5021c
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,135 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $node_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $node_standby = make_archive_standby($node_master, 'my_backup');
+
+	foreach my $param_item (@$recovery_params)
+	{
+		$node_standby->appendConf('recovery.conf',
+					   qq($param_item
+));
+	}
+
+	$node_standby->startNode();
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	poll_query_until($node_standby, $caughtup_query)
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = psql $node_standby->getConnStr(),
+		"SELECT count(*) FROM tab_int";
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	teardown_node($node_standby);
+}
+
+# Initialize master node
+my $node_master = make_master();
+enable_archiving($node_master);
+
+# Start it
+$node_master->startNode();
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $lsn1 = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# Take backup from which all operations will be run
+$node_master->backupNode('my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+my $recovery_txid = psql $node_master->getConnStr(),
+	"SELECT txid_current()";
+my $lsn2 = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# More data, with recovery target timestamp
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(2001,3000))";
+my $recovery_time = psql $node_master->getConnStr(), "SELECT now()";
+my $lsn3 = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# Even more data, this time with a recovery target name
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))";
+my $recovery_name = "my_target";
+my $lsn4 = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+psql $node_master->getConnStr(),
+	"SELECT pg_create_restore_point('$recovery_name')";
+
+# Force archiving of WAL file
+psql $node_master->getConnStr(), "SELECT pg_switch_xlog()";
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $node_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+teardown_node($node_master);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..f0ddebb
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,76 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use PostgresNode;
+use TestBase;
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->startNode();
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backupNode($backup_name);
+
+# Create two standbys linking to it
+my $node_standby_1 = make_stream_standby($node_master, $backup_name);
+$node_standby_1->startNode();
+my $node_standby_2 = make_stream_standby($node_master, $backup_name);
+$node_standby_2->startNode();
+
+# Create some content on master
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a";
+my $until_lsn = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($node_standby_1, $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+teardown_node($node_master);
+system_or_bail('pg_ctl', '-w', '-D', $node_standby_1->getDataDir(),
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree($node_standby_2->getDataDir() . '/recovery.conf');
+my $connstr_1 = $node_standby_1->getConnStr();
+$node_standby_2->appendConf('recovery.conf', qq(
+primary_conninfo='$connstr_1'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+$node_standby_2->restartNode();
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done. Standby 1 needs
+# to exit recovery first before moving on with the test.
+poll_query_until($node_standby_1, "SELECT pg_is_in_recovery() <> true",
+				 );
+psql $node_standby_1->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(1001,2000))";
+$until_lsn = psql $node_standby_1->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($node_standby_2, $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = psql $node_standby_2->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+is($result, qq(2000), 'check content of standby 2');
+
+# Stop nodes
+teardown_node($node_standby_2);
+teardown_node($node_standby_1);
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..f00e93b
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,49 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->startNode();
+
+# And some content
+psql $node_master->getConnStr(),
+	"CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a";
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backupNode($backup_name);
+
+# Create streaming standby from backup
+my $node_standby = make_stream_standby($node_master, $backup_name);
+$node_standby->appendConf('recovery.conf', qq(
+recovery_min_apply_delay = '2s'
+));
+$node_standby->startNode();
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+psql $node_master->getConnStr(),
+	"INSERT INTO tab_int VALUES (generate_series(11,20))";
+sleep 1;
+# Here we should have only 10 rows
+my $result = psql $node_standby->getConnStr(),
+	"SELECT count(*) FROM tab_int";
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = psql $node_master->getConnStr(),
+	"SELECT pg_current_xlog_location();";
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+poll_query_until($node_standby, $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+$result = psql $node_standby->getConnStr(), "SELECT count(*) FROM tab_int";
+is($result, qq(20), 'check content with delay of 2s');
+
+# Stop nodes
+teardown_node($node_standby);
+teardown_node($node_master);
#82Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#81)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sun, Nov 29, 2015 at 12:13 AM, Alvaro Herrera wrote:

Here's your recovery test patch rebased, for your (and others'!)
perusal. It passes for me. (Test 003 is unchanged.)

Are you planning to push that as well? It does not have much coverage
but I guess that's quite good for a first shot, and that can serve as
example for future tests.

Still, the first patch adds enough infrastructure to allow any other
module to have more complex regression test scenarios, the first two
targets coming immediately to my mind being the quorum syncrep patch
and pg_rewind and its timeline switch manipulation. So that's more
than welcome!
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#83Noah Misch
noah@leadboat.com
In reply to: Alvaro Herrera (#79)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Nov 27, 2015 at 07:53:10PM -0300, Alvaro Herrera wrote:

Michael Paquier wrote:

The result of a couple of hours of hacking is attached:
- 0001 is the refactoring adding PostgresNode and RecursiveCopy. I have
also found that it is quite advantageous to move some of the routines that
are synonyms of system() and the stuff used for logging into another
low-level library that PostgresNode depends on, that I called TestBase in
this patch.

Here's another version of this. I changed the packages a bit more. For
starters, I moved the routines around a bit; some of your choices seemed
more about keeping stuff where it was originally rather than moving it
to where it made sense. These are the routines in each module:

TestBase: system_or_bail system_log run_log slurp_dir slurp_file
append_to_file

TestLib: get_new_node teardown_node psql poll_query_until command_ok
command_fails command_exit_is program_help_ok program_version_ok
program_options_handling_ok command_like issues_sql_like

The proposed code is short on guidance about when to put a function in TestLib
versus TestBase. TestLib has no header comment. The TestBase header comment
would permit, for example, command_ok() in that module. I would try instead
keeping TestLib as the base module and moving into PostgresNode the functions
that deal with PostgreSQL clusters (get_new_node teardown_node psql
poll_query_until issues_sql_like).

I tried to get rid of teardown_node by having a DESTROY method for
PostgresNode; that method would call "pg_ctl stop -m immediate". That
would have been much cleaner. However, when a test fails this doesn't
work sanely because the END block for File::Temp runs earlier than that
DESTROY block, which means the datadir is already gone by the time
pg_ctl stop runs, so the node stop doesn't work at all. (Perhaps we
could fix this by noting postmaster's PID at start time, and then
sending a signal directly instead of relying on pg_ctl).

You could disable File::Temp cleanup and handle cleanup yourself at the
desired time. (I haven't reviewed whether the goal of removing teardown_node
is otherwise good.)

+my $node = get_new_node();
+# Initialize node without replication settings
+$node->initNode(0);
+$node->startNode();
+my $pgdata = $node->getDataDir();
+
+$ENV{PGPORT} = $node->getPort();

Starting a value retrieval method name with "get" is not Perlish. The TAP
suites currently follow "man perlstyle" in using underscored_lower_case method
names. No PostgreSQL Perl code uses lowerFirstCamelCase, though some uses
CamelCase. The word "Node" is redundant. Use this style:

$node->init(0);
$node->start;
my $pgdata = $node->data_dir;
$ENV{PGPORT} = $node->port;

As a matter of opinion, I recommend giving "init" key/value arguments instead
of the single Boolean argument. The method could easily need more options in
the future, and this makes the call site self-documenting:

$node->init(hba_permit_replication => 0);

-	'pg_controldata with nonexistent directory fails');
+			  'pg_controldata with nonexistent directory fails');

perltidy will undo this whitespace-only change.

--- a/src/bin/pg_rewind/t/001_basic.pl
+++ b/src/bin/pg_rewind/t/001_basic.pl
@@ -1,9 +1,11 @@
+# Basic pg_rewind test.
+
use strict;
use warnings;
-use TestLib;
-use Test::More tests => 8;
use RewindTest;
+use TestLib;
+use Test::More tests => 8;

Revert all changes to this file. Audit the rest of the patch for whitespace
change unrelated to the subject.

-	'fails with nonexistent table');
+			  'fails with nonexistent table');
-'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';
+	'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';

perltidy will undo these whitespace-only changes.

+# cluster -a is not compatible with -d, hence enforce environment variables

s/cluster -a/clusterdb -a/

-issues_sql_like(
+$ENV{PGPORT} = $node->getPort();
+
+issues_sql_like($node,

perltidy will move $node to its own line.

-command_fails([ 'createuser', 'user1' ], 'fails if role already exists');
+command_fails([ 'createuser', 'user1' ],
+			  'fails if role already exists');

perltidy will undo this whitespace-only change.

@@ -0,0 +1,252 @@
+# PostgresNode, simple node representation for regression tests.
+#
+# Regression tests should use this basic class infrastructure to define nodes
+# that need used in the test modules/scripts.
+package PostgresNode;

Consider just saying, "Class representing a data directory and postmaster."

+	my $self   = {
+		_port     => undef,
+		_host     => undef,
+		_basedir  => undef,
+		_applname => undef,
+		_logfile  => undef };
+
+	# Set up each field
+	$self->{_port}     = $pgport;
+	$self->{_host}     = $pghost;
+	$self->{_basedir}  = TestBase::tempdir;
+	$self->{_applname} = "node_$pgport";
+	$self->{_logfile}  = "$TestBase::log_path/node_$pgport.log";

Why set fields to undef immediately before filling them?

@@ -0,0 +1,143 @@
+# Set of low-level routines dedicated to base tasks for regression tests, like
+# command execution and logging.
+#
+# This module should not depend on any other PostgreSQL regression test
+# modules.
+package TestBase;

This is no mere set of routines. Just "use"-ing this module creates some
directories and alters stdin/stdout/stderr.

+BEGIN
+{
+	$windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
+
+	# Determine output directories, and create them.  The base path is the
+	# TESTDIR environment variable, which is normally set by the invoking
+	# Makefile.
+	$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
+	$log_path = "$tmp_check/log";
+
+	mkdir $tmp_check;
+	mkdir $log_path;

Never mutate the filesystem in a BEGIN block, because "perl -c" runs BEGIN
blocks. (Likewise for the BEGIN block this patch adds to TestLib.)

nm

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#84Andrew Dunstan
andrew@dunslane.net
In reply to: Noah Misch (#83)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 11/29/2015 04:28 PM, Noah Misch wrote:

+BEGIN
+{
+	$windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
+
+	# Determine output directories, and create them.  The base path is the
+	# TESTDIR environment variable, which is normally set by the invoking
+	# Makefile.
+	$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
+	$log_path = "$tmp_check/log";
+
+	mkdir $tmp_check;
+	mkdir $log_path;
Never mutate the filesystem in a BEGIN block, because "perl -c" runs BEGIN
blocks.  (Likewise for the BEGIN block this patch adds to TestLib.)

Yeah, those two lines might belong in an INIT block. "perldoc perlmod"
for details.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#85Michael Paquier
michael.paquier@gmail.com
In reply to: Noah Misch (#83)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Mon, Nov 30, 2015 at 6:28 AM, Noah Misch <noah@leadboat.com> wrote:

On Fri, Nov 27, 2015 at 07:53:10PM -0300, Alvaro Herrera wrote:

Michael Paquier wrote:

The result of a couple of hours of hacking is attached:
- 0001 is the refactoring adding PostgresNode and RecursiveCopy. I have
also found that it is quite advantageous to move some of the routines that
are synonyms of system() and the stuff used for logging into another
low-level library that PostgresNode depends on, that I called TestBase in
this patch.

Here's another version of this. I changed the packages a bit more. For
starters, I moved the routines around a bit; some of your choices seemed
more about keeping stuff where it was originally rather than moving it
to where it made sense. These are the routines in each module:

TestBase: system_or_bail system_log run_log slurp_dir slurp_file
append_to_file

TestLib: get_new_node teardown_node psql poll_query_until command_ok
command_fails command_exit_is program_help_ok program_version_ok
program_options_handling_ok command_like issues_sql_like

The proposed code is short on guidance about when to put a function in TestLib
versus TestBase. TestLib has no header comment. The TestBase header comment
would permit, for example, command_ok() in that module. I would try instead
keeping TestLib as the base module and moving into PostgresNode the functions
that deal with PostgreSQL clusters (get_new_node teardown_node psql
poll_query_until issues_sql_like).

PostgresNode is wanted to be a base representation of how of node is,
not of how to operate on it. The ways to perform the tests, which
works on a node, is wanted as a higher-level operation.

Logging and base configuration of a test set is a lower level of
operations than PostgresNode, because cluster nodes need actually to
perform system calls, some of those system calls like run_log allowing
to log in the centralized log file. I have tried to make the headers
of those modules more verbose, please see attached.

+my $node = get_new_node();
+# Initialize node without replication settings
+$node->initNode(0);
+$node->startNode();
+my $pgdata = $node->getDataDir();
+
+$ENV{PGPORT} = $node->getPort();

Starting a value retrieval method name with "get" is not Perlish. The TAP
suites currently follow "man perlstyle" in using underscored_lower_case method
names. No PostgreSQL Perl code uses lowerFirstCamelCase, though some uses
CamelCase. The word "Node" is redundant. Use this style:

$node->init(0);
$node->start;
my $pgdata = $node->data_dir;
$ENV{PGPORT} = $node->port;

I have switched the style this way.

As a matter of opinion, I recommend giving "init" key/value arguments instead
of the single Boolean argument. The method could easily need more options in
the future, and this makes the call site self-documenting:

$node->init(hba_permit_replication => 0);

Done.

-     'pg_controldata with nonexistent directory fails');
+                       'pg_controldata with nonexistent directory fails');

perltidy will undo this whitespace-only change.

Cleaned up.

--- a/src/bin/pg_rewind/t/001_basic.pl
+++ b/src/bin/pg_rewind/t/001_basic.pl
@@ -1,9 +1,11 @@
+# Basic pg_rewind test.
+
use strict;
use warnings;
-use TestLib;
-use Test::More tests => 8;
use RewindTest;
+use TestLib;
+use Test::More tests => 8;

Revert all changes to this file. Audit the rest of the patch for whitespace
change unrelated to the subject.

Done.

-     'fails with nonexistent table');
+                       'fails with nonexistent table');
-'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';
+     'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';

perltidy will undo these whitespace-only changes.

Cleaned up.

+# cluster -a is not compatible with -d, hence enforce environment variables

s/cluster -a/clusterdb -a/

Fixed.

-command_fails([ 'createuser', 'user1' ], 'fails if role already exists');
+command_fails([ 'createuser', 'user1' ],
+                       'fails if role already exists');

perltidy will undo this whitespace-only change.

@@ -0,0 +1,252 @@
+# PostgresNode, simple node representation for regression tests.
+#
+# Regression tests should use this basic class infrastructure to define nodes
+# that need used in the test modules/scripts.
+package PostgresNode;

Consider just saying, "Class representing a data directory and postmaster."

OK, I have changed this description:
+# PostgresNode, class representing a data directory and postmaster.
+#
+# This contains a basic set of routines able to work on a PostgreSQL node,
+# allowing to start, stop, backup and initialize it with various options.
+     my $self   = {
+             _port     => undef,
+             _host     => undef,
+             _basedir  => undef,
+             _applname => undef,
+             _logfile  => undef };
+
+     # Set up each field
+     $self->{_port}     = $pgport;
+     $self->{_host}     = $pghost;
+     $self->{_basedir}  = TestBase::tempdir;
+     $self->{_applname} = "node_$pgport";
+     $self->{_logfile}  = "$TestBase::log_path/node_$pgport.log";

Why set fields to undef immediately before filling them?

Fixed.

@@ -0,0 +1,143 @@
+# Set of low-level routines dedicated to base tasks for regression tests, like
+# command execution and logging.
+#
+# This module should not depend on any other PostgreSQL regression test
+# modules.
+package TestBase;

This is no mere set of routines. Just "use"-ing this module creates some
directories and alters stdin/stdout/stderr.

I have updated the description of this file.

+BEGIN
+{
+     $windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
+
+     # Determine output directories, and create them.  The base path is the
+     # TESTDIR environment variable, which is normally set by the invoking
+     # Makefile.
+     $tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
+     $log_path = "$tmp_check/log";
+
+     mkdir $tmp_check;
+     mkdir $log_path;

Never mutate the filesystem in a BEGIN block, because "perl -c" runs BEGIN
blocks. (Likewise for the BEGIN block this patch adds to TestLib.)

Hm. It seems to me that the whole block should be part of INIT then,
because the log file where STDERR and STDOUT is recaptured depends on
those to be created as well. By doing this change, please note that
compilation errors are not recaptured into the log file (thanks Andrew
for the pointers to perlmod).

I have as well updated pg_rewind tests to remove PGDATABASE. Patch to
address those issues is attached.
Regards,
--
Michael

Attachments:

20151130_tapcheck.patchtext/x-patch; charset=US-ASCII; name=20151130_tapcheck.patchDownload
diff --git a/src/bin/initdb/t/001_initdb.pl b/src/bin/initdb/t/001_initdb.pl
index 299dcf5..3b5d7af 100644
--- a/src/bin/initdb/t/001_initdb.pl
+++ b/src/bin/initdb/t/001_initdb.pl
@@ -4,10 +4,11 @@
 
 use strict;
 use warnings;
+use TestBase;
 use TestLib;
 use Test::More tests => 14;
 
-my $tempdir = TestLib::tempdir;
+my $tempdir = TestBase::tempdir;
 my $xlogdir = "$tempdir/pgxlog";
 my $datadir = "$tempdir/data";
 
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index dc96bbf..c4e1d16 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -2,6 +2,8 @@ use strict;
 use warnings;
 use Cwd;
 use Config;
+use PostgresNode;
+use TestBase;
 use TestLib;
 use Test::More tests => 51;
 
@@ -9,8 +11,15 @@ program_help_ok('pg_basebackup');
 program_version_ok('pg_basebackup');
 program_options_handling_ok('pg_basebackup');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $tempdir = TestBase::tempdir;
+
+my $node = get_new_node();
+# Initialize node without replication settings
+$node->init(hba_permit_replication => 0);
+$node->start;
+my $pgdata = $node->data_dir;
+
+$ENV{PGPORT} = $node->port;
 
 command_fails(['pg_basebackup'],
 	'pg_basebackup needs target directory specified');
@@ -26,19 +35,19 @@ if (open BADCHARS, ">>$tempdir/pgdata/FOO\xe0\xe0\xe0BAR")
 	close BADCHARS;
 }
 
-configure_hba_for_replication "$tempdir/pgdata";
-system_or_bail 'pg_ctl', '-D', "$tempdir/pgdata", 'reload';
+$node->set_replication_conf();
+system_or_bail 'pg_ctl', '-D', $pgdata, 'reload';
 
 command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup fails because of WAL configuration');
 
-open CONF, ">>$tempdir/pgdata/postgresql.conf";
+open CONF, ">>$pgdata/postgresql.conf";
 print CONF "max_replication_slots = 10\n";
 print CONF "max_wal_senders = 10\n";
 print CONF "wal_level = archive\n";
 close CONF;
-restart_test_server;
+$node->restart;
 
 command_ok([ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup runs');
@@ -81,13 +90,13 @@ command_fails(
 
 # Tar format doesn't support filenames longer than 100 bytes.
 my $superlongname = "superlongname_" . ("x" x 100);
-my $superlongpath = "$tempdir/pgdata/$superlongname";
+my $superlongpath = "$pgdata/$superlongname";
 
 open FILE, ">$superlongpath" or die "unable to create file $superlongpath";
 close FILE;
 command_fails([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l1", '-Ft' ],
 	'pg_basebackup tar with long name fails');
-unlink "$tempdir/pgdata/$superlongname";
+unlink "$pgdata/$superlongname";
 
 # The following tests test symlinks. Windows doesn't have symlinks, so
 # skip on Windows.
@@ -98,7 +107,7 @@ SKIP: {
 	# to our physical temp location.  That way we can use shorter names
 	# for the tablespace directories, which hopefully won't run afoul of
 	# the 99 character length limit.
-	my $shorter_tempdir = tempdir_short . "/tempdir";
+	my $shorter_tempdir = TestBase::tempdir_short . "/tempdir";
 	symlink "$tempdir", $shorter_tempdir;
 
 	mkdir "$tempdir/tblspc1";
@@ -120,7 +129,7 @@ SKIP: {
 			"-T$shorter_tempdir/tblspc1=$tempdir/tbackup/tblspc1" ],
 		'plain format with tablespaces succeeds with tablespace mapping');
 	ok(-d "$tempdir/tbackup/tblspc1", 'tablespace was relocated');
-	opendir(my $dh, "$tempdir/pgdata/pg_tblspc") or die;
+	opendir(my $dh, "$pgdata/pg_tblspc") or die;
 	ok( (   grep {
 		-l "$tempdir/backup1/pg_tblspc/$_"
 			and readlink "$tempdir/backup1/pg_tblspc/$_" eq
diff --git a/src/bin/pg_controldata/t/001_pg_controldata.pl b/src/bin/pg_controldata/t/001_pg_controldata.pl
index e2b0d42..ae45f41 100644
--- a/src/bin/pg_controldata/t/001_pg_controldata.pl
+++ b/src/bin/pg_controldata/t/001_pg_controldata.pl
@@ -1,16 +1,19 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
-my $tempdir = TestLib::tempdir;
-
 program_help_ok('pg_controldata');
 program_version_ok('pg_controldata');
 program_options_handling_ok('pg_controldata');
 command_fails(['pg_controldata'], 'pg_controldata without arguments fails');
 command_fails([ 'pg_controldata', 'nonexistent' ],
 	'pg_controldata with nonexistent directory fails');
-standard_initdb "$tempdir/data";
-command_like([ 'pg_controldata', "$tempdir/data" ],
+
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+command_like([ 'pg_controldata', $node->data_dir ],
 	qr/checkpoint/, 'pg_controldata produces output');
diff --git a/src/bin/pg_ctl/t/001_start_stop.pl b/src/bin/pg_ctl/t/001_start_stop.pl
index f57abce..d76fe80 100644
--- a/src/bin/pg_ctl/t/001_start_stop.pl
+++ b/src/bin/pg_ctl/t/001_start_stop.pl
@@ -1,11 +1,12 @@
 use strict;
 use warnings;
 use Config;
+use TestBase;
 use TestLib;
 use Test::More tests => 17;
 
-my $tempdir       = TestLib::tempdir;
-my $tempdir_short = TestLib::tempdir_short;
+my $tempdir       = TestBase::tempdir;
+my $tempdir_short = TestBase::tempdir_short;
 
 program_help_ok('pg_ctl');
 program_version_ok('pg_ctl');
diff --git a/src/bin/pg_ctl/t/002_status.pl b/src/bin/pg_ctl/t/002_status.pl
index 31f7c72..2a9e0a5 100644
--- a/src/bin/pg_ctl/t/002_status.pl
+++ b/src/bin/pg_ctl/t/002_status.pl
@@ -1,22 +1,25 @@
 use strict;
 use warnings;
+use PostgresNode;
+use TestBase;
 use TestLib;
 use Test::More tests => 3;
 
-my $tempdir       = TestLib::tempdir;
-my $tempdir_short = TestLib::tempdir_short;
+my $tempdir       = TestBase::tempdir;
+my $tempdir_short = TestBase::tempdir_short;
 
 command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/nonexistent" ],
 	4, 'pg_ctl status with nonexistent directory');
 
-standard_initdb "$tempdir/data";
+my $node = get_new_node();
+$node->init;
 
-command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/data" ],
+command_exit_is([ 'pg_ctl', 'status', '-D', $node->data_dir ],
 	3, 'pg_ctl status with server not running');
 
 system_or_bail 'pg_ctl', '-l', "$tempdir/logfile", '-D',
-  "$tempdir/data", '-w', 'start';
-command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/data" ],
+  $node->data_dir, '-w', 'start';
+command_exit_is([ 'pg_ctl', 'status', '-D', $node->data_dir ],
 	0, 'pg_ctl status with server running');
 
-system_or_bail 'pg_ctl', 'stop', '-D', "$tempdir/data", '-m', 'fast';
+system_or_bail 'pg_ctl', 'stop', '-D', $node->data_dir, '-m', 'fast';
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..24fd19c 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -9,22 +9,20 @@ package RewindTest;
 # To run a test, the test script (in t/ subdirectory) calls the functions
 # in this module. These functions should be called in this sequence:
 #
-# 1. init_rewind_test - sets up log file etc.
+# 1. setup_cluster - creates a PostgreSQL cluster that runs as the master
 #
-# 2. setup_cluster - creates a PostgreSQL cluster that runs as the master
+# 2. start_master - starts the master server
 #
-# 3. start_master - starts the master server
-#
-# 4. create_standby - runs pg_basebackup to initialize a standby server, and
+# 3. create_standby - runs pg_basebackup to initialize a standby server, and
 #    sets it up to follow the master.
 #
-# 5. promote_standby - runs "pg_ctl promote" to promote the standby server.
+# 4. promote_standby - runs "pg_ctl promote" to promote the standby server.
 # The old master keeps running.
 #
-# 6. run_pg_rewind - stops the old master (if it's still running) and runs
+# 5. run_pg_rewind - stops the old master (if it's still running) and runs
 # pg_rewind to synchronize it with the now-promoted standby server.
 #
-# 7. clean_rewind_test - stops both servers used in the test, if they're
+# 6. clean_rewind_test - stops both servers used in the test, if they're
 # still running.
 #
 # The test script can use the helper functions master_psql and standby_psql
@@ -37,27 +35,23 @@ package RewindTest;
 use strict;
 use warnings;
 
-use TestLib;
-use Test::More;
-
 use Config;
+use Exporter 'import';
 use File::Copy;
 use File::Path qw(rmtree);
 use IPC::Run qw(run start);
+use TestBase;
+use TestLib;
+use Test::More;
 
-use Exporter 'import';
 our @EXPORT = qw(
-  $connstr_master
-  $connstr_standby
-  $test_master_datadir
-  $test_standby_datadir
+  $node_master
+  $node_standby
 
-  append_to_file
   master_psql
   standby_psql
   check_query
 
-  init_rewind_test
   setup_cluster
   start_master
   create_standby
@@ -66,32 +60,24 @@ our @EXPORT = qw(
   clean_rewind_test
 );
 
-our $test_master_datadir  = "$tmp_check/data_master";
-our $test_standby_datadir = "$tmp_check/data_standby";
-
-# Define non-conflicting ports for both nodes.
-my $port_master  = $ENV{PGPORT};
-my $port_standby = $port_master + 1;
-
-my $connstr_master  = "port=$port_master";
-my $connstr_standby = "port=$port_standby";
-
-$ENV{PGDATABASE} = "postgres";
+# Our nodes.
+our $node_master;
+our $node_standby;
 
 sub master_psql
 {
 	my $cmd = shift;
 
-	system_or_bail 'psql', '-q', '--no-psqlrc', '-d', $connstr_master,
-	  '-c', "$cmd";
+	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
+	  $node_master->connstr('postgres'), '-c', "$cmd";
 }
 
 sub standby_psql
 {
 	my $cmd = shift;
 
-	system_or_bail 'psql', '-q', '--no-psqlrc', '-d', $connstr_standby,
-	  '-c', "$cmd";
+	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
+      $node_standby->connstr('postgres'), '-c', "$cmd";
 }
 
 # Run a query against the master, and check that the output matches what's
@@ -104,7 +90,7 @@ sub check_query
 	# we want just the output, no formatting
 	my $result = run [
 		'psql',          '-q', '-A', '-t', '--no-psqlrc', '-d',
-		$connstr_master, '-c', $query ],
+		$node_master->connstr('postgres'), '-c', $query ],
 	  '>', \$stdout, '2>', \$stderr;
 
 	# We don't use ok() for the exit code and stderr, because we want this
@@ -125,56 +111,14 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
-sub append_to_file
-{
-	my ($filename, $str) = @_;
-
-	open my $fh, ">>", $filename or die "could not open file $filename";
-	print $fh $str;
-	close $fh;
-}
-
 sub setup_cluster
 {
 	# Initialize master, data checksums are mandatory
-	rmtree($test_master_datadir);
-	standard_initdb($test_master_datadir);
+	$node_master = get_new_node();
+	$node_master->init;
 
 	# Custom parameters for master's postgresql.conf
-	append_to_file(
-		"$test_master_datadir/postgresql.conf", qq(
+	$node_master->append_conf("postgresql.conf", qq(
 wal_level = hot_standby
 max_wal_senders = 2
 wal_keep_segments = 20
@@ -185,17 +129,11 @@ hot_standby = on
 autovacuum = off
 max_connections = 10
 ));
-
-	# Accept replication connections on master
-	configure_hba_for_replication $test_master_datadir;
 }
 
 sub start_master
 {
-	system_or_bail('pg_ctl' , '-w',
-				   '-D' , $test_master_datadir,
-				   '-l',  "$log_path/master.log",
-				   "-o", "-p $port_master", 'start');
+	$node_master->start;
 
 	#### Now run the test-specific parts to initialize the master before setting
 	# up standby
@@ -203,24 +141,19 @@ sub start_master
 
 sub create_standby
 {
+	$node_standby = get_new_node();
+	$node_master->backup('my_backup');
+	$node_standby->init_from_backup($node_master, 'my_backup');
+	my $connstr_master = $node_master->connstr('postgres');
 
-	# Set up standby with necessary parameter
-	rmtree $test_standby_datadir;
-
-	# Base backup is taken with xlog files included
-	system_or_bail('pg_basebackup', '-D', $test_standby_datadir,
-				   '-p', $port_master, '-x');
-	append_to_file(
-		"$test_standby_datadir/recovery.conf", qq(
+	$node_standby->append_conf("recovery.conf", qq(
 primary_conninfo='$connstr_master application_name=rewind_standby'
 standby_mode=on
 recovery_target_timeline='latest'
 ));
 
 	# Start standby
-	system_or_bail('pg_ctl', '-w', '-D', $test_standby_datadir,
-				   '-l', "$log_path/standby.log",
-				   '-o', "-p $port_standby", 'start');
+	$node_standby->start;
 
 	# The standby may have WAL to apply before it matches the primary.  That
 	# is fine, because no test examines the standby before promotion.
@@ -234,14 +167,14 @@ sub promote_standby
 	# Wait for the standby to receive and write all WAL.
 	my $wal_received_query =
 "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = 'rewind_standby';";
-	poll_query_until($wal_received_query, $connstr_master)
+	poll_query_until($node_master, $wal_received_query)
 	  or die "Timed out while waiting for standby to receive and write WAL";
 
 	# Now promote slave and insert some new data on master, this will put
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
-	system_or_bail('pg_ctl', '-w', '-D', $test_standby_datadir, 'promote');
-	poll_query_until("SELECT NOT pg_is_in_recovery()", $connstr_standby)
+	system_or_bail('pg_ctl', '-w', '-D', $node_standby->data_dir, 'promote');
+	poll_query_until($node_standby, "SELECT NOT pg_is_in_recovery()")
 	  or die "Timed out while waiting for promotion of standby";
 
 	# Force a checkpoint after the promotion. pg_rewind looks at the control
@@ -256,9 +189,13 @@ sub promote_standby
 sub run_pg_rewind
 {
 	my $test_mode = shift;
+	my $master_pgdata = $node_master->data_dir;
+	my $standby_pgdata = $node_standby->data_dir;
+	my $standby_connstr = $node_standby->connstr('postgres');
+	my $tmp_folder = TestBase::tempdir;
 
 	# Stop the master and be ready to perform the rewind
-	system_or_bail('pg_ctl', '-D', $test_master_datadir, '-m', 'fast', 'stop');
+	$node_master->stop;
 
 	# At this point, the rewind processing is ready to run.
 	# We now have a very simple scenario with a few diverged WAL record.
@@ -267,20 +204,19 @@ sub run_pg_rewind
 
 	# Keep a temporary postgresql.conf for master node or it would be
 	# overwritten during the rewind.
-	copy("$test_master_datadir/postgresql.conf",
-		 "$tmp_check/master-postgresql.conf.tmp");
+	copy("$master_pgdata/postgresql.conf",
+		 "$tmp_folder/master-postgresql.conf.tmp");
 
 	# Now run pg_rewind
 	if ($test_mode eq "local")
 	{
 		# Do rewind using a local pgdata as source
 		# Stop the master and be ready to perform the rewind
-		system_or_bail('pg_ctl', '-D', $test_standby_datadir,
-					   '-m', 'fast', 'stop');
+		$node_standby->stop;
 		command_ok(['pg_rewind',
 					"--debug",
-					"--source-pgdata=$test_standby_datadir",
-					"--target-pgdata=$test_master_datadir"],
+					"--source-pgdata=$standby_pgdata",
+					"--target-pgdata=$master_pgdata"],
 				   'pg_rewind local');
 	}
 	elsif ($test_mode eq "remote")
@@ -289,33 +225,30 @@ sub run_pg_rewind
 		command_ok(['pg_rewind',
 					"--debug",
 					"--source-server",
-					"port=$port_standby dbname=postgres",
-					"--target-pgdata=$test_master_datadir"],
+					$standby_connstr,
+					"--target-pgdata=$master_pgdata"],
 				   'pg_rewind remote');
 	}
 	else
 	{
-
 		# Cannot come here normally
 		die("Incorrect test mode specified");
 	}
 
 	# Now move back postgresql.conf with old settings
-	move("$tmp_check/master-postgresql.conf.tmp",
-		 "$test_master_datadir/postgresql.conf");
+	move("$tmp_folder/master-postgresql.conf.tmp",
+		 "$master_pgdata/postgresql.conf");
 
 	# Plug-in rewound node to the now-promoted standby node
-	append_to_file(
-		"$test_master_datadir/recovery.conf", qq(
+	my $port_standby = $node_standby->port;
+	$node_master->append_conf('recovery.conf', qq(
 primary_conninfo='port=$port_standby'
 standby_mode=on
 recovery_target_timeline='latest'
 ));
 
 	# Restart the master to check that rewind went correctly
-	system_or_bail('pg_ctl', '-w', '-D', $test_master_datadir,
-				   '-l', "$log_path/master.log",
-				   '-o', "-p $port_master", 'start');
+	$node_master->restart;
 
 	#### Now run the test-specific parts to check the result
 }
@@ -323,22 +256,8 @@ recovery_target_timeline='latest'
 # Clean up after the test. Stop both servers, if they're still running.
 sub clean_rewind_test
 {
-	if ($test_master_datadir)
-	{
-		system
-		  'pg_ctl', '-D', $test_master_datadir, '-m', 'immediate', 'stop';
-	}
-	if ($test_standby_datadir)
-	{
-		system
-		  'pg_ctl', '-D', $test_standby_datadir, '-m', 'immediate', 'stop';
-	}
+	teardown_node($node_master) if (defined($node_master));
+	teardown_node($node_standby) if (defined($node_standby));
 }
 
-# Stop the test servers, just in case they're still running.
-END
-{
-	my $save_rc = $?;
-	clean_rewind_test();
-	$? = $save_rc;
-}
+1;
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index d317f53..d196367 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -2,14 +2,13 @@
 
 use strict;
 use warnings;
+use TestBase;
 use TestLib;
 use Test::More tests => 4;
 
 use File::Find;
-
 use RewindTest;
 
-
 sub run_test
 {
 	my $test_mode = shift;
@@ -17,7 +16,7 @@ sub run_test
 	RewindTest::setup_cluster();
 	RewindTest::start_master();
 
-	my $test_master_datadir = $RewindTest::test_master_datadir;
+	my $test_master_datadir = $node_master->data_dir;
 
 	# Create a subdir and files that will be present in both
 	mkdir "$test_master_datadir/tst_both_dir";
@@ -30,6 +29,7 @@ sub run_test
 	RewindTest::create_standby();
 
 	# Create different subdirs and files in master and standby
+	my $test_standby_datadir = $node_standby->data_dir;
 
 	mkdir "$test_standby_datadir/tst_standby_dir";
 	append_to_file "$test_standby_datadir/tst_standby_dir/standby_file1",
diff --git a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
index c5f72e2..e05f55e 100644
--- a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
+++ b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
@@ -5,7 +5,7 @@ use strict;
 use warnings;
 use File::Copy;
 use File::Path qw(rmtree);
-use TestLib;
+use TestBase;
 use Test::More;
 if ($windows_os)
 {
@@ -23,11 +23,13 @@ sub run_test
 {
 	my $test_mode = shift;
 
-	my $master_xlogdir = "$tmp_check/xlog_master";
+	my $master_xlogdir = "${TestBase::tmp_check}/xlog_master";
 
 	rmtree($master_xlogdir);
 	RewindTest::setup_cluster();
 
+	my $test_master_datadir = $node_master->data_dir;
+
 	# turn pg_xlog into a symlink
 	print("moving $test_master_datadir/pg_xlog to $master_xlogdir\n");
 	move("$test_master_datadir/pg_xlog", $master_xlogdir) or die;
diff --git a/src/bin/scripts/t/010_clusterdb.pl b/src/bin/scripts/t/010_clusterdb.pl
index dc0d78a..1db7fdb 100644
--- a/src/bin/scripts/t/010_clusterdb.pl
+++ b/src/bin/scripts/t/010_clusterdb.pl
@@ -7,20 +7,26 @@ program_help_ok('clusterdb');
 program_version_ok('clusterdb');
 program_options_handling_ok('clusterdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
+$ENV{PGDATABASE} = 'postgres';
 
 issues_sql_like(
-	[ 'clusterdb', 'postgres' ],
+	$node,
+	[ 'clusterdb' ],
 	qr/statement: CLUSTER;/,
 	'SQL CLUSTER run');
 
-command_fails([ 'clusterdb', '-t', 'nonexistent', 'postgres' ],
-	'fails with nonexistent table');
+command_fails([ 'clusterdb', '-t', 'nonexistent' ],
+			  'fails with nonexistent table');
 
 psql 'postgres',
-'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';
+	'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';
 issues_sql_like(
-	[ 'clusterdb', '-t', 'test1', 'postgres' ],
+	$node,
+	[ 'clusterdb', '-t', 'test1' ],
 	qr/statement: CLUSTER test1;/,
 	'cluster specific table');
diff --git a/src/bin/scripts/t/011_clusterdb_all.pl b/src/bin/scripts/t/011_clusterdb_all.pl
index 7769f70..22095ae 100644
--- a/src/bin/scripts/t/011_clusterdb_all.pl
+++ b/src/bin/scripts/t/011_clusterdb_all.pl
@@ -3,10 +3,17 @@ use warnings;
 use TestLib;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+# clusterdb -a is not compatible with -d, hence enforce environment variables
+# correctly.
+$ENV{PGDATABASE} = 'postgres';
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'clusterdb', '-a' ],
 	qr/statement: CLUSTER.*statement: CLUSTER/s,
 	'cluster all databases');
diff --git a/src/bin/scripts/t/020_createdb.pl b/src/bin/scripts/t/020_createdb.pl
index a44283c..01088ed 100644
--- a/src/bin/scripts/t/020_createdb.pl
+++ b/src/bin/scripts/t/020_createdb.pl
@@ -7,14 +7,19 @@ program_help_ok('createdb');
 program_version_ok('createdb');
 program_options_handling_ok('createdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'createdb', 'foobar1' ],
 	qr/statement: CREATE DATABASE foobar1/,
 	'SQL CREATE DATABASE run');
 issues_sql_like(
+	$node,
 	[ 'createdb', '-l', 'C', '-E', 'LATIN1', '-T', 'template0', 'foobar2' ],
 	qr/statement: CREATE DATABASE foobar2 ENCODING 'LATIN1'/,
 	'create database with encoding');
diff --git a/src/bin/scripts/t/030_createlang.pl b/src/bin/scripts/t/030_createlang.pl
index 7ff0a3e..9cbe781 100644
--- a/src/bin/scripts/t/030_createlang.pl
+++ b/src/bin/scripts/t/030_createlang.pl
@@ -7,18 +7,23 @@ program_help_ok('createlang');
 program_version_ok('createlang');
 program_options_handling_ok('createlang');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
+$ENV{PGDATABASE} = 'postgres';
 
 command_fails(
-	[ 'createlang', 'plpgsql', 'postgres' ],
+	[ 'createlang', 'plpgsql' ],
 	'fails if language already exists');
 
-psql 'postgres', 'DROP EXTENSION plpgsql';
+psql $node->connstr('postgres'), 'DROP EXTENSION plpgsql';
 issues_sql_like(
-	[ 'createlang', 'plpgsql', 'postgres' ],
+	$node,
+	[ 'createlang', 'plpgsql' ],
 	qr/statement: CREATE EXTENSION "plpgsql"/,
 	'SQL CREATE EXTENSION run');
 
-command_like([ 'createlang', '--list', 'postgres' ],
+command_like([ 'createlang', '--list' ],
 	qr/plpgsql/, 'list output');
diff --git a/src/bin/scripts/t/040_createuser.pl b/src/bin/scripts/t/040_createuser.pl
index 4d44e14..aa93247 100644
--- a/src/bin/scripts/t/040_createuser.pl
+++ b/src/bin/scripts/t/040_createuser.pl
@@ -7,22 +7,30 @@ program_help_ok('createuser');
 program_version_ok('createuser');
 program_options_handling_ok('createuser');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGDATABASE} = 'postgres';
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'createuser', 'user1' ],
 qr/statement: CREATE ROLE user1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN;/,
 	'SQL CREATE USER run');
 issues_sql_like(
+	$node,
 	[ 'createuser', '-L', 'role1' ],
 qr/statement: CREATE ROLE role1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT NOLOGIN;/,
 	'create a non-login role');
 issues_sql_like(
+	$node,
 	[ 'createuser', '-r', 'user2' ],
 qr/statement: CREATE ROLE user2 NOSUPERUSER NOCREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a CREATEROLE user');
 issues_sql_like(
+	$node,
 	[ 'createuser', '-s', 'user3' ],
 qr/statement: CREATE ROLE user3 SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a superuser');
diff --git a/src/bin/scripts/t/050_dropdb.pl b/src/bin/scripts/t/050_dropdb.pl
index 3065e50..c59319b 100644
--- a/src/bin/scripts/t/050_dropdb.pl
+++ b/src/bin/scripts/t/050_dropdb.pl
@@ -7,11 +7,15 @@ program_help_ok('dropdb');
 program_version_ok('dropdb');
 program_options_handling_ok('dropdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-psql 'postgres', 'CREATE DATABASE foobar1';
+$ENV{PGPORT} = $node->port;
+
+psql $node->connstr('postgres'), 'CREATE DATABASE foobar1';
 issues_sql_like(
+	$node,
 	[ 'dropdb', 'foobar1' ],
 	qr/statement: DROP DATABASE foobar1/,
 	'SQL DROP DATABASE run');
diff --git a/src/bin/scripts/t/060_droplang.pl b/src/bin/scripts/t/060_droplang.pl
index 6a21d7e..b59ac93 100644
--- a/src/bin/scripts/t/060_droplang.pl
+++ b/src/bin/scripts/t/060_droplang.pl
@@ -7,10 +7,14 @@ program_help_ok('droplang');
 program_version_ok('droplang');
 program_options_handling_ok('droplang');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'droplang', 'plpgsql', 'postgres' ],
 	qr/statement: DROP EXTENSION "plpgsql"/,
 	'SQL DROP EXTENSION run');
diff --git a/src/bin/scripts/t/070_dropuser.pl b/src/bin/scripts/t/070_dropuser.pl
index bbb3b79..9331f3e 100644
--- a/src/bin/scripts/t/070_dropuser.pl
+++ b/src/bin/scripts/t/070_dropuser.pl
@@ -7,11 +7,15 @@ program_help_ok('dropuser');
 program_version_ok('dropuser');
 program_options_handling_ok('dropuser');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 psql 'postgres', 'CREATE ROLE foobar1';
 issues_sql_like(
+	$node,
 	[ 'dropuser', 'foobar1' ],
 	qr/statement: DROP ROLE foobar1/,
 	'SQL DROP ROLE run');
diff --git a/src/bin/scripts/t/080_pg_isready.pl b/src/bin/scripts/t/080_pg_isready.pl
index f432505..73429f4 100644
--- a/src/bin/scripts/t/080_pg_isready.pl
+++ b/src/bin/scripts/t/080_pg_isready.pl
@@ -9,7 +9,10 @@ program_options_handling_ok('pg_isready');
 
 command_fails(['pg_isready'], 'fails with no server running');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 command_ok(['pg_isready'], 'succeeds with server running');
diff --git a/src/bin/scripts/t/090_reindexdb.pl b/src/bin/scripts/t/090_reindexdb.pl
index 42628c2..2d9fee1 100644
--- a/src/bin/scripts/t/090_reindexdb.pl
+++ b/src/bin/scripts/t/090_reindexdb.pl
@@ -7,12 +7,15 @@ program_help_ok('reindexdb');
 program_version_ok('reindexdb');
 program_options_handling_ok('reindexdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
+$ENV{PGPORT} = $node->port;
 $ENV{PGOPTIONS} = '--client-min-messages=WARNING';
 
 issues_sql_like(
+	$node,
 	[ 'reindexdb', 'postgres' ],
 	qr/statement: REINDEX DATABASE postgres;/,
 	'SQL REINDEX run');
@@ -20,22 +23,27 @@ issues_sql_like(
 psql 'postgres',
   'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a);';
 issues_sql_like(
+	$node,
 	[ 'reindexdb', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX TABLE test1;/,
 	'reindex specific table');
 issues_sql_like(
+	$node,
 	[ 'reindexdb', '-i', 'test1x', 'postgres' ],
 	qr/statement: REINDEX INDEX test1x;/,
 	'reindex specific index');
 issues_sql_like(
+	$node,
 	[ 'reindexdb', '-S', 'pg_catalog', 'postgres' ],
 	qr/statement: REINDEX SCHEMA pg_catalog;/,
 	'reindex specific schema');
 issues_sql_like(
+	$node,
 	[ 'reindexdb', '-s', 'postgres' ],
 	qr/statement: REINDEX SYSTEM postgres;/,
 	'reindex system tables');
 issues_sql_like(
+	$node,
 	[ 'reindexdb', '-v', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX \(VERBOSE\) TABLE test1;/,
 	'reindex with verbose output');
diff --git a/src/bin/scripts/t/091_reindexdb_all.pl b/src/bin/scripts/t/091_reindexdb_all.pl
index ffadf29..f6643d5 100644
--- a/src/bin/scripts/t/091_reindexdb_all.pl
+++ b/src/bin/scripts/t/091_reindexdb_all.pl
@@ -3,12 +3,15 @@ use warnings;
 use TestLib;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
+$ENV{PGPORT} = $node->port;
 $ENV{PGOPTIONS} = '--client-min-messages=WARNING';
 
 issues_sql_like(
+	$node,
 	[ 'reindexdb', '-a' ],
 	qr/statement: REINDEX.*statement: REINDEX/s,
 	'reindex all databases');
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index ac160ba..6329b34 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -7,26 +7,34 @@ program_help_ok('vacuumdb');
 program_version_ok('vacuumdb');
 program_options_handling_ok('vacuumdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', 'postgres' ],
 	qr/statement: VACUUM;/,
 	'SQL VACUUM run');
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '-f', 'postgres' ],
 	qr/statement: VACUUM \(FULL\);/,
 	'vacuumdb -f');
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '-F', 'postgres' ],
 	qr/statement: VACUUM \(FREEZE\);/,
 	'vacuumdb -F');
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '-z', 'postgres' ],
 	qr/statement: VACUUM \(ANALYZE\);/,
 	'vacuumdb -z');
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '-Z', 'postgres' ],
 	qr/statement: ANALYZE;/,
 	'vacuumdb -Z');
diff --git a/src/bin/scripts/t/101_vacuumdb_all.pl b/src/bin/scripts/t/101_vacuumdb_all.pl
index e90f321..cde6b3c 100644
--- a/src/bin/scripts/t/101_vacuumdb_all.pl
+++ b/src/bin/scripts/t/101_vacuumdb_all.pl
@@ -3,10 +3,14 @@ use warnings;
 use TestLib;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '-a' ],
 	qr/statement: VACUUM.*statement: VACUUM/s,
 	'vacuum all databases');
diff --git a/src/bin/scripts/t/102_vacuumdb_stages.pl b/src/bin/scripts/t/102_vacuumdb_stages.pl
index 57b980e..c7ba021 100644
--- a/src/bin/scripts/t/102_vacuumdb_stages.pl
+++ b/src/bin/scripts/t/102_vacuumdb_stages.pl
@@ -3,10 +3,14 @@ use warnings;
 use TestLib;
 use Test::More tests => 4;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '--analyze-in-stages', 'postgres' ],
 qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE.*
@@ -18,6 +22,7 @@ qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
 
 
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '--analyze-in-stages', '--all' ],
 qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE.*
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
new file mode 100644
index 0000000..e1d5353
--- /dev/null
+++ b/src/test/perl/PostgresNode.pm
@@ -0,0 +1,251 @@
+# PostgresNode, class representing a data directory and postmaster.
+#
+# This contains a basic set of routines able to work on a PostgreSQL node,
+# allowing to start, stop, backup and initialize it with various options.
+
+package PostgresNode;
+
+use strict;
+use warnings;
+
+use RecursiveCopy;
+use TestBase;
+use Test::More;
+
+sub new
+{
+	my $class  = shift;
+	my $pghost = shift;
+	my $pgport = shift;
+	my $self   = {
+		_port     => $pgport,
+		_host     => $pghost,
+		_basedir  => TestBase::tempdir,
+		_applname => "node_$pgport",
+		_logfile  => "$TestBase::log_path/node_$pgport.log" };
+
+	bless $self, $class;
+	$self->dump_info;
+
+	return $self;
+}
+
+sub port
+{
+	my ($self) = @_;
+	return $self->{_port};
+}
+
+sub host
+{
+	my ($self) = @_;
+	return $self->{_host};
+}
+
+sub basedir
+{
+	my ($self) = @_;
+	return $self->{_basedir};
+}
+
+sub applname
+{
+	my ($self) = @_;
+	return $self->{_applname};
+}
+
+sub logfile
+{
+	my ($self) = @_;
+	return $self->{_logfile};
+}
+
+sub connstr
+{
+	my ($self, $dbname) = @_;
+	my $pgport = $self->port;
+	my $pghost = $self->host;
+	if (!defined($dbname))
+	{
+		return "port=$pgport host=$pghost";
+	}
+	return "port=$pgport host=$pghost dbname=$dbname";
+}
+
+sub data_dir
+{
+	my ($self) = @_;
+	my $res = $self->basedir;
+	return "$res/pgdata";
+}
+
+sub archive_dir
+{
+	my ($self) = @_;
+	my $basedir = $self->basedir;
+	return "$basedir/archives";
+}
+
+sub backup_dir
+{
+	my ($self) = @_;
+	my $basedir = $self->basedir;
+	return "$basedir/backup";
+}
+
+# Dump node information
+sub dump_info
+{
+	my ($self) = @_;
+	print 'Data directory: ' . $self->data_dir . "\n";
+	print 'Backup directory: ' . $self->backup_dir . "\n";
+	print 'Archive directory: ' . $self->archive_dir . "\n";
+	print 'Connection string: ' . $self->connstr . "\n";
+	print 'Application name: ' . $self->applname . "\n";
+	print 'Log file: ' . $self->logfile . "\n";
+}
+
+sub set_replication_conf
+{
+	my ($self) = @_;
+	my $pgdata = $self->data_dir;
+
+	open my $hba, ">>$pgdata/pg_hba.conf";
+	print $hba "\n# Allow replication (set up by PostgresNode.pm)\n";
+	if (!$windows_os)
+	{
+		print $hba "local replication all trust\n";
+	}
+	else
+	{
+		print $hba
+"host replication all 127.0.0.1/32 sspi include_realm=1 map=regress\n";
+	}
+	close $hba;
+}
+
+# Initialize a new cluster for testing.
+#
+# Authentication is set up so that only the current OS user can access the
+# cluster. On Unix, we use Unix domain socket connections, with the socket in
+# a directory that's only accessible to the current user to ensure that.
+# On Windows, we use SSPI authentication to ensure the same (by pg_regress
+# --config-auth).
+sub init
+{
+	my ($self, %params) = @_;
+	my $port   = $self->port;
+	my $pgdata = $self->data_dir;
+	my $host   = $self->host;
+
+	$params{hba_permit_replication} = 1 if (!defined($params{hba_permit_replication}));
+
+	mkdir $self->backup_dir;
+	mkdir $self->archive_dir;
+
+	system_or_bail('initdb', '-D', $pgdata, '-A', 'trust', '-N');
+	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
+
+	open my $conf, ">>$pgdata/postgresql.conf";
+	print $conf "\n# Added by TestLib.pm)\n";
+	print $conf "fsync = off\n";
+	print $conf "log_statement = all\n";
+	print $conf "port = $port\n";
+	if ($windows_os)
+	{
+		print $conf "listen_addresses = '$host'\n";
+	}
+	else
+	{
+		print $conf "unix_socket_directories = '$host'\n";
+		print $conf "listen_addresses = ''\n";
+	}
+	close $conf;
+
+	$self->set_replication_conf if ($params{hba_permit_replication});
+}
+
+sub append_conf
+{
+	my ($self, $filename, $str) = @_;
+
+	my $conffile = $self->data_dir . '/' . $filename;
+
+	append_to_file($conffile, $str);
+}
+
+sub backup
+{
+	my ($self, $backup_name) = @_;
+	my $backup_path = $self->backup_dir . '/' . $backup_name;
+	my $port        = $self->port;
+
+	print "# Taking backup $backup_name from node with port $port\n";
+	system_or_bail("pg_basebackup -D $backup_path -p $port -x");
+	print "# Backup finished\n";
+}
+
+sub init_from_backup
+{
+	my ($self, $root_node, $backup_name) = @_;
+	my $backup_path = $root_node->backup_dir . '/' . $backup_name;
+	my $port        = $self->port;
+	my $root_port   = $root_node->port;
+
+	print
+"Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	die "Backup $backup_path does not exist" unless -d $backup_path;
+
+	mkdir $self->backup_dir;
+	mkdir $self->archive_dir;
+
+	my $data_path = $self->data_dir;
+	rmdir($data_path);
+	RecursiveCopy::copypath($backup_path, $data_path);
+	chmod(0700, $data_path);
+
+	# Base configuration for this node
+	$self->append_conf('postgresql.conf',
+		qq(
+port = $port
+));
+	$self->set_replication_conf;
+}
+
+sub start
+{
+	my ($self) = @_;
+	my $port   = $self->port;
+	my $pgdata = $self->data_dir;
+	print("### Starting test server in $pgdata\n");
+	my $ret = system_log('pg_ctl', '-w', '-D', $self->data_dir,
+		'-l', $self->logfile, 'start');
+
+	if ($ret != 0)
+	{
+		print "# pg_ctl failed; logfile:\n";
+		print slurp_file($self->logfile);
+		BAIL_OUT("pg_ctl failed");
+	}
+}
+
+sub stop
+{
+	my ($self, $mode) = @_;
+	my $port   = $self->port;
+	my $pgdata = $self->data_dir;
+	$mode = 'fast' if (!defined($mode));
+	print "### Stopping node in $pgdata with port $port using mode $mode\n";
+	system_log('pg_ctl', '-D', $pgdata, '-m', $mode, 'stop');
+}
+
+sub restart
+{
+	my ($self)  = @_;
+	my $port    = $self->port;
+	my $pgdata  = $self->data_dir;
+	my $logfile = $self->logfile;
+	system_log('pg_ctl', '-D', $pgdata, '-w', '-l', $logfile, 'restart');
+}
+
+1;
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
new file mode 100644
index 0000000..4e58ad3
--- /dev/null
+++ b/src/test/perl/RecursiveCopy.pm
@@ -0,0 +1,42 @@
+# RecursiveCopy, a simple recursive copy implementation
+package RecursiveCopy;
+
+use strict;
+use warnings;
+
+use File::Basename;
+use File::Copy;
+
+sub copypath
+{
+	my $srcpath  = shift;
+	my $destpath = shift;
+
+	die "Cannot operate on symlinks" if -l $srcpath or -l $destpath;
+
+	# This source path is a file, simply copy it to destination with the
+	# same name.
+	die "Destination path $destpath exists as file" if -f $destpath;
+	if (-f $srcpath)
+	{
+		copy($srcpath, $destpath)
+			or die "copy $srcpath -> $destpath failed: $!";
+		return 1;
+	}
+
+	die "Destination needs to be a directory" unless -d $srcpath;
+	mkdir($destpath) or die "mkdir($destpath) failed: $!";
+
+	# Scan existing source directory and recursively copy everything.
+	opendir(my $directory, $srcpath) or die "could not opendir($srcpath): $!";
+	while (my $entry = readdir($directory))
+	{
+		next if ($entry eq '.' || $entry eq '..');
+		RecursiveCopy::copypath("$srcpath/$entry", "$destpath/$entry")
+			or die "copypath $srcpath/$entry -> $destpath/$entry failed";
+	}
+	closedir($directory);
+	return 1;
+}
+
+1;
diff --git a/src/test/perl/TestBase.pm b/src/test/perl/TestBase.pm
new file mode 100644
index 0000000..73c25c1
--- /dev/null
+++ b/src/test/perl/TestBase.pm
@@ -0,0 +1,145 @@
+# testBase, low-level routines and actions regression tests.
+#
+# This module contains a set of routines dedicated to environment setup for
+# a PostgreSQL regression test sun, and includes some low-level routines
+# aimed at controlling command execution and logging. This module should
+# never depend on any other PostgreSQL regression test modules.
+
+package TestBase;
+
+use strict;
+use warnings;
+
+use Config;
+use Exporter 'import';
+use File::Basename;
+use File::Spec;
+use File::Temp ();
+use IPC::Run qw(run);
+use SimpleTee;
+use Test::More;
+
+our @EXPORT = qw(
+  system_or_bail
+  system_log
+  run_log
+  slurp_dir
+  slurp_file
+  append_to_file
+
+  $windows_os
+);
+
+our ($windows_os, $tmp_check, $log_path, $test_logfile);
+
+INIT
+{
+	$windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
+
+	# Determine output directories, and create them.  The base path is the
+	# TESTDIR environment variable, which is normally set by the invoking
+	# Makefile.
+	$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
+	$log_path = "$tmp_check/log";
+
+	mkdir $tmp_check;
+	mkdir $log_path;
+
+	# Open the test log file, whose name depends on the test name.
+	$test_logfile = basename($0);
+	$test_logfile =~ s/\.[^.]+$//;
+	$test_logfile = "$log_path/regress_log_$test_logfile";
+	open TESTLOG, '>', $test_logfile
+	  or die "could not open STDOUT to logfile \"$test_logfile\": $!";
+
+	# Hijack STDOUT and STDERR to the log file
+	open(ORIG_STDOUT, ">&STDOUT");
+	open(ORIG_STDERR, ">&STDERR");
+	open(STDOUT,      ">&TESTLOG");
+	open(STDERR,      ">&TESTLOG");
+
+	# The test output (ok ...) needs to be printed to the original STDOUT so
+	# that the 'prove' program can parse it, and display it to the user in
+	# real time. But also copy it to the log file, to provide more context
+	# in the log.
+	my $builder = Test::More->builder;
+	my $fh      = $builder->output;
+	tie *$fh, "SimpleTee", *ORIG_STDOUT, *TESTLOG;
+	$fh = $builder->failure_output;
+	tie *$fh, "SimpleTee", *ORIG_STDERR, *TESTLOG;
+
+	# Enable auto-flushing for all the file handles. Stderr and stdout are
+	# redirected to the same file, and buffering causes the lines to appear
+	# in the log in confusing order.
+	autoflush STDOUT 1;
+	autoflush STDERR 1;
+	autoflush TESTLOG 1;
+}
+
+#
+# Helper functions
+#
+sub tempdir
+{
+	return File::Temp::tempdir(
+		'tmp_testXXXX',
+		DIR => $ENV{TESTDIR} || cwd(),
+		CLEANUP => 1);
+}
+
+sub tempdir_short
+{
+	# Use a separate temp dir outside the build tree for the
+	# Unix-domain socket, to avoid file name length issues.
+	return File::Temp::tempdir(CLEANUP => 1);
+}
+
+sub system_log
+{
+	print("# Running: " . join(" ", @_) . "\n");
+	return system(@_);
+}
+
+sub system_or_bail
+{
+	if (system_log(@_) != 0)
+	{
+		BAIL_OUT("system $_[0] failed");
+	}
+}
+
+sub run_log
+{
+	print("# Running: " . join(" ", @{ $_[0] }) . "\n");
+	return run(@_);
+}
+
+sub slurp_dir
+{
+	my ($dir) = @_;
+	opendir(my $dh, $dir)
+	  or die "could not opendir \"$dir\": $!";
+	my @direntries = readdir $dh;
+	closedir $dh;
+	return @direntries;
+}
+
+sub slurp_file
+{
+	local $/;
+	local @ARGV = @_;
+	my $contents = <>;
+	$contents =~ s/\r//g if $Config{osname} eq 'msys';
+	return $contents;
+}
+
+sub append_to_file
+{
+	my ($filename, $str) = @_;
+
+	open my $fh, ">>", $filename or die "could not open \"$filename\": $!";
+	print $fh $str;
+	close $fh;
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..1ed98f2 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -4,20 +4,21 @@ use strict;
 use warnings;
 
 use Config;
+use Cwd;
 use Exporter 'import';
+use File::Basename;
+use File::Spec;
+use File::Temp ();
+use IPC::Run qw(run start);
+use PostgresNode;
+use Test::More;
+use TestBase;
+
 our @EXPORT = qw(
-  tempdir
-  tempdir_short
-  standard_initdb
-  configure_hba_for_replication
-  start_test_server
-  restart_test_server
+  get_new_node
+  teardown_node
   psql
-  slurp_dir
-  slurp_file
-  system_or_bail
-  system_log
-  run_log
+  poll_query_until
 
   command_ok
   command_fails
@@ -27,197 +28,91 @@ our @EXPORT = qw(
   program_options_handling_ok
   command_like
   issues_sql_like
-
-  $tmp_check
-  $log_path
-  $windows_os
 );
 
-use Cwd;
-use File::Basename;
-use File::Spec;
-use File::Temp ();
-use IPC::Run qw(run start);
-
-use SimpleTee;
+our ($test_pghost, $last_port_assigned);
+our (@all_nodes,   @active_nodes);
 
-use Test::More;
-
-our $windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
-
-# Open log file. For each test, the log file name uses the name of the
-# file launching this module, without the .pl suffix.
-our ($tmp_check, $log_path);
-$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
-$log_path = "$tmp_check/log";
-mkdir $tmp_check;
-mkdir $log_path;
-my $test_logfile = basename($0);
-$test_logfile =~ s/\.[^.]+$//;
-$test_logfile = "$log_path/regress_log_$test_logfile";
-open TESTLOG, '>', $test_logfile or die "Cannot open STDOUT to logfile: $!";
-
-# Hijack STDOUT and STDERR to the log file
-open(ORIG_STDOUT, ">&STDOUT");
-open(ORIG_STDERR, ">&STDERR");
-open(STDOUT, ">&TESTLOG");
-open(STDERR, ">&TESTLOG");
-
-# The test output (ok ...) needs to be printed to the original STDOUT so
-# that the 'prove' program can parse it, and display it to the user in
-# real time. But also copy it to the log file, to provide more context
-# in the log.
-my $builder = Test::More->builder;
-my $fh = $builder->output;
-tie *$fh, "SimpleTee", *ORIG_STDOUT, *TESTLOG;
-$fh = $builder->failure_output;
-tie *$fh, "SimpleTee", *ORIG_STDERR, *TESTLOG;
-
-# Enable auto-flushing for all the file handles. Stderr and stdout are
-# redirected to the same file, and buffering causes the lines to appear
-# in the log in confusing order.
-autoflush STDOUT 1;
-autoflush STDERR 1;
-autoflush TESTLOG 1;
-
-# Set to untranslated messages, to be able to compare program output
-# with expected strings.
-delete $ENV{LANGUAGE};
-delete $ENV{LC_ALL};
-$ENV{LC_MESSAGES} = 'C';
-
-delete $ENV{PGCONNECT_TIMEOUT};
-delete $ENV{PGDATA};
-delete $ENV{PGDATABASE};
-delete $ENV{PGHOSTADDR};
-delete $ENV{PGREQUIRESSL};
-delete $ENV{PGSERVICE};
-delete $ENV{PGSSLMODE};
-delete $ENV{PGUSER};
-
-if (!$ENV{PGPORT})
+BEGIN
 {
-	$ENV{PGPORT} = 65432;
+	# Set to untranslated messages, to be able to compare program output
+	# with expected strings.
+	delete $ENV{LANGUAGE};
+	delete $ENV{LC_ALL};
+	$ENV{LC_MESSAGES} = 'C';
+
+	delete $ENV{PGCONNECT_TIMEOUT};
+	delete $ENV{PGDATA};
+	delete $ENV{PGDATABASE};
+	delete $ENV{PGHOSTADDR};
+	delete $ENV{PGREQUIRESSL};
+	delete $ENV{PGSERVICE};
+	delete $ENV{PGSSLMODE};
+	delete $ENV{PGUSER};
+	delete $ENV{PGPORT};
+	delete $ENV{PGHOST};
+
+	# PGHOST is set once and for all through a single series of tests when
+	# this module is loaded.
+	$test_pghost = $windows_os ? "127.0.0.1" : TestBase::tempdir_short();
+	$ENV{PGHOST} = $test_pghost;
+	$ENV{PGDATABASE} = 'postgres';
+
+	# Tracking of last port value assigned to accelerate free port lookup.
+	# XXX: Should this use PG_VERSION_NUM?
+	$last_port_assigned = 90600 % 16384 + 49152;
+
+	# Tracker of active nodes
+	@all_nodes    = ();
+	@active_nodes = ();
 }
 
-$ENV{PGPORT} = int($ENV{PGPORT}) % 65536;
-
-
+# Build a new PostgresNode object, assigning a free port number.
 #
-# Helper functions
-#
-
-
-sub tempdir
+# We also register the node, to avoid the port number from being reused
+# for another node even when this one is not active.
+sub get_new_node
 {
-	return File::Temp::tempdir(
-		'tmp_testXXXX',
-		DIR => $ENV{TESTDIR} || cwd(),
-		CLEANUP => 1);
-}
+	my $found = 0;
+	my $port  = $last_port_assigned;
 
-sub tempdir_short
-{
-
-	# Use a separate temp dir outside the build tree for the
-	# Unix-domain socket, to avoid file name length issues.
-	return File::Temp::tempdir(CLEANUP => 1);
-}
-
-# Initialize a new cluster for testing.
-#
-# The PGHOST environment variable is set to connect to the new cluster.
-#
-# Authentication is set up so that only the current OS user can access the
-# cluster. On Unix, we use Unix domain socket connections, with the socket in
-# a directory that's only accessible to the current user to ensure that.
-# On Windows, we use SSPI authentication to ensure the same (by pg_regress
-# --config-auth).
-sub standard_initdb
-{
-	my $pgdata = shift;
-	system_or_bail('initdb', '-D', "$pgdata", '-A' , 'trust', '-N');
-	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
-
-	my $tempdir_short = tempdir_short;
-
-	open CONF, ">>$pgdata/postgresql.conf";
-	print CONF "\n# Added by TestLib.pm)\n";
-	print CONF "fsync = off\n";
-	if ($windows_os)
+	while ($found == 0)
 	{
-		print CONF "listen_addresses = '127.0.0.1'\n";
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		if (!run_log([ 'pg_isready', '-p', $port ]))
+		{
+			$found = 1;
+
+			# Found a potential candidate port number.  Check first that it is
+			# not included in the list of registered nodes.
+			foreach my $node (@all_nodes)
+			{
+				$found = 0 if ($node->port == $port);
+			}
+		}
 	}
-	else
-	{
-		print CONF "unix_socket_directories = '$tempdir_short'\n";
-		print CONF "listen_addresses = ''\n";
-	}
-	close CONF;
 
-	$ENV{PGHOST}         = $windows_os ? "127.0.0.1" : $tempdir_short;
-}
+	print "# Found free port $port\n";
 
-# Set up the cluster to allow replication connections, in the same way that
-# standard_initdb does for normal connections.
-sub configure_hba_for_replication
-{
-	my $pgdata = shift;
+	# Lock port number found by creating a new node
+	my $node = new PostgresNode($test_pghost, $port);
 
-	open HBA, ">>$pgdata/pg_hba.conf";
-	print HBA "\n# Allow replication (set up by TestLib.pm)\n";
-	if (! $windows_os)
-	{
-		print HBA "local replication all trust\n";
-	}
-	else
-	{
-		print HBA "host replication all 127.0.0.1/32 sspi include_realm=1 map=regress\n";
-	}
-	close HBA;
-}
-
-my ($test_server_datadir, $test_server_logfile);
-
-
-# Initialize a new cluster for testing in given directory, and start it.
-sub start_test_server
-{
-	my ($tempdir) = @_;
-	my $ret;
+	# Add node to list of nodes currently in use
+	push(@all_nodes,    $node);
+	push(@active_nodes, $node);
+	$last_port_assigned = $port;
 
-	print("### Starting test server in $tempdir\n");
-	standard_initdb "$tempdir/pgdata";
-
-	$ret = system_log('pg_ctl', '-D', "$tempdir/pgdata", '-w', '-l',
-	  "$log_path/postmaster.log", '-o', "--log-statement=all",
-	  'start');
-
-	if ($ret != 0)
-	{
-		print "# pg_ctl failed; logfile:\n";
-		system('cat', "$log_path/postmaster.log");
-		BAIL_OUT("pg_ctl failed");
-	}
-
-	$test_server_datadir = "$tempdir/pgdata";
-	$test_server_logfile = "$log_path/postmaster.log";
+	return $node;
 }
 
-sub restart_test_server
+sub teardown_node
 {
-	print("### Restarting test server\n");
-	system_log('pg_ctl', '-D', $test_server_datadir, '-w', '-l',
-	  $test_server_logfile, 'restart');
-}
+	my $node = shift;
 
-END
-{
-	if ($test_server_datadir)
-	{
-		system_log('pg_ctl', '-D', $test_server_datadir, '-m',
-		  'immediate', 'stop');
-	}
+	$node->stop('immediate');
+	@active_nodes = grep { $_ ne $node } @active_nodes;
 }
 
 sub psql
@@ -225,56 +120,55 @@ sub psql
 	my ($dbname, $sql) = @_;
 	my ($stdout, $stderr);
 	print("# Running SQL command: $sql\n");
-	run [ 'psql', '-X', '-A', '-t', '-q', '-d', $dbname, '-f', '-' ], '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
+	run [ 'psql', '-X', '-A', '-t', '-q', '-d', $dbname, '-f', '-' ],
+	  '<', \$sql, '>', \$stdout, '2>', \$stderr
+	  or die;
+	if ($stderr ne "")
+	{
+		print "#### Begin standard error\n";
+		print $stderr;
+		print "#### End standard error\n";
+	}
 	chomp $stdout;
 	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
 	return $stdout;
 }
 
-sub slurp_dir
+# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
+sub poll_query_until
 {
-	my ($dir) = @_;
-	opendir(my $dh, $dir) or die;
-	my @direntries = readdir $dh;
-	closedir $dh;
-	return @direntries;
-}
+	my ($node, $query) = @_;
 
-sub slurp_file
-{
-	local $/;
-	local @ARGV = @_;
-	my $contents = <>;
-	$contents =~ s/\r//g if $Config{osname} eq 'msys';
-	return $contents;
-}
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
 
-sub system_or_bail
-{
-	if (system_log(@_) != 0)
+	while ($attempts < $max_attempts)
 	{
-		BAIL_OUT("system $_[0] failed: $?");
+		my $cmd = [ 'psql', '-At', '-c', $query, '-d', $node->connstr() ];
+		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
 	}
-}
 
-sub system_log
-{
-	print("# Running: " . join(" ", @_) ."\n");
-	return system(@_);
-}
-
-sub run_log
-{
-	print("# Running: " . join(" ", @{$_[0]}) ."\n");
-	return run (@_);
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
 }
 
-
 #
 # Test functions
 #
-
-
 sub command_ok
 {
 	my ($cmd, $test_name) = @_;
@@ -292,7 +186,7 @@ sub command_fails
 sub command_exit_is
 {
 	my ($cmd, $expected, $test_name) = @_;
-	print("# Running: " . join(" ", @{$cmd}) ."\n");
+	print("# Running: " . join(" ", @{$cmd}) . "\n");
 	my $h = start $cmd;
 	$h->finish();
 
@@ -303,8 +197,10 @@ sub command_exit_is
 	# assuming the Unix convention, which will always return 0 on Windows as
 	# long as the process was not terminated by an exception. To work around
 	# that, use $h->full_result on Windows instead.
-	my $result = ($Config{osname} eq "MSWin32") ?
-		($h->full_results)[0] : $h->result(0);
+	my $result =
+	    ($Config{osname} eq "MSWin32")
+	  ? ($h->full_results)[0]
+	  : $h->result(0);
 	is($result, $expected, $test_name);
 }
 
@@ -335,8 +231,8 @@ sub program_options_handling_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --not-a-valid-option\n");
-	my $result = run [ $cmd, '--not-a-valid-option' ], '>', \$stdout, '2>',
-	  \$stderr;
+	my $result = run [ $cmd, '--not-a-valid-option' ], '>', \$stdout,
+	  '2>', \$stderr;
 	ok(!$result, "$cmd with invalid option nonzero exit code");
 	isnt($stderr, '', "$cmd with invalid option prints error message");
 }
@@ -354,11 +250,11 @@ sub command_like
 
 sub issues_sql_like
 {
-	my ($cmd, $expected_sql, $test_name) = @_;
-	truncate $test_server_logfile, 0;
+	my ($node, $cmd, $expected_sql, $test_name) = @_;
+	truncate $node->logfile, 0;
 	my $result = run_log($cmd);
 	ok($result, "@$cmd exit code 0");
-	my $log = slurp_file($test_server_logfile);
+	my $log = slurp_file($node->logfile);
 	like($log, $expected_sql, "$test_name: SQL found in server log");
 }
 
diff --git a/src/test/ssl/ServerSetup.pm b/src/test/ssl/ServerSetup.pm
index a6c77b5..c6b9f9b 100644
--- a/src/test/ssl/ServerSetup.pm
+++ b/src/test/ssl/ServerSetup.pm
@@ -18,6 +18,7 @@ package ServerSetup;
 
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use File::Basename;
 use File::Copy;
@@ -45,7 +46,7 @@ sub copy_files
 
 sub configure_test_server_for_ssl
 {
-	my $tempdir    = $_[0];
+	my $pgdata     = $_[0];
 	my $serverhost = $_[1];
 
 	# Create test users and databases
@@ -55,7 +56,7 @@ sub configure_test_server_for_ssl
 	psql 'postgres', "CREATE DATABASE certdb";
 
 	# enable logging etc.
-	open CONF, ">>$tempdir/pgdata/postgresql.conf";
+	open CONF, ">>$pgdata/postgresql.conf";
 	print CONF "fsync=off\n";
 	print CONF "log_connections=on\n";
 	print CONF "log_hostname=on\n";
@@ -68,17 +69,17 @@ sub configure_test_server_for_ssl
 	close CONF;
 
 # Copy all server certificates and keys, and client root cert, to the data dir
-	copy_files("ssl/server-*.crt", "$tempdir/pgdata");
-	copy_files("ssl/server-*.key", "$tempdir/pgdata");
-	chmod(0600, glob "$tempdir/pgdata/server-*.key") or die $!;
-	copy_files("ssl/root+client_ca.crt", "$tempdir/pgdata");
-	copy_files("ssl/root+client.crl",    "$tempdir/pgdata");
+	copy_files("ssl/server-*.crt", $pgdata);
+	copy_files("ssl/server-*.key", $pgdata);
+	chmod(0600, glob "$pgdata/server-*.key") or die $!;
+	copy_files("ssl/root+client_ca.crt", $pgdata);
+	copy_files("ssl/root+client.crl",    $pgdata);
 
   # Only accept SSL connections from localhost. Our tests don't depend on this
   # but seems best to keep it as narrow as possible for security reasons.
   #
   # When connecting to certdb, also check the client certificate.
-	open HBA, ">$tempdir/pgdata/pg_hba.conf";
+	open HBA, ">$pgdata/pg_hba.conf";
 	print HBA
 "# TYPE  DATABASE        USER            ADDRESS                 METHOD\n";
 	print HBA
@@ -96,12 +97,13 @@ sub configure_test_server_for_ssl
 # the server so that the configuration takes effect.
 sub switch_server_cert
 {
-	my $tempdir  = $_[0];
+	my $node     = $_[0];
 	my $certfile = $_[1];
+	my $pgdata   = $node->data_dir;
 
 	diag "Restarting server with certfile \"$certfile\"...";
 
-	open SSLCONF, ">$tempdir/pgdata/sslconfig.conf";
+	open SSLCONF, ">$pgdata/sslconfig.conf";
 	print SSLCONF "ssl=on\n";
 	print SSLCONF "ssl_ca_file='root+client_ca.crt'\n";
 	print SSLCONF "ssl_cert_file='$certfile.crt'\n";
@@ -110,5 +112,5 @@ sub switch_server_cert
 	close SSLCONF;
 
 	# Stop and restart server to reload the new config.
-	restart_test_server();
+	$node->restart;
 }
diff --git a/src/test/ssl/t/001_ssltests.pl b/src/test/ssl/t/001_ssltests.pl
index 0d6f339..5fd936b 100644
--- a/src/test/ssl/t/001_ssltests.pl
+++ b/src/test/ssl/t/001_ssltests.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+use PostgresNode;
+use TestBase;
 use TestLib;
 use Test::More tests => 38;
 use ServerSetup;
@@ -25,8 +27,6 @@ BEGIN
 # postgresql-ssl-regression.test.
 my $SERVERHOSTADDR = '127.0.0.1';
 
-my $tempdir = TestLib::tempdir;
-
 # Define a couple of helper functions to test connecting to the server.
 
 my $common_connstr;
@@ -74,10 +74,16 @@ chmod 0600, "ssl/client.key";
 
 #### Part 0. Set up the server.
 
-diag "setting up data directory in \"$tempdir\"...";
-start_test_server($tempdir);
-configure_test_server_for_ssl($tempdir, $SERVERHOSTADDR);
-switch_server_cert($tempdir, 'server-cn-only');
+diag "setting up data directory...";
+my $node = get_new_node();
+$node->init;
+# PGHOST is enforced here to set up the node, subsequent connections
+# will use a dedicated connection string.
+$ENV{PGHOST} = $node->host;
+$ENV{PGPORT} = $node->port;
+$node->start;
+configure_test_server_for_ssl($node->data_dir, $SERVERHOSTADDR);
+switch_server_cert($node, 'server-cn-only');
 
 ### Part 1. Run client-side tests.
 ###
@@ -150,7 +156,7 @@ test_connect_ok("sslmode=verify-ca host=wronghost.test");
 test_connect_fails("sslmode=verify-full host=wronghost.test");
 
 # Test Subject Alternative Names.
-switch_server_cert($tempdir, 'server-multiple-alt-names');
+switch_server_cert($node, 'server-multiple-alt-names');
 
 diag "test hostname matching with X509 Subject Alternative Names";
 $common_connstr =
@@ -165,7 +171,7 @@ test_connect_fails("host=deep.subdomain.wildcard.pg-ssltest.test");
 
 # Test certificate with a single Subject Alternative Name. (this gives a
 # slightly different error message, that's all)
-switch_server_cert($tempdir, 'server-single-alt-name');
+switch_server_cert($node, 'server-single-alt-name');
 
 diag "test hostname matching with a single X509 Subject Alternative Name";
 $common_connstr =
@@ -178,7 +184,7 @@ test_connect_fails("host=deep.subdomain.wildcard.pg-ssltest.test");
 
 # Test server certificate with a CN and SANs. Per RFCs 2818 and 6125, the CN
 # should be ignored when the certificate has both.
-switch_server_cert($tempdir, 'server-cn-and-alt-names');
+switch_server_cert($node, 'server-cn-and-alt-names');
 
 diag "test certificate with both a CN and SANs";
 $common_connstr =
@@ -190,7 +196,7 @@ test_connect_fails("host=common-name.pg-ssltest.test");
 
 # Finally, test a server certificate that has no CN or SANs. Of course, that's
 # not a very sensible certificate, but libpq should handle it gracefully.
-switch_server_cert($tempdir, 'server-no-names');
+switch_server_cert($node, 'server-no-names');
 $common_connstr =
 "user=ssltestuser dbname=trustdb sslcert=invalid sslrootcert=ssl/root+server_ca.crt hostaddr=$SERVERHOSTADDR";
 
@@ -199,7 +205,7 @@ test_connect_fails("sslmode=verify-full host=common-name.pg-ssltest.test");
 
 # Test that the CRL works
 diag "Testing client-side CRL";
-switch_server_cert($tempdir, 'server-revoked');
+switch_server_cert($node, 'server-revoked');
 
 $common_connstr =
 "user=ssltestuser dbname=trustdb sslcert=invalid hostaddr=$SERVERHOSTADDR host=common-name.pg-ssltest.test";
@@ -233,7 +239,3 @@ test_connect_fails(
 test_connect_fails(
 "user=ssltestuser sslcert=ssl/client-revoked.crt sslkey=ssl/client-revoked.key"
 );
-
-
-# All done! Save the log, before the temporary installation is deleted
-copy("$tempdir/client-log", "./client-log");
#86Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#85)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

On Mon, Nov 30, 2015 at 6:28 AM, Noah Misch <noah@leadboat.com> wrote:

The proposed code is short on guidance about when to put a function in TestLib
versus TestBase. TestLib has no header comment. The TestBase header comment
would permit, for example, command_ok() in that module. I would try instead
keeping TestLib as the base module and moving into PostgresNode the functions
that deal with PostgreSQL clusters (get_new_node teardown_node psql
poll_query_until issues_sql_like).

PostgresNode is wanted to be a base representation of how of node is,
not of how to operate on it. The ways to perform the tests, which
works on a node, is wanted as a higher-level operation.

Logging and base configuration of a test set is a lower level of
operations than PostgresNode, because cluster nodes need actually to
perform system calls, some of those system calls like run_log allowing
to log in the centralized log file. I have tried to make the headers
of those modules more verbose, please see attached.

I'm not terribly convinced by this argument TBH. Perhaps we can have
PostgresNode be one package, and the logging routines be another
package, and we create a higher-level package whose @ISA=(PostgresNode,
LoggingWhatever) and then we move the routines suggested by Noah into
that new package. Then the tests use that instead of PostgresNode
directly.

--- a/src/bin/pg_rewind/t/001_basic.pl
+++ b/src/bin/pg_rewind/t/001_basic.pl
@@ -1,9 +1,11 @@
+# Basic pg_rewind test.
+
use strict;
use warnings;
-use TestLib;
-use Test::More tests => 8;
use RewindTest;
+use TestLib;
+use Test::More tests => 8;

Revert all changes to this file. Audit the rest of the patch for whitespace
change unrelated to the subject.

Done.

I perltidied several files, though not consistently. Regarding this
particular hunk, what is going on here is that I moved "use strict;use
warnings" as one stanza, followed by all the other "use" lines as
another stanza, alphabetically. It was previously a bit messy, with
@EXPORTS and other stuff in between "use" lines.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#87Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#86)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Mon, Nov 30, 2015 at 11:51 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Michael Paquier wrote:

On Mon, Nov 30, 2015 at 6:28 AM, Noah Misch <noah@leadboat.com> wrote:

The proposed code is short on guidance about when to put a function in TestLib
versus TestBase. TestLib has no header comment. The TestBase header comment
would permit, for example, command_ok() in that module. I would try instead
keeping TestLib as the base module and moving into PostgresNode the functions
that deal with PostgreSQL clusters (get_new_node teardown_node psql
poll_query_until issues_sql_like).

PostgresNode is wanted to be a base representation of how of node is,
not of how to operate on it. The ways to perform the tests, which
works on a node, is wanted as a higher-level operation.

Logging and base configuration of a test set is a lower level of
operations than PostgresNode, because cluster nodes need actually to
perform system calls, some of those system calls like run_log allowing
to log in the centralized log file. I have tried to make the headers
of those modules more verbose, please see attached.

I'm not terribly convinced by this argument TBH. Perhaps we can have
PostgresNode be one package, and the logging routines be another
package, and we create a higher-level package whose @ISA=(PostgresNode,
LoggingWhatever) and then we move the routines suggested by Noah into
that new package. Then the tests use that instead of PostgresNode
directly.

OK... I have merged TestLib and PostgresNode of the previous patch
into PostgresNode into the way suggested by Noah. TestBase has been
renamed back to TestLib, and includes as well the base test functions
like command_ok.
Regards,
--
Michael

Attachments:

20151201_tapcheck_v12.patchtext/x-patch; charset=US-ASCII; name=20151201_tapcheck_v12.patchDownload
diff --git a/src/bin/initdb/t/001_initdb.pl b/src/bin/initdb/t/001_initdb.pl
index 299dcf5..f64186d 100644
--- a/src/bin/initdb/t/001_initdb.pl
+++ b/src/bin/initdb/t/001_initdb.pl
@@ -4,6 +4,7 @@
 
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 14;
 
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index dc96bbf..c7eb250 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -2,6 +2,7 @@ use strict;
 use warnings;
 use Cwd;
 use Config;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 51;
 
@@ -9,8 +10,15 @@ program_help_ok('pg_basebackup');
 program_version_ok('pg_basebackup');
 program_options_handling_ok('pg_basebackup');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $tempdir = TestLib::tempdir;
+
+my $node = get_new_node();
+# Initialize node without replication settings
+$node->init(hba_permit_replication => 0);
+$node->start;
+my $pgdata = $node->data_dir;
+
+$ENV{PGPORT} = $node->port;
 
 command_fails(['pg_basebackup'],
 	'pg_basebackup needs target directory specified');
@@ -26,19 +34,19 @@ if (open BADCHARS, ">>$tempdir/pgdata/FOO\xe0\xe0\xe0BAR")
 	close BADCHARS;
 }
 
-configure_hba_for_replication "$tempdir/pgdata";
-system_or_bail 'pg_ctl', '-D', "$tempdir/pgdata", 'reload';
+$node->set_replication_conf();
+system_or_bail 'pg_ctl', '-D', $pgdata, 'reload';
 
 command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup fails because of WAL configuration');
 
-open CONF, ">>$tempdir/pgdata/postgresql.conf";
+open CONF, ">>$pgdata/postgresql.conf";
 print CONF "max_replication_slots = 10\n";
 print CONF "max_wal_senders = 10\n";
 print CONF "wal_level = archive\n";
 close CONF;
-restart_test_server;
+$node->restart;
 
 command_ok([ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup runs');
@@ -81,13 +89,13 @@ command_fails(
 
 # Tar format doesn't support filenames longer than 100 bytes.
 my $superlongname = "superlongname_" . ("x" x 100);
-my $superlongpath = "$tempdir/pgdata/$superlongname";
+my $superlongpath = "$pgdata/$superlongname";
 
 open FILE, ">$superlongpath" or die "unable to create file $superlongpath";
 close FILE;
 command_fails([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l1", '-Ft' ],
 	'pg_basebackup tar with long name fails');
-unlink "$tempdir/pgdata/$superlongname";
+unlink "$pgdata/$superlongname";
 
 # The following tests test symlinks. Windows doesn't have symlinks, so
 # skip on Windows.
@@ -98,7 +106,7 @@ SKIP: {
 	# to our physical temp location.  That way we can use shorter names
 	# for the tablespace directories, which hopefully won't run afoul of
 	# the 99 character length limit.
-	my $shorter_tempdir = tempdir_short . "/tempdir";
+	my $shorter_tempdir = TestLib::tempdir_short . "/tempdir";
 	symlink "$tempdir", $shorter_tempdir;
 
 	mkdir "$tempdir/tblspc1";
@@ -120,7 +128,7 @@ SKIP: {
 			"-T$shorter_tempdir/tblspc1=$tempdir/tbackup/tblspc1" ],
 		'plain format with tablespaces succeeds with tablespace mapping');
 	ok(-d "$tempdir/tbackup/tblspc1", 'tablespace was relocated');
-	opendir(my $dh, "$tempdir/pgdata/pg_tblspc") or die;
+	opendir(my $dh, "$pgdata/pg_tblspc") or die;
 	ok( (   grep {
 		-l "$tempdir/backup1/pg_tblspc/$_"
 			and readlink "$tempdir/backup1/pg_tblspc/$_" eq
diff --git a/src/bin/pg_controldata/t/001_pg_controldata.pl b/src/bin/pg_controldata/t/001_pg_controldata.pl
index e2b0d42..ae45f41 100644
--- a/src/bin/pg_controldata/t/001_pg_controldata.pl
+++ b/src/bin/pg_controldata/t/001_pg_controldata.pl
@@ -1,16 +1,19 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
-my $tempdir = TestLib::tempdir;
-
 program_help_ok('pg_controldata');
 program_version_ok('pg_controldata');
 program_options_handling_ok('pg_controldata');
 command_fails(['pg_controldata'], 'pg_controldata without arguments fails');
 command_fails([ 'pg_controldata', 'nonexistent' ],
 	'pg_controldata with nonexistent directory fails');
-standard_initdb "$tempdir/data";
-command_like([ 'pg_controldata', "$tempdir/data" ],
+
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+command_like([ 'pg_controldata', $node->data_dir ],
 	qr/checkpoint/, 'pg_controldata produces output');
diff --git a/src/bin/pg_ctl/t/001_start_stop.pl b/src/bin/pg_ctl/t/001_start_stop.pl
index f57abce..d1e271f 100644
--- a/src/bin/pg_ctl/t/001_start_stop.pl
+++ b/src/bin/pg_ctl/t/001_start_stop.pl
@@ -1,6 +1,7 @@
 use strict;
 use warnings;
 use Config;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 17;
 
diff --git a/src/bin/pg_ctl/t/002_status.pl b/src/bin/pg_ctl/t/002_status.pl
index 31f7c72..bdbacfe 100644
--- a/src/bin/pg_ctl/t/002_status.pl
+++ b/src/bin/pg_ctl/t/002_status.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 3;
 
@@ -9,14 +10,15 @@ my $tempdir_short = TestLib::tempdir_short;
 command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/nonexistent" ],
 	4, 'pg_ctl status with nonexistent directory');
 
-standard_initdb "$tempdir/data";
+my $node = get_new_node();
+$node->init;
 
-command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/data" ],
+command_exit_is([ 'pg_ctl', 'status', '-D', $node->data_dir ],
 	3, 'pg_ctl status with server not running');
 
 system_or_bail 'pg_ctl', '-l', "$tempdir/logfile", '-D',
-  "$tempdir/data", '-w', 'start';
-command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/data" ],
+  $node->data_dir, '-w', 'start';
+command_exit_is([ 'pg_ctl', 'status', '-D', $node->data_dir ],
 	0, 'pg_ctl status with server running');
 
-system_or_bail 'pg_ctl', 'stop', '-D', "$tempdir/data", '-m', 'fast';
+system_or_bail 'pg_ctl', 'stop', '-D', $node->data_dir, '-m', 'fast';
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..1e5adf8 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -9,22 +9,20 @@ package RewindTest;
 # To run a test, the test script (in t/ subdirectory) calls the functions
 # in this module. These functions should be called in this sequence:
 #
-# 1. init_rewind_test - sets up log file etc.
+# 1. setup_cluster - creates a PostgreSQL cluster that runs as the master
 #
-# 2. setup_cluster - creates a PostgreSQL cluster that runs as the master
+# 2. start_master - starts the master server
 #
-# 3. start_master - starts the master server
-#
-# 4. create_standby - runs pg_basebackup to initialize a standby server, and
+# 3. create_standby - runs pg_basebackup to initialize a standby server, and
 #    sets it up to follow the master.
 #
-# 5. promote_standby - runs "pg_ctl promote" to promote the standby server.
+# 4. promote_standby - runs "pg_ctl promote" to promote the standby server.
 # The old master keeps running.
 #
-# 6. run_pg_rewind - stops the old master (if it's still running) and runs
+# 5. run_pg_rewind - stops the old master (if it's still running) and runs
 # pg_rewind to synchronize it with the now-promoted standby server.
 #
-# 7. clean_rewind_test - stops both servers used in the test, if they're
+# 6. clean_rewind_test - stops both servers used in the test, if they're
 # still running.
 #
 # The test script can use the helper functions master_psql and standby_psql
@@ -37,27 +35,23 @@ package RewindTest;
 use strict;
 use warnings;
 
-use TestLib;
-use Test::More;
-
 use Config;
+use Exporter 'import';
 use File::Copy;
 use File::Path qw(rmtree);
-use IPC::Run qw(run start);
+use IPC::Run;
+use PostgresNode;
+use TestLib;
+use Test::More;
 
-use Exporter 'import';
 our @EXPORT = qw(
-  $connstr_master
-  $connstr_standby
-  $test_master_datadir
-  $test_standby_datadir
+  $node_master
+  $node_standby
 
-  append_to_file
   master_psql
   standby_psql
   check_query
 
-  init_rewind_test
   setup_cluster
   start_master
   create_standby
@@ -66,32 +60,24 @@ our @EXPORT = qw(
   clean_rewind_test
 );
 
-our $test_master_datadir  = "$tmp_check/data_master";
-our $test_standby_datadir = "$tmp_check/data_standby";
-
-# Define non-conflicting ports for both nodes.
-my $port_master  = $ENV{PGPORT};
-my $port_standby = $port_master + 1;
-
-my $connstr_master  = "port=$port_master";
-my $connstr_standby = "port=$port_standby";
-
-$ENV{PGDATABASE} = "postgres";
+# Our nodes.
+our $node_master;
+our $node_standby;
 
 sub master_psql
 {
 	my $cmd = shift;
 
-	system_or_bail 'psql', '-q', '--no-psqlrc', '-d', $connstr_master,
-	  '-c', "$cmd";
+	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
+	  $node_master->connstr('postgres'), '-c', "$cmd";
 }
 
 sub standby_psql
 {
 	my $cmd = shift;
 
-	system_or_bail 'psql', '-q', '--no-psqlrc', '-d', $connstr_standby,
-	  '-c', "$cmd";
+	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
+      $node_standby->connstr('postgres'), '-c', "$cmd";
 }
 
 # Run a query against the master, and check that the output matches what's
@@ -102,9 +88,9 @@ sub check_query
 	my ($stdout, $stderr);
 
 	# we want just the output, no formatting
-	my $result = run [
+	my $result = IPC::Run::run [
 		'psql',          '-q', '-A', '-t', '--no-psqlrc', '-d',
-		$connstr_master, '-c', $query ],
+		$node_master->connstr('postgres'), '-c', $query ],
 	  '>', \$stdout, '2>', \$stderr;
 
 	# We don't use ok() for the exit code and stderr, because we want this
@@ -125,56 +111,14 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
-sub append_to_file
-{
-	my ($filename, $str) = @_;
-
-	open my $fh, ">>", $filename or die "could not open file $filename";
-	print $fh $str;
-	close $fh;
-}
-
 sub setup_cluster
 {
 	# Initialize master, data checksums are mandatory
-	rmtree($test_master_datadir);
-	standard_initdb($test_master_datadir);
+	$node_master = get_new_node();
+	$node_master->init;
 
 	# Custom parameters for master's postgresql.conf
-	append_to_file(
-		"$test_master_datadir/postgresql.conf", qq(
+	$node_master->append_conf("postgresql.conf", qq(
 wal_level = hot_standby
 max_wal_senders = 2
 wal_keep_segments = 20
@@ -185,17 +129,11 @@ hot_standby = on
 autovacuum = off
 max_connections = 10
 ));
-
-	# Accept replication connections on master
-	configure_hba_for_replication $test_master_datadir;
 }
 
 sub start_master
 {
-	system_or_bail('pg_ctl' , '-w',
-				   '-D' , $test_master_datadir,
-				   '-l',  "$log_path/master.log",
-				   "-o", "-p $port_master", 'start');
+	$node_master->start;
 
 	#### Now run the test-specific parts to initialize the master before setting
 	# up standby
@@ -203,24 +141,19 @@ sub start_master
 
 sub create_standby
 {
+	$node_standby = get_new_node();
+	$node_master->backup('my_backup');
+	$node_standby->init_from_backup($node_master, 'my_backup');
+	my $connstr_master = $node_master->connstr('postgres');
 
-	# Set up standby with necessary parameter
-	rmtree $test_standby_datadir;
-
-	# Base backup is taken with xlog files included
-	system_or_bail('pg_basebackup', '-D', $test_standby_datadir,
-				   '-p', $port_master, '-x');
-	append_to_file(
-		"$test_standby_datadir/recovery.conf", qq(
+	$node_standby->append_conf("recovery.conf", qq(
 primary_conninfo='$connstr_master application_name=rewind_standby'
 standby_mode=on
 recovery_target_timeline='latest'
 ));
 
 	# Start standby
-	system_or_bail('pg_ctl', '-w', '-D', $test_standby_datadir,
-				   '-l', "$log_path/standby.log",
-				   '-o', "-p $port_standby", 'start');
+	$node_standby->start;
 
 	# The standby may have WAL to apply before it matches the primary.  That
 	# is fine, because no test examines the standby before promotion.
@@ -234,14 +167,14 @@ sub promote_standby
 	# Wait for the standby to receive and write all WAL.
 	my $wal_received_query =
 "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = 'rewind_standby';";
-	poll_query_until($wal_received_query, $connstr_master)
+	poll_query_until($node_master, $wal_received_query)
 	  or die "Timed out while waiting for standby to receive and write WAL";
 
 	# Now promote slave and insert some new data on master, this will put
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
-	system_or_bail('pg_ctl', '-w', '-D', $test_standby_datadir, 'promote');
-	poll_query_until("SELECT NOT pg_is_in_recovery()", $connstr_standby)
+	system_or_bail('pg_ctl', '-w', '-D', $node_standby->data_dir, 'promote');
+	poll_query_until($node_standby, "SELECT NOT pg_is_in_recovery()")
 	  or die "Timed out while waiting for promotion of standby";
 
 	# Force a checkpoint after the promotion. pg_rewind looks at the control
@@ -256,9 +189,13 @@ sub promote_standby
 sub run_pg_rewind
 {
 	my $test_mode = shift;
+	my $master_pgdata = $node_master->data_dir;
+	my $standby_pgdata = $node_standby->data_dir;
+	my $standby_connstr = $node_standby->connstr('postgres');
+	my $tmp_folder = TestLib::tempdir;
 
 	# Stop the master and be ready to perform the rewind
-	system_or_bail('pg_ctl', '-D', $test_master_datadir, '-m', 'fast', 'stop');
+	$node_master->stop;
 
 	# At this point, the rewind processing is ready to run.
 	# We now have a very simple scenario with a few diverged WAL record.
@@ -267,20 +204,19 @@ sub run_pg_rewind
 
 	# Keep a temporary postgresql.conf for master node or it would be
 	# overwritten during the rewind.
-	copy("$test_master_datadir/postgresql.conf",
-		 "$tmp_check/master-postgresql.conf.tmp");
+	copy("$master_pgdata/postgresql.conf",
+		 "$tmp_folder/master-postgresql.conf.tmp");
 
 	# Now run pg_rewind
 	if ($test_mode eq "local")
 	{
 		# Do rewind using a local pgdata as source
 		# Stop the master and be ready to perform the rewind
-		system_or_bail('pg_ctl', '-D', $test_standby_datadir,
-					   '-m', 'fast', 'stop');
+		$node_standby->stop;
 		command_ok(['pg_rewind',
 					"--debug",
-					"--source-pgdata=$test_standby_datadir",
-					"--target-pgdata=$test_master_datadir"],
+					"--source-pgdata=$standby_pgdata",
+					"--target-pgdata=$master_pgdata"],
 				   'pg_rewind local');
 	}
 	elsif ($test_mode eq "remote")
@@ -289,33 +225,30 @@ sub run_pg_rewind
 		command_ok(['pg_rewind',
 					"--debug",
 					"--source-server",
-					"port=$port_standby dbname=postgres",
-					"--target-pgdata=$test_master_datadir"],
+					$standby_connstr,
+					"--target-pgdata=$master_pgdata"],
 				   'pg_rewind remote');
 	}
 	else
 	{
-
 		# Cannot come here normally
 		die("Incorrect test mode specified");
 	}
 
 	# Now move back postgresql.conf with old settings
-	move("$tmp_check/master-postgresql.conf.tmp",
-		 "$test_master_datadir/postgresql.conf");
+	move("$tmp_folder/master-postgresql.conf.tmp",
+		 "$master_pgdata/postgresql.conf");
 
 	# Plug-in rewound node to the now-promoted standby node
-	append_to_file(
-		"$test_master_datadir/recovery.conf", qq(
+	my $port_standby = $node_standby->port;
+	$node_master->append_conf('recovery.conf', qq(
 primary_conninfo='port=$port_standby'
 standby_mode=on
 recovery_target_timeline='latest'
 ));
 
 	# Restart the master to check that rewind went correctly
-	system_or_bail('pg_ctl', '-w', '-D', $test_master_datadir,
-				   '-l', "$log_path/master.log",
-				   '-o', "-p $port_master", 'start');
+	$node_master->restart;
 
 	#### Now run the test-specific parts to check the result
 }
@@ -323,22 +256,8 @@ recovery_target_timeline='latest'
 # Clean up after the test. Stop both servers, if they're still running.
 sub clean_rewind_test
 {
-	if ($test_master_datadir)
-	{
-		system
-		  'pg_ctl', '-D', $test_master_datadir, '-m', 'immediate', 'stop';
-	}
-	if ($test_standby_datadir)
-	{
-		system
-		  'pg_ctl', '-D', $test_standby_datadir, '-m', 'immediate', 'stop';
-	}
+	teardown_node($node_master) if (defined($node_master));
+	teardown_node($node_standby) if (defined($node_standby));
 }
 
-# Stop the test servers, just in case they're still running.
-END
-{
-	my $save_rc = $?;
-	clean_rewind_test();
-	$? = $save_rc;
-}
+1;
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index d317f53..cedde14 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -17,7 +17,7 @@ sub run_test
 	RewindTest::setup_cluster();
 	RewindTest::start_master();
 
-	my $test_master_datadir = $RewindTest::test_master_datadir;
+	my $test_master_datadir = $node_master->data_dir;
 
 	# Create a subdir and files that will be present in both
 	mkdir "$test_master_datadir/tst_both_dir";
@@ -30,6 +30,7 @@ sub run_test
 	RewindTest::create_standby();
 
 	# Create different subdirs and files in master and standby
+	my $test_standby_datadir = $node_standby->data_dir;
 
 	mkdir "$test_standby_datadir/tst_standby_dir";
 	append_to_file "$test_standby_datadir/tst_standby_dir/standby_file1",
diff --git a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
index c5f72e2..bdcab56 100644
--- a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
+++ b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
@@ -23,11 +23,13 @@ sub run_test
 {
 	my $test_mode = shift;
 
-	my $master_xlogdir = "$tmp_check/xlog_master";
+	my $master_xlogdir = "${TestLib::tmp_check}/xlog_master";
 
 	rmtree($master_xlogdir);
 	RewindTest::setup_cluster();
 
+	my $test_master_datadir = $node_master->data_dir;
+
 	# turn pg_xlog into a symlink
 	print("moving $test_master_datadir/pg_xlog to $master_xlogdir\n");
 	move("$test_master_datadir/pg_xlog", $master_xlogdir) or die;
diff --git a/src/bin/scripts/t/010_clusterdb.pl b/src/bin/scripts/t/010_clusterdb.pl
index dc0d78a..34382f5 100644
--- a/src/bin/scripts/t/010_clusterdb.pl
+++ b/src/bin/scripts/t/010_clusterdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
@@ -7,20 +8,26 @@ program_help_ok('clusterdb');
 program_version_ok('clusterdb');
 program_options_handling_ok('clusterdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
+$ENV{PGDATABASE} = 'postgres';
 
 issues_sql_like(
-	[ 'clusterdb', 'postgres' ],
+	$node,
+	[ 'clusterdb' ],
 	qr/statement: CLUSTER;/,
 	'SQL CLUSTER run');
 
-command_fails([ 'clusterdb', '-t', 'nonexistent', 'postgres' ],
-	'fails with nonexistent table');
+command_fails([ 'clusterdb', '-t', 'nonexistent' ],
+			  'fails with nonexistent table');
 
 psql 'postgres',
-'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';
+	'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';
 issues_sql_like(
-	[ 'clusterdb', '-t', 'test1', 'postgres' ],
+	$node,
+	[ 'clusterdb', '-t', 'test1' ],
 	qr/statement: CLUSTER test1;/,
 	'cluster specific table');
diff --git a/src/bin/scripts/t/011_clusterdb_all.pl b/src/bin/scripts/t/011_clusterdb_all.pl
index 7769f70..ff5eb97 100644
--- a/src/bin/scripts/t/011_clusterdb_all.pl
+++ b/src/bin/scripts/t/011_clusterdb_all.pl
@@ -1,12 +1,20 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+# clusterdb -a is not compatible with -d, hence enforce environment variables
+# correctly.
+$ENV{PGDATABASE} = 'postgres';
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'clusterdb', '-a' ],
 	qr/statement: CLUSTER.*statement: CLUSTER/s,
 	'cluster all databases');
diff --git a/src/bin/scripts/t/020_createdb.pl b/src/bin/scripts/t/020_createdb.pl
index a44283c..6b79655 100644
--- a/src/bin/scripts/t/020_createdb.pl
+++ b/src/bin/scripts/t/020_createdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
@@ -7,14 +8,19 @@ program_help_ok('createdb');
 program_version_ok('createdb');
 program_options_handling_ok('createdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'createdb', 'foobar1' ],
 	qr/statement: CREATE DATABASE foobar1/,
 	'SQL CREATE DATABASE run');
 issues_sql_like(
+	$node,
 	[ 'createdb', '-l', 'C', '-E', 'LATIN1', '-T', 'template0', 'foobar2' ],
 	qr/statement: CREATE DATABASE foobar2 ENCODING 'LATIN1'/,
 	'create database with encoding');
diff --git a/src/bin/scripts/t/030_createlang.pl b/src/bin/scripts/t/030_createlang.pl
index 7ff0a3e..5fd7613 100644
--- a/src/bin/scripts/t/030_createlang.pl
+++ b/src/bin/scripts/t/030_createlang.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 14;
 
@@ -7,18 +8,23 @@ program_help_ok('createlang');
 program_version_ok('createlang');
 program_options_handling_ok('createlang');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
+$ENV{PGDATABASE} = 'postgres';
 
 command_fails(
-	[ 'createlang', 'plpgsql', 'postgres' ],
+	[ 'createlang', 'plpgsql' ],
 	'fails if language already exists');
 
-psql 'postgres', 'DROP EXTENSION plpgsql';
+psql $node->connstr('postgres'), 'DROP EXTENSION plpgsql';
 issues_sql_like(
-	[ 'createlang', 'plpgsql', 'postgres' ],
+	$node,
+	[ 'createlang', 'plpgsql' ],
 	qr/statement: CREATE EXTENSION "plpgsql"/,
 	'SQL CREATE EXTENSION run');
 
-command_like([ 'createlang', '--list', 'postgres' ],
+command_like([ 'createlang', '--list' ],
 	qr/plpgsql/, 'list output');
diff --git a/src/bin/scripts/t/040_createuser.pl b/src/bin/scripts/t/040_createuser.pl
index 4d44e14..a87d659 100644
--- a/src/bin/scripts/t/040_createuser.pl
+++ b/src/bin/scripts/t/040_createuser.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 17;
 
@@ -7,22 +8,30 @@ program_help_ok('createuser');
 program_version_ok('createuser');
 program_options_handling_ok('createuser');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGDATABASE} = 'postgres';
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'createuser', 'user1' ],
 qr/statement: CREATE ROLE user1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN;/,
 	'SQL CREATE USER run');
 issues_sql_like(
+	$node,
 	[ 'createuser', '-L', 'role1' ],
 qr/statement: CREATE ROLE role1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT NOLOGIN;/,
 	'create a non-login role');
 issues_sql_like(
+	$node,
 	[ 'createuser', '-r', 'user2' ],
 qr/statement: CREATE ROLE user2 NOSUPERUSER NOCREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a CREATEROLE user');
 issues_sql_like(
+	$node,
 	[ 'createuser', '-s', 'user3' ],
 qr/statement: CREATE ROLE user3 SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a superuser');
diff --git a/src/bin/scripts/t/050_dropdb.pl b/src/bin/scripts/t/050_dropdb.pl
index 3065e50..3260faa 100644
--- a/src/bin/scripts/t/050_dropdb.pl
+++ b/src/bin/scripts/t/050_dropdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,11 +8,15 @@ program_help_ok('dropdb');
 program_version_ok('dropdb');
 program_options_handling_ok('dropdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-psql 'postgres', 'CREATE DATABASE foobar1';
+$ENV{PGPORT} = $node->port;
+
+psql $node->connstr('postgres'), 'CREATE DATABASE foobar1';
 issues_sql_like(
+	$node,
 	[ 'dropdb', 'foobar1' ],
 	qr/statement: DROP DATABASE foobar1/,
 	'SQL DROP DATABASE run');
diff --git a/src/bin/scripts/t/060_droplang.pl b/src/bin/scripts/t/060_droplang.pl
index 6a21d7e..64d36ce 100644
--- a/src/bin/scripts/t/060_droplang.pl
+++ b/src/bin/scripts/t/060_droplang.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,10 +8,14 @@ program_help_ok('droplang');
 program_version_ok('droplang');
 program_options_handling_ok('droplang');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'droplang', 'plpgsql', 'postgres' ],
 	qr/statement: DROP EXTENSION "plpgsql"/,
 	'SQL DROP EXTENSION run');
diff --git a/src/bin/scripts/t/070_dropuser.pl b/src/bin/scripts/t/070_dropuser.pl
index bbb3b79..0802a66 100644
--- a/src/bin/scripts/t/070_dropuser.pl
+++ b/src/bin/scripts/t/070_dropuser.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,11 +8,15 @@ program_help_ok('dropuser');
 program_version_ok('dropuser');
 program_options_handling_ok('dropuser');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 psql 'postgres', 'CREATE ROLE foobar1';
 issues_sql_like(
+	$node,
 	[ 'dropuser', 'foobar1' ],
 	qr/statement: DROP ROLE foobar1/,
 	'SQL DROP ROLE run');
diff --git a/src/bin/scripts/t/080_pg_isready.pl b/src/bin/scripts/t/080_pg_isready.pl
index f432505..5ac9614 100644
--- a/src/bin/scripts/t/080_pg_isready.pl
+++ b/src/bin/scripts/t/080_pg_isready.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 10;
 
@@ -9,7 +10,10 @@ program_options_handling_ok('pg_isready');
 
 command_fails(['pg_isready'], 'fails with no server running');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 command_ok(['pg_isready'], 'succeeds with server running');
diff --git a/src/bin/scripts/t/090_reindexdb.pl b/src/bin/scripts/t/090_reindexdb.pl
index 42628c2..58788fd 100644
--- a/src/bin/scripts/t/090_reindexdb.pl
+++ b/src/bin/scripts/t/090_reindexdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 20;
 
@@ -7,12 +8,15 @@ program_help_ok('reindexdb');
 program_version_ok('reindexdb');
 program_options_handling_ok('reindexdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
+$ENV{PGPORT} = $node->port;
 $ENV{PGOPTIONS} = '--client-min-messages=WARNING';
 
 issues_sql_like(
+	$node,
 	[ 'reindexdb', 'postgres' ],
 	qr/statement: REINDEX DATABASE postgres;/,
 	'SQL REINDEX run');
@@ -20,22 +24,27 @@ issues_sql_like(
 psql 'postgres',
   'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a);';
 issues_sql_like(
+	$node,
 	[ 'reindexdb', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX TABLE test1;/,
 	'reindex specific table');
 issues_sql_like(
+	$node,
 	[ 'reindexdb', '-i', 'test1x', 'postgres' ],
 	qr/statement: REINDEX INDEX test1x;/,
 	'reindex specific index');
 issues_sql_like(
+	$node,
 	[ 'reindexdb', '-S', 'pg_catalog', 'postgres' ],
 	qr/statement: REINDEX SCHEMA pg_catalog;/,
 	'reindex specific schema');
 issues_sql_like(
+	$node,
 	[ 'reindexdb', '-s', 'postgres' ],
 	qr/statement: REINDEX SYSTEM postgres;/,
 	'reindex system tables');
 issues_sql_like(
+	$node,
 	[ 'reindexdb', '-v', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX \(VERBOSE\) TABLE test1;/,
 	'reindex with verbose output');
diff --git a/src/bin/scripts/t/091_reindexdb_all.pl b/src/bin/scripts/t/091_reindexdb_all.pl
index ffadf29..1278f07 100644
--- a/src/bin/scripts/t/091_reindexdb_all.pl
+++ b/src/bin/scripts/t/091_reindexdb_all.pl
@@ -1,14 +1,17 @@
 use strict;
 use warnings;
-use TestLib;
+use PostgresNode;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
+$ENV{PGPORT} = $node->port;
 $ENV{PGOPTIONS} = '--client-min-messages=WARNING';
 
 issues_sql_like(
+	$node,
 	[ 'reindexdb', '-a' ],
 	qr/statement: REINDEX.*statement: REINDEX/s,
 	'reindex all databases');
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index ac160ba..b32eed4 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -1,5 +1,6 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 18;
 
@@ -7,26 +8,34 @@ program_help_ok('vacuumdb');
 program_version_ok('vacuumdb');
 program_options_handling_ok('vacuumdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', 'postgres' ],
 	qr/statement: VACUUM;/,
 	'SQL VACUUM run');
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '-f', 'postgres' ],
 	qr/statement: VACUUM \(FULL\);/,
 	'vacuumdb -f');
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '-F', 'postgres' ],
 	qr/statement: VACUUM \(FREEZE\);/,
 	'vacuumdb -F');
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '-z', 'postgres' ],
 	qr/statement: VACUUM \(ANALYZE\);/,
 	'vacuumdb -z');
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '-Z', 'postgres' ],
 	qr/statement: ANALYZE;/,
 	'vacuumdb -Z');
diff --git a/src/bin/scripts/t/101_vacuumdb_all.pl b/src/bin/scripts/t/101_vacuumdb_all.pl
index e90f321..c277787 100644
--- a/src/bin/scripts/t/101_vacuumdb_all.pl
+++ b/src/bin/scripts/t/101_vacuumdb_all.pl
@@ -1,12 +1,16 @@
 use strict;
 use warnings;
-use TestLib;
+use PostgresNode;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '-a' ],
 	qr/statement: VACUUM.*statement: VACUUM/s,
 	'vacuum all databases');
diff --git a/src/bin/scripts/t/102_vacuumdb_stages.pl b/src/bin/scripts/t/102_vacuumdb_stages.pl
index 57b980e..9a83cd5 100644
--- a/src/bin/scripts/t/102_vacuumdb_stages.pl
+++ b/src/bin/scripts/t/102_vacuumdb_stages.pl
@@ -1,12 +1,16 @@
 use strict;
 use warnings;
-use TestLib;
+use PostgresNode;
 use Test::More tests => 4;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '--analyze-in-stages', 'postgres' ],
 qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE.*
@@ -18,6 +22,7 @@ qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
 
 
 issues_sql_like(
+	$node,
 	[ 'vacuumdb', '--analyze-in-stages', '--all' ],
 qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE.*
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
new file mode 100644
index 0000000..e87dc75
--- /dev/null
+++ b/src/test/perl/PostgresNode.pm
@@ -0,0 +1,427 @@
+# PostgresNode, class representing a data directory and postmaster.
+#
+# This contains a basic set of routines able to work on a PostgreSQL node,
+# allowing to start, stop, backup and initialize it with various options.
+# The set of nodes managed by a given test is also managed by this module.
+
+package PostgresNode;
+
+use strict;
+use warnings;
+
+use Config;
+use Cwd;
+use Exporter 'import';
+use File::Basename;
+use File::Spec;
+use File::Temp ();
+use IPC::Run;
+use PostgresNode;
+use RecursiveCopy;
+use Test::More;
+use TestLib;
+
+our ($test_pghost, $last_port_assigned);
+our (@all_nodes,   @active_nodes);
+
+our @EXPORT = qw(
+  get_new_node
+  issues_sql_like
+  poll_query_until
+  psql
+  teardown_node
+);
+
+
+BEGIN
+{
+	# Set to untranslated messages, to be able to compare program output
+	# with expected strings.
+	delete $ENV{LANGUAGE};
+	delete $ENV{LC_ALL};
+	$ENV{LC_MESSAGES} = 'C';
+
+	delete $ENV{PGCONNECT_TIMEOUT};
+	delete $ENV{PGDATA};
+	delete $ENV{PGDATABASE};
+	delete $ENV{PGHOSTADDR};
+	delete $ENV{PGREQUIRESSL};
+	delete $ENV{PGSERVICE};
+	delete $ENV{PGSSLMODE};
+	delete $ENV{PGUSER};
+	delete $ENV{PGPORT};
+	delete $ENV{PGHOST};
+
+	# PGHOST is set once and for all through a single series of tests when
+	# this module is loaded.
+	$test_pghost = $windows_os ? "127.0.0.1" : TestLib::tempdir_short();
+	$ENV{PGHOST} = $test_pghost;
+	$ENV{PGDATABASE} = 'postgres';
+
+	# Tracking of last port value assigned to accelerate free port lookup.
+	# XXX: Should this use PG_VERSION_NUM?
+	$last_port_assigned = 90600 % 16384 + 49152;
+
+	# Tracker of active nodes
+	@all_nodes    = ();
+	@active_nodes = ();
+}
+
+sub new
+{
+	my $class  = shift;
+	my $pghost = shift;
+	my $pgport = shift;
+	my $self   = {
+		_port     => $pgport,
+		_host     => $pghost,
+		_basedir  => TestLib::tempdir,
+		_applname => "node_$pgport",
+		_logfile  => "$TestLib::log_path/node_$pgport.log" };
+
+	bless $self, $class;
+	$self->dump_info;
+
+	return $self;
+}
+
+sub port
+{
+	my ($self) = @_;
+	return $self->{_port};
+}
+
+sub host
+{
+	my ($self) = @_;
+	return $self->{_host};
+}
+
+sub basedir
+{
+	my ($self) = @_;
+	return $self->{_basedir};
+}
+
+sub applname
+{
+	my ($self) = @_;
+	return $self->{_applname};
+}
+
+sub logfile
+{
+	my ($self) = @_;
+	return $self->{_logfile};
+}
+
+sub connstr
+{
+	my ($self, $dbname) = @_;
+	my $pgport = $self->port;
+	my $pghost = $self->host;
+	if (!defined($dbname))
+	{
+		return "port=$pgport host=$pghost";
+	}
+	return "port=$pgport host=$pghost dbname=$dbname";
+}
+
+sub data_dir
+{
+	my ($self) = @_;
+	my $res = $self->basedir;
+	return "$res/pgdata";
+}
+
+sub archive_dir
+{
+	my ($self) = @_;
+	my $basedir = $self->basedir;
+	return "$basedir/archives";
+}
+
+sub backup_dir
+{
+	my ($self) = @_;
+	my $basedir = $self->basedir;
+	return "$basedir/backup";
+}
+
+# Dump node information
+sub dump_info
+{
+	my ($self) = @_;
+	print 'Data directory: ' . $self->data_dir . "\n";
+	print 'Backup directory: ' . $self->backup_dir . "\n";
+	print 'Archive directory: ' . $self->archive_dir . "\n";
+	print 'Connection string: ' . $self->connstr . "\n";
+	print 'Application name: ' . $self->applname . "\n";
+	print 'Log file: ' . $self->logfile . "\n";
+}
+
+sub set_replication_conf
+{
+	my ($self) = @_;
+	my $pgdata = $self->data_dir;
+
+	open my $hba, ">>$pgdata/pg_hba.conf";
+	print $hba "\n# Allow replication (set up by PostgresNode.pm)\n";
+	if (!$windows_os)
+	{
+		print $hba "local replication all trust\n";
+	}
+	else
+	{
+		print $hba
+"host replication all 127.0.0.1/32 sspi include_realm=1 map=regress\n";
+	}
+	close $hba;
+}
+
+# Initialize a new cluster for testing.
+#
+# Authentication is set up so that only the current OS user can access the
+# cluster. On Unix, we use Unix domain socket connections, with the socket in
+# a directory that's only accessible to the current user to ensure that.
+# On Windows, we use SSPI authentication to ensure the same (by pg_regress
+# --config-auth).
+sub init
+{
+	my ($self, %params) = @_;
+	my $port   = $self->port;
+	my $pgdata = $self->data_dir;
+	my $host   = $self->host;
+
+	$params{hba_permit_replication} = 1 if (!defined($params{hba_permit_replication}));
+
+	mkdir $self->backup_dir;
+	mkdir $self->archive_dir;
+
+	system_or_bail('initdb', '-D', $pgdata, '-A', 'trust', '-N');
+	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
+
+	open my $conf, ">>$pgdata/postgresql.conf";
+	print $conf "\n# Added by TestLib.pm)\n";
+	print $conf "fsync = off\n";
+	print $conf "log_statement = all\n";
+	print $conf "port = $port\n";
+	if ($windows_os)
+	{
+		print $conf "listen_addresses = '$host'\n";
+	}
+	else
+	{
+		print $conf "unix_socket_directories = '$host'\n";
+		print $conf "listen_addresses = ''\n";
+	}
+	close $conf;
+
+	$self->set_replication_conf if ($params{hba_permit_replication});
+}
+
+sub append_conf
+{
+	my ($self, $filename, $str) = @_;
+
+	my $conffile = $self->data_dir . '/' . $filename;
+
+	append_to_file($conffile, $str);
+}
+
+sub backup
+{
+	my ($self, $backup_name) = @_;
+	my $backup_path = $self->backup_dir . '/' . $backup_name;
+	my $port        = $self->port;
+
+	print "# Taking backup $backup_name from node with port $port\n";
+	system_or_bail("pg_basebackup -D $backup_path -p $port -x");
+	print "# Backup finished\n";
+}
+
+sub init_from_backup
+{
+	my ($self, $root_node, $backup_name) = @_;
+	my $backup_path = $root_node->backup_dir . '/' . $backup_name;
+	my $port        = $self->port;
+	my $root_port   = $root_node->port;
+
+	print
+"Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	die "Backup $backup_path does not exist" unless -d $backup_path;
+
+	mkdir $self->backup_dir;
+	mkdir $self->archive_dir;
+
+	my $data_path = $self->data_dir;
+	rmdir($data_path);
+	RecursiveCopy::copypath($backup_path, $data_path);
+	chmod(0700, $data_path);
+
+	# Base configuration for this node
+	$self->append_conf('postgresql.conf',
+		qq(
+port = $port
+));
+	$self->set_replication_conf;
+}
+
+sub start
+{
+	my ($self) = @_;
+	my $port   = $self->port;
+	my $pgdata = $self->data_dir;
+	print("### Starting test server in $pgdata\n");
+	my $ret = system_log('pg_ctl', '-w', '-D', $self->data_dir,
+		'-l', $self->logfile, 'start');
+
+	if ($ret != 0)
+	{
+		print "# pg_ctl failed; logfile:\n";
+		print slurp_file($self->logfile);
+		BAIL_OUT("pg_ctl failed");
+	}
+}
+
+sub stop
+{
+	my ($self, $mode) = @_;
+	my $port   = $self->port;
+	my $pgdata = $self->data_dir;
+	$mode = 'fast' if (!defined($mode));
+	print "### Stopping node in $pgdata with port $port using mode $mode\n";
+	system_log('pg_ctl', '-D', $pgdata, '-m', $mode, 'stop');
+}
+
+sub restart
+{
+	my ($self)  = @_;
+	my $port    = $self->port;
+	my $pgdata  = $self->data_dir;
+	my $logfile = $self->logfile;
+	system_log('pg_ctl', '-D', $pgdata, '-w', '-l', $logfile, 'restart');
+}
+
+
+#
+# Cluster management functions
+#
+
+# Build a new PostgresNode object, assigning a free port number.
+#
+# We also register the node, to avoid the port number from being reused
+# for another node even when this one is not active.
+sub get_new_node
+{
+	my $found = 0;
+	my $port  = $last_port_assigned;
+
+	while ($found == 0)
+	{
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		if (!run_log([ 'pg_isready', '-p', $port ]))
+		{
+			$found = 1;
+
+			# Found a potential candidate port number.  Check first that it is
+			# not included in the list of registered nodes.
+			foreach my $node (@all_nodes)
+			{
+				$found = 0 if ($node->port == $port);
+			}
+		}
+	}
+
+	print "# Found free port $port\n";
+
+	# Lock port number found by creating a new node
+	my $node = new PostgresNode($test_pghost, $port);
+
+	# Add node to list of nodes currently in use
+	push(@all_nodes,    $node);
+	push(@active_nodes, $node);
+	$last_port_assigned = $port;
+
+	return $node;
+}
+
+sub teardown_node
+{
+	my $node = shift;
+
+	$node->stop('immediate');
+	@active_nodes = grep { $_ ne $node } @active_nodes;
+}
+
+#
+# Utility functions
+#
+
+sub psql
+{
+	my ($dbname, $sql) = @_;
+	my ($stdout, $stderr);
+	print("# Running SQL command: $sql\n");
+	IPC::Run::run [ 'psql', '-X', '-A', '-t', '-q', '-d', $dbname, '-f', '-' ],
+	  '<', \$sql, '>', \$stdout, '2>', \$stderr
+	  or die;
+	if ($stderr ne "")
+	{
+		print "#### Begin standard error\n";
+		print $stderr;
+		print "#### End standard error\n";
+	}
+	chomp $stdout;
+	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+	return $stdout;
+}
+
+# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
+sub poll_query_until
+{
+	my ($node, $query) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', $query, '-d', $node->connstr() ];
+		my $result = IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
+#
+# Test function working on a node.
+#
+sub issues_sql_like
+{
+	my ($node, $cmd, $expected_sql, $test_name) = @_;
+	truncate $node->logfile, 0;
+	my $result = run_log($cmd);
+	ok($result, "@$cmd exit code 0");
+	my $log = slurp_file($node->logfile);
+	like($log, $expected_sql, "$test_name: SQL found in server log");
+}
+
+1;
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
new file mode 100644
index 0000000..4e58ad3
--- /dev/null
+++ b/src/test/perl/RecursiveCopy.pm
@@ -0,0 +1,42 @@
+# RecursiveCopy, a simple recursive copy implementation
+package RecursiveCopy;
+
+use strict;
+use warnings;
+
+use File::Basename;
+use File::Copy;
+
+sub copypath
+{
+	my $srcpath  = shift;
+	my $destpath = shift;
+
+	die "Cannot operate on symlinks" if -l $srcpath or -l $destpath;
+
+	# This source path is a file, simply copy it to destination with the
+	# same name.
+	die "Destination path $destpath exists as file" if -f $destpath;
+	if (-f $srcpath)
+	{
+		copy($srcpath, $destpath)
+			or die "copy $srcpath -> $destpath failed: $!";
+		return 1;
+	}
+
+	die "Destination needs to be a directory" unless -d $srcpath;
+	mkdir($destpath) or die "mkdir($destpath) failed: $!";
+
+	# Scan existing source directory and recursively copy everything.
+	opendir(my $directory, $srcpath) or die "could not opendir($srcpath): $!";
+	while (my $entry = readdir($directory))
+	{
+		next if ($entry eq '.' || $entry eq '..');
+		RecursiveCopy::copypath("$srcpath/$entry", "$destpath/$entry")
+			or die "copypath $srcpath/$entry -> $destpath/$entry failed";
+	}
+	closedir($directory);
+	return 1;
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..19f9727 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -1,3 +1,10 @@
+# TestLib, low-level routines and actions regression tests.
+#
+# This module contains a set of routines dedicated to environment setup for
+# a PostgreSQL regression test tun, and includes some low-level routines
+# aimed at controlling command execution, logging and test functions. This
+# module should never depend on any other PostgreSQL regression test modules.
+
 package TestLib;
 
 use strict;
@@ -5,19 +12,20 @@ use warnings;
 
 use Config;
 use Exporter 'import';
+use File::Basename;
+use File::Spec;
+use File::Temp ();
+use IPC::Run qw(run);
+use SimpleTee;
+use Test::More;
+
 our @EXPORT = qw(
-  tempdir
-  tempdir_short
-  standard_initdb
-  configure_hba_for_replication
-  start_test_server
-  restart_test_server
-  psql
-  slurp_dir
-  slurp_file
   system_or_bail
   system_log
   run_log
+  slurp_dir
+  slurp_file
+  append_to_file
 
   command_ok
   command_fails
@@ -26,88 +34,59 @@ our @EXPORT = qw(
   program_version_ok
   program_options_handling_ok
   command_like
-  issues_sql_like
 
-  $tmp_check
-  $log_path
   $windows_os
 );
 
-use Cwd;
-use File::Basename;
-use File::Spec;
-use File::Temp ();
-use IPC::Run qw(run start);
+our ($windows_os, $tmp_check, $log_path, $test_logfile);
 
-use SimpleTee;
+INIT
+{
+	$windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
 
-use Test::More;
+	# Determine output directories, and create them.  The base path is the
+	# TESTDIR environment variable, which is normally set by the invoking
+	# Makefile.
+	$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
+	$log_path = "$tmp_check/log";
 
-our $windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
-
-# Open log file. For each test, the log file name uses the name of the
-# file launching this module, without the .pl suffix.
-our ($tmp_check, $log_path);
-$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
-$log_path = "$tmp_check/log";
-mkdir $tmp_check;
-mkdir $log_path;
-my $test_logfile = basename($0);
-$test_logfile =~ s/\.[^.]+$//;
-$test_logfile = "$log_path/regress_log_$test_logfile";
-open TESTLOG, '>', $test_logfile or die "Cannot open STDOUT to logfile: $!";
-
-# Hijack STDOUT and STDERR to the log file
-open(ORIG_STDOUT, ">&STDOUT");
-open(ORIG_STDERR, ">&STDERR");
-open(STDOUT, ">&TESTLOG");
-open(STDERR, ">&TESTLOG");
-
-# The test output (ok ...) needs to be printed to the original STDOUT so
-# that the 'prove' program can parse it, and display it to the user in
-# real time. But also copy it to the log file, to provide more context
-# in the log.
-my $builder = Test::More->builder;
-my $fh = $builder->output;
-tie *$fh, "SimpleTee", *ORIG_STDOUT, *TESTLOG;
-$fh = $builder->failure_output;
-tie *$fh, "SimpleTee", *ORIG_STDERR, *TESTLOG;
-
-# Enable auto-flushing for all the file handles. Stderr and stdout are
-# redirected to the same file, and buffering causes the lines to appear
-# in the log in confusing order.
-autoflush STDOUT 1;
-autoflush STDERR 1;
-autoflush TESTLOG 1;
-
-# Set to untranslated messages, to be able to compare program output
-# with expected strings.
-delete $ENV{LANGUAGE};
-delete $ENV{LC_ALL};
-$ENV{LC_MESSAGES} = 'C';
-
-delete $ENV{PGCONNECT_TIMEOUT};
-delete $ENV{PGDATA};
-delete $ENV{PGDATABASE};
-delete $ENV{PGHOSTADDR};
-delete $ENV{PGREQUIRESSL};
-delete $ENV{PGSERVICE};
-delete $ENV{PGSSLMODE};
-delete $ENV{PGUSER};
-
-if (!$ENV{PGPORT})
-{
-	$ENV{PGPORT} = 65432;
-}
+	mkdir $tmp_check;
+	mkdir $log_path;
+
+	# Open the test log file, whose name depends on the test name.
+	$test_logfile = basename($0);
+	$test_logfile =~ s/\.[^.]+$//;
+	$test_logfile = "$log_path/regress_log_$test_logfile";
+	open TESTLOG, '>', $test_logfile
+	  or die "could not open STDOUT to logfile \"$test_logfile\": $!";
 
-$ENV{PGPORT} = int($ENV{PGPORT}) % 65536;
+	# Hijack STDOUT and STDERR to the log file
+	open(ORIG_STDOUT, ">&STDOUT");
+	open(ORIG_STDERR, ">&STDERR");
+	open(STDOUT,      ">&TESTLOG");
+	open(STDERR,      ">&TESTLOG");
 
+	# The test output (ok ...) needs to be printed to the original STDOUT so
+	# that the 'prove' program can parse it, and display it to the user in
+	# real time. But also copy it to the log file, to provide more context
+	# in the log.
+	my $builder = Test::More->builder;
+	my $fh      = $builder->output;
+	tie *$fh, "SimpleTee", *ORIG_STDOUT, *TESTLOG;
+	$fh = $builder->failure_output;
+	tie *$fh, "SimpleTee", *ORIG_STDERR, *TESTLOG;
+
+	# Enable auto-flushing for all the file handles. Stderr and stdout are
+	# redirected to the same file, and buffering causes the lines to appear
+	# in the log in confusing order.
+	autoflush STDOUT 1;
+	autoflush STDERR 1;
+	autoflush TESTLOG 1;
+}
 
 #
 # Helper functions
 #
-
-
 sub tempdir
 {
 	return File::Temp::tempdir(
@@ -118,123 +97,36 @@ sub tempdir
 
 sub tempdir_short
 {
-
 	# Use a separate temp dir outside the build tree for the
 	# Unix-domain socket, to avoid file name length issues.
 	return File::Temp::tempdir(CLEANUP => 1);
 }
 
-# Initialize a new cluster for testing.
-#
-# The PGHOST environment variable is set to connect to the new cluster.
-#
-# Authentication is set up so that only the current OS user can access the
-# cluster. On Unix, we use Unix domain socket connections, with the socket in
-# a directory that's only accessible to the current user to ensure that.
-# On Windows, we use SSPI authentication to ensure the same (by pg_regress
-# --config-auth).
-sub standard_initdb
-{
-	my $pgdata = shift;
-	system_or_bail('initdb', '-D', "$pgdata", '-A' , 'trust', '-N');
-	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
-
-	my $tempdir_short = tempdir_short;
-
-	open CONF, ">>$pgdata/postgresql.conf";
-	print CONF "\n# Added by TestLib.pm)\n";
-	print CONF "fsync = off\n";
-	if ($windows_os)
-	{
-		print CONF "listen_addresses = '127.0.0.1'\n";
-	}
-	else
-	{
-		print CONF "unix_socket_directories = '$tempdir_short'\n";
-		print CONF "listen_addresses = ''\n";
-	}
-	close CONF;
-
-	$ENV{PGHOST}         = $windows_os ? "127.0.0.1" : $tempdir_short;
-}
-
-# Set up the cluster to allow replication connections, in the same way that
-# standard_initdb does for normal connections.
-sub configure_hba_for_replication
-{
-	my $pgdata = shift;
-
-	open HBA, ">>$pgdata/pg_hba.conf";
-	print HBA "\n# Allow replication (set up by TestLib.pm)\n";
-	if (! $windows_os)
-	{
-		print HBA "local replication all trust\n";
-	}
-	else
-	{
-		print HBA "host replication all 127.0.0.1/32 sspi include_realm=1 map=regress\n";
-	}
-	close HBA;
-}
-
-my ($test_server_datadir, $test_server_logfile);
-
-
-# Initialize a new cluster for testing in given directory, and start it.
-sub start_test_server
-{
-	my ($tempdir) = @_;
-	my $ret;
-
-	print("### Starting test server in $tempdir\n");
-	standard_initdb "$tempdir/pgdata";
-
-	$ret = system_log('pg_ctl', '-D', "$tempdir/pgdata", '-w', '-l',
-	  "$log_path/postmaster.log", '-o', "--log-statement=all",
-	  'start');
-
-	if ($ret != 0)
-	{
-		print "# pg_ctl failed; logfile:\n";
-		system('cat', "$log_path/postmaster.log");
-		BAIL_OUT("pg_ctl failed");
-	}
-
-	$test_server_datadir = "$tempdir/pgdata";
-	$test_server_logfile = "$log_path/postmaster.log";
-}
-
-sub restart_test_server
+sub system_log
 {
-	print("### Restarting test server\n");
-	system_log('pg_ctl', '-D', $test_server_datadir, '-w', '-l',
-	  $test_server_logfile, 'restart');
+	print("# Running: " . join(" ", @_) . "\n");
+	return system(@_);
 }
 
-END
+sub system_or_bail
 {
-	if ($test_server_datadir)
+	if (system_log(@_) != 0)
 	{
-		system_log('pg_ctl', '-D', $test_server_datadir, '-m',
-		  'immediate', 'stop');
+		BAIL_OUT("system $_[0] failed");
 	}
 }
 
-sub psql
+sub run_log
 {
-	my ($dbname, $sql) = @_;
-	my ($stdout, $stderr);
-	print("# Running SQL command: $sql\n");
-	run [ 'psql', '-X', '-A', '-t', '-q', '-d', $dbname, '-f', '-' ], '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
-	chomp $stdout;
-	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-	return $stdout;
+	print("# Running: " . join(" ", @{ $_[0] }) . "\n");
+	return run(@_);
 }
 
 sub slurp_dir
 {
 	my ($dir) = @_;
-	opendir(my $dh, $dir) or die;
+	opendir(my $dh, $dir)
+	  or die "could not opendir \"$dir\": $!";
 	my @direntries = readdir $dh;
 	closedir $dh;
 	return @direntries;
@@ -249,32 +141,18 @@ sub slurp_file
 	return $contents;
 }
 
-sub system_or_bail
+sub append_to_file
 {
-	if (system_log(@_) != 0)
-	{
-		BAIL_OUT("system $_[0] failed: $?");
-	}
-}
+	my ($filename, $str) = @_;
 
-sub system_log
-{
-	print("# Running: " . join(" ", @_) ."\n");
-	return system(@_);
+	open my $fh, ">>", $filename or die "could not open \"$filename\": $!";
+	print $fh $str;
+	close $fh;
 }
 
-sub run_log
-{
-	print("# Running: " . join(" ", @{$_[0]}) ."\n");
-	return run (@_);
-}
-
-
 #
 # Test functions
 #
-
-
 sub command_ok
 {
 	my ($cmd, $test_name) = @_;
@@ -292,8 +170,8 @@ sub command_fails
 sub command_exit_is
 {
 	my ($cmd, $expected, $test_name) = @_;
-	print("# Running: " . join(" ", @{$cmd}) ."\n");
-	my $h = start $cmd;
+	print("# Running: " . join(" ", @{$cmd}) . "\n");
+	my $h = IPC::Run::start $cmd;
 	$h->finish();
 
 	# On Windows, the exit status of the process is returned directly as the
@@ -303,8 +181,10 @@ sub command_exit_is
 	# assuming the Unix convention, which will always return 0 on Windows as
 	# long as the process was not terminated by an exception. To work around
 	# that, use $h->full_result on Windows instead.
-	my $result = ($Config{osname} eq "MSWin32") ?
-		($h->full_results)[0] : $h->result(0);
+	my $result =
+	    ($Config{osname} eq "MSWin32")
+	  ? ($h->full_results)[0]
+	  : $h->result(0);
 	is($result, $expected, $test_name);
 }
 
@@ -313,7 +193,7 @@ sub program_help_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --help\n");
-	my $result = run [ $cmd, '--help' ], '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--help' ], '>', \$stdout, '2>', \$stderr;
 	ok($result, "$cmd --help exit code 0");
 	isnt($stdout, '', "$cmd --help goes to stdout");
 	is($stderr, '', "$cmd --help nothing to stderr");
@@ -324,7 +204,7 @@ sub program_version_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --version\n");
-	my $result = run [ $cmd, '--version' ], '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--version' ], '>', \$stdout, '2>', \$stderr;
 	ok($result, "$cmd --version exit code 0");
 	isnt($stdout, '', "$cmd --version goes to stdout");
 	is($stderr, '', "$cmd --version nothing to stderr");
@@ -335,8 +215,8 @@ sub program_options_handling_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --not-a-valid-option\n");
-	my $result = run [ $cmd, '--not-a-valid-option' ], '>', \$stdout, '2>',
-	  \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--not-a-valid-option' ], '>', \$stdout,
+	  '2>', \$stderr;
 	ok(!$result, "$cmd with invalid option nonzero exit code");
 	isnt($stderr, '', "$cmd with invalid option prints error message");
 }
@@ -346,20 +226,10 @@ sub command_like
 	my ($cmd, $expected_stdout, $test_name) = @_;
 	my ($stdout, $stderr);
 	print("# Running: " . join(" ", @{$cmd}) . "\n");
-	my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr;
 	ok($result, "@$cmd exit code 0");
 	is($stderr, '', "@$cmd no stderr");
 	like($stdout, $expected_stdout, "$test_name: matches");
 }
 
-sub issues_sql_like
-{
-	my ($cmd, $expected_sql, $test_name) = @_;
-	truncate $test_server_logfile, 0;
-	my $result = run_log($cmd);
-	ok($result, "@$cmd exit code 0");
-	my $log = slurp_file($test_server_logfile);
-	like($log, $expected_sql, "$test_name: SQL found in server log");
-}
-
 1;
diff --git a/src/test/ssl/ServerSetup.pm b/src/test/ssl/ServerSetup.pm
index a6c77b5..c6b9f9b 100644
--- a/src/test/ssl/ServerSetup.pm
+++ b/src/test/ssl/ServerSetup.pm
@@ -18,6 +18,7 @@ package ServerSetup;
 
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use File::Basename;
 use File::Copy;
@@ -45,7 +46,7 @@ sub copy_files
 
 sub configure_test_server_for_ssl
 {
-	my $tempdir    = $_[0];
+	my $pgdata     = $_[0];
 	my $serverhost = $_[1];
 
 	# Create test users and databases
@@ -55,7 +56,7 @@ sub configure_test_server_for_ssl
 	psql 'postgres', "CREATE DATABASE certdb";
 
 	# enable logging etc.
-	open CONF, ">>$tempdir/pgdata/postgresql.conf";
+	open CONF, ">>$pgdata/postgresql.conf";
 	print CONF "fsync=off\n";
 	print CONF "log_connections=on\n";
 	print CONF "log_hostname=on\n";
@@ -68,17 +69,17 @@ sub configure_test_server_for_ssl
 	close CONF;
 
 # Copy all server certificates and keys, and client root cert, to the data dir
-	copy_files("ssl/server-*.crt", "$tempdir/pgdata");
-	copy_files("ssl/server-*.key", "$tempdir/pgdata");
-	chmod(0600, glob "$tempdir/pgdata/server-*.key") or die $!;
-	copy_files("ssl/root+client_ca.crt", "$tempdir/pgdata");
-	copy_files("ssl/root+client.crl",    "$tempdir/pgdata");
+	copy_files("ssl/server-*.crt", $pgdata);
+	copy_files("ssl/server-*.key", $pgdata);
+	chmod(0600, glob "$pgdata/server-*.key") or die $!;
+	copy_files("ssl/root+client_ca.crt", $pgdata);
+	copy_files("ssl/root+client.crl",    $pgdata);
 
   # Only accept SSL connections from localhost. Our tests don't depend on this
   # but seems best to keep it as narrow as possible for security reasons.
   #
   # When connecting to certdb, also check the client certificate.
-	open HBA, ">$tempdir/pgdata/pg_hba.conf";
+	open HBA, ">$pgdata/pg_hba.conf";
 	print HBA
 "# TYPE  DATABASE        USER            ADDRESS                 METHOD\n";
 	print HBA
@@ -96,12 +97,13 @@ sub configure_test_server_for_ssl
 # the server so that the configuration takes effect.
 sub switch_server_cert
 {
-	my $tempdir  = $_[0];
+	my $node     = $_[0];
 	my $certfile = $_[1];
+	my $pgdata   = $node->data_dir;
 
 	diag "Restarting server with certfile \"$certfile\"...";
 
-	open SSLCONF, ">$tempdir/pgdata/sslconfig.conf";
+	open SSLCONF, ">$pgdata/sslconfig.conf";
 	print SSLCONF "ssl=on\n";
 	print SSLCONF "ssl_ca_file='root+client_ca.crt'\n";
 	print SSLCONF "ssl_cert_file='$certfile.crt'\n";
@@ -110,5 +112,5 @@ sub switch_server_cert
 	close SSLCONF;
 
 	# Stop and restart server to reload the new config.
-	restart_test_server();
+	$node->restart;
 }
diff --git a/src/test/ssl/t/001_ssltests.pl b/src/test/ssl/t/001_ssltests.pl
index 0d6f339..269ccf8 100644
--- a/src/test/ssl/t/001_ssltests.pl
+++ b/src/test/ssl/t/001_ssltests.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+use PostgresNode;
+use TestLib;
 use TestLib;
 use Test::More tests => 38;
 use ServerSetup;
@@ -25,8 +27,6 @@ BEGIN
 # postgresql-ssl-regression.test.
 my $SERVERHOSTADDR = '127.0.0.1';
 
-my $tempdir = TestLib::tempdir;
-
 # Define a couple of helper functions to test connecting to the server.
 
 my $common_connstr;
@@ -74,10 +74,16 @@ chmod 0600, "ssl/client.key";
 
 #### Part 0. Set up the server.
 
-diag "setting up data directory in \"$tempdir\"...";
-start_test_server($tempdir);
-configure_test_server_for_ssl($tempdir, $SERVERHOSTADDR);
-switch_server_cert($tempdir, 'server-cn-only');
+diag "setting up data directory...";
+my $node = get_new_node();
+$node->init;
+# PGHOST is enforced here to set up the node, subsequent connections
+# will use a dedicated connection string.
+$ENV{PGHOST} = $node->host;
+$ENV{PGPORT} = $node->port;
+$node->start;
+configure_test_server_for_ssl($node->data_dir, $SERVERHOSTADDR);
+switch_server_cert($node, 'server-cn-only');
 
 ### Part 1. Run client-side tests.
 ###
@@ -150,7 +156,7 @@ test_connect_ok("sslmode=verify-ca host=wronghost.test");
 test_connect_fails("sslmode=verify-full host=wronghost.test");
 
 # Test Subject Alternative Names.
-switch_server_cert($tempdir, 'server-multiple-alt-names');
+switch_server_cert($node, 'server-multiple-alt-names');
 
 diag "test hostname matching with X509 Subject Alternative Names";
 $common_connstr =
@@ -165,7 +171,7 @@ test_connect_fails("host=deep.subdomain.wildcard.pg-ssltest.test");
 
 # Test certificate with a single Subject Alternative Name. (this gives a
 # slightly different error message, that's all)
-switch_server_cert($tempdir, 'server-single-alt-name');
+switch_server_cert($node, 'server-single-alt-name');
 
 diag "test hostname matching with a single X509 Subject Alternative Name";
 $common_connstr =
@@ -178,7 +184,7 @@ test_connect_fails("host=deep.subdomain.wildcard.pg-ssltest.test");
 
 # Test server certificate with a CN and SANs. Per RFCs 2818 and 6125, the CN
 # should be ignored when the certificate has both.
-switch_server_cert($tempdir, 'server-cn-and-alt-names');
+switch_server_cert($node, 'server-cn-and-alt-names');
 
 diag "test certificate with both a CN and SANs";
 $common_connstr =
@@ -190,7 +196,7 @@ test_connect_fails("host=common-name.pg-ssltest.test");
 
 # Finally, test a server certificate that has no CN or SANs. Of course, that's
 # not a very sensible certificate, but libpq should handle it gracefully.
-switch_server_cert($tempdir, 'server-no-names');
+switch_server_cert($node, 'server-no-names');
 $common_connstr =
 "user=ssltestuser dbname=trustdb sslcert=invalid sslrootcert=ssl/root+server_ca.crt hostaddr=$SERVERHOSTADDR";
 
@@ -199,7 +205,7 @@ test_connect_fails("sslmode=verify-full host=common-name.pg-ssltest.test");
 
 # Test that the CRL works
 diag "Testing client-side CRL";
-switch_server_cert($tempdir, 'server-revoked');
+switch_server_cert($node, 'server-revoked');
 
 $common_connstr =
 "user=ssltestuser dbname=trustdb sslcert=invalid hostaddr=$SERVERHOSTADDR host=common-name.pg-ssltest.test";
@@ -233,7 +239,3 @@ test_connect_fails(
 test_connect_fails(
 "user=ssltestuser sslcert=ssl/client-revoked.crt sslkey=ssl/client-revoked.key"
 );
-
-
-# All done! Save the log, before the temporary installation is deleted
-copy("$tempdir/client-log", "./client-log");
#88Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#87)
2 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

OK... I have merged TestLib and PostgresNode of the previous patch
into PostgresNode into the way suggested by Noah. TestBase has been
renamed back to TestLib, and includes as well the base test functions
like command_ok.

Great, thanks. Here's one more version, hopefully the last one.

- I discovered that not setting PGPORT was causing some of the tests
that fail (using command_fails) to fail to test what was being tested.
The problem is that the command was failing with "could not connect to
server" instead of failing because of trying to cluster a nonexistant
table, etc. Unfortunately the only way to verify this is by looking
at the regress_log_xxx_foobar file. Not ideal, but not this patch's
fault.

- I changed the routines moved to PostgresNode so that they are instance
methods, i.e. $node->poll_until_query; also psql and
issues_query_like. The latter also sets "local $PGPORT" so that the
command is run with the correct port.

- It would be nice to have command_ok and command_fails in PostgresNode
too; that would remove the need for setting $ENV{PGPORT} but it's
possible to run commands outside a node too, so we'd need duplicates,
which would be worse.

- I changed start/stop/restart so that they keep track of the postmaster
PID; also added a DESTROY sub to PostgresNode that sends SIGQUIT.
This means that when the test finishes, the server gets an immediate
stop signal. We were getting a lot of errors in the server log about
failing to write to the stats file otherwise, until the node noticed
that the datadir was gone.

- I removed the @active_nodes array, which is now unnecessary (per
above).

- I moved the "delete $ENV{PGxxx}" back to a BEGIN block in TestLib.
I did it because it's possible to write test scripts without
PostgresNode, and it's not nice to have those pick up the environment.
This still affects anything using PostgresNode because that one uses
TestLib.

Finally, I ran perltidy on all the files, which strangely changed stuff
that I didn't expect it to change. I wonder if this is related to the
perltidy version. Do you see further changes if you run perltidy on the
patched tree? This is my version:

$ perltidy --version
This is perltidy, v20140328

Copyright 2000-2014, Steve Hancock

Perltidy is free software and may be copied under the terms of the GNU
General Public License, which is included in the distribution files.

Complete documentation for perltidy can be found using 'man perltidy'
or on the internet at http://perltidy.sourceforge.net.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

20151201_tapcheck_v13.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/bin/initdb/t/001_initdb.pl b/src/bin/initdb/t/001_initdb.pl
index 299dcf5..f64186d 100644
--- a/src/bin/initdb/t/001_initdb.pl
+++ b/src/bin/initdb/t/001_initdb.pl
@@ -4,6 +4,7 @@
 
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 14;
 
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index dc96bbf..86cc14e 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -2,6 +2,7 @@ use strict;
 use warnings;
 use Cwd;
 use Config;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 51;
 
@@ -9,8 +10,15 @@ program_help_ok('pg_basebackup');
 program_version_ok('pg_basebackup');
 program_options_handling_ok('pg_basebackup');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $tempdir = TestLib::tempdir;
+
+my $node = get_new_node();
+# Initialize node without replication settings
+$node->init(hba_permit_replication => 0);
+$node->start;
+my $pgdata = $node->data_dir;
+
+$ENV{PGPORT} = $node->port;
 
 command_fails(['pg_basebackup'],
 	'pg_basebackup needs target directory specified');
@@ -26,19 +34,19 @@ if (open BADCHARS, ">>$tempdir/pgdata/FOO\xe0\xe0\xe0BAR")
 	close BADCHARS;
 }
 
-configure_hba_for_replication "$tempdir/pgdata";
-system_or_bail 'pg_ctl', '-D', "$tempdir/pgdata", 'reload';
+$node->set_replication_conf();
+system_or_bail 'pg_ctl', '-D', $pgdata, 'reload';
 
 command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup fails because of WAL configuration');
 
-open CONF, ">>$tempdir/pgdata/postgresql.conf";
+open CONF, ">>$pgdata/postgresql.conf";
 print CONF "max_replication_slots = 10\n";
 print CONF "max_wal_senders = 10\n";
 print CONF "wal_level = archive\n";
 close CONF;
-restart_test_server;
+$node->restart;
 
 command_ok([ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup runs');
@@ -81,13 +89,13 @@ command_fails(
 
 # Tar format doesn't support filenames longer than 100 bytes.
 my $superlongname = "superlongname_" . ("x" x 100);
-my $superlongpath = "$tempdir/pgdata/$superlongname";
+my $superlongpath = "$pgdata/$superlongname";
 
 open FILE, ">$superlongpath" or die "unable to create file $superlongpath";
 close FILE;
 command_fails([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l1", '-Ft' ],
 	'pg_basebackup tar with long name fails');
-unlink "$tempdir/pgdata/$superlongname";
+unlink "$pgdata/$superlongname";
 
 # The following tests test symlinks. Windows doesn't have symlinks, so
 # skip on Windows.
@@ -98,13 +106,13 @@ SKIP: {
 	# to our physical temp location.  That way we can use shorter names
 	# for the tablespace directories, which hopefully won't run afoul of
 	# the 99 character length limit.
-	my $shorter_tempdir = tempdir_short . "/tempdir";
+	my $shorter_tempdir = TestLib::tempdir_short . "/tempdir";
 	symlink "$tempdir", $shorter_tempdir;
 
 	mkdir "$tempdir/tblspc1";
-	psql 'postgres',
-	"CREATE TABLESPACE tblspc1 LOCATION '$shorter_tempdir/tblspc1';";
-	psql 'postgres', "CREATE TABLE test1 (a int) TABLESPACE tblspc1;";
+	$node->psql('postgres',
+	"CREATE TABLESPACE tblspc1 LOCATION '$shorter_tempdir/tblspc1';");
+	$node->psql('postgres', "CREATE TABLE test1 (a int) TABLESPACE tblspc1;");
 	command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup2", '-Ft' ],
 			   'tar format with tablespaces');
 	ok(-f "$tempdir/tarbackup2/base.tar", 'backup tar was created');
@@ -120,7 +128,7 @@ SKIP: {
 			"-T$shorter_tempdir/tblspc1=$tempdir/tbackup/tblspc1" ],
 		'plain format with tablespaces succeeds with tablespace mapping');
 	ok(-d "$tempdir/tbackup/tblspc1", 'tablespace was relocated');
-	opendir(my $dh, "$tempdir/pgdata/pg_tblspc") or die;
+	opendir(my $dh, "$pgdata/pg_tblspc") or die;
 	ok( (   grep {
 		-l "$tempdir/backup1/pg_tblspc/$_"
 			and readlink "$tempdir/backup1/pg_tblspc/$_" eq
@@ -130,23 +138,23 @@ SKIP: {
 	closedir $dh;
 
 	mkdir "$tempdir/tbl=spc2";
-	psql 'postgres', "DROP TABLE test1;";
-	psql 'postgres', "DROP TABLESPACE tblspc1;";
-	psql 'postgres',
-	"CREATE TABLESPACE tblspc2 LOCATION '$shorter_tempdir/tbl=spc2';";
+	$node->psql('postgres', "DROP TABLE test1;");
+	$node->psql('postgres', "DROP TABLESPACE tblspc1;");
+	$node->psql('postgres',
+	"CREATE TABLESPACE tblspc2 LOCATION '$shorter_tempdir/tbl=spc2';");
 	command_ok(
 		[   'pg_basebackup', '-D', "$tempdir/backup3", '-Fp',
 			"-T$shorter_tempdir/tbl\\=spc2=$tempdir/tbackup/tbl\\=spc2" ],
 		'mapping tablespace with = sign in path');
 	ok(-d "$tempdir/tbackup/tbl=spc2", 'tablespace with = sign was relocated');
-	psql 'postgres', "DROP TABLESPACE tblspc2;";
+	$node->psql('postgres', "DROP TABLESPACE tblspc2;");
 
 	mkdir "$tempdir/$superlongname";
-	psql 'postgres',
-	"CREATE TABLESPACE tblspc3 LOCATION '$tempdir/$superlongname';";
+	$node->psql('postgres',
+	"CREATE TABLESPACE tblspc3 LOCATION '$tempdir/$superlongname';");
 	command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l3", '-Ft' ],
 			   'pg_basebackup tar with long symlink target');
-	psql 'postgres', "DROP TABLESPACE tblspc3;";
+	$node->psql('postgres', "DROP TABLESPACE tblspc3;");
 }
 
 command_ok([ 'pg_basebackup', '-D', "$tempdir/backupR", '-R' ],
@@ -170,12 +178,12 @@ command_fails([ 'pg_basebackup', '-D', "$tempdir/fail", '-S', 'slot1' ],
 command_fails([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl_fail", '-X', 'stream', '-S', 'slot1' ],
 	'pg_basebackup fails with nonexistent replication slot');
 
-psql 'postgres', q{SELECT * FROM pg_create_physical_replication_slot('slot1')};
-my $lsn = psql 'postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'};
+$node->psql('postgres', q{SELECT * FROM pg_create_physical_replication_slot('slot1')});
+my $lsn = $node->psql('postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'});
 is($lsn, '', 'restart LSN of new slot is null');
 command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl", '-X', 'stream', '-S', 'slot1' ],
 	'pg_basebackup -X stream with replication slot runs');
-$lsn = psql 'postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'};
+$lsn = $node->psql('postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'});
 like($lsn, qr!^0/[0-9A-Z]{7,8}$!, 'restart LSN of slot has advanced');
 
 command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl_R", '-X', 'stream', '-S', 'slot1', '-R' ],
diff --git a/src/bin/pg_controldata/t/001_pg_controldata.pl b/src/bin/pg_controldata/t/001_pg_controldata.pl
index e2b0d42..ae45f41 100644
--- a/src/bin/pg_controldata/t/001_pg_controldata.pl
+++ b/src/bin/pg_controldata/t/001_pg_controldata.pl
@@ -1,16 +1,19 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
-my $tempdir = TestLib::tempdir;
-
 program_help_ok('pg_controldata');
 program_version_ok('pg_controldata');
 program_options_handling_ok('pg_controldata');
 command_fails(['pg_controldata'], 'pg_controldata without arguments fails');
 command_fails([ 'pg_controldata', 'nonexistent' ],
 	'pg_controldata with nonexistent directory fails');
-standard_initdb "$tempdir/data";
-command_like([ 'pg_controldata', "$tempdir/data" ],
+
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+command_like([ 'pg_controldata', $node->data_dir ],
 	qr/checkpoint/, 'pg_controldata produces output');
diff --git a/src/bin/pg_ctl/t/001_start_stop.pl b/src/bin/pg_ctl/t/001_start_stop.pl
index f57abce..a224ece 100644
--- a/src/bin/pg_ctl/t/001_start_stop.pl
+++ b/src/bin/pg_ctl/t/001_start_stop.pl
@@ -1,6 +1,8 @@
 use strict;
 use warnings;
+
 use Config;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 17;
 
diff --git a/src/bin/pg_ctl/t/002_status.pl b/src/bin/pg_ctl/t/002_status.pl
index 31f7c72..f1c131b 100644
--- a/src/bin/pg_ctl/t/002_status.pl
+++ b/src/bin/pg_ctl/t/002_status.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 3;
 
@@ -9,14 +11,15 @@ my $tempdir_short = TestLib::tempdir_short;
 command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/nonexistent" ],
 	4, 'pg_ctl status with nonexistent directory');
 
-standard_initdb "$tempdir/data";
+my $node = get_new_node();
+$node->init;
 
-command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/data" ],
+command_exit_is([ 'pg_ctl', 'status', '-D', $node->data_dir ],
 	3, 'pg_ctl status with server not running');
 
 system_or_bail 'pg_ctl', '-l', "$tempdir/logfile", '-D',
-  "$tempdir/data", '-w', 'start';
-command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/data" ],
+  $node->data_dir, '-w', 'start';
+command_exit_is([ 'pg_ctl', 'status', '-D', $node->data_dir ],
 	0, 'pg_ctl status with server running');
 
-system_or_bail 'pg_ctl', 'stop', '-D', "$tempdir/data", '-m', 'fast';
+system_or_bail 'pg_ctl', 'stop', '-D', $node->data_dir, '-m', 'fast';
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..55bbc9c 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -9,22 +9,20 @@ package RewindTest;
 # To run a test, the test script (in t/ subdirectory) calls the functions
 # in this module. These functions should be called in this sequence:
 #
-# 1. init_rewind_test - sets up log file etc.
+# 1. setup_cluster - creates a PostgreSQL cluster that runs as the master
 #
-# 2. setup_cluster - creates a PostgreSQL cluster that runs as the master
+# 2. start_master - starts the master server
 #
-# 3. start_master - starts the master server
-#
-# 4. create_standby - runs pg_basebackup to initialize a standby server, and
+# 3. create_standby - runs pg_basebackup to initialize a standby server, and
 #    sets it up to follow the master.
 #
-# 5. promote_standby - runs "pg_ctl promote" to promote the standby server.
+# 4. promote_standby - runs "pg_ctl promote" to promote the standby server.
 # The old master keeps running.
 #
-# 6. run_pg_rewind - stops the old master (if it's still running) and runs
+# 5. run_pg_rewind - stops the old master (if it's still running) and runs
 # pg_rewind to synchronize it with the now-promoted standby server.
 #
-# 7. clean_rewind_test - stops both servers used in the test, if they're
+# 6. clean_rewind_test - stops both servers used in the test, if they're
 # still running.
 #
 # The test script can use the helper functions master_psql and standby_psql
@@ -37,27 +35,23 @@ package RewindTest;
 use strict;
 use warnings;
 
+use Config;
+use Exporter 'import';
+use File::Copy;
+use File::Path qw(rmtree);
+use IPC::Run;
+use PostgresNode;
 use TestLib;
 use Test::More;
 
-use Config;
-use File::Copy;
-use File::Path qw(rmtree);
-use IPC::Run qw(run start);
-
-use Exporter 'import';
 our @EXPORT = qw(
-  $connstr_master
-  $connstr_standby
-  $test_master_datadir
-  $test_standby_datadir
+  $node_master
+  $node_standby
 
-  append_to_file
   master_psql
   standby_psql
   check_query
 
-  init_rewind_test
   setup_cluster
   start_master
   create_standby
@@ -66,32 +60,24 @@ our @EXPORT = qw(
   clean_rewind_test
 );
 
-our $test_master_datadir  = "$tmp_check/data_master";
-our $test_standby_datadir = "$tmp_check/data_standby";
-
-# Define non-conflicting ports for both nodes.
-my $port_master  = $ENV{PGPORT};
-my $port_standby = $port_master + 1;
-
-my $connstr_master  = "port=$port_master";
-my $connstr_standby = "port=$port_standby";
-
-$ENV{PGDATABASE} = "postgres";
+# Our nodes.
+our $node_master;
+our $node_standby;
 
 sub master_psql
 {
 	my $cmd = shift;
 
-	system_or_bail 'psql', '-q', '--no-psqlrc', '-d', $connstr_master,
-	  '-c', "$cmd";
+	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
+	  $node_master->connstr('postgres'), '-c', "$cmd";
 }
 
 sub standby_psql
 {
 	my $cmd = shift;
 
-	system_or_bail 'psql', '-q', '--no-psqlrc', '-d', $connstr_standby,
-	  '-c', "$cmd";
+	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
+      $node_standby->connstr('postgres'), '-c', "$cmd";
 }
 
 # Run a query against the master, and check that the output matches what's
@@ -102,9 +88,9 @@ sub check_query
 	my ($stdout, $stderr);
 
 	# we want just the output, no formatting
-	my $result = run [
+	my $result = IPC::Run::run [
 		'psql',          '-q', '-A', '-t', '--no-psqlrc', '-d',
-		$connstr_master, '-c', $query ],
+		$node_master->connstr('postgres'), '-c', $query ],
 	  '>', \$stdout, '2>', \$stderr;
 
 	# We don't use ok() for the exit code and stderr, because we want this
@@ -125,56 +111,14 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
-sub append_to_file
-{
-	my ($filename, $str) = @_;
-
-	open my $fh, ">>", $filename or die "could not open file $filename";
-	print $fh $str;
-	close $fh;
-}
-
 sub setup_cluster
 {
 	# Initialize master, data checksums are mandatory
-	rmtree($test_master_datadir);
-	standard_initdb($test_master_datadir);
+	$node_master = get_new_node();
+	$node_master->init;
 
 	# Custom parameters for master's postgresql.conf
-	append_to_file(
-		"$test_master_datadir/postgresql.conf", qq(
+	$node_master->append_conf("postgresql.conf", qq(
 wal_level = hot_standby
 max_wal_senders = 2
 wal_keep_segments = 20
@@ -185,17 +129,11 @@ hot_standby = on
 autovacuum = off
 max_connections = 10
 ));
-
-	# Accept replication connections on master
-	configure_hba_for_replication $test_master_datadir;
 }
 
 sub start_master
 {
-	system_or_bail('pg_ctl' , '-w',
-				   '-D' , $test_master_datadir,
-				   '-l',  "$log_path/master.log",
-				   "-o", "-p $port_master", 'start');
+	$node_master->start;
 
 	#### Now run the test-specific parts to initialize the master before setting
 	# up standby
@@ -203,24 +141,19 @@ sub start_master
 
 sub create_standby
 {
+	$node_standby = get_new_node();
+	$node_master->backup('my_backup');
+	$node_standby->init_from_backup($node_master, 'my_backup');
+	my $connstr_master = $node_master->connstr('postgres');
 
-	# Set up standby with necessary parameter
-	rmtree $test_standby_datadir;
-
-	# Base backup is taken with xlog files included
-	system_or_bail('pg_basebackup', '-D', $test_standby_datadir,
-				   '-p', $port_master, '-x');
-	append_to_file(
-		"$test_standby_datadir/recovery.conf", qq(
+	$node_standby->append_conf("recovery.conf", qq(
 primary_conninfo='$connstr_master application_name=rewind_standby'
 standby_mode=on
 recovery_target_timeline='latest'
 ));
 
 	# Start standby
-	system_or_bail('pg_ctl', '-w', '-D', $test_standby_datadir,
-				   '-l', "$log_path/standby.log",
-				   '-o', "-p $port_standby", 'start');
+	$node_standby->start;
 
 	# The standby may have WAL to apply before it matches the primary.  That
 	# is fine, because no test examines the standby before promotion.
@@ -234,14 +167,14 @@ sub promote_standby
 	# Wait for the standby to receive and write all WAL.
 	my $wal_received_query =
 "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = 'rewind_standby';";
-	poll_query_until($wal_received_query, $connstr_master)
+	$node_master->poll_query_until('postgres', $wal_received_query)
 	  or die "Timed out while waiting for standby to receive and write WAL";
 
 	# Now promote slave and insert some new data on master, this will put
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
-	system_or_bail('pg_ctl', '-w', '-D', $test_standby_datadir, 'promote');
-	poll_query_until("SELECT NOT pg_is_in_recovery()", $connstr_standby)
+	system_or_bail('pg_ctl', '-w', '-D', $node_standby->data_dir, 'promote');
+	$node_standby->poll_query_until('postgres', "SELECT NOT pg_is_in_recovery()")
 	  or die "Timed out while waiting for promotion of standby";
 
 	# Force a checkpoint after the promotion. pg_rewind looks at the control
@@ -256,9 +189,13 @@ sub promote_standby
 sub run_pg_rewind
 {
 	my $test_mode = shift;
+	my $master_pgdata = $node_master->data_dir;
+	my $standby_pgdata = $node_standby->data_dir;
+	my $standby_connstr = $node_standby->connstr('postgres');
+	my $tmp_folder = TestLib::tempdir;
 
 	# Stop the master and be ready to perform the rewind
-	system_or_bail('pg_ctl', '-D', $test_master_datadir, '-m', 'fast', 'stop');
+	$node_master->stop;
 
 	# At this point, the rewind processing is ready to run.
 	# We now have a very simple scenario with a few diverged WAL record.
@@ -267,20 +204,19 @@ sub run_pg_rewind
 
 	# Keep a temporary postgresql.conf for master node or it would be
 	# overwritten during the rewind.
-	copy("$test_master_datadir/postgresql.conf",
-		 "$tmp_check/master-postgresql.conf.tmp");
+	copy("$master_pgdata/postgresql.conf",
+		 "$tmp_folder/master-postgresql.conf.tmp");
 
 	# Now run pg_rewind
 	if ($test_mode eq "local")
 	{
 		# Do rewind using a local pgdata as source
 		# Stop the master and be ready to perform the rewind
-		system_or_bail('pg_ctl', '-D', $test_standby_datadir,
-					   '-m', 'fast', 'stop');
+		$node_standby->stop;
 		command_ok(['pg_rewind',
 					"--debug",
-					"--source-pgdata=$test_standby_datadir",
-					"--target-pgdata=$test_master_datadir"],
+					"--source-pgdata=$standby_pgdata",
+					"--target-pgdata=$master_pgdata"],
 				   'pg_rewind local');
 	}
 	elsif ($test_mode eq "remote")
@@ -289,33 +225,30 @@ sub run_pg_rewind
 		command_ok(['pg_rewind',
 					"--debug",
 					"--source-server",
-					"port=$port_standby dbname=postgres",
-					"--target-pgdata=$test_master_datadir"],
+					$standby_connstr,
+					"--target-pgdata=$master_pgdata"],
 				   'pg_rewind remote');
 	}
 	else
 	{
-
 		# Cannot come here normally
 		die("Incorrect test mode specified");
 	}
 
 	# Now move back postgresql.conf with old settings
-	move("$tmp_check/master-postgresql.conf.tmp",
-		 "$test_master_datadir/postgresql.conf");
+	move("$tmp_folder/master-postgresql.conf.tmp",
+		 "$master_pgdata/postgresql.conf");
 
 	# Plug-in rewound node to the now-promoted standby node
-	append_to_file(
-		"$test_master_datadir/recovery.conf", qq(
+	my $port_standby = $node_standby->port;
+	$node_master->append_conf('recovery.conf', qq(
 primary_conninfo='port=$port_standby'
 standby_mode=on
 recovery_target_timeline='latest'
 ));
 
 	# Restart the master to check that rewind went correctly
-	system_or_bail('pg_ctl', '-w', '-D', $test_master_datadir,
-				   '-l', "$log_path/master.log",
-				   '-o', "-p $port_master", 'start');
+	$node_master->restart;
 
 	#### Now run the test-specific parts to check the result
 }
@@ -323,22 +256,8 @@ recovery_target_timeline='latest'
 # Clean up after the test. Stop both servers, if they're still running.
 sub clean_rewind_test
 {
-	if ($test_master_datadir)
-	{
-		system
-		  'pg_ctl', '-D', $test_master_datadir, '-m', 'immediate', 'stop';
-	}
-	if ($test_standby_datadir)
-	{
-		system
-		  'pg_ctl', '-D', $test_standby_datadir, '-m', 'immediate', 'stop';
-	}
+	$node_master->teardown_node if defined $node_master;
+	$node_standby->teardown_node if defined $node_standby;
 }
 
-# Stop the test servers, just in case they're still running.
-END
-{
-	my $save_rc = $?;
-	clean_rewind_test();
-	$? = $save_rc;
-}
+1;
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index d317f53..cedde14 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -17,7 +17,7 @@ sub run_test
 	RewindTest::setup_cluster();
 	RewindTest::start_master();
 
-	my $test_master_datadir = $RewindTest::test_master_datadir;
+	my $test_master_datadir = $node_master->data_dir;
 
 	# Create a subdir and files that will be present in both
 	mkdir "$test_master_datadir/tst_both_dir";
@@ -30,6 +30,7 @@ sub run_test
 	RewindTest::create_standby();
 
 	# Create different subdirs and files in master and standby
+	my $test_standby_datadir = $node_standby->data_dir;
 
 	mkdir "$test_standby_datadir/tst_standby_dir";
 	append_to_file "$test_standby_datadir/tst_standby_dir/standby_file1",
diff --git a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
index c5f72e2..bdcab56 100644
--- a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
+++ b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
@@ -23,11 +23,13 @@ sub run_test
 {
 	my $test_mode = shift;
 
-	my $master_xlogdir = "$tmp_check/xlog_master";
+	my $master_xlogdir = "${TestLib::tmp_check}/xlog_master";
 
 	rmtree($master_xlogdir);
 	RewindTest::setup_cluster();
 
+	my $test_master_datadir = $node_master->data_dir;
+
 	# turn pg_xlog into a symlink
 	print("moving $test_master_datadir/pg_xlog to $master_xlogdir\n");
 	move("$test_master_datadir/pg_xlog", $master_xlogdir) or die;
diff --git a/src/bin/scripts/t/010_clusterdb.pl b/src/bin/scripts/t/010_clusterdb.pl
index dc0d78a..756ae38 100644
--- a/src/bin/scripts/t/010_clusterdb.pl
+++ b/src/bin/scripts/t/010_clusterdb.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
@@ -7,20 +9,23 @@ program_help_ok('clusterdb');
 program_version_ok('clusterdb');
 program_options_handling_ok('clusterdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
-	[ 'clusterdb', 'postgres' ],
+$ENV{PGPORT} = $node->port;
+
+$node->issues_sql_like(
+	[ 'clusterdb' ],
 	qr/statement: CLUSTER;/,
 	'SQL CLUSTER run');
 
-command_fails([ 'clusterdb', '-t', 'nonexistent', 'postgres' ],
-	'fails with nonexistent table');
+command_fails([ 'clusterdb', '-t', 'nonexistent' ],
+			  'fails with nonexistent table');
 
-psql 'postgres',
-'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';
-issues_sql_like(
-	[ 'clusterdb', '-t', 'test1', 'postgres' ],
+$node->psql('postgres',
+	'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x');
+$node->issues_sql_like(
+	[ 'clusterdb', '-t', 'test1' ],
 	qr/statement: CLUSTER test1;/,
 	'cluster specific table');
diff --git a/src/bin/scripts/t/011_clusterdb_all.pl b/src/bin/scripts/t/011_clusterdb_all.pl
index 7769f70..f0c4e19 100644
--- a/src/bin/scripts/t/011_clusterdb_all.pl
+++ b/src/bin/scripts/t/011_clusterdb_all.pl
@@ -1,12 +1,19 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
+# clusterdb -a is not compatible with -d, hence enforce environment variables
+# correctly.
+$ENV{PGDATABASE} = 'postgres';
+
+$node->issues_sql_like(
 	[ 'clusterdb', '-a' ],
 	qr/statement: CLUSTER.*statement: CLUSTER/s,
 	'cluster all databases');
diff --git a/src/bin/scripts/t/020_createdb.pl b/src/bin/scripts/t/020_createdb.pl
index a44283c..09efdc6 100644
--- a/src/bin/scripts/t/020_createdb.pl
+++ b/src/bin/scripts/t/020_createdb.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
@@ -7,14 +9,16 @@ program_help_ok('createdb');
 program_version_ok('createdb');
 program_options_handling_ok('createdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+$ENV{PGPORT} = $node->port;
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'createdb', 'foobar1' ],
 	qr/statement: CREATE DATABASE foobar1/,
 	'SQL CREATE DATABASE run');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'createdb', '-l', 'C', '-E', 'LATIN1', '-T', 'template0', 'foobar2' ],
 	qr/statement: CREATE DATABASE foobar2 ENCODING 'LATIN1'/,
 	'create database with encoding');
diff --git a/src/bin/scripts/t/030_createlang.pl b/src/bin/scripts/t/030_createlang.pl
index 7ff0a3e..867f8b2 100644
--- a/src/bin/scripts/t/030_createlang.pl
+++ b/src/bin/scripts/t/030_createlang.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 14;
 
@@ -7,18 +9,22 @@ program_help_ok('createlang');
 program_version_ok('createlang');
 program_options_handling_ok('createlang');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGDATABASE} = 'postgres';
+$ENV{PGPORT} = $node->port;
 
 command_fails(
-	[ 'createlang', 'plpgsql', 'postgres' ],
+	[ 'createlang', 'plpgsql' ],
 	'fails if language already exists');
 
-psql 'postgres', 'DROP EXTENSION plpgsql';
-issues_sql_like(
-	[ 'createlang', 'plpgsql', 'postgres' ],
+$node->psql('postgres', 'DROP EXTENSION plpgsql');
+$node->issues_sql_like(
+	[ 'createlang', 'plpgsql' ],
 	qr/statement: CREATE EXTENSION "plpgsql"/,
 	'SQL CREATE EXTENSION run');
 
-command_like([ 'createlang', '--list', 'postgres' ],
+command_like([ 'createlang', '--list' ],
 	qr/plpgsql/, 'list output');
diff --git a/src/bin/scripts/t/040_createuser.pl b/src/bin/scripts/t/040_createuser.pl
index 4d44e14..760b437 100644
--- a/src/bin/scripts/t/040_createuser.pl
+++ b/src/bin/scripts/t/040_createuser.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 17;
 
@@ -7,22 +9,25 @@ program_help_ok('createuser');
 program_version_ok('createuser');
 program_options_handling_ok('createuser');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
+$ENV{PGPORT} = $node->port;
+
+$node->issues_sql_like(
 	[ 'createuser', 'user1' ],
 qr/statement: CREATE ROLE user1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN;/,
 	'SQL CREATE USER run');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'createuser', '-L', 'role1' ],
 qr/statement: CREATE ROLE role1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT NOLOGIN;/,
 	'create a non-login role');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'createuser', '-r', 'user2' ],
 qr/statement: CREATE ROLE user2 NOSUPERUSER NOCREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a CREATEROLE user');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'createuser', '-s', 'user3' ],
 qr/statement: CREATE ROLE user3 SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a superuser');
diff --git a/src/bin/scripts/t/050_dropdb.pl b/src/bin/scripts/t/050_dropdb.pl
index 3065e50..50ee604 100644
--- a/src/bin/scripts/t/050_dropdb.pl
+++ b/src/bin/scripts/t/050_dropdb.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,11 +9,13 @@ program_help_ok('dropdb');
 program_version_ok('dropdb');
 program_options_handling_ok('dropdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+$ENV{PGPORT} = $node->port;
 
-psql 'postgres', 'CREATE DATABASE foobar1';
-issues_sql_like(
+$node->psql('postgres', 'CREATE DATABASE foobar1');
+$node->issues_sql_like(
 	[ 'dropdb', 'foobar1' ],
 	qr/statement: DROP DATABASE foobar1/,
 	'SQL DROP DATABASE run');
diff --git a/src/bin/scripts/t/060_droplang.pl b/src/bin/scripts/t/060_droplang.pl
index 6a21d7e..43720fb 100644
--- a/src/bin/scripts/t/060_droplang.pl
+++ b/src/bin/scripts/t/060_droplang.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,10 +9,12 @@ program_help_ok('droplang');
 program_version_ok('droplang');
 program_options_handling_ok('droplang');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+$ENV{PGPORT} = $node->port;
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'droplang', 'plpgsql', 'postgres' ],
 	qr/statement: DROP EXTENSION "plpgsql"/,
 	'SQL DROP EXTENSION run');
diff --git a/src/bin/scripts/t/070_dropuser.pl b/src/bin/scripts/t/070_dropuser.pl
index bbb3b79..5a452c4 100644
--- a/src/bin/scripts/t/070_dropuser.pl
+++ b/src/bin/scripts/t/070_dropuser.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,11 +9,13 @@ program_help_ok('dropuser');
 program_version_ok('dropuser');
 program_options_handling_ok('dropuser');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+$ENV{PGPORT} = $node->port;
 
-psql 'postgres', 'CREATE ROLE foobar1';
-issues_sql_like(
+$node->psql('postgres', 'CREATE ROLE foobar1');
+$node->issues_sql_like(
 	[ 'dropuser', 'foobar1' ],
 	qr/statement: DROP ROLE foobar1/,
 	'SQL DROP ROLE run');
diff --git a/src/bin/scripts/t/080_pg_isready.pl b/src/bin/scripts/t/080_pg_isready.pl
index f432505..ca512c1 100644
--- a/src/bin/scripts/t/080_pg_isready.pl
+++ b/src/bin/scripts/t/080_pg_isready.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 10;
 
@@ -9,7 +11,10 @@ program_options_handling_ok('pg_isready');
 
 command_fails(['pg_isready'], 'fails with no server running');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGPORT} = $node->port;
 
 command_ok(['pg_isready'], 'succeeds with server running');
diff --git a/src/bin/scripts/t/090_reindexdb.pl b/src/bin/scripts/t/090_reindexdb.pl
index 42628c2..68b2291 100644
--- a/src/bin/scripts/t/090_reindexdb.pl
+++ b/src/bin/scripts/t/090_reindexdb.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 20;
 
@@ -7,35 +9,36 @@ program_help_ok('reindexdb');
 program_version_ok('reindexdb');
 program_options_handling_ok('reindexdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
 $ENV{PGOPTIONS} = '--client-min-messages=WARNING';
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'reindexdb', 'postgres' ],
 	qr/statement: REINDEX DATABASE postgres;/,
 	'SQL REINDEX run');
 
-psql 'postgres',
-  'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a);';
-issues_sql_like(
+$node->psql('postgres',
+  'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a);');
+$node->issues_sql_like(
 	[ 'reindexdb', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX TABLE test1;/,
 	'reindex specific table');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'reindexdb', '-i', 'test1x', 'postgres' ],
 	qr/statement: REINDEX INDEX test1x;/,
 	'reindex specific index');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'reindexdb', '-S', 'pg_catalog', 'postgres' ],
 	qr/statement: REINDEX SCHEMA pg_catalog;/,
 	'reindex specific schema');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'reindexdb', '-s', 'postgres' ],
 	qr/statement: REINDEX SYSTEM postgres;/,
 	'reindex system tables');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'reindexdb', '-v', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX \(VERBOSE\) TABLE test1;/,
 	'reindex with verbose output');
diff --git a/src/bin/scripts/t/091_reindexdb_all.pl b/src/bin/scripts/t/091_reindexdb_all.pl
index ffadf29..d47b18b 100644
--- a/src/bin/scripts/t/091_reindexdb_all.pl
+++ b/src/bin/scripts/t/091_reindexdb_all.pl
@@ -1,14 +1,16 @@
 use strict;
 use warnings;
-use TestLib;
+
+use PostgresNode;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
 $ENV{PGOPTIONS} = '--client-min-messages=WARNING';
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'reindexdb', '-a' ],
 	qr/statement: REINDEX.*statement: REINDEX/s,
 	'reindex all databases');
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index ac160ba..387d2b4 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 18;
 
@@ -7,26 +9,27 @@ program_help_ok('vacuumdb');
 program_version_ok('vacuumdb');
 program_options_handling_ok('vacuumdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', 'postgres' ],
 	qr/statement: VACUUM;/,
 	'SQL VACUUM run');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '-f', 'postgres' ],
 	qr/statement: VACUUM \(FULL\);/,
 	'vacuumdb -f');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '-F', 'postgres' ],
 	qr/statement: VACUUM \(FREEZE\);/,
 	'vacuumdb -F');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '-z', 'postgres' ],
 	qr/statement: VACUUM \(ANALYZE\);/,
 	'vacuumdb -z');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '-Z', 'postgres' ],
 	qr/statement: ANALYZE;/,
 	'vacuumdb -Z');
diff --git a/src/bin/scripts/t/101_vacuumdb_all.pl b/src/bin/scripts/t/101_vacuumdb_all.pl
index e90f321..8f1536f 100644
--- a/src/bin/scripts/t/101_vacuumdb_all.pl
+++ b/src/bin/scripts/t/101_vacuumdb_all.pl
@@ -1,12 +1,14 @@
 use strict;
 use warnings;
-use TestLib;
+
+use PostgresNode;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '-a' ],
 	qr/statement: VACUUM.*statement: VACUUM/s,
 	'vacuum all databases');
diff --git a/src/bin/scripts/t/102_vacuumdb_stages.pl b/src/bin/scripts/t/102_vacuumdb_stages.pl
index 57b980e..4cb5b64 100644
--- a/src/bin/scripts/t/102_vacuumdb_stages.pl
+++ b/src/bin/scripts/t/102_vacuumdb_stages.pl
@@ -1,12 +1,14 @@
 use strict;
 use warnings;
-use TestLib;
+
+use PostgresNode;
 use Test::More tests => 4;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '--analyze-in-stages', 'postgres' ],
 qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE.*
@@ -16,8 +18,7 @@ qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE/sx,
 	'analyze three times');
 
-
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '--analyze-in-stages', '--all' ],
 qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE.*
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
new file mode 100644
index 0000000..0d03fdb
--- /dev/null
+++ b/src/test/perl/PostgresNode.pm
@@ -0,0 +1,437 @@
+# PostgresNode, class representing a data directory and postmaster.
+#
+# This contains a basic set of routines able to work on a PostgreSQL node,
+# allowing to start, stop, backup and initialize it with various options.
+# The set of nodes managed by a given test is also managed by this module.
+
+package PostgresNode;
+
+use strict;
+use warnings;
+
+use Config;
+use Cwd;
+use Exporter 'import';
+use File::Basename;
+use File::Spec;
+use File::Temp ();
+use IPC::Run;
+use PostgresNode;
+use RecursiveCopy;
+use Test::More;
+use TestLib;
+
+our @EXPORT = qw(
+  get_new_node
+);
+
+our ($test_pghost, $last_port_assigned, @all_nodes);
+
+BEGIN
+{
+	# PGHOST is set once and for all through a single series of tests when
+	# this module is loaded.
+	$test_pghost = $windows_os ? "127.0.0.1" : TestLib::tempdir_short();
+	$ENV{PGHOST} = $test_pghost;
+	$ENV{PGDATABASE} = 'postgres';
+
+	# Tracking of last port value assigned to accelerate free port lookup.
+	# XXX: Should this use PG_VERSION_NUM?
+	$last_port_assigned = 90600 % 16384 + 49152;
+
+	# Node tracking
+	@all_nodes = ();
+}
+
+sub new
+{
+	my $class  = shift;
+	my $pghost = shift;
+	my $pgport = shift;
+	my $self   = {
+		_port     => $pgport,
+		_host     => $pghost,
+		_basedir  => TestLib::tempdir,
+		_applname => "node_$pgport",
+		_logfile  => "$TestLib::log_path/node_$pgport.log" };
+
+	bless $self, $class;
+	$self->dump_info;
+
+	return $self;
+}
+
+sub port
+{
+	my ($self) = @_;
+	return $self->{_port};
+}
+
+sub host
+{
+	my ($self) = @_;
+	return $self->{_host};
+}
+
+sub basedir
+{
+	my ($self) = @_;
+	return $self->{_basedir};
+}
+
+sub applname
+{
+	my ($self) = @_;
+	return $self->{_applname};
+}
+
+sub logfile
+{
+	my ($self) = @_;
+	return $self->{_logfile};
+}
+
+sub connstr
+{
+	my ($self, $dbname) = @_;
+	my $pgport = $self->port;
+	my $pghost = $self->host;
+	if (!defined($dbname))
+	{
+		return "port=$pgport host=$pghost";
+	}
+	return "port=$pgport host=$pghost dbname=$dbname";
+}
+
+sub data_dir
+{
+	my ($self) = @_;
+	my $res = $self->basedir;
+	return "$res/pgdata";
+}
+
+sub archive_dir
+{
+	my ($self) = @_;
+	my $basedir = $self->basedir;
+	return "$basedir/archives";
+}
+
+sub backup_dir
+{
+	my ($self) = @_;
+	my $basedir = $self->basedir;
+	return "$basedir/backup";
+}
+
+# Dump node information
+sub dump_info
+{
+	my ($self) = @_;
+	print "Data directory: " . $self->data_dir . "\n";
+	print "Backup directory: " . $self->backup_dir . "\n";
+	print "Archive directory: " . $self->archive_dir . "\n";
+	print "Connection string: " . $self->connstr . "\n";
+	print "Application name: " . $self->applname . "\n";
+	print "Log file: " . $self->logfile . "\n";
+}
+
+sub set_replication_conf
+{
+	my ($self) = @_;
+	my $pgdata = $self->data_dir;
+
+	open my $hba, ">>$pgdata/pg_hba.conf";
+	print $hba "\n# Allow replication (set up by PostgresNode.pm)\n";
+	if (!$windows_os)
+	{
+		print $hba "local replication all trust\n";
+	}
+	else
+	{
+		print $hba
+"host replication all 127.0.0.1/32 sspi include_realm=1 map=regress\n";
+	}
+	close $hba;
+}
+
+# Initialize a new cluster for testing.
+#
+# Authentication is set up so that only the current OS user can access the
+# cluster. On Unix, we use Unix domain socket connections, with the socket in
+# a directory that's only accessible to the current user to ensure that.
+# On Windows, we use SSPI authentication to ensure the same (by pg_regress
+# --config-auth).
+sub init
+{
+	my ($self, %params) = @_;
+	my $port   = $self->port;
+	my $pgdata = $self->data_dir;
+	my $host   = $self->host;
+
+	$params{hba_permit_replication} = 1 if (!defined($params{hba_permit_replication}));
+
+	mkdir $self->backup_dir;
+	mkdir $self->archive_dir;
+
+	system_or_bail('initdb', '-D', $pgdata, '-A', 'trust', '-N');
+	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
+
+	open my $conf, ">>$pgdata/postgresql.conf";
+	print $conf "\n# Added by TestLib.pm)\n";
+	print $conf "fsync = off\n";
+	print $conf "log_statement = all\n";
+	print $conf "port = $port\n";
+	if ($windows_os)
+	{
+		print $conf "listen_addresses = '$host'\n";
+	}
+	else
+	{
+		print $conf "unix_socket_directories = '$host'\n";
+		print $conf "listen_addresses = ''\n";
+	}
+	close $conf;
+
+	$self->set_replication_conf if ($params{hba_permit_replication});
+}
+
+sub append_conf
+{
+	my ($self, $filename, $str) = @_;
+
+	my $conffile = $self->data_dir . '/' . $filename;
+
+	append_to_file($conffile, $str);
+}
+
+sub backup
+{
+	my ($self, $backup_name) = @_;
+	my $backup_path = $self->backup_dir . '/' . $backup_name;
+	my $port        = $self->port;
+
+	print "# Taking backup $backup_name from node with port $port\n";
+	system_or_bail("pg_basebackup -D $backup_path -p $port -x");
+	print "# Backup finished\n";
+}
+
+sub init_from_backup
+{
+	my ($self, $root_node, $backup_name) = @_;
+	my $backup_path = $root_node->backup_dir . '/' . $backup_name;
+	my $port        = $self->port;
+	my $root_port   = $root_node->port;
+
+	print
+"Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	die "Backup $backup_path does not exist" unless -d $backup_path;
+
+	mkdir $self->backup_dir;
+	mkdir $self->archive_dir;
+
+	my $data_path = $self->data_dir;
+	rmdir($data_path);
+	RecursiveCopy::copypath($backup_path, $data_path);
+	chmod(0700, $data_path);
+
+	# Base configuration for this node
+	$self->append_conf('postgresql.conf',
+		qq(
+port = $port
+));
+	$self->set_replication_conf;
+}
+
+sub start
+{
+	my ($self) = @_;
+	my $port   = $self->port;
+	my $pgdata = $self->data_dir;
+	print("### Starting test server in $pgdata\n");
+	my $ret = system_log('pg_ctl', '-w', '-D', $self->data_dir,
+		'-l', $self->logfile, 'start');
+
+	if ($ret != 0)
+	{
+		print "# pg_ctl failed; logfile:\n";
+		print slurp_file($self->logfile);
+		BAIL_OUT("pg_ctl failed");
+	}
+
+	$self->_update_pid;
+
+}
+
+sub stop
+{
+	my ($self, $mode) = @_;
+	my $port   = $self->port;
+	my $pgdata = $self->data_dir;
+	$mode = 'fast' if (!defined($mode));
+	print "### Stopping node in $pgdata with port $port using mode $mode\n";
+	system_log('pg_ctl', '-D', $pgdata, '-m', $mode, 'stop');
+	$self->{_pid} = undef;
+	$self->_update_pid;
+}
+
+sub restart
+{
+	my ($self)  = @_;
+	my $port    = $self->port;
+	my $pgdata  = $self->data_dir;
+	my $logfile = $self->logfile;
+	system_log('pg_ctl', '-D', $pgdata, '-w', '-l', $logfile, 'restart');
+	$self->_update_pid;
+}
+
+sub _update_pid
+{
+	my $self = shift;
+
+	# If we can open the PID file, read its first line and that's the PID we
+	# want.  If the file cannot be opened, presumably the server is not
+	# running; don't be noisy in that case.
+	open my $pidfile, $self->data_dir . "/postmaster.pid";
+	if (not defined $pidfile)
+	{
+		$self->{_pid} = undef;
+		print "# No postmaster PID\n";
+		return;
+	}
+
+	$self->{_pid} = <$pidfile>;
+	print "# Postmaster PID is $self->{_pid}\n";
+	close $pidfile;
+}
+
+
+#
+# Cluster management functions
+#
+
+# Build a new PostgresNode object, assigning a free port number.
+#
+# We also register the node, to avoid the port number from being reused
+# for another node even when this one is not active.
+sub get_new_node
+{
+	my $found = 0;
+	my $port  = $last_port_assigned;
+
+	while ($found == 0)
+	{
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		if (!run_log([ 'pg_isready', '-p', $port ]))
+		{
+			$found = 1;
+
+			# Found a potential candidate port number.  Check first that it is
+			# not included in the list of registered nodes.
+			foreach my $node (@all_nodes)
+			{
+				$found = 0 if ($node->port == $port);
+			}
+		}
+	}
+
+	print "# Found free port $port\n";
+
+	# Lock port number found by creating a new node
+	my $node = new PostgresNode($test_pghost, $port);
+
+	# Add node to list of nodes
+	push(@all_nodes, $node);
+
+	# And update port for next time
+	$last_port_assigned = $port;
+
+	return $node;
+}
+
+sub DESTROY
+{
+	my $self = shift;
+	return if not defined $self->{_pid};
+	print "# signalling QUIT to $self->{_pid}\n";
+	kill 'QUIT', $self->{_pid};
+}
+
+sub teardown_node
+{
+	my $self = shift;
+
+	$self->stop('immediate');
+}
+
+sub psql
+{
+	my ($self, $dbname, $sql) = @_;
+
+	my ($stdout, $stderr);
+	print("# Running SQL command: $sql\n");
+
+	IPC::Run::run [ 'psql', '-XAtq', '-d', $self->connstr($dbname), '-f', '-' ],
+	  '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
+
+	if ($stderr ne "")
+	{
+		print "#### Begin standard error\n";
+		print $stderr;
+		print "#### End standard error\n";
+	}
+	chomp $stdout;
+	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+	return $stdout;
+}
+
+# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
+sub poll_query_until
+{
+	my ($self, $dbname, $query) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', $query, '-d', $self->connstr($dbname) ];
+		my $result = IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
+# Run a command on the node, then verify that $expected_sql appears in the
+# server log file.
+sub issues_sql_like
+{
+	my ($self, $cmd, $expected_sql, $test_name) = @_;
+
+	local $ENV{PGPORT} = $self->port;
+
+	truncate $self->logfile, 0;
+	my $result = run_log($cmd);
+	ok($result, "@$cmd exit code 0");
+	my $log = slurp_file($self->logfile);
+	like($log, $expected_sql, "$test_name: SQL found in server log");
+}
+
+1;
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
new file mode 100644
index 0000000..4e58ad3
--- /dev/null
+++ b/src/test/perl/RecursiveCopy.pm
@@ -0,0 +1,42 @@
+# RecursiveCopy, a simple recursive copy implementation
+package RecursiveCopy;
+
+use strict;
+use warnings;
+
+use File::Basename;
+use File::Copy;
+
+sub copypath
+{
+	my $srcpath  = shift;
+	my $destpath = shift;
+
+	die "Cannot operate on symlinks" if -l $srcpath or -l $destpath;
+
+	# This source path is a file, simply copy it to destination with the
+	# same name.
+	die "Destination path $destpath exists as file" if -f $destpath;
+	if (-f $srcpath)
+	{
+		copy($srcpath, $destpath)
+			or die "copy $srcpath -> $destpath failed: $!";
+		return 1;
+	}
+
+	die "Destination needs to be a directory" unless -d $srcpath;
+	mkdir($destpath) or die "mkdir($destpath) failed: $!";
+
+	# Scan existing source directory and recursively copy everything.
+	opendir(my $directory, $srcpath) or die "could not opendir($srcpath): $!";
+	while (my $entry = readdir($directory))
+	{
+		next if ($entry eq '.' || $entry eq '..');
+		RecursiveCopy::copypath("$srcpath/$entry", "$destpath/$entry")
+			or die "copypath $srcpath/$entry -> $destpath/$entry failed";
+	}
+	closedir($directory);
+	return 1;
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..3a01b7a 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -1,3 +1,10 @@
+# TestLib, low-level routines and actions regression tests.
+#
+# This module contains a set of routines dedicated to environment setup for
+# a PostgreSQL regression test tun, and includes some low-level routines
+# aimed at controlling command execution, logging and test functions. This
+# module should never depend on any other PostgreSQL regression test modules.
+
 package TestLib;
 
 use strict;
@@ -5,16 +12,17 @@ use warnings;
 
 use Config;
 use Exporter 'import';
+use File::Basename;
+use File::Spec;
+use File::Temp ();
+use IPC::Run qw(run);
+use SimpleTee;
+use Test::More;
+
 our @EXPORT = qw(
-  tempdir
-  tempdir_short
-  standard_initdb
-  configure_hba_for_replication
-  start_test_server
-  restart_test_server
-  psql
   slurp_dir
   slurp_file
+  append_to_file
   system_or_bail
   system_log
   run_log
@@ -26,88 +34,80 @@ our @EXPORT = qw(
   program_version_ok
   program_options_handling_ok
   command_like
-  issues_sql_like
 
-  $tmp_check
-  $log_path
   $windows_os
 );
 
-use Cwd;
-use File::Basename;
-use File::Spec;
-use File::Temp ();
-use IPC::Run qw(run start);
+our ($windows_os, $tmp_check, $log_path, $test_logfile);
 
-use SimpleTee;
-
-use Test::More;
-
-our $windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
-
-# Open log file. For each test, the log file name uses the name of the
-# file launching this module, without the .pl suffix.
-our ($tmp_check, $log_path);
-$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
-$log_path = "$tmp_check/log";
-mkdir $tmp_check;
-mkdir $log_path;
-my $test_logfile = basename($0);
-$test_logfile =~ s/\.[^.]+$//;
-$test_logfile = "$log_path/regress_log_$test_logfile";
-open TESTLOG, '>', $test_logfile or die "Cannot open STDOUT to logfile: $!";
-
-# Hijack STDOUT and STDERR to the log file
-open(ORIG_STDOUT, ">&STDOUT");
-open(ORIG_STDERR, ">&STDERR");
-open(STDOUT, ">&TESTLOG");
-open(STDERR, ">&TESTLOG");
-
-# The test output (ok ...) needs to be printed to the original STDOUT so
-# that the 'prove' program can parse it, and display it to the user in
-# real time. But also copy it to the log file, to provide more context
-# in the log.
-my $builder = Test::More->builder;
-my $fh = $builder->output;
-tie *$fh, "SimpleTee", *ORIG_STDOUT, *TESTLOG;
-$fh = $builder->failure_output;
-tie *$fh, "SimpleTee", *ORIG_STDERR, *TESTLOG;
-
-# Enable auto-flushing for all the file handles. Stderr and stdout are
-# redirected to the same file, and buffering causes the lines to appear
-# in the log in confusing order.
-autoflush STDOUT 1;
-autoflush STDERR 1;
-autoflush TESTLOG 1;
-
-# Set to untranslated messages, to be able to compare program output
-# with expected strings.
-delete $ENV{LANGUAGE};
-delete $ENV{LC_ALL};
-$ENV{LC_MESSAGES} = 'C';
-
-delete $ENV{PGCONNECT_TIMEOUT};
-delete $ENV{PGDATA};
-delete $ENV{PGDATABASE};
-delete $ENV{PGHOSTADDR};
-delete $ENV{PGREQUIRESSL};
-delete $ENV{PGSERVICE};
-delete $ENV{PGSSLMODE};
-delete $ENV{PGUSER};
-
-if (!$ENV{PGPORT})
+BEGIN
 {
-	$ENV{PGPORT} = 65432;
+	# Set to untranslated messages, to be able to compare program output
+	# with expected strings.
+	delete $ENV{LANGUAGE};
+	delete $ENV{LC_ALL};
+	$ENV{LC_MESSAGES} = 'C';
+
+	delete $ENV{PGCONNECT_TIMEOUT};
+	delete $ENV{PGDATA};
+	delete $ENV{PGDATABASE};
+	delete $ENV{PGHOSTADDR};
+	delete $ENV{PGREQUIRESSL};
+	delete $ENV{PGSERVICE};
+	delete $ENV{PGSSLMODE};
+	delete $ENV{PGUSER};
+	delete $ENV{PGPORT};
+	delete $ENV{PGHOST};
+
+	# Must be set early
+	$windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
 }
 
-$ENV{PGPORT} = int($ENV{PGPORT}) % 65536;
+INIT
+{
+	# Determine output directories, and create them.  The base path is the
+	# TESTDIR environment variable, which is normally set by the invoking
+	# Makefile.
+	$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
+	$log_path = "$tmp_check/log";
 
+	mkdir $tmp_check;
+	mkdir $log_path;
+
+	# Open the test log file, whose name depends on the test name.
+	$test_logfile = basename($0);
+	$test_logfile =~ s/\.[^.]+$//;
+	$test_logfile = "$log_path/regress_log_$test_logfile";
+	open TESTLOG, '>', $test_logfile
+	  or die "could not open STDOUT to logfile \"$test_logfile\": $!";
+
+	# Hijack STDOUT and STDERR to the log file
+	open(ORIG_STDOUT, ">&STDOUT");
+	open(ORIG_STDERR, ">&STDERR");
+	open(STDOUT,      ">&TESTLOG");
+	open(STDERR,      ">&TESTLOG");
+
+	# The test output (ok ...) needs to be printed to the original STDOUT so
+	# that the 'prove' program can parse it, and display it to the user in
+	# real time. But also copy it to the log file, to provide more context
+	# in the log.
+	my $builder = Test::More->builder;
+	my $fh      = $builder->output;
+	tie *$fh, "SimpleTee", *ORIG_STDOUT, *TESTLOG;
+	$fh = $builder->failure_output;
+	tie *$fh, "SimpleTee", *ORIG_STDERR, *TESTLOG;
+
+	# Enable auto-flushing for all the file handles. Stderr and stdout are
+	# redirected to the same file, and buffering causes the lines to appear
+	# in the log in confusing order.
+	autoflush STDOUT 1;
+	autoflush STDERR 1;
+	autoflush TESTLOG 1;
+}
 
 #
 # Helper functions
 #
-
-
 sub tempdir
 {
 	return File::Temp::tempdir(
@@ -118,123 +118,36 @@ sub tempdir
 
 sub tempdir_short
 {
-
 	# Use a separate temp dir outside the build tree for the
 	# Unix-domain socket, to avoid file name length issues.
 	return File::Temp::tempdir(CLEANUP => 1);
 }
 
-# Initialize a new cluster for testing.
-#
-# The PGHOST environment variable is set to connect to the new cluster.
-#
-# Authentication is set up so that only the current OS user can access the
-# cluster. On Unix, we use Unix domain socket connections, with the socket in
-# a directory that's only accessible to the current user to ensure that.
-# On Windows, we use SSPI authentication to ensure the same (by pg_regress
-# --config-auth).
-sub standard_initdb
+sub system_log
 {
-	my $pgdata = shift;
-	system_or_bail('initdb', '-D', "$pgdata", '-A' , 'trust', '-N');
-	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
-
-	my $tempdir_short = tempdir_short;
-
-	open CONF, ">>$pgdata/postgresql.conf";
-	print CONF "\n# Added by TestLib.pm)\n";
-	print CONF "fsync = off\n";
-	if ($windows_os)
-	{
-		print CONF "listen_addresses = '127.0.0.1'\n";
-	}
-	else
-	{
-		print CONF "unix_socket_directories = '$tempdir_short'\n";
-		print CONF "listen_addresses = ''\n";
-	}
-	close CONF;
-
-	$ENV{PGHOST}         = $windows_os ? "127.0.0.1" : $tempdir_short;
+	print("# Running: " . join(" ", @_) . "\n");
+	return system(@_);
 }
 
-# Set up the cluster to allow replication connections, in the same way that
-# standard_initdb does for normal connections.
-sub configure_hba_for_replication
+sub system_or_bail
 {
-	my $pgdata = shift;
-
-	open HBA, ">>$pgdata/pg_hba.conf";
-	print HBA "\n# Allow replication (set up by TestLib.pm)\n";
-	if (! $windows_os)
+	if (system_log(@_) != 0)
 	{
-		print HBA "local replication all trust\n";
-	}
-	else
-	{
-		print HBA "host replication all 127.0.0.1/32 sspi include_realm=1 map=regress\n";
-	}
-	close HBA;
-}
-
-my ($test_server_datadir, $test_server_logfile);
-
-
-# Initialize a new cluster for testing in given directory, and start it.
-sub start_test_server
-{
-	my ($tempdir) = @_;
-	my $ret;
-
-	print("### Starting test server in $tempdir\n");
-	standard_initdb "$tempdir/pgdata";
-
-	$ret = system_log('pg_ctl', '-D', "$tempdir/pgdata", '-w', '-l',
-	  "$log_path/postmaster.log", '-o', "--log-statement=all",
-	  'start');
-
-	if ($ret != 0)
-	{
-		print "# pg_ctl failed; logfile:\n";
-		system('cat', "$log_path/postmaster.log");
-		BAIL_OUT("pg_ctl failed");
-	}
-
-	$test_server_datadir = "$tempdir/pgdata";
-	$test_server_logfile = "$log_path/postmaster.log";
-}
-
-sub restart_test_server
-{
-	print("### Restarting test server\n");
-	system_log('pg_ctl', '-D', $test_server_datadir, '-w', '-l',
-	  $test_server_logfile, 'restart');
-}
-
-END
-{
-	if ($test_server_datadir)
-	{
-		system_log('pg_ctl', '-D', $test_server_datadir, '-m',
-		  'immediate', 'stop');
+		BAIL_OUT("system $_[0] failed");
 	}
 }
 
-sub psql
+sub run_log
 {
-	my ($dbname, $sql) = @_;
-	my ($stdout, $stderr);
-	print("# Running SQL command: $sql\n");
-	run [ 'psql', '-X', '-A', '-t', '-q', '-d', $dbname, '-f', '-' ], '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
-	chomp $stdout;
-	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-	return $stdout;
+	print("# Running: " . join(" ", @{ $_[0] }) . "\n");
+	return run(@_);
 }
 
 sub slurp_dir
 {
 	my ($dir) = @_;
-	opendir(my $dh, $dir) or die;
+	opendir(my $dh, $dir)
+	  or die "could not opendir \"$dir\": $!";
 	my @direntries = readdir $dh;
 	closedir $dh;
 	return @direntries;
@@ -249,32 +162,18 @@ sub slurp_file
 	return $contents;
 }
 
-sub system_or_bail
+sub append_to_file
 {
-	if (system_log(@_) != 0)
-	{
-		BAIL_OUT("system $_[0] failed: $?");
-	}
-}
+	my ($filename, $str) = @_;
 
-sub system_log
-{
-	print("# Running: " . join(" ", @_) ."\n");
-	return system(@_);
+	open my $fh, ">>", $filename or die "could not open \"$filename\": $!";
+	print $fh $str;
+	close $fh;
 }
 
-sub run_log
-{
-	print("# Running: " . join(" ", @{$_[0]}) ."\n");
-	return run (@_);
-}
-
-
 #
 # Test functions
 #
-
-
 sub command_ok
 {
 	my ($cmd, $test_name) = @_;
@@ -292,8 +191,8 @@ sub command_fails
 sub command_exit_is
 {
 	my ($cmd, $expected, $test_name) = @_;
-	print("# Running: " . join(" ", @{$cmd}) ."\n");
-	my $h = start $cmd;
+	print("# Running: " . join(" ", @{$cmd}) . "\n");
+	my $h = IPC::Run::start $cmd;
 	$h->finish();
 
 	# On Windows, the exit status of the process is returned directly as the
@@ -303,8 +202,10 @@ sub command_exit_is
 	# assuming the Unix convention, which will always return 0 on Windows as
 	# long as the process was not terminated by an exception. To work around
 	# that, use $h->full_result on Windows instead.
-	my $result = ($Config{osname} eq "MSWin32") ?
-		($h->full_results)[0] : $h->result(0);
+	my $result =
+	    ($Config{osname} eq "MSWin32")
+	  ? ($h->full_results)[0]
+	  : $h->result(0);
 	is($result, $expected, $test_name);
 }
 
@@ -313,7 +214,7 @@ sub program_help_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --help\n");
-	my $result = run [ $cmd, '--help' ], '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--help' ], '>', \$stdout, '2>', \$stderr;
 	ok($result, "$cmd --help exit code 0");
 	isnt($stdout, '', "$cmd --help goes to stdout");
 	is($stderr, '', "$cmd --help nothing to stderr");
@@ -324,7 +225,7 @@ sub program_version_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --version\n");
-	my $result = run [ $cmd, '--version' ], '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--version' ], '>', \$stdout, '2>', \$stderr;
 	ok($result, "$cmd --version exit code 0");
 	isnt($stdout, '', "$cmd --version goes to stdout");
 	is($stderr, '', "$cmd --version nothing to stderr");
@@ -335,8 +236,8 @@ sub program_options_handling_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --not-a-valid-option\n");
-	my $result = run [ $cmd, '--not-a-valid-option' ], '>', \$stdout, '2>',
-	  \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--not-a-valid-option' ], '>', \$stdout,
+	  '2>', \$stderr;
 	ok(!$result, "$cmd with invalid option nonzero exit code");
 	isnt($stderr, '', "$cmd with invalid option prints error message");
 }
@@ -346,20 +247,10 @@ sub command_like
 	my ($cmd, $expected_stdout, $test_name) = @_;
 	my ($stdout, $stderr);
 	print("# Running: " . join(" ", @{$cmd}) . "\n");
-	my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr;
 	ok($result, "@$cmd exit code 0");
 	is($stderr, '', "@$cmd no stderr");
 	like($stdout, $expected_stdout, "$test_name: matches");
 }
 
-sub issues_sql_like
-{
-	my ($cmd, $expected_sql, $test_name) = @_;
-	truncate $test_server_logfile, 0;
-	my $result = run_log($cmd);
-	ok($result, "@$cmd exit code 0");
-	my $log = slurp_file($test_server_logfile);
-	like($log, $expected_sql, "$test_name: SQL found in server log");
-}
-
 1;
diff --git a/src/test/ssl/ServerSetup.pm b/src/test/ssl/ServerSetup.pm
index a6c77b5..4e93184 100644
--- a/src/test/ssl/ServerSetup.pm
+++ b/src/test/ssl/ServerSetup.pm
@@ -18,6 +18,7 @@ package ServerSetup;
 
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use File::Basename;
 use File::Copy;
@@ -45,17 +46,19 @@ sub copy_files
 
 sub configure_test_server_for_ssl
 {
-	my $tempdir    = $_[0];
+	my $node       = $_[0];
 	my $serverhost = $_[1];
 
+	my $pgdata = $node->data_dir;
+
 	# Create test users and databases
-	psql 'postgres', "CREATE USER ssltestuser";
-	psql 'postgres', "CREATE USER anotheruser";
-	psql 'postgres', "CREATE DATABASE trustdb";
-	psql 'postgres', "CREATE DATABASE certdb";
+	$node->psql('postgres', "CREATE USER ssltestuser");
+	$node->psql('postgres', "CREATE USER anotheruser");
+	$node->psql('postgres', "CREATE DATABASE trustdb");
+	$node->psql('postgres', "CREATE DATABASE certdb");
 
 	# enable logging etc.
-	open CONF, ">>$tempdir/pgdata/postgresql.conf";
+	open CONF, ">>$pgdata/postgresql.conf";
 	print CONF "fsync=off\n";
 	print CONF "log_connections=on\n";
 	print CONF "log_hostname=on\n";
@@ -68,17 +71,17 @@ sub configure_test_server_for_ssl
 	close CONF;
 
 # Copy all server certificates and keys, and client root cert, to the data dir
-	copy_files("ssl/server-*.crt", "$tempdir/pgdata");
-	copy_files("ssl/server-*.key", "$tempdir/pgdata");
-	chmod(0600, glob "$tempdir/pgdata/server-*.key") or die $!;
-	copy_files("ssl/root+client_ca.crt", "$tempdir/pgdata");
-	copy_files("ssl/root+client.crl",    "$tempdir/pgdata");
+	copy_files("ssl/server-*.crt", $pgdata);
+	copy_files("ssl/server-*.key", $pgdata);
+	chmod(0600, glob "$pgdata/server-*.key") or die $!;
+	copy_files("ssl/root+client_ca.crt", $pgdata);
+	copy_files("ssl/root+client.crl",    $pgdata);
 
   # Only accept SSL connections from localhost. Our tests don't depend on this
   # but seems best to keep it as narrow as possible for security reasons.
   #
   # When connecting to certdb, also check the client certificate.
-	open HBA, ">$tempdir/pgdata/pg_hba.conf";
+	open HBA, ">$pgdata/pg_hba.conf";
 	print HBA
 "# TYPE  DATABASE        USER            ADDRESS                 METHOD\n";
 	print HBA
@@ -96,12 +99,13 @@ sub configure_test_server_for_ssl
 # the server so that the configuration takes effect.
 sub switch_server_cert
 {
-	my $tempdir  = $_[0];
+	my $node     = $_[0];
 	my $certfile = $_[1];
+	my $pgdata   = $node->data_dir;
 
 	diag "Restarting server with certfile \"$certfile\"...";
 
-	open SSLCONF, ">$tempdir/pgdata/sslconfig.conf";
+	open SSLCONF, ">$pgdata/sslconfig.conf";
 	print SSLCONF "ssl=on\n";
 	print SSLCONF "ssl_ca_file='root+client_ca.crt'\n";
 	print SSLCONF "ssl_cert_file='$certfile.crt'\n";
@@ -110,5 +114,5 @@ sub switch_server_cert
 	close SSLCONF;
 
 	# Stop and restart server to reload the new config.
-	restart_test_server();
+	$node->restart;
 }
diff --git a/src/test/ssl/t/001_ssltests.pl b/src/test/ssl/t/001_ssltests.pl
index 0d6f339..dcc541d 100644
--- a/src/test/ssl/t/001_ssltests.pl
+++ b/src/test/ssl/t/001_ssltests.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+use PostgresNode;
+use TestLib;
 use TestLib;
 use Test::More tests => 38;
 use ServerSetup;
@@ -25,8 +27,6 @@ BEGIN
 # postgresql-ssl-regression.test.
 my $SERVERHOSTADDR = '127.0.0.1';
 
-my $tempdir = TestLib::tempdir;
-
 # Define a couple of helper functions to test connecting to the server.
 
 my $common_connstr;
@@ -74,10 +74,16 @@ chmod 0600, "ssl/client.key";
 
 #### Part 0. Set up the server.
 
-diag "setting up data directory in \"$tempdir\"...";
-start_test_server($tempdir);
-configure_test_server_for_ssl($tempdir, $SERVERHOSTADDR);
-switch_server_cert($tempdir, 'server-cn-only');
+diag "setting up data directory...";
+my $node = get_new_node();
+$node->init;
+# PGHOST is enforced here to set up the node, subsequent connections
+# will use a dedicated connection string.
+$ENV{PGHOST} = $node->host;
+$ENV{PGPORT} = $node->port;
+$node->start;
+configure_test_server_for_ssl($node, $SERVERHOSTADDR);
+switch_server_cert($node, 'server-cn-only');
 
 ### Part 1. Run client-side tests.
 ###
@@ -150,7 +156,7 @@ test_connect_ok("sslmode=verify-ca host=wronghost.test");
 test_connect_fails("sslmode=verify-full host=wronghost.test");
 
 # Test Subject Alternative Names.
-switch_server_cert($tempdir, 'server-multiple-alt-names');
+switch_server_cert($node, 'server-multiple-alt-names');
 
 diag "test hostname matching with X509 Subject Alternative Names";
 $common_connstr =
@@ -165,7 +171,7 @@ test_connect_fails("host=deep.subdomain.wildcard.pg-ssltest.test");
 
 # Test certificate with a single Subject Alternative Name. (this gives a
 # slightly different error message, that's all)
-switch_server_cert($tempdir, 'server-single-alt-name');
+switch_server_cert($node, 'server-single-alt-name');
 
 diag "test hostname matching with a single X509 Subject Alternative Name";
 $common_connstr =
@@ -178,7 +184,7 @@ test_connect_fails("host=deep.subdomain.wildcard.pg-ssltest.test");
 
 # Test server certificate with a CN and SANs. Per RFCs 2818 and 6125, the CN
 # should be ignored when the certificate has both.
-switch_server_cert($tempdir, 'server-cn-and-alt-names');
+switch_server_cert($node, 'server-cn-and-alt-names');
 
 diag "test certificate with both a CN and SANs";
 $common_connstr =
@@ -190,7 +196,7 @@ test_connect_fails("host=common-name.pg-ssltest.test");
 
 # Finally, test a server certificate that has no CN or SANs. Of course, that's
 # not a very sensible certificate, but libpq should handle it gracefully.
-switch_server_cert($tempdir, 'server-no-names');
+switch_server_cert($node, 'server-no-names');
 $common_connstr =
 "user=ssltestuser dbname=trustdb sslcert=invalid sslrootcert=ssl/root+server_ca.crt hostaddr=$SERVERHOSTADDR";
 
@@ -199,7 +205,7 @@ test_connect_fails("sslmode=verify-full host=common-name.pg-ssltest.test");
 
 # Test that the CRL works
 diag "Testing client-side CRL";
-switch_server_cert($tempdir, 'server-revoked');
+switch_server_cert($node, 'server-revoked');
 
 $common_connstr =
 "user=ssltestuser dbname=trustdb sslcert=invalid hostaddr=$SERVERHOSTADDR host=common-name.pg-ssltest.test";
@@ -233,7 +239,3 @@ test_connect_fails(
 test_connect_fails(
 "user=ssltestuser sslcert=ssl/client-revoked.crt sslkey=ssl/client-revoked.key"
 );
-
-
-# All done! Save the log, before the temporary installation is deleted
-copy("$tempdir/client-log", "./client-log");
20151201_tapcheck_v13-2.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 86cc14e..bc4afce 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -13,6 +13,7 @@ program_options_handling_ok('pg_basebackup');
 my $tempdir = TestLib::tempdir;
 
 my $node = get_new_node();
+
 # Initialize node without replication settings
 $node->init(hba_permit_replication => 0);
 $node->start;
@@ -52,9 +53,10 @@ command_ok([ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup runs');
 ok(-f "$tempdir/backup/PG_VERSION", 'backup was created');
 
-is_deeply([sort(slurp_dir("$tempdir/backup/pg_xlog/"))],
-		  [sort qw(. .. archive_status)],
-		  'no WAL files copied');
+is_deeply(
+	[ sort(slurp_dir("$tempdir/backup/pg_xlog/")) ],
+	[ sort qw(. .. archive_status) ],
+	'no WAL files copied');
 
 command_ok(
 	[   'pg_basebackup', '-D', "$tempdir/backup2", '--xlogdir',
@@ -99,8 +101,9 @@ unlink "$pgdata/$superlongname";
 
 # The following tests test symlinks. Windows doesn't have symlinks, so
 # skip on Windows.
-SKIP: {
-    skip "symlinks not supported on Windows", 10 if ($windows_os);
+SKIP:
+{
+	skip "symlinks not supported on Windows", 10 if ($windows_os);
 
 	# Create a temporary directory in the system location and symlink it
 	# to our physical temp location.  That way we can use shorter names
@@ -111,10 +114,10 @@ SKIP: {
 
 	mkdir "$tempdir/tblspc1";
 	$node->psql('postgres',
-	"CREATE TABLESPACE tblspc1 LOCATION '$shorter_tempdir/tblspc1';");
+		"CREATE TABLESPACE tblspc1 LOCATION '$shorter_tempdir/tblspc1';");
 	$node->psql('postgres', "CREATE TABLE test1 (a int) TABLESPACE tblspc1;");
 	command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup2", '-Ft' ],
-			   'tar format with tablespaces');
+		'tar format with tablespaces');
 	ok(-f "$tempdir/tarbackup2/base.tar", 'backup tar was created');
 	my @tblspc_tars = glob "$tempdir/tarbackup2/[0-9]*.tar";
 	is(scalar(@tblspc_tars), 1, 'one tablespace tar was created');
@@ -130,9 +133,9 @@ SKIP: {
 	ok(-d "$tempdir/tbackup/tblspc1", 'tablespace was relocated');
 	opendir(my $dh, "$pgdata/pg_tblspc") or die;
 	ok( (   grep {
-		-l "$tempdir/backup1/pg_tblspc/$_"
-			and readlink "$tempdir/backup1/pg_tblspc/$_" eq
-			"$tempdir/tbackup/tblspc1"
+				-l "$tempdir/backup1/pg_tblspc/$_"
+				  and readlink "$tempdir/backup1/pg_tblspc/$_" eq
+				  "$tempdir/tbackup/tblspc1"
 			} readdir($dh)),
 		"tablespace symlink was updated");
 	closedir $dh;
@@ -141,19 +144,20 @@ SKIP: {
 	$node->psql('postgres', "DROP TABLE test1;");
 	$node->psql('postgres', "DROP TABLESPACE tblspc1;");
 	$node->psql('postgres',
-	"CREATE TABLESPACE tblspc2 LOCATION '$shorter_tempdir/tbl=spc2';");
+		"CREATE TABLESPACE tblspc2 LOCATION '$shorter_tempdir/tbl=spc2';");
 	command_ok(
 		[   'pg_basebackup', '-D', "$tempdir/backup3", '-Fp',
 			"-T$shorter_tempdir/tbl\\=spc2=$tempdir/tbackup/tbl\\=spc2" ],
 		'mapping tablespace with = sign in path');
-	ok(-d "$tempdir/tbackup/tbl=spc2", 'tablespace with = sign was relocated');
+	ok(-d "$tempdir/tbackup/tbl=spc2",
+		'tablespace with = sign was relocated');
 	$node->psql('postgres', "DROP TABLESPACE tblspc2;");
 
 	mkdir "$tempdir/$superlongname";
 	$node->psql('postgres',
-	"CREATE TABLESPACE tblspc3 LOCATION '$tempdir/$superlongname';");
+		"CREATE TABLESPACE tblspc3 LOCATION '$tempdir/$superlongname';");
 	command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l3", '-Ft' ],
-			   'pg_basebackup tar with long symlink target');
+		'pg_basebackup tar with long symlink target');
 	$node->psql('postgres', "DROP TABLESPACE tblspc3;");
 }
 
@@ -161,33 +165,57 @@ command_ok([ 'pg_basebackup', '-D', "$tempdir/backupR", '-R' ],
 	'pg_basebackup -R runs');
 ok(-f "$tempdir/backupR/recovery.conf", 'recovery.conf was created');
 my $recovery_conf = slurp_file "$tempdir/backupR/recovery.conf";
+
 # using a character class for the final "'" here works around an apparent
 # bug in several version of the Msys DTK perl
-like($recovery_conf, qr/^standby_mode = 'on[']$/m, 'recovery.conf sets standby_mode');
-like($recovery_conf, qr/^primary_conninfo = '.*port=$ENV{PGPORT}.*'$/m, 'recovery.conf sets primary_conninfo');
+like(
+	$recovery_conf,
+	qr/^standby_mode = 'on[']$/m,
+	'recovery.conf sets standby_mode');
+like(
+	$recovery_conf,
+	qr/^primary_conninfo = '.*port=$ENV{PGPORT}.*'$/m,
+	'recovery.conf sets primary_conninfo');
 
 command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxf", '-X', 'fetch' ],
 	'pg_basebackup -X fetch runs');
-ok(grep(/^[0-9A-F]{24}$/, slurp_dir("$tempdir/backupxf/pg_xlog")), 'WAL files copied');
+ok(grep(/^[0-9A-F]{24}$/, slurp_dir("$tempdir/backupxf/pg_xlog")),
+	'WAL files copied');
 command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs", '-X', 'stream' ],
 	'pg_basebackup -X stream runs');
-ok(grep(/^[0-9A-F]{24}$/, slurp_dir("$tempdir/backupxf/pg_xlog")), 'WAL files copied');
+ok(grep(/^[0-9A-F]{24}$/, slurp_dir("$tempdir/backupxf/pg_xlog")),
+	'WAL files copied');
 
-command_fails([ 'pg_basebackup', '-D', "$tempdir/fail", '-S', 'slot1' ],
+command_fails(
+	[ 'pg_basebackup', '-D', "$tempdir/fail", '-S', 'slot1' ],
 	'pg_basebackup with replication slot fails without -X stream');
-command_fails([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl_fail", '-X', 'stream', '-S', 'slot1' ],
+command_fails(
+	[   'pg_basebackup',             '-D',
+		"$tempdir/backupxs_sl_fail", '-X',
+		'stream',                    '-S',
+		'slot1' ],
 	'pg_basebackup fails with nonexistent replication slot');
 
-$node->psql('postgres', q{SELECT * FROM pg_create_physical_replication_slot('slot1')});
-my $lsn = $node->psql('postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'});
+$node->psql('postgres',
+	q{SELECT * FROM pg_create_physical_replication_slot('slot1')});
+my $lsn = $node->psql('postgres',
+	q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'}
+);
 is($lsn, '', 'restart LSN of new slot is null');
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl", '-X', 'stream', '-S', 'slot1' ],
+command_ok(
+	[   'pg_basebackup', '-D', "$tempdir/backupxs_sl", '-X',
+		'stream',        '-S', 'slot1' ],
 	'pg_basebackup -X stream with replication slot runs');
-$lsn = $node->psql('postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'});
+$lsn = $node->psql('postgres',
+	q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'}
+);
 like($lsn, qr!^0/[0-9A-Z]{7,8}$!, 'restart LSN of slot has advanced');
 
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl_R", '-X', 'stream', '-S', 'slot1', '-R' ],
+command_ok(
+	[   'pg_basebackup', '-D', "$tempdir/backupxs_sl_R", '-X',
+		'stream',        '-S', 'slot1',                  '-R' ],
 	'pg_basebackup with replication slot and -R runs');
-like(slurp_file("$tempdir/backupxs_sl_R/recovery.conf"),
-	 qr/^primary_slot_name = 'slot1'$/m,
-	 'recovery.conf sets primary_slot_name');
+like(
+	slurp_file("$tempdir/backupxs_sl_R/recovery.conf"),
+	qr/^primary_slot_name = 'slot1'$/m,
+	'recovery.conf sets primary_slot_name');
diff --git a/src/bin/pg_ctl/t/001_start_stop.pl b/src/bin/pg_ctl/t/001_start_stop.pl
index a224ece..cbe99d7 100644
--- a/src/bin/pg_ctl/t/001_start_stop.pl
+++ b/src/bin/pg_ctl/t/001_start_stop.pl
@@ -18,13 +18,11 @@ command_exit_is([ 'pg_ctl', 'start', '-D', "$tempdir/nonexistent" ],
 
 command_ok([ 'pg_ctl', 'initdb', '-D', "$tempdir/data", '-o', '-N' ],
 	'pg_ctl initdb');
-command_ok(
-	[ $ENV{PG_REGRESS}, '--config-auth',
-		"$tempdir/data" ],
+command_ok([ $ENV{PG_REGRESS}, '--config-auth', "$tempdir/data" ],
 	'configure authentication');
 open CONF, ">>$tempdir/data/postgresql.conf";
 print CONF "fsync = off\n";
-if (! $windows_os)
+if (!$windows_os)
 {
 	print CONF "listen_addresses = ''\n";
 	print CONF "unix_socket_directories = '$tempdir_short'\n";
@@ -36,6 +34,7 @@ else
 close CONF;
 command_ok([ 'pg_ctl', 'start', '-D', "$tempdir/data", '-w' ],
 	'pg_ctl start -w');
+
 # sleep here is because Windows builds can't check postmaster.pid exactly,
 # so they may mistake a pre-existing postmaster.pid for one created by the
 # postmaster they start.  Waiting more than the 2 seconds slop time allowed
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index 55bbc9c..f2c0ca9 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -77,7 +77,7 @@ sub standby_psql
 	my $cmd = shift;
 
 	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
-      $node_standby->connstr('postgres'), '-c', "$cmd";
+	  $node_standby->connstr('postgres'), '-c', "$cmd";
 }
 
 # Run a query against the master, and check that the output matches what's
@@ -89,8 +89,9 @@ sub check_query
 
 	# we want just the output, no formatting
 	my $result = IPC::Run::run [
-		'psql',          '-q', '-A', '-t', '--no-psqlrc', '-d',
-		$node_master->connstr('postgres'), '-c', $query ],
+		'psql', '-q', '-A', '-t', '--no-psqlrc', '-d',
+		$node_master->connstr('postgres'),
+		'-c', $query ],
 	  '>', \$stdout, '2>', \$stderr;
 
 	# We don't use ok() for the exit code and stderr, because we want this
@@ -118,7 +119,8 @@ sub setup_cluster
 	$node_master->init;
 
 	# Custom parameters for master's postgresql.conf
-	$node_master->append_conf("postgresql.conf", qq(
+	$node_master->append_conf(
+		"postgresql.conf", qq(
 wal_level = hot_standby
 max_wal_senders = 2
 wal_keep_segments = 20
@@ -146,7 +148,8 @@ sub create_standby
 	$node_standby->init_from_backup($node_master, 'my_backup');
 	my $connstr_master = $node_master->connstr('postgres');
 
-	$node_standby->append_conf("recovery.conf", qq(
+	$node_standby->append_conf(
+		"recovery.conf", qq(
 primary_conninfo='$connstr_master application_name=rewind_standby'
 standby_mode=on
 recovery_target_timeline='latest'
@@ -174,7 +177,8 @@ sub promote_standby
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
 	system_or_bail('pg_ctl', '-w', '-D', $node_standby->data_dir, 'promote');
-	$node_standby->poll_query_until('postgres', "SELECT NOT pg_is_in_recovery()")
+	$node_standby->poll_query_until('postgres',
+		"SELECT NOT pg_is_in_recovery()")
 	  or die "Timed out while waiting for promotion of standby";
 
 	# Force a checkpoint after the promotion. pg_rewind looks at the control
@@ -188,11 +192,11 @@ sub promote_standby
 
 sub run_pg_rewind
 {
-	my $test_mode = shift;
-	my $master_pgdata = $node_master->data_dir;
-	my $standby_pgdata = $node_standby->data_dir;
+	my $test_mode       = shift;
+	my $master_pgdata   = $node_master->data_dir;
+	my $standby_pgdata  = $node_standby->data_dir;
 	my $standby_connstr = $node_standby->connstr('postgres');
-	my $tmp_folder = TestLib::tempdir;
+	my $tmp_folder      = TestLib::tempdir;
 
 	# Stop the master and be ready to perform the rewind
 	$node_master->stop;
@@ -204,8 +208,9 @@ sub run_pg_rewind
 
 	# Keep a temporary postgresql.conf for master node or it would be
 	# overwritten during the rewind.
-	copy("$master_pgdata/postgresql.conf",
-		 "$tmp_folder/master-postgresql.conf.tmp");
+	copy(
+		"$master_pgdata/postgresql.conf",
+		"$tmp_folder/master-postgresql.conf.tmp");
 
 	# Now run pg_rewind
 	if ($test_mode eq "local")
@@ -213,21 +218,21 @@ sub run_pg_rewind
 		# Do rewind using a local pgdata as source
 		# Stop the master and be ready to perform the rewind
 		$node_standby->stop;
-		command_ok(['pg_rewind',
-					"--debug",
-					"--source-pgdata=$standby_pgdata",
-					"--target-pgdata=$master_pgdata"],
-				   'pg_rewind local');
+		command_ok(
+			[   'pg_rewind',
+				"--debug",
+				"--source-pgdata=$standby_pgdata",
+				"--target-pgdata=$master_pgdata" ],
+			'pg_rewind local');
 	}
 	elsif ($test_mode eq "remote")
 	{
 		# Do rewind using a remote connection as source
-		command_ok(['pg_rewind',
-					"--debug",
-					"--source-server",
-					$standby_connstr,
-					"--target-pgdata=$master_pgdata"],
-				   'pg_rewind remote');
+		command_ok(
+			[   'pg_rewind',       "--debug",
+				"--source-server", $standby_connstr,
+				"--target-pgdata=$master_pgdata" ],
+			'pg_rewind remote');
 	}
 	else
 	{
@@ -236,12 +241,14 @@ sub run_pg_rewind
 	}
 
 	# Now move back postgresql.conf with old settings
-	move("$tmp_folder/master-postgresql.conf.tmp",
-		 "$master_pgdata/postgresql.conf");
+	move(
+		"$tmp_folder/master-postgresql.conf.tmp",
+		"$master_pgdata/postgresql.conf");
 
 	# Plug-in rewound node to the now-promoted standby node
 	my $port_standby = $node_standby->port;
-	$node_master->append_conf('recovery.conf', qq(
+	$node_master->append_conf(
+		'recovery.conf', qq(
 primary_conninfo='port=$port_standby'
 standby_mode=on
 recovery_target_timeline='latest'
@@ -256,7 +263,7 @@ recovery_target_timeline='latest'
 # Clean up after the test. Stop both servers, if they're still running.
 sub clean_rewind_test
 {
-	$node_master->teardown_node if defined $node_master;
+	$node_master->teardown_node  if defined $node_master;
 	$node_standby->teardown_node if defined $node_standby;
 }
 
diff --git a/src/bin/scripts/t/010_clusterdb.pl b/src/bin/scripts/t/010_clusterdb.pl
index 756ae38..b1d4185 100644
--- a/src/bin/scripts/t/010_clusterdb.pl
+++ b/src/bin/scripts/t/010_clusterdb.pl
@@ -16,15 +16,16 @@ $node->start;
 $ENV{PGPORT} = $node->port;
 
 $node->issues_sql_like(
-	[ 'clusterdb' ],
+	['clusterdb'],
 	qr/statement: CLUSTER;/,
 	'SQL CLUSTER run');
 
 command_fails([ 'clusterdb', '-t', 'nonexistent' ],
-			  'fails with nonexistent table');
+	'fails with nonexistent table');
 
 $node->psql('postgres',
-	'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x');
+'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x'
+);
 $node->issues_sql_like(
 	[ 'clusterdb', '-t', 'test1' ],
 	qr/statement: CLUSTER test1;/,
diff --git a/src/bin/scripts/t/030_createlang.pl b/src/bin/scripts/t/030_createlang.pl
index 867f8b2..a323bbf 100644
--- a/src/bin/scripts/t/030_createlang.pl
+++ b/src/bin/scripts/t/030_createlang.pl
@@ -14,10 +14,9 @@ $node->init;
 $node->start;
 
 $ENV{PGDATABASE} = 'postgres';
-$ENV{PGPORT} = $node->port;
+$ENV{PGPORT}     = $node->port;
 
-command_fails(
-	[ 'createlang', 'plpgsql' ],
+command_fails([ 'createlang', 'plpgsql' ],
 	'fails if language already exists');
 
 $node->psql('postgres', 'DROP EXTENSION plpgsql');
@@ -26,5 +25,4 @@ $node->issues_sql_like(
 	qr/statement: CREATE EXTENSION "plpgsql"/,
 	'SQL CREATE EXTENSION run');
 
-command_like([ 'createlang', '--list' ],
-	qr/plpgsql/, 'list output');
+command_like([ 'createlang', '--list' ], qr/plpgsql/, 'list output');
diff --git a/src/bin/scripts/t/090_reindexdb.pl b/src/bin/scripts/t/090_reindexdb.pl
index 68b2291..fd4eac3 100644
--- a/src/bin/scripts/t/090_reindexdb.pl
+++ b/src/bin/scripts/t/090_reindexdb.pl
@@ -21,7 +21,7 @@ $node->issues_sql_like(
 	'SQL REINDEX run');
 
 $node->psql('postgres',
-  'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a);');
+	'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a);');
 $node->issues_sql_like(
 	[ 'reindexdb', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX TABLE test1;/,
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 0d03fdb..76a935d 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -169,7 +169,8 @@ sub init
 	my $pgdata = $self->data_dir;
 	my $host   = $self->host;
 
-	$params{hba_permit_replication} = 1 if (!defined($params{hba_permit_replication}));
+	$params{hba_permit_replication} = 1
+	  if (!defined($params{hba_permit_replication}));
 
 	mkdir $self->backup_dir;
 	mkdir $self->archive_dir;
@@ -236,7 +237,8 @@ sub init_from_backup
 	chmod(0700, $data_path);
 
 	# Base configuration for this node
-	$self->append_conf('postgresql.conf',
+	$self->append_conf(
+		'postgresql.conf',
 		qq(
 port = $port
 ));
@@ -373,8 +375,9 @@ sub psql
 	my ($stdout, $stderr);
 	print("# Running SQL command: $sql\n");
 
-	IPC::Run::run [ 'psql', '-XAtq', '-d', $self->connstr($dbname), '-f', '-' ],
-	  '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
+	IPC::Run::run [ 'psql', '-XAtq', '-d', $self->connstr($dbname), '-f',
+		'-' ], '<', \$sql, '>', \$stdout, '2>', \$stderr
+	  or die;
 
 	if ($stderr ne "")
 	{
@@ -398,7 +401,8 @@ sub poll_query_until
 
 	while ($attempts < $max_attempts)
 	{
-		my $cmd = [ 'psql', '-At', '-c', $query, '-d', $self->connstr($dbname) ];
+		my $cmd =
+		  [ 'psql', '-At', '-c', $query, '-d', $self->connstr($dbname) ];
 		my $result = IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr;
 
 		chomp($stdout);
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index 4e58ad3..9362aa8 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -20,7 +20,7 @@ sub copypath
 	if (-f $srcpath)
 	{
 		copy($srcpath, $destpath)
-			or die "copy $srcpath -> $destpath failed: $!";
+		  or die "copy $srcpath -> $destpath failed: $!";
 		return 1;
 	}
 
@@ -33,7 +33,7 @@ sub copypath
 	{
 		next if ($entry eq '.' || $entry eq '..');
 		RecursiveCopy::copypath("$srcpath/$entry", "$destpath/$entry")
-			or die "copypath $srcpath/$entry -> $destpath/$entry failed";
+		  or die "copypath $srcpath/$entry -> $destpath/$entry failed";
 	}
 	closedir($directory);
 	return 1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 3a01b7a..b4f1080 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -214,7 +214,8 @@ sub program_help_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --help\n");
-	my $result = IPC::Run::run [ $cmd, '--help' ], '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--help' ], '>', \$stdout, '2>',
+	  \$stderr;
 	ok($result, "$cmd --help exit code 0");
 	isnt($stdout, '', "$cmd --help goes to stdout");
 	is($stderr, '', "$cmd --help nothing to stderr");
@@ -225,7 +226,8 @@ sub program_version_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --version\n");
-	my $result = IPC::Run::run [ $cmd, '--version' ], '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--version' ], '>', \$stdout, '2>',
+	  \$stderr;
 	ok($result, "$cmd --version exit code 0");
 	isnt($stdout, '', "$cmd --version goes to stdout");
 	is($stderr, '', "$cmd --version nothing to stderr");
@@ -236,7 +238,8 @@ sub program_options_handling_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --not-a-valid-option\n");
-	my $result = IPC::Run::run [ $cmd, '--not-a-valid-option' ], '>', \$stdout,
+	my $result = IPC::Run::run [ $cmd, '--not-a-valid-option' ], '>',
+	  \$stdout,
 	  '2>', \$stderr;
 	ok(!$result, "$cmd with invalid option nonzero exit code");
 	isnt($stderr, '', "$cmd with invalid option prints error message");
diff --git a/src/test/ssl/t/001_ssltests.pl b/src/test/ssl/t/001_ssltests.pl
index dcc541d..92f16e4 100644
--- a/src/test/ssl/t/001_ssltests.pl
+++ b/src/test/ssl/t/001_ssltests.pl
@@ -77,6 +77,7 @@ chmod 0600, "ssl/client.key";
 diag "setting up data directory...";
 my $node = get_new_node();
 $node->init;
+
 # PGHOST is enforced here to set up the node, subsequent connections
 # will use a dedicated connection string.
 $ENV{PGHOST} = $node->host;
#89Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#88)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Dec 2, 2015 at 8:11 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

- I discovered that not setting PGPORT was causing some of the tests
that fail (using command_fails) to fail to test what was being tested.
The problem is that the command was failing with "could not connect to
server" instead of failing because of trying to cluster a nonexistant
table, etc. Unfortunately the only way to verify this is by looking
at the regress_log_xxx_foobar file. Not ideal, but not this patch's
fault.

Nice catch.

- I changed the routines moved to PostgresNode so that they are instance
methods, i.e. $node->poll_until_query; also psql and
issues_query_like. The latter also sets "local $PGPORT" so that the
command is run with the correct port.

OK.

- It would be nice to have command_ok and command_fails in PostgresNode
too; that would remove the need for setting $ENV{PGPORT} but it's
possible to run commands outside a node too, so we'd need duplicates,
which would be worse.

I am fine to let it the way your patch does it. There are already many changes.

- I removed the @active_nodes array, which is now unnecessary (per
above).

So that's basically replaced by @all_nodes.

- I moved the "delete $ENV{PGxxx}" back to a BEGIN block in TestLib.
I did it because it's possible to write test scripts without
PostgresNode, and it's not nice to have those pick up the environment.
This still affects anything using PostgresNode because that one uses
TestLib.

OK.

Finally, I ran perltidy on all the files, which strangely changed stuff
that I didn't expect it to change. I wonder if this is related to the
perltidy version. Do you see further changes if you run perltidy on the
patched tree?

SimpleTee.pm shows some diffs, though it doesn't seem that this patch
should care about that. The rest is showing no diffs. And I have used
perltidy v20140711.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#90Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#89)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

On Wed, Dec 2, 2015 at 8:11 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

- It would be nice to have command_ok and command_fails in PostgresNode
too; that would remove the need for setting $ENV{PGPORT} but it's
possible to run commands outside a node too, so we'd need duplicates,
which would be worse.

I am fine to let it the way your patch does it. There are already many changes.

Idea: we can have a bare command_ok exported by TestLib just as
currently, and instance method PostgresNode->command_ok that first sets
local $ENV{PGPORT} and then calls the other one.

- I removed the @active_nodes array, which is now unnecessary (per
above).

So that's basically replaced by @all_nodes.

@all_nodes is only used to look for unused port numbers.

Finally, I ran perltidy on all the files, which strangely changed stuff
that I didn't expect it to change. I wonder if this is related to the
perltidy version. Do you see further changes if you run perltidy on the
patched tree?

SimpleTee.pm shows some diffs, though it doesn't seem that this patch
should care about that. The rest is showing no diffs. And I have used
perltidy v20140711.

Yes, the patch doesn't modify SimpleTee -- I just used "find" to indent
perl files. What I don't understand is why doesn't my perltidy run
match what was in master currently. It should be a no-op ...

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#91Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#90)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Dec 2, 2015 at 12:01 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Michael Paquier wrote:

On Wed, Dec 2, 2015 at 8:11 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

- It would be nice to have command_ok and command_fails in PostgresNode
too; that would remove the need for setting $ENV{PGPORT} but it's
possible to run commands outside a node too, so we'd need duplicates,
which would be worse.

I am fine to let it the way your patch does it. There are already many changes.

Idea: we can have a bare command_ok exported by TestLib just as
currently, and instance method PostgresNode->command_ok that first sets
local $ENV{PGPORT} and then calls the other one.

Hm. That would be cleaner and make the code more consistent. Now as
TestLib exports command_ok, command_like and command_fails, we would
get redefinition errors when compiling the code if those routines are
not named differently in PostgresNode. If you want to have the names
consistent, then I guess that the only way would be to remove those
routines from the export list of TestLib and call them directly as for
example TestLib::command_ok(). See for example the patch attached that
applies on top on your patch 2 that adds a set of routines in
PostgresNode with a slightly different name.

Finally, I ran perltidy on all the files, which strangely changed stuff
that I didn't expect it to change. I wonder if this is related to the
perltidy version. Do you see further changes if you run perltidy on the
patched tree?

SimpleTee.pm shows some diffs, though it doesn't seem that this patch
should care about that. The rest is showing no diffs. And I have used
perltidy v20140711.

Yes, the patch doesn't modify SimpleTee -- I just used "find" to indent
perl files. What I don't understand is why doesn't my perltidy run
match what was in master currently. It should be a no-op ...

Well I don't get it either :)
I did a run on top of your two patches and saw no differences.
--
Michael

Attachments:

20151202_taptests-3.patchapplication/x-patch; name=20151202_taptests-3.patchDownload
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index bc4afce..96dd103 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -19,11 +19,10 @@ $node->init(hba_permit_replication => 0);
 $node->start;
 my $pgdata = $node->data_dir;
 
-$ENV{PGPORT} = $node->port;
-
-command_fails(['pg_basebackup'],
+$node->node_command_fails(
+	['pg_basebackup'],
 	'pg_basebackup needs target directory specified');
-command_fails(
+$node->node_command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup fails because of hba');
 
@@ -38,7 +37,7 @@ if (open BADCHARS, ">>$tempdir/pgdata/FOO\xe0\xe0\xe0BAR")
 $node->set_replication_conf();
 system_or_bail 'pg_ctl', '-D', $pgdata, 'reload';
 
-command_fails(
+$node->node_command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup fails because of WAL configuration');
 
@@ -49,7 +48,7 @@ print CONF "wal_level = archive\n";
 close CONF;
 $node->restart;
 
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backup" ],
+$node->node_command_ok([ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup runs');
 ok(-f "$tempdir/backup/PG_VERSION", 'backup was created');
 
@@ -58,34 +57,34 @@ is_deeply(
 	[ sort qw(. .. archive_status) ],
 	'no WAL files copied');
 
-command_ok(
+$node->node_command_ok(
 	[   'pg_basebackup', '-D', "$tempdir/backup2", '--xlogdir',
 		"$tempdir/xlog2" ],
 	'separate xlog directory');
 ok(-f "$tempdir/backup2/PG_VERSION", 'backup was created');
 ok(-d "$tempdir/xlog2/",             'xlog directory was created');
 
-command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup", '-Ft' ],
+$node->node_command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup", '-Ft' ],
 	'tar format');
 ok(-f "$tempdir/tarbackup/base.tar", 'backup tar was created');
 
-command_fails(
+$node->node_command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup_foo", '-Fp', "-T=/foo" ],
 	'-T with empty old directory fails');
-command_fails(
+$node->node_command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup_foo", '-Fp', "-T/foo=" ],
 	'-T with empty new directory fails');
-command_fails(
+$node->node_command_fails(
 	[   'pg_basebackup', '-D', "$tempdir/backup_foo", '-Fp',
 		"-T/foo=/bar=/baz" ],
 	'-T with multiple = fails');
-command_fails(
+$node->node_command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup_foo", '-Fp', "-Tfoo=/bar" ],
 	'-T with old directory not absolute fails');
-command_fails(
+$node->node_command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup_foo", '-Fp', "-T/foo=bar" ],
 	'-T with new directory not absolute fails');
-command_fails(
+$node->node_command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup_foo", '-Fp', "-Tfoo" ],
 	'-T with invalid format fails');
 
@@ -95,7 +94,7 @@ my $superlongpath = "$pgdata/$superlongname";
 
 open FILE, ">$superlongpath" or die "unable to create file $superlongpath";
 close FILE;
-command_fails([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l1", '-Ft' ],
+$node->node_command_fails([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l1", '-Ft' ],
 	'pg_basebackup tar with long name fails');
 unlink "$pgdata/$superlongname";
 
@@ -116,17 +115,17 @@ SKIP:
 	$node->psql('postgres',
 		"CREATE TABLESPACE tblspc1 LOCATION '$shorter_tempdir/tblspc1';");
 	$node->psql('postgres', "CREATE TABLE test1 (a int) TABLESPACE tblspc1;");
-	command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup2", '-Ft' ],
+	$node->node_command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup2", '-Ft' ],
 		'tar format with tablespaces');
 	ok(-f "$tempdir/tarbackup2/base.tar", 'backup tar was created');
 	my @tblspc_tars = glob "$tempdir/tarbackup2/[0-9]*.tar";
 	is(scalar(@tblspc_tars), 1, 'one tablespace tar was created');
 
-	command_fails(
+	$node->node_command_fails(
 		[ 'pg_basebackup', '-D', "$tempdir/backup1", '-Fp' ],
 		'plain format with tablespaces fails without tablespace mapping');
 
-	command_ok(
+	$node->node_command_ok(
 		[   'pg_basebackup', '-D', "$tempdir/backup1", '-Fp',
 			"-T$shorter_tempdir/tblspc1=$tempdir/tbackup/tblspc1" ],
 		'plain format with tablespaces succeeds with tablespace mapping');
@@ -145,7 +144,7 @@ SKIP:
 	$node->psql('postgres', "DROP TABLESPACE tblspc1;");
 	$node->psql('postgres',
 		"CREATE TABLESPACE tblspc2 LOCATION '$shorter_tempdir/tbl=spc2';");
-	command_ok(
+	$node->node_command_ok(
 		[   'pg_basebackup', '-D', "$tempdir/backup3", '-Fp',
 			"-T$shorter_tempdir/tbl\\=spc2=$tempdir/tbackup/tbl\\=spc2" ],
 		'mapping tablespace with = sign in path');
@@ -156,12 +155,12 @@ SKIP:
 	mkdir "$tempdir/$superlongname";
 	$node->psql('postgres',
 		"CREATE TABLESPACE tblspc3 LOCATION '$tempdir/$superlongname';");
-	command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l3", '-Ft' ],
+	$node->node_command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l3", '-Ft' ],
 		'pg_basebackup tar with long symlink target');
 	$node->psql('postgres', "DROP TABLESPACE tblspc3;");
 }
 
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backupR", '-R' ],
+$node->node_command_ok([ 'pg_basebackup', '-D', "$tempdir/backupR", '-R' ],
 	'pg_basebackup -R runs');
 ok(-f "$tempdir/backupR/recovery.conf", 'recovery.conf was created');
 my $recovery_conf = slurp_file "$tempdir/backupR/recovery.conf";
@@ -177,19 +176,19 @@ like(
 	qr/^primary_conninfo = '.*port=$ENV{PGPORT}.*'$/m,
 	'recovery.conf sets primary_conninfo');
 
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxf", '-X', 'fetch' ],
+$node->node_command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxf", '-X', 'fetch' ],
 	'pg_basebackup -X fetch runs');
 ok(grep(/^[0-9A-F]{24}$/, slurp_dir("$tempdir/backupxf/pg_xlog")),
 	'WAL files copied');
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs", '-X', 'stream' ],
+$node->node_command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs", '-X', 'stream' ],
 	'pg_basebackup -X stream runs');
 ok(grep(/^[0-9A-F]{24}$/, slurp_dir("$tempdir/backupxf/pg_xlog")),
 	'WAL files copied');
 
-command_fails(
+$node->node_command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/fail", '-S', 'slot1' ],
 	'pg_basebackup with replication slot fails without -X stream');
-command_fails(
+$node->node_command_fails(
 	[   'pg_basebackup',             '-D',
 		"$tempdir/backupxs_sl_fail", '-X',
 		'stream',                    '-S',
@@ -202,7 +201,7 @@ my $lsn = $node->psql('postgres',
 	q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'}
 );
 is($lsn, '', 'restart LSN of new slot is null');
-command_ok(
+$node->node_command_ok(
 	[   'pg_basebackup', '-D', "$tempdir/backupxs_sl", '-X',
 		'stream',        '-S', 'slot1' ],
 	'pg_basebackup -X stream with replication slot runs');
@@ -211,7 +210,7 @@ $lsn = $node->psql('postgres',
 );
 like($lsn, qr!^0/[0-9A-Z]{7,8}$!, 'restart LSN of slot has advanced');
 
-command_ok(
+$node->node_command_ok(
 	[   'pg_basebackup', '-D', "$tempdir/backupxs_sl_R", '-X',
 		'stream',        '-S', 'slot1',                  '-R' ],
 	'pg_basebackup with replication slot and -R runs');
diff --git a/src/bin/scripts/t/010_clusterdb.pl b/src/bin/scripts/t/010_clusterdb.pl
index b1d4185..5d8c24e 100644
--- a/src/bin/scripts/t/010_clusterdb.pl
+++ b/src/bin/scripts/t/010_clusterdb.pl
@@ -13,14 +13,13 @@ my $node = get_new_node();
 $node->init;
 $node->start;
 
-$ENV{PGPORT} = $node->port;
-
 $node->issues_sql_like(
 	['clusterdb'],
 	qr/statement: CLUSTER;/,
 	'SQL CLUSTER run');
 
-command_fails([ 'clusterdb', '-t', 'nonexistent' ],
+$node->node_command_fails(
+	[ 'clusterdb', '-t', 'nonexistent' ],
 	'fails with nonexistent table');
 
 $node->psql('postgres',
diff --git a/src/bin/scripts/t/020_createdb.pl b/src/bin/scripts/t/020_createdb.pl
index 09efdc6..7fd5893 100644
--- a/src/bin/scripts/t/020_createdb.pl
+++ b/src/bin/scripts/t/020_createdb.pl
@@ -12,7 +12,6 @@ program_options_handling_ok('createdb');
 my $node = get_new_node();
 $node->init;
 $node->start;
-$ENV{PGPORT} = $node->port;
 
 $node->issues_sql_like(
 	[ 'createdb', 'foobar1' ],
@@ -23,4 +22,6 @@ $node->issues_sql_like(
 	qr/statement: CREATE DATABASE foobar2 ENCODING 'LATIN1'/,
 	'create database with encoding');
 
-command_fails([ 'createdb', 'foobar1' ], 'fails if database already exists');
+$node->node_command_fails(
+	[ 'createdb', 'foobar1' ],
+	'fails if database already exists');
diff --git a/src/bin/scripts/t/030_createlang.pl b/src/bin/scripts/t/030_createlang.pl
index a323bbf..34d7cac 100644
--- a/src/bin/scripts/t/030_createlang.pl
+++ b/src/bin/scripts/t/030_createlang.pl
@@ -14,9 +14,8 @@ $node->init;
 $node->start;
 
 $ENV{PGDATABASE} = 'postgres';
-$ENV{PGPORT}     = $node->port;
 
-command_fails([ 'createlang', 'plpgsql' ],
+$node->node_command_fails([ 'createlang', 'plpgsql' ],
 	'fails if language already exists');
 
 $node->psql('postgres', 'DROP EXTENSION plpgsql');
@@ -25,4 +24,7 @@ $node->issues_sql_like(
 	qr/statement: CREATE EXTENSION "plpgsql"/,
 	'SQL CREATE EXTENSION run');
 
-command_like([ 'createlang', '--list' ], qr/plpgsql/, 'list output');
+$node->node_command_like(
+	[ 'createlang', '--list' ],
+	qr/plpgsql/,
+	'list output');
diff --git a/src/bin/scripts/t/040_createuser.pl b/src/bin/scripts/t/040_createuser.pl
index 760b437..1642f45 100644
--- a/src/bin/scripts/t/040_createuser.pl
+++ b/src/bin/scripts/t/040_createuser.pl
@@ -13,8 +13,6 @@ my $node = get_new_node();
 $node->init;
 $node->start;
 
-$ENV{PGPORT} = $node->port;
-
 $node->issues_sql_like(
 	[ 'createuser', 'user1' ],
 qr/statement: CREATE ROLE user1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN;/,
@@ -32,4 +30,6 @@ $node->issues_sql_like(
 qr/statement: CREATE ROLE user3 SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a superuser');
 
-command_fails([ 'createuser', 'user1' ], 'fails if role already exists');
+$node->node_command_fails(
+	[ 'createuser', 'user1' ],
+	'fails if role already exists');
diff --git a/src/bin/scripts/t/050_dropdb.pl b/src/bin/scripts/t/050_dropdb.pl
index 50ee604..67ff243 100644
--- a/src/bin/scripts/t/050_dropdb.pl
+++ b/src/bin/scripts/t/050_dropdb.pl
@@ -12,7 +12,6 @@ program_options_handling_ok('dropdb');
 my $node = get_new_node();
 $node->init;
 $node->start;
-$ENV{PGPORT} = $node->port;
 
 $node->psql('postgres', 'CREATE DATABASE foobar1');
 $node->issues_sql_like(
@@ -20,4 +19,6 @@ $node->issues_sql_like(
 	qr/statement: DROP DATABASE foobar1/,
 	'SQL DROP DATABASE run');
 
-command_fails([ 'dropdb', 'nonexistent' ], 'fails with nonexistent database');
+$node->node_command_fails(
+	[ 'dropdb', 'nonexistent' ],
+	'fails with nonexistent database');
diff --git a/src/bin/scripts/t/060_droplang.pl b/src/bin/scripts/t/060_droplang.pl
index 43720fb..46a9eee 100644
--- a/src/bin/scripts/t/060_droplang.pl
+++ b/src/bin/scripts/t/060_droplang.pl
@@ -12,13 +12,12 @@ program_options_handling_ok('droplang');
 my $node = get_new_node();
 $node->init;
 $node->start;
-$ENV{PGPORT} = $node->port;
 
 $node->issues_sql_like(
 	[ 'droplang', 'plpgsql', 'postgres' ],
 	qr/statement: DROP EXTENSION "plpgsql"/,
 	'SQL DROP EXTENSION run');
 
-command_fails(
+$node->node_command_fails(
 	[ 'droplang', 'nonexistent', 'postgres' ],
 	'fails with nonexistent language');
diff --git a/src/bin/scripts/t/070_dropuser.pl b/src/bin/scripts/t/070_dropuser.pl
index 5a452c4..47913ce 100644
--- a/src/bin/scripts/t/070_dropuser.pl
+++ b/src/bin/scripts/t/070_dropuser.pl
@@ -12,7 +12,6 @@ program_options_handling_ok('dropuser');
 my $node = get_new_node();
 $node->init;
 $node->start;
-$ENV{PGPORT} = $node->port;
 
 $node->psql('postgres', 'CREATE ROLE foobar1');
 $node->issues_sql_like(
@@ -20,4 +19,6 @@ $node->issues_sql_like(
 	qr/statement: DROP ROLE foobar1/,
 	'SQL DROP ROLE run');
 
-command_fails([ 'dropuser', 'nonexistent' ], 'fails with nonexistent user');
+$node->node_command_fails(
+	[ 'dropuser', 'nonexistent' ],
+	'fails with nonexistent user');
diff --git a/src/bin/scripts/t/080_pg_isready.pl b/src/bin/scripts/t/080_pg_isready.pl
index ca512c1..5f107bc 100644
--- a/src/bin/scripts/t/080_pg_isready.pl
+++ b/src/bin/scripts/t/080_pg_isready.pl
@@ -15,6 +15,6 @@ my $node = get_new_node();
 $node->init;
 $node->start;
 
-$ENV{PGPORT} = $node->port;
-
-command_ok(['pg_isready'], 'succeeds with server running');
+$node->node_command_ok(
+	['pg_isready'],
+	'succeeds with server running');
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 76a935d..c1fd45f 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -423,6 +423,33 @@ sub poll_query_until
 	return 0;
 }
 
+sub node_command_ok
+{
+	my ($self, $cmd, $test_name) = @_;
+
+	local $ENV{PGPORT} = $self->port;
+
+	TestLib::command_ok($cmd, $test_name);
+}
+
+sub node_command_fails
+{
+	my ($self, $cmd, $test_name) = @_;
+
+	local $ENV{PGPORT} = $self->port;
+
+	TestLib::command_fails($cmd, $test_name);
+}
+
+sub node_command_like
+{
+	my ($self, $cmd, $test_name) = @_;
+
+	local $ENV{PGPORT} = $self->port;
+
+	TestLib::command_like($cmd, $test_name);
+}
+
 # Run a command on the node, then verify that $expected_sql appears in the
 # server log file.
 sub issues_sql_like
#92Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#91)
2 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Dec 2, 2015 at 1:04 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Dec 2, 2015 at 12:01 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Michael Paquier wrote:

On Wed, Dec 2, 2015 at 8:11 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

- It would be nice to have command_ok and command_fails in PostgresNode
too; that would remove the need for setting $ENV{PGPORT} but it's
possible to run commands outside a node too, so we'd need duplicates,
which would be worse.

I am fine to let it the way your patch does it. There are already many changes.

Idea: we can have a bare command_ok exported by TestLib just as
currently, and instance method PostgresNode->command_ok that first sets
local $ENV{PGPORT} and then calls the other one.

Hm. That would be cleaner and make the code more consistent. Now as
TestLib exports command_ok, command_like and command_fails, we would
get redefinition errors when compiling the code if those routines are
not named differently in PostgresNode. If you want to have the names
consistent, then I guess that the only way would be to remove those
routines from the export list of TestLib and call them directly as for
example TestLib::command_ok(). See for example the patch attached that
applies on top on your patch 2 that adds a set of routines in
PostgresNode with a slightly different name.

Well, Alvaro has whispered me a more elegant method by using TestLib()
to only import a portion of the routines and avoid the redefinition
errors. Hence, patch 0001 attached creates equivalents of command_*
for PostgresNode and tests use it without setting PGPORT. Patch 0002
is a run of perltidy on the whole.
--
Michael

Attachments:

0001-Rework-TAP-test-infrastructure-to-ease-node-manageme.patchapplication/x-patch; name=0001-Rework-TAP-test-infrastructure-to-ease-node-manageme.patchDownload
From 970423c6e5c3c30f6ecd82d8691437a2aa4acc87 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Wed, 2 Dec 2015 15:12:03 +0900
Subject: [PATCH 1/2] Rework TAP test infrastructure to ease node management

This is aimed to being used extensively by modules and features to
test features that require a bit more than a standalone node.
---
 src/bin/initdb/t/001_initdb.pl                 |   1 +
 src/bin/pg_basebackup/t/010_pg_basebackup.pl   | 104 +++---
 src/bin/pg_controldata/t/001_pg_controldata.pl |  11 +-
 src/bin/pg_ctl/t/001_start_stop.pl             |   2 +
 src/bin/pg_ctl/t/002_status.pl                 |  13 +-
 src/bin/pg_rewind/RewindTest.pm                | 191 +++-------
 src/bin/pg_rewind/t/003_extrafiles.pl          |   3 +-
 src/bin/pg_rewind/t/004_pg_xlog_symlink.pl     |   4 +-
 src/bin/scripts/t/010_clusterdb.pl             |  23 +-
 src/bin/scripts/t/011_clusterdb_all.pl         |  13 +-
 src/bin/scripts/t/020_createdb.pl              |  13 +-
 src/bin/scripts/t/030_createlang.pl            |  21 +-
 src/bin/scripts/t/040_createuser.pl            |  17 +-
 src/bin/scripts/t/050_dropdb.pl                |  13 +-
 src/bin/scripts/t/060_droplang.pl              |  11 +-
 src/bin/scripts/t/070_dropuser.pl              |  13 +-
 src/bin/scripts/t/080_pg_isready.pl            |   9 +-
 src/bin/scripts/t/090_reindexdb.pl             |  23 +-
 src/bin/scripts/t/091_reindexdb_all.pl         |  10 +-
 src/bin/scripts/t/100_vacuumdb.pl              |  17 +-
 src/bin/scripts/t/101_vacuumdb_all.pl          |  10 +-
 src/bin/scripts/t/102_vacuumdb_stages.pl       |  13 +-
 src/test/perl/PostgresNode.pm                  | 471 +++++++++++++++++++++++++
 src/test/perl/RecursiveCopy.pm                 |  42 +++
 src/test/perl/TestLib.pm                       | 317 ++++++-----------
 src/test/ssl/ServerSetup.pm                    |  34 +-
 src/test/ssl/t/001_ssltests.pl                 |  32 +-
 27 files changed, 911 insertions(+), 520 deletions(-)
 create mode 100644 src/test/perl/PostgresNode.pm
 create mode 100644 src/test/perl/RecursiveCopy.pm

diff --git a/src/bin/initdb/t/001_initdb.pl b/src/bin/initdb/t/001_initdb.pl
index 299dcf5..f64186d 100644
--- a/src/bin/initdb/t/001_initdb.pl
+++ b/src/bin/initdb/t/001_initdb.pl
@@ -4,6 +4,7 @@
 
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 14;
 
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index dc96bbf..46c8a71 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -2,6 +2,7 @@ use strict;
 use warnings;
 use Cwd;
 use Config;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 51;
 
@@ -9,12 +10,17 @@ program_help_ok('pg_basebackup');
 program_version_ok('pg_basebackup');
 program_options_handling_ok('pg_basebackup');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $tempdir = TestLib::tempdir;
 
-command_fails(['pg_basebackup'],
+my $node = get_new_node();
+# Initialize node without replication settings
+$node->init(hba_permit_replication => 0);
+$node->start;
+my $pgdata = $node->data_dir;
+
+$node->command_fails(['pg_basebackup'],
 	'pg_basebackup needs target directory specified');
-command_fails(
+$node->command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup fails because of hba');
 
@@ -26,21 +32,21 @@ if (open BADCHARS, ">>$tempdir/pgdata/FOO\xe0\xe0\xe0BAR")
 	close BADCHARS;
 }
 
-configure_hba_for_replication "$tempdir/pgdata";
-system_or_bail 'pg_ctl', '-D', "$tempdir/pgdata", 'reload';
+$node->set_replication_conf();
+system_or_bail 'pg_ctl', '-D', $pgdata, 'reload';
 
-command_fails(
+$node->command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup fails because of WAL configuration');
 
-open CONF, ">>$tempdir/pgdata/postgresql.conf";
+open CONF, ">>$pgdata/postgresql.conf";
 print CONF "max_replication_slots = 10\n";
 print CONF "max_wal_senders = 10\n";
 print CONF "wal_level = archive\n";
 close CONF;
-restart_test_server;
+$node->restart;
 
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backup" ],
+$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup runs');
 ok(-f "$tempdir/backup/PG_VERSION", 'backup was created');
 
@@ -48,46 +54,46 @@ is_deeply([sort(slurp_dir("$tempdir/backup/pg_xlog/"))],
 		  [sort qw(. .. archive_status)],
 		  'no WAL files copied');
 
-command_ok(
+$node->command_ok(
 	[   'pg_basebackup', '-D', "$tempdir/backup2", '--xlogdir',
 		"$tempdir/xlog2" ],
 	'separate xlog directory');
 ok(-f "$tempdir/backup2/PG_VERSION", 'backup was created');
 ok(-d "$tempdir/xlog2/",             'xlog directory was created');
 
-command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup", '-Ft' ],
+$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup", '-Ft' ],
 	'tar format');
 ok(-f "$tempdir/tarbackup/base.tar", 'backup tar was created');
 
-command_fails(
+$node->command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup_foo", '-Fp', "-T=/foo" ],
 	'-T with empty old directory fails');
-command_fails(
+$node->command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup_foo", '-Fp', "-T/foo=" ],
 	'-T with empty new directory fails');
-command_fails(
+$node->command_fails(
 	[   'pg_basebackup', '-D', "$tempdir/backup_foo", '-Fp',
 		"-T/foo=/bar=/baz" ],
 	'-T with multiple = fails');
-command_fails(
+$node->command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup_foo", '-Fp', "-Tfoo=/bar" ],
 	'-T with old directory not absolute fails');
-command_fails(
+$node->command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup_foo", '-Fp', "-T/foo=bar" ],
 	'-T with new directory not absolute fails');
-command_fails(
+$node->command_fails(
 	[ 'pg_basebackup', '-D', "$tempdir/backup_foo", '-Fp', "-Tfoo" ],
 	'-T with invalid format fails');
 
 # Tar format doesn't support filenames longer than 100 bytes.
 my $superlongname = "superlongname_" . ("x" x 100);
-my $superlongpath = "$tempdir/pgdata/$superlongname";
+my $superlongpath = "$pgdata/$superlongname";
 
 open FILE, ">$superlongpath" or die "unable to create file $superlongpath";
 close FILE;
-command_fails([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l1", '-Ft' ],
+$node->command_fails([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l1", '-Ft' ],
 	'pg_basebackup tar with long name fails');
-unlink "$tempdir/pgdata/$superlongname";
+unlink "$pgdata/$superlongname";
 
 # The following tests test symlinks. Windows doesn't have symlinks, so
 # skip on Windows.
@@ -98,29 +104,29 @@ SKIP: {
 	# to our physical temp location.  That way we can use shorter names
 	# for the tablespace directories, which hopefully won't run afoul of
 	# the 99 character length limit.
-	my $shorter_tempdir = tempdir_short . "/tempdir";
+	my $shorter_tempdir = TestLib::tempdir_short . "/tempdir";
 	symlink "$tempdir", $shorter_tempdir;
 
 	mkdir "$tempdir/tblspc1";
-	psql 'postgres',
-	"CREATE TABLESPACE tblspc1 LOCATION '$shorter_tempdir/tblspc1';";
-	psql 'postgres', "CREATE TABLE test1 (a int) TABLESPACE tblspc1;";
-	command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup2", '-Ft' ],
+	$node->psql('postgres',
+	"CREATE TABLESPACE tblspc1 LOCATION '$shorter_tempdir/tblspc1';");
+	$node->psql('postgres', "CREATE TABLE test1 (a int) TABLESPACE tblspc1;");
+	$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup2", '-Ft' ],
 			   'tar format with tablespaces');
 	ok(-f "$tempdir/tarbackup2/base.tar", 'backup tar was created');
 	my @tblspc_tars = glob "$tempdir/tarbackup2/[0-9]*.tar";
 	is(scalar(@tblspc_tars), 1, 'one tablespace tar was created');
 
-	command_fails(
+	$node->command_fails(
 		[ 'pg_basebackup', '-D', "$tempdir/backup1", '-Fp' ],
 		'plain format with tablespaces fails without tablespace mapping');
 
-	command_ok(
+	$node->command_ok(
 		[   'pg_basebackup', '-D', "$tempdir/backup1", '-Fp',
 			"-T$shorter_tempdir/tblspc1=$tempdir/tbackup/tblspc1" ],
 		'plain format with tablespaces succeeds with tablespace mapping');
 	ok(-d "$tempdir/tbackup/tblspc1", 'tablespace was relocated');
-	opendir(my $dh, "$tempdir/pgdata/pg_tblspc") or die;
+	opendir(my $dh, "$pgdata/pg_tblspc") or die;
 	ok( (   grep {
 		-l "$tempdir/backup1/pg_tblspc/$_"
 			and readlink "$tempdir/backup1/pg_tblspc/$_" eq
@@ -130,26 +136,26 @@ SKIP: {
 	closedir $dh;
 
 	mkdir "$tempdir/tbl=spc2";
-	psql 'postgres', "DROP TABLE test1;";
-	psql 'postgres', "DROP TABLESPACE tblspc1;";
-	psql 'postgres',
-	"CREATE TABLESPACE tblspc2 LOCATION '$shorter_tempdir/tbl=spc2';";
-	command_ok(
+	$node->psql('postgres', "DROP TABLE test1;");
+	$node->psql('postgres', "DROP TABLESPACE tblspc1;");
+	$node->psql('postgres',
+	"CREATE TABLESPACE tblspc2 LOCATION '$shorter_tempdir/tbl=spc2';");
+	$node->command_ok(
 		[   'pg_basebackup', '-D', "$tempdir/backup3", '-Fp',
 			"-T$shorter_tempdir/tbl\\=spc2=$tempdir/tbackup/tbl\\=spc2" ],
 		'mapping tablespace with = sign in path');
 	ok(-d "$tempdir/tbackup/tbl=spc2", 'tablespace with = sign was relocated');
-	psql 'postgres', "DROP TABLESPACE tblspc2;";
+	$node->psql('postgres', "DROP TABLESPACE tblspc2;");
 
 	mkdir "$tempdir/$superlongname";
-	psql 'postgres',
-	"CREATE TABLESPACE tblspc3 LOCATION '$tempdir/$superlongname';";
-	command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l3", '-Ft' ],
+	$node->psql('postgres',
+	"CREATE TABLESPACE tblspc3 LOCATION '$tempdir/$superlongname';");
+	$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l3", '-Ft' ],
 			   'pg_basebackup tar with long symlink target');
-	psql 'postgres', "DROP TABLESPACE tblspc3;";
+	$node->psql('postgres', "DROP TABLESPACE tblspc3;");
 }
 
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backupR", '-R' ],
+$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backupR", '-R' ],
 	'pg_basebackup -R runs');
 ok(-f "$tempdir/backupR/recovery.conf", 'recovery.conf was created');
 my $recovery_conf = slurp_file "$tempdir/backupR/recovery.conf";
@@ -158,27 +164,27 @@ my $recovery_conf = slurp_file "$tempdir/backupR/recovery.conf";
 like($recovery_conf, qr/^standby_mode = 'on[']$/m, 'recovery.conf sets standby_mode');
 like($recovery_conf, qr/^primary_conninfo = '.*port=$ENV{PGPORT}.*'$/m, 'recovery.conf sets primary_conninfo');
 
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxf", '-X', 'fetch' ],
+$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxf", '-X', 'fetch' ],
 	'pg_basebackup -X fetch runs');
 ok(grep(/^[0-9A-F]{24}$/, slurp_dir("$tempdir/backupxf/pg_xlog")), 'WAL files copied');
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs", '-X', 'stream' ],
+$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs", '-X', 'stream' ],
 	'pg_basebackup -X stream runs');
 ok(grep(/^[0-9A-F]{24}$/, slurp_dir("$tempdir/backupxf/pg_xlog")), 'WAL files copied');
 
-command_fails([ 'pg_basebackup', '-D', "$tempdir/fail", '-S', 'slot1' ],
+$node->command_fails([ 'pg_basebackup', '-D', "$tempdir/fail", '-S', 'slot1' ],
 	'pg_basebackup with replication slot fails without -X stream');
-command_fails([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl_fail", '-X', 'stream', '-S', 'slot1' ],
+$node->command_fails([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl_fail", '-X', 'stream', '-S', 'slot1' ],
 	'pg_basebackup fails with nonexistent replication slot');
 
-psql 'postgres', q{SELECT * FROM pg_create_physical_replication_slot('slot1')};
-my $lsn = psql 'postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'};
+$node->psql('postgres', q{SELECT * FROM pg_create_physical_replication_slot('slot1')});
+my $lsn = $node->psql('postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'});
 is($lsn, '', 'restart LSN of new slot is null');
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl", '-X', 'stream', '-S', 'slot1' ],
+$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl", '-X', 'stream', '-S', 'slot1' ],
 	'pg_basebackup -X stream with replication slot runs');
-$lsn = psql 'postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'};
+$lsn = $node->psql('postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'});
 like($lsn, qr!^0/[0-9A-Z]{7,8}$!, 'restart LSN of slot has advanced');
 
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl_R", '-X', 'stream', '-S', 'slot1', '-R' ],
+$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl_R", '-X', 'stream', '-S', 'slot1', '-R' ],
 	'pg_basebackup with replication slot and -R runs');
 like(slurp_file("$tempdir/backupxs_sl_R/recovery.conf"),
 	 qr/^primary_slot_name = 'slot1'$/m,
diff --git a/src/bin/pg_controldata/t/001_pg_controldata.pl b/src/bin/pg_controldata/t/001_pg_controldata.pl
index e2b0d42..ae45f41 100644
--- a/src/bin/pg_controldata/t/001_pg_controldata.pl
+++ b/src/bin/pg_controldata/t/001_pg_controldata.pl
@@ -1,16 +1,19 @@
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
-my $tempdir = TestLib::tempdir;
-
 program_help_ok('pg_controldata');
 program_version_ok('pg_controldata');
 program_options_handling_ok('pg_controldata');
 command_fails(['pg_controldata'], 'pg_controldata without arguments fails');
 command_fails([ 'pg_controldata', 'nonexistent' ],
 	'pg_controldata with nonexistent directory fails');
-standard_initdb "$tempdir/data";
-command_like([ 'pg_controldata', "$tempdir/data" ],
+
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+command_like([ 'pg_controldata', $node->data_dir ],
 	qr/checkpoint/, 'pg_controldata produces output');
diff --git a/src/bin/pg_ctl/t/001_start_stop.pl b/src/bin/pg_ctl/t/001_start_stop.pl
index f57abce..a224ece 100644
--- a/src/bin/pg_ctl/t/001_start_stop.pl
+++ b/src/bin/pg_ctl/t/001_start_stop.pl
@@ -1,6 +1,8 @@
 use strict;
 use warnings;
+
 use Config;
+use PostgresNode;
 use TestLib;
 use Test::More tests => 17;
 
diff --git a/src/bin/pg_ctl/t/002_status.pl b/src/bin/pg_ctl/t/002_status.pl
index 31f7c72..f1c131b 100644
--- a/src/bin/pg_ctl/t/002_status.pl
+++ b/src/bin/pg_ctl/t/002_status.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 3;
 
@@ -9,14 +11,15 @@ my $tempdir_short = TestLib::tempdir_short;
 command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/nonexistent" ],
 	4, 'pg_ctl status with nonexistent directory');
 
-standard_initdb "$tempdir/data";
+my $node = get_new_node();
+$node->init;
 
-command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/data" ],
+command_exit_is([ 'pg_ctl', 'status', '-D', $node->data_dir ],
 	3, 'pg_ctl status with server not running');
 
 system_or_bail 'pg_ctl', '-l', "$tempdir/logfile", '-D',
-  "$tempdir/data", '-w', 'start';
-command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/data" ],
+  $node->data_dir, '-w', 'start';
+command_exit_is([ 'pg_ctl', 'status', '-D', $node->data_dir ],
 	0, 'pg_ctl status with server running');
 
-system_or_bail 'pg_ctl', 'stop', '-D', "$tempdir/data", '-m', 'fast';
+system_or_bail 'pg_ctl', 'stop', '-D', $node->data_dir, '-m', 'fast';
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index a4c1737..55bbc9c 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -9,22 +9,20 @@ package RewindTest;
 # To run a test, the test script (in t/ subdirectory) calls the functions
 # in this module. These functions should be called in this sequence:
 #
-# 1. init_rewind_test - sets up log file etc.
+# 1. setup_cluster - creates a PostgreSQL cluster that runs as the master
 #
-# 2. setup_cluster - creates a PostgreSQL cluster that runs as the master
+# 2. start_master - starts the master server
 #
-# 3. start_master - starts the master server
-#
-# 4. create_standby - runs pg_basebackup to initialize a standby server, and
+# 3. create_standby - runs pg_basebackup to initialize a standby server, and
 #    sets it up to follow the master.
 #
-# 5. promote_standby - runs "pg_ctl promote" to promote the standby server.
+# 4. promote_standby - runs "pg_ctl promote" to promote the standby server.
 # The old master keeps running.
 #
-# 6. run_pg_rewind - stops the old master (if it's still running) and runs
+# 5. run_pg_rewind - stops the old master (if it's still running) and runs
 # pg_rewind to synchronize it with the now-promoted standby server.
 #
-# 7. clean_rewind_test - stops both servers used in the test, if they're
+# 6. clean_rewind_test - stops both servers used in the test, if they're
 # still running.
 #
 # The test script can use the helper functions master_psql and standby_psql
@@ -37,27 +35,23 @@ package RewindTest;
 use strict;
 use warnings;
 
-use TestLib;
-use Test::More;
-
 use Config;
+use Exporter 'import';
 use File::Copy;
 use File::Path qw(rmtree);
-use IPC::Run qw(run start);
+use IPC::Run;
+use PostgresNode;
+use TestLib;
+use Test::More;
 
-use Exporter 'import';
 our @EXPORT = qw(
-  $connstr_master
-  $connstr_standby
-  $test_master_datadir
-  $test_standby_datadir
+  $node_master
+  $node_standby
 
-  append_to_file
   master_psql
   standby_psql
   check_query
 
-  init_rewind_test
   setup_cluster
   start_master
   create_standby
@@ -66,32 +60,24 @@ our @EXPORT = qw(
   clean_rewind_test
 );
 
-our $test_master_datadir  = "$tmp_check/data_master";
-our $test_standby_datadir = "$tmp_check/data_standby";
-
-# Define non-conflicting ports for both nodes.
-my $port_master  = $ENV{PGPORT};
-my $port_standby = $port_master + 1;
-
-my $connstr_master  = "port=$port_master";
-my $connstr_standby = "port=$port_standby";
-
-$ENV{PGDATABASE} = "postgres";
+# Our nodes.
+our $node_master;
+our $node_standby;
 
 sub master_psql
 {
 	my $cmd = shift;
 
-	system_or_bail 'psql', '-q', '--no-psqlrc', '-d', $connstr_master,
-	  '-c', "$cmd";
+	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
+	  $node_master->connstr('postgres'), '-c', "$cmd";
 }
 
 sub standby_psql
 {
 	my $cmd = shift;
 
-	system_or_bail 'psql', '-q', '--no-psqlrc', '-d', $connstr_standby,
-	  '-c', "$cmd";
+	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
+      $node_standby->connstr('postgres'), '-c', "$cmd";
 }
 
 # Run a query against the master, and check that the output matches what's
@@ -102,9 +88,9 @@ sub check_query
 	my ($stdout, $stderr);
 
 	# we want just the output, no formatting
-	my $result = run [
+	my $result = IPC::Run::run [
 		'psql',          '-q', '-A', '-t', '--no-psqlrc', '-d',
-		$connstr_master, '-c', $query ],
+		$node_master->connstr('postgres'), '-c', $query ],
 	  '>', \$stdout, '2>', \$stderr;
 
 	# We don't use ok() for the exit code and stderr, because we want this
@@ -125,56 +111,14 @@ sub check_query
 	}
 }
 
-# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
-sub poll_query_until
-{
-	my ($query, $connstr) = @_;
-
-	my $max_attempts = 30;
-	my $attempts     = 0;
-	my ($stdout, $stderr);
-
-	while ($attempts < $max_attempts)
-	{
-		my $cmd = [ 'psql', '-At', '-c', "$query", '-d', "$connstr" ];
-		my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
-
-		chomp($stdout);
-		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-		if ($stdout eq "t")
-		{
-			return 1;
-		}
-
-		# Wait a second before retrying.
-		sleep 1;
-		$attempts++;
-	}
-
-	# The query result didn't change in 30 seconds. Give up. Print the stderr
-	# from the last attempt, hopefully that's useful for debugging.
-	diag $stderr;
-	return 0;
-}
-
-sub append_to_file
-{
-	my ($filename, $str) = @_;
-
-	open my $fh, ">>", $filename or die "could not open file $filename";
-	print $fh $str;
-	close $fh;
-}
-
 sub setup_cluster
 {
 	# Initialize master, data checksums are mandatory
-	rmtree($test_master_datadir);
-	standard_initdb($test_master_datadir);
+	$node_master = get_new_node();
+	$node_master->init;
 
 	# Custom parameters for master's postgresql.conf
-	append_to_file(
-		"$test_master_datadir/postgresql.conf", qq(
+	$node_master->append_conf("postgresql.conf", qq(
 wal_level = hot_standby
 max_wal_senders = 2
 wal_keep_segments = 20
@@ -185,17 +129,11 @@ hot_standby = on
 autovacuum = off
 max_connections = 10
 ));
-
-	# Accept replication connections on master
-	configure_hba_for_replication $test_master_datadir;
 }
 
 sub start_master
 {
-	system_or_bail('pg_ctl' , '-w',
-				   '-D' , $test_master_datadir,
-				   '-l',  "$log_path/master.log",
-				   "-o", "-p $port_master", 'start');
+	$node_master->start;
 
 	#### Now run the test-specific parts to initialize the master before setting
 	# up standby
@@ -203,24 +141,19 @@ sub start_master
 
 sub create_standby
 {
+	$node_standby = get_new_node();
+	$node_master->backup('my_backup');
+	$node_standby->init_from_backup($node_master, 'my_backup');
+	my $connstr_master = $node_master->connstr('postgres');
 
-	# Set up standby with necessary parameter
-	rmtree $test_standby_datadir;
-
-	# Base backup is taken with xlog files included
-	system_or_bail('pg_basebackup', '-D', $test_standby_datadir,
-				   '-p', $port_master, '-x');
-	append_to_file(
-		"$test_standby_datadir/recovery.conf", qq(
+	$node_standby->append_conf("recovery.conf", qq(
 primary_conninfo='$connstr_master application_name=rewind_standby'
 standby_mode=on
 recovery_target_timeline='latest'
 ));
 
 	# Start standby
-	system_or_bail('pg_ctl', '-w', '-D', $test_standby_datadir,
-				   '-l', "$log_path/standby.log",
-				   '-o', "-p $port_standby", 'start');
+	$node_standby->start;
 
 	# The standby may have WAL to apply before it matches the primary.  That
 	# is fine, because no test examines the standby before promotion.
@@ -234,14 +167,14 @@ sub promote_standby
 	# Wait for the standby to receive and write all WAL.
 	my $wal_received_query =
 "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = 'rewind_standby';";
-	poll_query_until($wal_received_query, $connstr_master)
+	$node_master->poll_query_until('postgres', $wal_received_query)
 	  or die "Timed out while waiting for standby to receive and write WAL";
 
 	# Now promote slave and insert some new data on master, this will put
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
-	system_or_bail('pg_ctl', '-w', '-D', $test_standby_datadir, 'promote');
-	poll_query_until("SELECT NOT pg_is_in_recovery()", $connstr_standby)
+	system_or_bail('pg_ctl', '-w', '-D', $node_standby->data_dir, 'promote');
+	$node_standby->poll_query_until('postgres', "SELECT NOT pg_is_in_recovery()")
 	  or die "Timed out while waiting for promotion of standby";
 
 	# Force a checkpoint after the promotion. pg_rewind looks at the control
@@ -256,9 +189,13 @@ sub promote_standby
 sub run_pg_rewind
 {
 	my $test_mode = shift;
+	my $master_pgdata = $node_master->data_dir;
+	my $standby_pgdata = $node_standby->data_dir;
+	my $standby_connstr = $node_standby->connstr('postgres');
+	my $tmp_folder = TestLib::tempdir;
 
 	# Stop the master and be ready to perform the rewind
-	system_or_bail('pg_ctl', '-D', $test_master_datadir, '-m', 'fast', 'stop');
+	$node_master->stop;
 
 	# At this point, the rewind processing is ready to run.
 	# We now have a very simple scenario with a few diverged WAL record.
@@ -267,20 +204,19 @@ sub run_pg_rewind
 
 	# Keep a temporary postgresql.conf for master node or it would be
 	# overwritten during the rewind.
-	copy("$test_master_datadir/postgresql.conf",
-		 "$tmp_check/master-postgresql.conf.tmp");
+	copy("$master_pgdata/postgresql.conf",
+		 "$tmp_folder/master-postgresql.conf.tmp");
 
 	# Now run pg_rewind
 	if ($test_mode eq "local")
 	{
 		# Do rewind using a local pgdata as source
 		# Stop the master and be ready to perform the rewind
-		system_or_bail('pg_ctl', '-D', $test_standby_datadir,
-					   '-m', 'fast', 'stop');
+		$node_standby->stop;
 		command_ok(['pg_rewind',
 					"--debug",
-					"--source-pgdata=$test_standby_datadir",
-					"--target-pgdata=$test_master_datadir"],
+					"--source-pgdata=$standby_pgdata",
+					"--target-pgdata=$master_pgdata"],
 				   'pg_rewind local');
 	}
 	elsif ($test_mode eq "remote")
@@ -289,33 +225,30 @@ sub run_pg_rewind
 		command_ok(['pg_rewind',
 					"--debug",
 					"--source-server",
-					"port=$port_standby dbname=postgres",
-					"--target-pgdata=$test_master_datadir"],
+					$standby_connstr,
+					"--target-pgdata=$master_pgdata"],
 				   'pg_rewind remote');
 	}
 	else
 	{
-
 		# Cannot come here normally
 		die("Incorrect test mode specified");
 	}
 
 	# Now move back postgresql.conf with old settings
-	move("$tmp_check/master-postgresql.conf.tmp",
-		 "$test_master_datadir/postgresql.conf");
+	move("$tmp_folder/master-postgresql.conf.tmp",
+		 "$master_pgdata/postgresql.conf");
 
 	# Plug-in rewound node to the now-promoted standby node
-	append_to_file(
-		"$test_master_datadir/recovery.conf", qq(
+	my $port_standby = $node_standby->port;
+	$node_master->append_conf('recovery.conf', qq(
 primary_conninfo='port=$port_standby'
 standby_mode=on
 recovery_target_timeline='latest'
 ));
 
 	# Restart the master to check that rewind went correctly
-	system_or_bail('pg_ctl', '-w', '-D', $test_master_datadir,
-				   '-l', "$log_path/master.log",
-				   '-o', "-p $port_master", 'start');
+	$node_master->restart;
 
 	#### Now run the test-specific parts to check the result
 }
@@ -323,22 +256,8 @@ recovery_target_timeline='latest'
 # Clean up after the test. Stop both servers, if they're still running.
 sub clean_rewind_test
 {
-	if ($test_master_datadir)
-	{
-		system
-		  'pg_ctl', '-D', $test_master_datadir, '-m', 'immediate', 'stop';
-	}
-	if ($test_standby_datadir)
-	{
-		system
-		  'pg_ctl', '-D', $test_standby_datadir, '-m', 'immediate', 'stop';
-	}
+	$node_master->teardown_node if defined $node_master;
+	$node_standby->teardown_node if defined $node_standby;
 }
 
-# Stop the test servers, just in case they're still running.
-END
-{
-	my $save_rc = $?;
-	clean_rewind_test();
-	$? = $save_rc;
-}
+1;
diff --git a/src/bin/pg_rewind/t/003_extrafiles.pl b/src/bin/pg_rewind/t/003_extrafiles.pl
index d317f53..cedde14 100644
--- a/src/bin/pg_rewind/t/003_extrafiles.pl
+++ b/src/bin/pg_rewind/t/003_extrafiles.pl
@@ -17,7 +17,7 @@ sub run_test
 	RewindTest::setup_cluster();
 	RewindTest::start_master();
 
-	my $test_master_datadir = $RewindTest::test_master_datadir;
+	my $test_master_datadir = $node_master->data_dir;
 
 	# Create a subdir and files that will be present in both
 	mkdir "$test_master_datadir/tst_both_dir";
@@ -30,6 +30,7 @@ sub run_test
 	RewindTest::create_standby();
 
 	# Create different subdirs and files in master and standby
+	my $test_standby_datadir = $node_standby->data_dir;
 
 	mkdir "$test_standby_datadir/tst_standby_dir";
 	append_to_file "$test_standby_datadir/tst_standby_dir/standby_file1",
diff --git a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
index c5f72e2..bdcab56 100644
--- a/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
+++ b/src/bin/pg_rewind/t/004_pg_xlog_symlink.pl
@@ -23,11 +23,13 @@ sub run_test
 {
 	my $test_mode = shift;
 
-	my $master_xlogdir = "$tmp_check/xlog_master";
+	my $master_xlogdir = "${TestLib::tmp_check}/xlog_master";
 
 	rmtree($master_xlogdir);
 	RewindTest::setup_cluster();
 
+	my $test_master_datadir = $node_master->data_dir;
+
 	# turn pg_xlog into a symlink
 	print("moving $test_master_datadir/pg_xlog to $master_xlogdir\n");
 	move("$test_master_datadir/pg_xlog", $master_xlogdir) or die;
diff --git a/src/bin/scripts/t/010_clusterdb.pl b/src/bin/scripts/t/010_clusterdb.pl
index dc0d78a..bdb140f 100644
--- a/src/bin/scripts/t/010_clusterdb.pl
+++ b/src/bin/scripts/t/010_clusterdb.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
@@ -7,20 +9,21 @@ program_help_ok('clusterdb');
 program_version_ok('clusterdb');
 program_options_handling_ok('clusterdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
-	[ 'clusterdb', 'postgres' ],
+$node->issues_sql_like(
+	[ 'clusterdb' ],
 	qr/statement: CLUSTER;/,
 	'SQL CLUSTER run');
 
-command_fails([ 'clusterdb', '-t', 'nonexistent', 'postgres' ],
-	'fails with nonexistent table');
+$node->command_fails([ 'clusterdb', '-t', 'nonexistent' ],
+			  'fails with nonexistent table');
 
-psql 'postgres',
-'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x';
-issues_sql_like(
-	[ 'clusterdb', '-t', 'test1', 'postgres' ],
+$node->psql('postgres',
+	'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x');
+$node->issues_sql_like(
+	[ 'clusterdb', '-t', 'test1' ],
 	qr/statement: CLUSTER test1;/,
 	'cluster specific table');
diff --git a/src/bin/scripts/t/011_clusterdb_all.pl b/src/bin/scripts/t/011_clusterdb_all.pl
index 7769f70..15cd30c 100644
--- a/src/bin/scripts/t/011_clusterdb_all.pl
+++ b/src/bin/scripts/t/011_clusterdb_all.pl
@@ -1,12 +1,19 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+# clusterdb -a is not compatible with -d, hence enforce environment variable
+# correctly.
+$ENV{PGDATABASE} = 'postgres';
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'clusterdb', '-a' ],
 	qr/statement: CLUSTER.*statement: CLUSTER/s,
 	'cluster all databases');
diff --git a/src/bin/scripts/t/020_createdb.pl b/src/bin/scripts/t/020_createdb.pl
index a44283c..840c1b3 100644
--- a/src/bin/scripts/t/020_createdb.pl
+++ b/src/bin/scripts/t/020_createdb.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 13;
 
@@ -7,16 +9,17 @@ program_help_ok('createdb');
 program_version_ok('createdb');
 program_options_handling_ok('createdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'createdb', 'foobar1' ],
 	qr/statement: CREATE DATABASE foobar1/,
 	'SQL CREATE DATABASE run');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'createdb', '-l', 'C', '-E', 'LATIN1', '-T', 'template0', 'foobar2' ],
 	qr/statement: CREATE DATABASE foobar2 ENCODING 'LATIN1'/,
 	'create database with encoding');
 
-command_fails([ 'createdb', 'foobar1' ], 'fails if database already exists');
+$node->command_fails([ 'createdb', 'foobar1' ], 'fails if database already exists');
diff --git a/src/bin/scripts/t/030_createlang.pl b/src/bin/scripts/t/030_createlang.pl
index 7ff0a3e..6d14c98 100644
--- a/src/bin/scripts/t/030_createlang.pl
+++ b/src/bin/scripts/t/030_createlang.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 14;
 
@@ -7,18 +9,21 @@ program_help_ok('createlang');
 program_version_ok('createlang');
 program_options_handling_ok('createlang');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
+
+$ENV{PGDATABASE} = 'postgres';
 
-command_fails(
-	[ 'createlang', 'plpgsql', 'postgres' ],
+$node->command_fails(
+	[ 'createlang', 'plpgsql' ],
 	'fails if language already exists');
 
-psql 'postgres', 'DROP EXTENSION plpgsql';
-issues_sql_like(
-	[ 'createlang', 'plpgsql', 'postgres' ],
+$node->psql('postgres', 'DROP EXTENSION plpgsql');
+$node->issues_sql_like(
+	[ 'createlang', 'plpgsql' ],
 	qr/statement: CREATE EXTENSION "plpgsql"/,
 	'SQL CREATE EXTENSION run');
 
-command_like([ 'createlang', '--list', 'postgres' ],
+$node->command_like([ 'createlang', '--list' ],
 	qr/plpgsql/, 'list output');
diff --git a/src/bin/scripts/t/040_createuser.pl b/src/bin/scripts/t/040_createuser.pl
index 4d44e14..ac08fde 100644
--- a/src/bin/scripts/t/040_createuser.pl
+++ b/src/bin/scripts/t/040_createuser.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 17;
 
@@ -7,24 +9,25 @@ program_help_ok('createuser');
 program_version_ok('createuser');
 program_options_handling_ok('createuser');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'createuser', 'user1' ],
 qr/statement: CREATE ROLE user1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN;/,
 	'SQL CREATE USER run');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'createuser', '-L', 'role1' ],
 qr/statement: CREATE ROLE role1 NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT NOLOGIN;/,
 	'create a non-login role');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'createuser', '-r', 'user2' ],
 qr/statement: CREATE ROLE user2 NOSUPERUSER NOCREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a CREATEROLE user');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'createuser', '-s', 'user3' ],
 qr/statement: CREATE ROLE user3 SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a superuser');
 
-command_fails([ 'createuser', 'user1' ], 'fails if role already exists');
+$node->command_fails([ 'createuser', 'user1' ], 'fails if role already exists');
diff --git a/src/bin/scripts/t/050_dropdb.pl b/src/bin/scripts/t/050_dropdb.pl
index 3065e50..fb4feb8 100644
--- a/src/bin/scripts/t/050_dropdb.pl
+++ b/src/bin/scripts/t/050_dropdb.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,13 +9,14 @@ program_help_ok('dropdb');
 program_version_ok('dropdb');
 program_options_handling_ok('dropdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-psql 'postgres', 'CREATE DATABASE foobar1';
-issues_sql_like(
+$node->psql('postgres', 'CREATE DATABASE foobar1');
+$node->issues_sql_like(
 	[ 'dropdb', 'foobar1' ],
 	qr/statement: DROP DATABASE foobar1/,
 	'SQL DROP DATABASE run');
 
-command_fails([ 'dropdb', 'nonexistent' ], 'fails with nonexistent database');
+$node->command_fails([ 'dropdb', 'nonexistent' ], 'fails with nonexistent database');
diff --git a/src/bin/scripts/t/060_droplang.pl b/src/bin/scripts/t/060_droplang.pl
index 6a21d7e..7228047 100644
--- a/src/bin/scripts/t/060_droplang.pl
+++ b/src/bin/scripts/t/060_droplang.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,14 +9,15 @@ program_help_ok('droplang');
 program_version_ok('droplang');
 program_options_handling_ok('droplang');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'droplang', 'plpgsql', 'postgres' ],
 	qr/statement: DROP EXTENSION "plpgsql"/,
 	'SQL DROP EXTENSION run');
 
-command_fails(
+$node->command_fails(
 	[ 'droplang', 'nonexistent', 'postgres' ],
 	'fails with nonexistent language');
diff --git a/src/bin/scripts/t/070_dropuser.pl b/src/bin/scripts/t/070_dropuser.pl
index bbb3b79..6e87f1e 100644
--- a/src/bin/scripts/t/070_dropuser.pl
+++ b/src/bin/scripts/t/070_dropuser.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 11;
 
@@ -7,13 +9,14 @@ program_help_ok('dropuser');
 program_version_ok('dropuser');
 program_options_handling_ok('dropuser');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-psql 'postgres', 'CREATE ROLE foobar1';
-issues_sql_like(
+$node->psql('postgres', 'CREATE ROLE foobar1');
+$node->issues_sql_like(
 	[ 'dropuser', 'foobar1' ],
 	qr/statement: DROP ROLE foobar1/,
 	'SQL DROP ROLE run');
 
-command_fails([ 'dropuser', 'nonexistent' ], 'fails with nonexistent user');
+$node->command_fails([ 'dropuser', 'nonexistent' ], 'fails with nonexistent user');
diff --git a/src/bin/scripts/t/080_pg_isready.pl b/src/bin/scripts/t/080_pg_isready.pl
index f432505..8f3f25c 100644
--- a/src/bin/scripts/t/080_pg_isready.pl
+++ b/src/bin/scripts/t/080_pg_isready.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 10;
 
@@ -9,7 +11,8 @@ program_options_handling_ok('pg_isready');
 
 command_fails(['pg_isready'], 'fails with no server running');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-command_ok(['pg_isready'], 'succeeds with server running');
+$node->command_ok(['pg_isready'], 'succeeds with server running');
diff --git a/src/bin/scripts/t/090_reindexdb.pl b/src/bin/scripts/t/090_reindexdb.pl
index 42628c2..68b2291 100644
--- a/src/bin/scripts/t/090_reindexdb.pl
+++ b/src/bin/scripts/t/090_reindexdb.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 20;
 
@@ -7,35 +9,36 @@ program_help_ok('reindexdb');
 program_version_ok('reindexdb');
 program_options_handling_ok('reindexdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
 $ENV{PGOPTIONS} = '--client-min-messages=WARNING';
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'reindexdb', 'postgres' ],
 	qr/statement: REINDEX DATABASE postgres;/,
 	'SQL REINDEX run');
 
-psql 'postgres',
-  'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a);';
-issues_sql_like(
+$node->psql('postgres',
+  'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a);');
+$node->issues_sql_like(
 	[ 'reindexdb', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX TABLE test1;/,
 	'reindex specific table');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'reindexdb', '-i', 'test1x', 'postgres' ],
 	qr/statement: REINDEX INDEX test1x;/,
 	'reindex specific index');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'reindexdb', '-S', 'pg_catalog', 'postgres' ],
 	qr/statement: REINDEX SCHEMA pg_catalog;/,
 	'reindex specific schema');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'reindexdb', '-s', 'postgres' ],
 	qr/statement: REINDEX SYSTEM postgres;/,
 	'reindex system tables');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'reindexdb', '-v', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX \(VERBOSE\) TABLE test1;/,
 	'reindex with verbose output');
diff --git a/src/bin/scripts/t/091_reindexdb_all.pl b/src/bin/scripts/t/091_reindexdb_all.pl
index ffadf29..d47b18b 100644
--- a/src/bin/scripts/t/091_reindexdb_all.pl
+++ b/src/bin/scripts/t/091_reindexdb_all.pl
@@ -1,14 +1,16 @@
 use strict;
 use warnings;
-use TestLib;
+
+use PostgresNode;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
 $ENV{PGOPTIONS} = '--client-min-messages=WARNING';
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'reindexdb', '-a' ],
 	qr/statement: REINDEX.*statement: REINDEX/s,
 	'reindex all databases');
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index ac160ba..387d2b4 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+
+use PostgresNode;
 use TestLib;
 use Test::More tests => 18;
 
@@ -7,26 +9,27 @@ program_help_ok('vacuumdb');
 program_version_ok('vacuumdb');
 program_options_handling_ok('vacuumdb');
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', 'postgres' ],
 	qr/statement: VACUUM;/,
 	'SQL VACUUM run');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '-f', 'postgres' ],
 	qr/statement: VACUUM \(FULL\);/,
 	'vacuumdb -f');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '-F', 'postgres' ],
 	qr/statement: VACUUM \(FREEZE\);/,
 	'vacuumdb -F');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '-z', 'postgres' ],
 	qr/statement: VACUUM \(ANALYZE\);/,
 	'vacuumdb -z');
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '-Z', 'postgres' ],
 	qr/statement: ANALYZE;/,
 	'vacuumdb -Z');
diff --git a/src/bin/scripts/t/101_vacuumdb_all.pl b/src/bin/scripts/t/101_vacuumdb_all.pl
index e90f321..8f1536f 100644
--- a/src/bin/scripts/t/101_vacuumdb_all.pl
+++ b/src/bin/scripts/t/101_vacuumdb_all.pl
@@ -1,12 +1,14 @@
 use strict;
 use warnings;
-use TestLib;
+
+use PostgresNode;
 use Test::More tests => 2;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '-a' ],
 	qr/statement: VACUUM.*statement: VACUUM/s,
 	'vacuum all databases');
diff --git a/src/bin/scripts/t/102_vacuumdb_stages.pl b/src/bin/scripts/t/102_vacuumdb_stages.pl
index 57b980e..4cb5b64 100644
--- a/src/bin/scripts/t/102_vacuumdb_stages.pl
+++ b/src/bin/scripts/t/102_vacuumdb_stages.pl
@@ -1,12 +1,14 @@
 use strict;
 use warnings;
-use TestLib;
+
+use PostgresNode;
 use Test::More tests => 4;
 
-my $tempdir = tempdir;
-start_test_server $tempdir;
+my $node = get_new_node();
+$node->init;
+$node->start;
 
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '--analyze-in-stages', 'postgres' ],
 qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE.*
@@ -16,8 +18,7 @@ qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE/sx,
 	'analyze three times');
 
-
-issues_sql_like(
+$node->issues_sql_like(
 	[ 'vacuumdb', '--analyze-in-stages', '--all' ],
 qr/.*statement:\ SET\ default_statistics_target=1;\ SET\ vacuum_cost_delay=0;
                    .*statement:\ ANALYZE.*
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
new file mode 100644
index 0000000..c182ce7
--- /dev/null
+++ b/src/test/perl/PostgresNode.pm
@@ -0,0 +1,471 @@
+# PostgresNode, class representing a data directory and postmaster.
+#
+# This contains a basic set of routines able to work on a PostgreSQL node,
+# allowing to start, stop, backup and initialize it with various options.
+# The set of nodes managed by a given test is also managed by this module.
+
+package PostgresNode;
+
+use strict;
+use warnings;
+
+use Config;
+use Cwd;
+use Exporter 'import';
+use File::Basename;
+use File::Spec;
+use File::Temp ();
+use IPC::Run;
+use PostgresNode;
+use RecursiveCopy;
+use Test::More;
+use TestLib qw(
+  $windows_os
+  slurp_file
+  run_log
+  system_or_bail
+  system_log
+  append_to_file
+);
+
+our @EXPORT = qw(
+  get_new_node
+);
+
+our ($test_pghost, $last_port_assigned, @all_nodes);
+
+BEGIN
+{
+	# PGHOST is set once and for all through a single series of tests when
+	# this module is loaded.
+	$test_pghost = $windows_os ? "127.0.0.1" : TestLib::tempdir_short();
+	$ENV{PGHOST} = $test_pghost;
+	$ENV{PGDATABASE} = 'postgres';
+
+	# Tracking of last port value assigned to accelerate free port lookup.
+	# XXX: Should this use PG_VERSION_NUM?
+	$last_port_assigned = 90600 % 16384 + 49152;
+
+	# Node tracking
+	@all_nodes = ();
+}
+
+sub new
+{
+	my $class  = shift;
+	my $pghost = shift;
+	my $pgport = shift;
+	my $self   = {
+		_port     => $pgport,
+		_host     => $pghost,
+		_basedir  => TestLib::tempdir,
+		_applname => "node_$pgport",
+		_logfile  => "$TestLib::log_path/node_$pgport.log" };
+
+	bless $self, $class;
+	$self->dump_info;
+
+	return $self;
+}
+
+sub port
+{
+	my ($self) = @_;
+	return $self->{_port};
+}
+
+sub host
+{
+	my ($self) = @_;
+	return $self->{_host};
+}
+
+sub basedir
+{
+	my ($self) = @_;
+	return $self->{_basedir};
+}
+
+sub applname
+{
+	my ($self) = @_;
+	return $self->{_applname};
+}
+
+sub logfile
+{
+	my ($self) = @_;
+	return $self->{_logfile};
+}
+
+sub connstr
+{
+	my ($self, $dbname) = @_;
+	my $pgport = $self->port;
+	my $pghost = $self->host;
+	if (!defined($dbname))
+	{
+		return "port=$pgport host=$pghost";
+	}
+	return "port=$pgport host=$pghost dbname=$dbname";
+}
+
+sub data_dir
+{
+	my ($self) = @_;
+	my $res = $self->basedir;
+	return "$res/pgdata";
+}
+
+sub archive_dir
+{
+	my ($self) = @_;
+	my $basedir = $self->basedir;
+	return "$basedir/archives";
+}
+
+sub backup_dir
+{
+	my ($self) = @_;
+	my $basedir = $self->basedir;
+	return "$basedir/backup";
+}
+
+# Dump node information
+sub dump_info
+{
+	my ($self) = @_;
+	print "Data directory: " . $self->data_dir . "\n";
+	print "Backup directory: " . $self->backup_dir . "\n";
+	print "Archive directory: " . $self->archive_dir . "\n";
+	print "Connection string: " . $self->connstr . "\n";
+	print "Application name: " . $self->applname . "\n";
+	print "Log file: " . $self->logfile . "\n";
+}
+
+sub set_replication_conf
+{
+	my ($self) = @_;
+	my $pgdata = $self->data_dir;
+
+	open my $hba, ">>$pgdata/pg_hba.conf";
+	print $hba "\n# Allow replication (set up by PostgresNode.pm)\n";
+	if (!$windows_os)
+	{
+		print $hba "local replication all trust\n";
+	}
+	else
+	{
+		print $hba
+"host replication all 127.0.0.1/32 sspi include_realm=1 map=regress\n";
+	}
+	close $hba;
+}
+
+# Initialize a new cluster for testing.
+#
+# Authentication is set up so that only the current OS user can access the
+# cluster. On Unix, we use Unix domain socket connections, with the socket in
+# a directory that's only accessible to the current user to ensure that.
+# On Windows, we use SSPI authentication to ensure the same (by pg_regress
+# --config-auth).
+sub init
+{
+	my ($self, %params) = @_;
+	my $port   = $self->port;
+	my $pgdata = $self->data_dir;
+	my $host   = $self->host;
+
+	$params{hba_permit_replication} = 1 if (!defined($params{hba_permit_replication}));
+
+	mkdir $self->backup_dir;
+	mkdir $self->archive_dir;
+
+	system_or_bail('initdb', '-D', $pgdata, '-A', 'trust', '-N');
+	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
+
+	open my $conf, ">>$pgdata/postgresql.conf";
+	print $conf "\n# Added by TestLib.pm)\n";
+	print $conf "fsync = off\n";
+	print $conf "log_statement = all\n";
+	print $conf "port = $port\n";
+	if ($windows_os)
+	{
+		print $conf "listen_addresses = '$host'\n";
+	}
+	else
+	{
+		print $conf "unix_socket_directories = '$host'\n";
+		print $conf "listen_addresses = ''\n";
+	}
+	close $conf;
+
+	$self->set_replication_conf if ($params{hba_permit_replication});
+}
+
+sub append_conf
+{
+	my ($self, $filename, $str) = @_;
+
+	my $conffile = $self->data_dir . '/' . $filename;
+
+	append_to_file($conffile, $str);
+}
+
+sub backup
+{
+	my ($self, $backup_name) = @_;
+	my $backup_path = $self->backup_dir . '/' . $backup_name;
+	my $port        = $self->port;
+
+	print "# Taking backup $backup_name from node with port $port\n";
+	system_or_bail("pg_basebackup -D $backup_path -p $port -x");
+	print "# Backup finished\n";
+}
+
+sub init_from_backup
+{
+	my ($self, $root_node, $backup_name) = @_;
+	my $backup_path = $root_node->backup_dir . '/' . $backup_name;
+	my $port        = $self->port;
+	my $root_port   = $root_node->port;
+
+	print
+"Initializing node $port from backup \"$backup_name\" of node $root_port\n";
+	die "Backup $backup_path does not exist" unless -d $backup_path;
+
+	mkdir $self->backup_dir;
+	mkdir $self->archive_dir;
+
+	my $data_path = $self->data_dir;
+	rmdir($data_path);
+	RecursiveCopy::copypath($backup_path, $data_path);
+	chmod(0700, $data_path);
+
+	# Base configuration for this node
+	$self->append_conf('postgresql.conf',
+		qq(
+port = $port
+));
+	$self->set_replication_conf;
+}
+
+sub start
+{
+	my ($self) = @_;
+	my $port   = $self->port;
+	my $pgdata = $self->data_dir;
+	print("### Starting test server in $pgdata\n");
+	my $ret = system_log('pg_ctl', '-w', '-D', $self->data_dir,
+		'-l', $self->logfile, 'start');
+
+	if ($ret != 0)
+	{
+		print "# pg_ctl failed; logfile:\n";
+		print slurp_file($self->logfile);
+		BAIL_OUT("pg_ctl failed");
+	}
+
+	$self->_update_pid;
+
+}
+
+sub stop
+{
+	my ($self, $mode) = @_;
+	my $port   = $self->port;
+	my $pgdata = $self->data_dir;
+	$mode = 'fast' if (!defined($mode));
+	print "### Stopping node in $pgdata with port $port using mode $mode\n";
+	system_log('pg_ctl', '-D', $pgdata, '-m', $mode, 'stop');
+	$self->{_pid} = undef;
+	$self->_update_pid;
+}
+
+sub restart
+{
+	my ($self)  = @_;
+	my $port    = $self->port;
+	my $pgdata  = $self->data_dir;
+	my $logfile = $self->logfile;
+	system_log('pg_ctl', '-D', $pgdata, '-w', '-l', $logfile, 'restart');
+	$self->_update_pid;
+}
+
+sub _update_pid
+{
+	my $self = shift;
+
+	# If we can open the PID file, read its first line and that's the PID we
+	# want.  If the file cannot be opened, presumably the server is not
+	# running; don't be noisy in that case.
+	open my $pidfile, $self->data_dir . "/postmaster.pid";
+	if (not defined $pidfile)
+	{
+		$self->{_pid} = undef;
+		print "# No postmaster PID\n";
+		return;
+	}
+
+	$self->{_pid} = <$pidfile>;
+	print "# Postmaster PID is $self->{_pid}\n";
+	close $pidfile;
+}
+
+
+#
+# Cluster management functions
+#
+
+# Build a new PostgresNode object, assigning a free port number.
+#
+# We also register the node, to avoid the port number from being reused
+# for another node even when this one is not active.
+sub get_new_node
+{
+	my $found = 0;
+	my $port  = $last_port_assigned;
+
+	while ($found == 0)
+	{
+		$port++;
+		print "# Checking for port $port\n";
+		my $devnull = $windows_os ? "nul" : "/dev/null";
+		if (!run_log([ 'pg_isready', '-p', $port ]))
+		{
+			$found = 1;
+
+			# Found a potential candidate port number.  Check first that it is
+			# not included in the list of registered nodes.
+			foreach my $node (@all_nodes)
+			{
+				$found = 0 if ($node->port == $port);
+			}
+		}
+	}
+
+	print "# Found free port $port\n";
+
+	# Lock port number found by creating a new node
+	my $node = new PostgresNode($test_pghost, $port);
+
+	# Add node to list of nodes
+	push(@all_nodes, $node);
+
+	# And update port for next time
+	$last_port_assigned = $port;
+
+	return $node;
+}
+
+sub DESTROY
+{
+	my $self = shift;
+	return if not defined $self->{_pid};
+	print "# signalling QUIT to $self->{_pid}\n";
+	kill 'QUIT', $self->{_pid};
+}
+
+sub teardown_node
+{
+	my $self = shift;
+
+	$self->stop('immediate');
+}
+
+sub psql
+{
+	my ($self, $dbname, $sql) = @_;
+
+	my ($stdout, $stderr);
+	print("# Running SQL command: $sql\n");
+
+	IPC::Run::run [ 'psql', '-XAtq', '-d', $self->connstr($dbname), '-f', '-' ],
+	  '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
+
+	if ($stderr ne "")
+	{
+		print "#### Begin standard error\n";
+		print $stderr;
+		print "#### End standard error\n";
+	}
+	chomp $stdout;
+	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+	return $stdout;
+}
+
+# Run a query once a second, until it returns 't' (i.e. SQL boolean true).
+sub poll_query_until
+{
+	my ($self, $dbname, $query) = @_;
+
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	my ($stdout, $stderr);
+
+	while ($attempts < $max_attempts)
+	{
+		my $cmd = [ 'psql', '-At', '-c', $query, '-d', $self->connstr($dbname) ];
+		my $result = IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr;
+
+		chomp($stdout);
+		$stdout =~ s/\r//g if $Config{osname} eq 'msys';
+		if ($stdout eq "t")
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+
+	# The query result didn't change in 30 seconds. Give up. Print the stderr
+	# from the last attempt, hopefully that's useful for debugging.
+	diag $stderr;
+	return 0;
+}
+
+sub command_ok
+{
+	my ($self, $cmd, $test_name) = @_;
+
+	local $ENV{PGPORT} = $self->port;
+
+	TestLib::command_ok($cmd, $test_name);
+}
+
+sub command_fails
+{
+	my ($self, $cmd, $test_name) = @_;
+
+	local $ENV{PGPORT} = $self->port;
+
+	TestLib::command_fails($cmd, $test_name);
+}
+
+sub command_like
+{
+	my ($self, $cmd, $test_name) = @_;
+
+	local $ENV{PGPORT} = $self->port;
+
+	TestLib::command_like($cmd, $test_name);
+}
+
+# Run a command on the node, then verify that $expected_sql appears in the
+# server log file.
+sub issues_sql_like
+{
+	my ($self, $cmd, $expected_sql, $test_name) = @_;
+
+	local $ENV{PGPORT} = $self->port;
+
+	truncate $self->logfile, 0;
+	my $result = run_log($cmd);
+	ok($result, "@$cmd exit code 0");
+	my $log = slurp_file($self->logfile);
+	like($log, $expected_sql, "$test_name: SQL found in server log");
+}
+
+1;
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
new file mode 100644
index 0000000..4e58ad3
--- /dev/null
+++ b/src/test/perl/RecursiveCopy.pm
@@ -0,0 +1,42 @@
+# RecursiveCopy, a simple recursive copy implementation
+package RecursiveCopy;
+
+use strict;
+use warnings;
+
+use File::Basename;
+use File::Copy;
+
+sub copypath
+{
+	my $srcpath  = shift;
+	my $destpath = shift;
+
+	die "Cannot operate on symlinks" if -l $srcpath or -l $destpath;
+
+	# This source path is a file, simply copy it to destination with the
+	# same name.
+	die "Destination path $destpath exists as file" if -f $destpath;
+	if (-f $srcpath)
+	{
+		copy($srcpath, $destpath)
+			or die "copy $srcpath -> $destpath failed: $!";
+		return 1;
+	}
+
+	die "Destination needs to be a directory" unless -d $srcpath;
+	mkdir($destpath) or die "mkdir($destpath) failed: $!";
+
+	# Scan existing source directory and recursively copy everything.
+	opendir(my $directory, $srcpath) or die "could not opendir($srcpath): $!";
+	while (my $entry = readdir($directory))
+	{
+		next if ($entry eq '.' || $entry eq '..');
+		RecursiveCopy::copypath("$srcpath/$entry", "$destpath/$entry")
+			or die "copypath $srcpath/$entry -> $destpath/$entry failed";
+	}
+	closedir($directory);
+	return 1;
+}
+
+1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 02533eb..3a01b7a 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -1,3 +1,10 @@
+# TestLib, low-level routines and actions regression tests.
+#
+# This module contains a set of routines dedicated to environment setup for
+# a PostgreSQL regression test tun, and includes some low-level routines
+# aimed at controlling command execution, logging and test functions. This
+# module should never depend on any other PostgreSQL regression test modules.
+
 package TestLib;
 
 use strict;
@@ -5,16 +12,17 @@ use warnings;
 
 use Config;
 use Exporter 'import';
+use File::Basename;
+use File::Spec;
+use File::Temp ();
+use IPC::Run qw(run);
+use SimpleTee;
+use Test::More;
+
 our @EXPORT = qw(
-  tempdir
-  tempdir_short
-  standard_initdb
-  configure_hba_for_replication
-  start_test_server
-  restart_test_server
-  psql
   slurp_dir
   slurp_file
+  append_to_file
   system_or_bail
   system_log
   run_log
@@ -26,88 +34,80 @@ our @EXPORT = qw(
   program_version_ok
   program_options_handling_ok
   command_like
-  issues_sql_like
 
-  $tmp_check
-  $log_path
   $windows_os
 );
 
-use Cwd;
-use File::Basename;
-use File::Spec;
-use File::Temp ();
-use IPC::Run qw(run start);
-
-use SimpleTee;
-
-use Test::More;
+our ($windows_os, $tmp_check, $log_path, $test_logfile);
 
-our $windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
-
-# Open log file. For each test, the log file name uses the name of the
-# file launching this module, without the .pl suffix.
-our ($tmp_check, $log_path);
-$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
-$log_path = "$tmp_check/log";
-mkdir $tmp_check;
-mkdir $log_path;
-my $test_logfile = basename($0);
-$test_logfile =~ s/\.[^.]+$//;
-$test_logfile = "$log_path/regress_log_$test_logfile";
-open TESTLOG, '>', $test_logfile or die "Cannot open STDOUT to logfile: $!";
-
-# Hijack STDOUT and STDERR to the log file
-open(ORIG_STDOUT, ">&STDOUT");
-open(ORIG_STDERR, ">&STDERR");
-open(STDOUT, ">&TESTLOG");
-open(STDERR, ">&TESTLOG");
-
-# The test output (ok ...) needs to be printed to the original STDOUT so
-# that the 'prove' program can parse it, and display it to the user in
-# real time. But also copy it to the log file, to provide more context
-# in the log.
-my $builder = Test::More->builder;
-my $fh = $builder->output;
-tie *$fh, "SimpleTee", *ORIG_STDOUT, *TESTLOG;
-$fh = $builder->failure_output;
-tie *$fh, "SimpleTee", *ORIG_STDERR, *TESTLOG;
-
-# Enable auto-flushing for all the file handles. Stderr and stdout are
-# redirected to the same file, and buffering causes the lines to appear
-# in the log in confusing order.
-autoflush STDOUT 1;
-autoflush STDERR 1;
-autoflush TESTLOG 1;
-
-# Set to untranslated messages, to be able to compare program output
-# with expected strings.
-delete $ENV{LANGUAGE};
-delete $ENV{LC_ALL};
-$ENV{LC_MESSAGES} = 'C';
-
-delete $ENV{PGCONNECT_TIMEOUT};
-delete $ENV{PGDATA};
-delete $ENV{PGDATABASE};
-delete $ENV{PGHOSTADDR};
-delete $ENV{PGREQUIRESSL};
-delete $ENV{PGSERVICE};
-delete $ENV{PGSSLMODE};
-delete $ENV{PGUSER};
-
-if (!$ENV{PGPORT})
+BEGIN
 {
-	$ENV{PGPORT} = 65432;
+	# Set to untranslated messages, to be able to compare program output
+	# with expected strings.
+	delete $ENV{LANGUAGE};
+	delete $ENV{LC_ALL};
+	$ENV{LC_MESSAGES} = 'C';
+
+	delete $ENV{PGCONNECT_TIMEOUT};
+	delete $ENV{PGDATA};
+	delete $ENV{PGDATABASE};
+	delete $ENV{PGHOSTADDR};
+	delete $ENV{PGREQUIRESSL};
+	delete $ENV{PGSERVICE};
+	delete $ENV{PGSSLMODE};
+	delete $ENV{PGUSER};
+	delete $ENV{PGPORT};
+	delete $ENV{PGHOST};
+
+	# Must be set early
+	$windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';
 }
 
-$ENV{PGPORT} = int($ENV{PGPORT}) % 65536;
-
+INIT
+{
+	# Determine output directories, and create them.  The base path is the
+	# TESTDIR environment variable, which is normally set by the invoking
+	# Makefile.
+	$tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
+	$log_path = "$tmp_check/log";
+
+	mkdir $tmp_check;
+	mkdir $log_path;
+
+	# Open the test log file, whose name depends on the test name.
+	$test_logfile = basename($0);
+	$test_logfile =~ s/\.[^.]+$//;
+	$test_logfile = "$log_path/regress_log_$test_logfile";
+	open TESTLOG, '>', $test_logfile
+	  or die "could not open STDOUT to logfile \"$test_logfile\": $!";
+
+	# Hijack STDOUT and STDERR to the log file
+	open(ORIG_STDOUT, ">&STDOUT");
+	open(ORIG_STDERR, ">&STDERR");
+	open(STDOUT,      ">&TESTLOG");
+	open(STDERR,      ">&TESTLOG");
+
+	# The test output (ok ...) needs to be printed to the original STDOUT so
+	# that the 'prove' program can parse it, and display it to the user in
+	# real time. But also copy it to the log file, to provide more context
+	# in the log.
+	my $builder = Test::More->builder;
+	my $fh      = $builder->output;
+	tie *$fh, "SimpleTee", *ORIG_STDOUT, *TESTLOG;
+	$fh = $builder->failure_output;
+	tie *$fh, "SimpleTee", *ORIG_STDERR, *TESTLOG;
+
+	# Enable auto-flushing for all the file handles. Stderr and stdout are
+	# redirected to the same file, and buffering causes the lines to appear
+	# in the log in confusing order.
+	autoflush STDOUT 1;
+	autoflush STDERR 1;
+	autoflush TESTLOG 1;
+}
 
 #
 # Helper functions
 #
-
-
 sub tempdir
 {
 	return File::Temp::tempdir(
@@ -118,123 +118,36 @@ sub tempdir
 
 sub tempdir_short
 {
-
 	# Use a separate temp dir outside the build tree for the
 	# Unix-domain socket, to avoid file name length issues.
 	return File::Temp::tempdir(CLEANUP => 1);
 }
 
-# Initialize a new cluster for testing.
-#
-# The PGHOST environment variable is set to connect to the new cluster.
-#
-# Authentication is set up so that only the current OS user can access the
-# cluster. On Unix, we use Unix domain socket connections, with the socket in
-# a directory that's only accessible to the current user to ensure that.
-# On Windows, we use SSPI authentication to ensure the same (by pg_regress
-# --config-auth).
-sub standard_initdb
-{
-	my $pgdata = shift;
-	system_or_bail('initdb', '-D', "$pgdata", '-A' , 'trust', '-N');
-	system_or_bail($ENV{PG_REGRESS}, '--config-auth', $pgdata);
-
-	my $tempdir_short = tempdir_short;
-
-	open CONF, ">>$pgdata/postgresql.conf";
-	print CONF "\n# Added by TestLib.pm)\n";
-	print CONF "fsync = off\n";
-	if ($windows_os)
-	{
-		print CONF "listen_addresses = '127.0.0.1'\n";
-	}
-	else
-	{
-		print CONF "unix_socket_directories = '$tempdir_short'\n";
-		print CONF "listen_addresses = ''\n";
-	}
-	close CONF;
-
-	$ENV{PGHOST}         = $windows_os ? "127.0.0.1" : $tempdir_short;
-}
-
-# Set up the cluster to allow replication connections, in the same way that
-# standard_initdb does for normal connections.
-sub configure_hba_for_replication
-{
-	my $pgdata = shift;
-
-	open HBA, ">>$pgdata/pg_hba.conf";
-	print HBA "\n# Allow replication (set up by TestLib.pm)\n";
-	if (! $windows_os)
-	{
-		print HBA "local replication all trust\n";
-	}
-	else
-	{
-		print HBA "host replication all 127.0.0.1/32 sspi include_realm=1 map=regress\n";
-	}
-	close HBA;
-}
-
-my ($test_server_datadir, $test_server_logfile);
-
-
-# Initialize a new cluster for testing in given directory, and start it.
-sub start_test_server
-{
-	my ($tempdir) = @_;
-	my $ret;
-
-	print("### Starting test server in $tempdir\n");
-	standard_initdb "$tempdir/pgdata";
-
-	$ret = system_log('pg_ctl', '-D', "$tempdir/pgdata", '-w', '-l',
-	  "$log_path/postmaster.log", '-o', "--log-statement=all",
-	  'start');
-
-	if ($ret != 0)
-	{
-		print "# pg_ctl failed; logfile:\n";
-		system('cat', "$log_path/postmaster.log");
-		BAIL_OUT("pg_ctl failed");
-	}
-
-	$test_server_datadir = "$tempdir/pgdata";
-	$test_server_logfile = "$log_path/postmaster.log";
-}
-
-sub restart_test_server
+sub system_log
 {
-	print("### Restarting test server\n");
-	system_log('pg_ctl', '-D', $test_server_datadir, '-w', '-l',
-	  $test_server_logfile, 'restart');
+	print("# Running: " . join(" ", @_) . "\n");
+	return system(@_);
 }
 
-END
+sub system_or_bail
 {
-	if ($test_server_datadir)
+	if (system_log(@_) != 0)
 	{
-		system_log('pg_ctl', '-D', $test_server_datadir, '-m',
-		  'immediate', 'stop');
+		BAIL_OUT("system $_[0] failed");
 	}
 }
 
-sub psql
+sub run_log
 {
-	my ($dbname, $sql) = @_;
-	my ($stdout, $stderr);
-	print("# Running SQL command: $sql\n");
-	run [ 'psql', '-X', '-A', '-t', '-q', '-d', $dbname, '-f', '-' ], '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
-	chomp $stdout;
-	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-	return $stdout;
+	print("# Running: " . join(" ", @{ $_[0] }) . "\n");
+	return run(@_);
 }
 
 sub slurp_dir
 {
 	my ($dir) = @_;
-	opendir(my $dh, $dir) or die;
+	opendir(my $dh, $dir)
+	  or die "could not opendir \"$dir\": $!";
 	my @direntries = readdir $dh;
 	closedir $dh;
 	return @direntries;
@@ -249,32 +162,18 @@ sub slurp_file
 	return $contents;
 }
 
-sub system_or_bail
+sub append_to_file
 {
-	if (system_log(@_) != 0)
-	{
-		BAIL_OUT("system $_[0] failed: $?");
-	}
-}
-
-sub system_log
-{
-	print("# Running: " . join(" ", @_) ."\n");
-	return system(@_);
-}
+	my ($filename, $str) = @_;
 
-sub run_log
-{
-	print("# Running: " . join(" ", @{$_[0]}) ."\n");
-	return run (@_);
+	open my $fh, ">>", $filename or die "could not open \"$filename\": $!";
+	print $fh $str;
+	close $fh;
 }
 
-
 #
 # Test functions
 #
-
-
 sub command_ok
 {
 	my ($cmd, $test_name) = @_;
@@ -292,8 +191,8 @@ sub command_fails
 sub command_exit_is
 {
 	my ($cmd, $expected, $test_name) = @_;
-	print("# Running: " . join(" ", @{$cmd}) ."\n");
-	my $h = start $cmd;
+	print("# Running: " . join(" ", @{$cmd}) . "\n");
+	my $h = IPC::Run::start $cmd;
 	$h->finish();
 
 	# On Windows, the exit status of the process is returned directly as the
@@ -303,8 +202,10 @@ sub command_exit_is
 	# assuming the Unix convention, which will always return 0 on Windows as
 	# long as the process was not terminated by an exception. To work around
 	# that, use $h->full_result on Windows instead.
-	my $result = ($Config{osname} eq "MSWin32") ?
-		($h->full_results)[0] : $h->result(0);
+	my $result =
+	    ($Config{osname} eq "MSWin32")
+	  ? ($h->full_results)[0]
+	  : $h->result(0);
 	is($result, $expected, $test_name);
 }
 
@@ -313,7 +214,7 @@ sub program_help_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --help\n");
-	my $result = run [ $cmd, '--help' ], '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--help' ], '>', \$stdout, '2>', \$stderr;
 	ok($result, "$cmd --help exit code 0");
 	isnt($stdout, '', "$cmd --help goes to stdout");
 	is($stderr, '', "$cmd --help nothing to stderr");
@@ -324,7 +225,7 @@ sub program_version_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --version\n");
-	my $result = run [ $cmd, '--version' ], '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--version' ], '>', \$stdout, '2>', \$stderr;
 	ok($result, "$cmd --version exit code 0");
 	isnt($stdout, '', "$cmd --version goes to stdout");
 	is($stderr, '', "$cmd --version nothing to stderr");
@@ -335,8 +236,8 @@ sub program_options_handling_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --not-a-valid-option\n");
-	my $result = run [ $cmd, '--not-a-valid-option' ], '>', \$stdout, '2>',
-	  \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--not-a-valid-option' ], '>', \$stdout,
+	  '2>', \$stderr;
 	ok(!$result, "$cmd with invalid option nonzero exit code");
 	isnt($stderr, '', "$cmd with invalid option prints error message");
 }
@@ -346,20 +247,10 @@ sub command_like
 	my ($cmd, $expected_stdout, $test_name) = @_;
 	my ($stdout, $stderr);
 	print("# Running: " . join(" ", @{$cmd}) . "\n");
-	my $result = run $cmd, '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr;
 	ok($result, "@$cmd exit code 0");
 	is($stderr, '', "@$cmd no stderr");
 	like($stdout, $expected_stdout, "$test_name: matches");
 }
 
-sub issues_sql_like
-{
-	my ($cmd, $expected_sql, $test_name) = @_;
-	truncate $test_server_logfile, 0;
-	my $result = run_log($cmd);
-	ok($result, "@$cmd exit code 0");
-	my $log = slurp_file($test_server_logfile);
-	like($log, $expected_sql, "$test_name: SQL found in server log");
-}
-
 1;
diff --git a/src/test/ssl/ServerSetup.pm b/src/test/ssl/ServerSetup.pm
index a6c77b5..4e93184 100644
--- a/src/test/ssl/ServerSetup.pm
+++ b/src/test/ssl/ServerSetup.pm
@@ -18,6 +18,7 @@ package ServerSetup;
 
 use strict;
 use warnings;
+use PostgresNode;
 use TestLib;
 use File::Basename;
 use File::Copy;
@@ -45,17 +46,19 @@ sub copy_files
 
 sub configure_test_server_for_ssl
 {
-	my $tempdir    = $_[0];
+	my $node       = $_[0];
 	my $serverhost = $_[1];
 
+	my $pgdata = $node->data_dir;
+
 	# Create test users and databases
-	psql 'postgres', "CREATE USER ssltestuser";
-	psql 'postgres', "CREATE USER anotheruser";
-	psql 'postgres', "CREATE DATABASE trustdb";
-	psql 'postgres', "CREATE DATABASE certdb";
+	$node->psql('postgres', "CREATE USER ssltestuser");
+	$node->psql('postgres', "CREATE USER anotheruser");
+	$node->psql('postgres', "CREATE DATABASE trustdb");
+	$node->psql('postgres', "CREATE DATABASE certdb");
 
 	# enable logging etc.
-	open CONF, ">>$tempdir/pgdata/postgresql.conf";
+	open CONF, ">>$pgdata/postgresql.conf";
 	print CONF "fsync=off\n";
 	print CONF "log_connections=on\n";
 	print CONF "log_hostname=on\n";
@@ -68,17 +71,17 @@ sub configure_test_server_for_ssl
 	close CONF;
 
 # Copy all server certificates and keys, and client root cert, to the data dir
-	copy_files("ssl/server-*.crt", "$tempdir/pgdata");
-	copy_files("ssl/server-*.key", "$tempdir/pgdata");
-	chmod(0600, glob "$tempdir/pgdata/server-*.key") or die $!;
-	copy_files("ssl/root+client_ca.crt", "$tempdir/pgdata");
-	copy_files("ssl/root+client.crl",    "$tempdir/pgdata");
+	copy_files("ssl/server-*.crt", $pgdata);
+	copy_files("ssl/server-*.key", $pgdata);
+	chmod(0600, glob "$pgdata/server-*.key") or die $!;
+	copy_files("ssl/root+client_ca.crt", $pgdata);
+	copy_files("ssl/root+client.crl",    $pgdata);
 
   # Only accept SSL connections from localhost. Our tests don't depend on this
   # but seems best to keep it as narrow as possible for security reasons.
   #
   # When connecting to certdb, also check the client certificate.
-	open HBA, ">$tempdir/pgdata/pg_hba.conf";
+	open HBA, ">$pgdata/pg_hba.conf";
 	print HBA
 "# TYPE  DATABASE        USER            ADDRESS                 METHOD\n";
 	print HBA
@@ -96,12 +99,13 @@ sub configure_test_server_for_ssl
 # the server so that the configuration takes effect.
 sub switch_server_cert
 {
-	my $tempdir  = $_[0];
+	my $node     = $_[0];
 	my $certfile = $_[1];
+	my $pgdata   = $node->data_dir;
 
 	diag "Restarting server with certfile \"$certfile\"...";
 
-	open SSLCONF, ">$tempdir/pgdata/sslconfig.conf";
+	open SSLCONF, ">$pgdata/sslconfig.conf";
 	print SSLCONF "ssl=on\n";
 	print SSLCONF "ssl_ca_file='root+client_ca.crt'\n";
 	print SSLCONF "ssl_cert_file='$certfile.crt'\n";
@@ -110,5 +114,5 @@ sub switch_server_cert
 	close SSLCONF;
 
 	# Stop and restart server to reload the new config.
-	restart_test_server();
+	$node->restart;
 }
diff --git a/src/test/ssl/t/001_ssltests.pl b/src/test/ssl/t/001_ssltests.pl
index 0d6f339..dcc541d 100644
--- a/src/test/ssl/t/001_ssltests.pl
+++ b/src/test/ssl/t/001_ssltests.pl
@@ -1,5 +1,7 @@
 use strict;
 use warnings;
+use PostgresNode;
+use TestLib;
 use TestLib;
 use Test::More tests => 38;
 use ServerSetup;
@@ -25,8 +27,6 @@ BEGIN
 # postgresql-ssl-regression.test.
 my $SERVERHOSTADDR = '127.0.0.1';
 
-my $tempdir = TestLib::tempdir;
-
 # Define a couple of helper functions to test connecting to the server.
 
 my $common_connstr;
@@ -74,10 +74,16 @@ chmod 0600, "ssl/client.key";
 
 #### Part 0. Set up the server.
 
-diag "setting up data directory in \"$tempdir\"...";
-start_test_server($tempdir);
-configure_test_server_for_ssl($tempdir, $SERVERHOSTADDR);
-switch_server_cert($tempdir, 'server-cn-only');
+diag "setting up data directory...";
+my $node = get_new_node();
+$node->init;
+# PGHOST is enforced here to set up the node, subsequent connections
+# will use a dedicated connection string.
+$ENV{PGHOST} = $node->host;
+$ENV{PGPORT} = $node->port;
+$node->start;
+configure_test_server_for_ssl($node, $SERVERHOSTADDR);
+switch_server_cert($node, 'server-cn-only');
 
 ### Part 1. Run client-side tests.
 ###
@@ -150,7 +156,7 @@ test_connect_ok("sslmode=verify-ca host=wronghost.test");
 test_connect_fails("sslmode=verify-full host=wronghost.test");
 
 # Test Subject Alternative Names.
-switch_server_cert($tempdir, 'server-multiple-alt-names');
+switch_server_cert($node, 'server-multiple-alt-names');
 
 diag "test hostname matching with X509 Subject Alternative Names";
 $common_connstr =
@@ -165,7 +171,7 @@ test_connect_fails("host=deep.subdomain.wildcard.pg-ssltest.test");
 
 # Test certificate with a single Subject Alternative Name. (this gives a
 # slightly different error message, that's all)
-switch_server_cert($tempdir, 'server-single-alt-name');
+switch_server_cert($node, 'server-single-alt-name');
 
 diag "test hostname matching with a single X509 Subject Alternative Name";
 $common_connstr =
@@ -178,7 +184,7 @@ test_connect_fails("host=deep.subdomain.wildcard.pg-ssltest.test");
 
 # Test server certificate with a CN and SANs. Per RFCs 2818 and 6125, the CN
 # should be ignored when the certificate has both.
-switch_server_cert($tempdir, 'server-cn-and-alt-names');
+switch_server_cert($node, 'server-cn-and-alt-names');
 
 diag "test certificate with both a CN and SANs";
 $common_connstr =
@@ -190,7 +196,7 @@ test_connect_fails("host=common-name.pg-ssltest.test");
 
 # Finally, test a server certificate that has no CN or SANs. Of course, that's
 # not a very sensible certificate, but libpq should handle it gracefully.
-switch_server_cert($tempdir, 'server-no-names');
+switch_server_cert($node, 'server-no-names');
 $common_connstr =
 "user=ssltestuser dbname=trustdb sslcert=invalid sslrootcert=ssl/root+server_ca.crt hostaddr=$SERVERHOSTADDR";
 
@@ -199,7 +205,7 @@ test_connect_fails("sslmode=verify-full host=common-name.pg-ssltest.test");
 
 # Test that the CRL works
 diag "Testing client-side CRL";
-switch_server_cert($tempdir, 'server-revoked');
+switch_server_cert($node, 'server-revoked');
 
 $common_connstr =
 "user=ssltestuser dbname=trustdb sslcert=invalid hostaddr=$SERVERHOSTADDR host=common-name.pg-ssltest.test";
@@ -233,7 +239,3 @@ test_connect_fails(
 test_connect_fails(
 "user=ssltestuser sslcert=ssl/client-revoked.crt sslkey=ssl/client-revoked.key"
 );
-
-
-# All done! Save the log, before the temporary installation is deleted
-copy("$tempdir/client-log", "./client-log");
-- 
2.6.3

0002-Run-of-perltidy-for-previous-patch.patchapplication/x-patch; name=0002-Run-of-perltidy-for-previous-patch.patchDownload
From 5474c9a8251462328c5879202b6f2d7adbb6f048 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Wed, 2 Dec 2015 15:16:38 +0000
Subject: [PATCH 2/2] Run of perltidy for previous patch

---
 src/bin/pg_basebackup/t/010_pg_basebackup.pl | 96 ++++++++++++++++++----------
 src/bin/pg_ctl/t/001_start_stop.pl           |  7 +-
 src/bin/pg_rewind/RewindTest.pm              | 61 ++++++++++--------
 src/bin/scripts/t/010_clusterdb.pl           |  7 +-
 src/bin/scripts/t/020_createdb.pl            |  3 +-
 src/bin/scripts/t/030_createlang.pl          |  6 +-
 src/bin/scripts/t/040_createuser.pl          |  3 +-
 src/bin/scripts/t/050_dropdb.pl              |  3 +-
 src/bin/scripts/t/070_dropuser.pl            |  3 +-
 src/bin/scripts/t/090_reindexdb.pl           |  2 +-
 src/test/perl/PostgresNode.pm                | 14 ++--
 src/test/perl/RecursiveCopy.pm               |  4 +-
 src/test/perl/TestLib.pm                     |  9 ++-
 src/test/ssl/t/001_ssltests.pl               |  1 +
 14 files changed, 134 insertions(+), 85 deletions(-)

diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 46c8a71..106606a 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -13,6 +13,7 @@ program_options_handling_ok('pg_basebackup');
 my $tempdir = TestLib::tempdir;
 
 my $node = get_new_node();
+
 # Initialize node without replication settings
 $node->init(hba_permit_replication => 0);
 $node->start;
@@ -50,9 +51,10 @@ $node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backup" ],
 	'pg_basebackup runs');
 ok(-f "$tempdir/backup/PG_VERSION", 'backup was created');
 
-is_deeply([sort(slurp_dir("$tempdir/backup/pg_xlog/"))],
-		  [sort qw(. .. archive_status)],
-		  'no WAL files copied');
+is_deeply(
+	[ sort(slurp_dir("$tempdir/backup/pg_xlog/")) ],
+	[ sort qw(. .. archive_status) ],
+	'no WAL files copied');
 
 $node->command_ok(
 	[   'pg_basebackup', '-D', "$tempdir/backup2", '--xlogdir',
@@ -91,14 +93,16 @@ my $superlongpath = "$pgdata/$superlongname";
 
 open FILE, ">$superlongpath" or die "unable to create file $superlongpath";
 close FILE;
-$node->command_fails([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l1", '-Ft' ],
+$node->command_fails(
+	[ 'pg_basebackup', '-D', "$tempdir/tarbackup_l1", '-Ft' ],
 	'pg_basebackup tar with long name fails');
 unlink "$pgdata/$superlongname";
 
 # The following tests test symlinks. Windows doesn't have symlinks, so
 # skip on Windows.
-SKIP: {
-    skip "symlinks not supported on Windows", 10 if ($windows_os);
+SKIP:
+{
+	skip "symlinks not supported on Windows", 10 if ($windows_os);
 
 	# Create a temporary directory in the system location and symlink it
 	# to our physical temp location.  That way we can use shorter names
@@ -109,10 +113,10 @@ SKIP: {
 
 	mkdir "$tempdir/tblspc1";
 	$node->psql('postgres',
-	"CREATE TABLESPACE tblspc1 LOCATION '$shorter_tempdir/tblspc1';");
+		"CREATE TABLESPACE tblspc1 LOCATION '$shorter_tempdir/tblspc1';");
 	$node->psql('postgres', "CREATE TABLE test1 (a int) TABLESPACE tblspc1;");
 	$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup2", '-Ft' ],
-			   'tar format with tablespaces');
+		'tar format with tablespaces');
 	ok(-f "$tempdir/tarbackup2/base.tar", 'backup tar was created');
 	my @tblspc_tars = glob "$tempdir/tarbackup2/[0-9]*.tar";
 	is(scalar(@tblspc_tars), 1, 'one tablespace tar was created');
@@ -128,9 +132,9 @@ SKIP: {
 	ok(-d "$tempdir/tbackup/tblspc1", 'tablespace was relocated');
 	opendir(my $dh, "$pgdata/pg_tblspc") or die;
 	ok( (   grep {
-		-l "$tempdir/backup1/pg_tblspc/$_"
-			and readlink "$tempdir/backup1/pg_tblspc/$_" eq
-			"$tempdir/tbackup/tblspc1"
+				-l "$tempdir/backup1/pg_tblspc/$_"
+				  and readlink "$tempdir/backup1/pg_tblspc/$_" eq
+				  "$tempdir/tbackup/tblspc1"
 			} readdir($dh)),
 		"tablespace symlink was updated");
 	closedir $dh;
@@ -139,19 +143,21 @@ SKIP: {
 	$node->psql('postgres', "DROP TABLE test1;");
 	$node->psql('postgres', "DROP TABLESPACE tblspc1;");
 	$node->psql('postgres',
-	"CREATE TABLESPACE tblspc2 LOCATION '$shorter_tempdir/tbl=spc2';");
+		"CREATE TABLESPACE tblspc2 LOCATION '$shorter_tempdir/tbl=spc2';");
 	$node->command_ok(
 		[   'pg_basebackup', '-D', "$tempdir/backup3", '-Fp',
 			"-T$shorter_tempdir/tbl\\=spc2=$tempdir/tbackup/tbl\\=spc2" ],
 		'mapping tablespace with = sign in path');
-	ok(-d "$tempdir/tbackup/tbl=spc2", 'tablespace with = sign was relocated');
+	ok(-d "$tempdir/tbackup/tbl=spc2",
+		'tablespace with = sign was relocated');
 	$node->psql('postgres', "DROP TABLESPACE tblspc2;");
 
 	mkdir "$tempdir/$superlongname";
 	$node->psql('postgres',
-	"CREATE TABLESPACE tblspc3 LOCATION '$tempdir/$superlongname';");
-	$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/tarbackup_l3", '-Ft' ],
-			   'pg_basebackup tar with long symlink target');
+		"CREATE TABLESPACE tblspc3 LOCATION '$tempdir/$superlongname';");
+	$node->command_ok(
+		[ 'pg_basebackup', '-D', "$tempdir/tarbackup_l3", '-Ft' ],
+		'pg_basebackup tar with long symlink target');
 	$node->psql('postgres', "DROP TABLESPACE tblspc3;");
 }
 
@@ -159,33 +165,59 @@ $node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backupR", '-R' ],
 	'pg_basebackup -R runs');
 ok(-f "$tempdir/backupR/recovery.conf", 'recovery.conf was created');
 my $recovery_conf = slurp_file "$tempdir/backupR/recovery.conf";
+
 # using a character class for the final "'" here works around an apparent
 # bug in several version of the Msys DTK perl
-like($recovery_conf, qr/^standby_mode = 'on[']$/m, 'recovery.conf sets standby_mode');
-like($recovery_conf, qr/^primary_conninfo = '.*port=$ENV{PGPORT}.*'$/m, 'recovery.conf sets primary_conninfo');
+like(
+	$recovery_conf,
+	qr/^standby_mode = 'on[']$/m,
+	'recovery.conf sets standby_mode');
+like(
+	$recovery_conf,
+	qr/^primary_conninfo = '.*port=$ENV{PGPORT}.*'$/m,
+	'recovery.conf sets primary_conninfo');
 
-$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxf", '-X', 'fetch' ],
+$node->command_ok(
+	[ 'pg_basebackup', '-D', "$tempdir/backupxf", '-X', 'fetch' ],
 	'pg_basebackup -X fetch runs');
-ok(grep(/^[0-9A-F]{24}$/, slurp_dir("$tempdir/backupxf/pg_xlog")), 'WAL files copied');
-$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs", '-X', 'stream' ],
+ok(grep(/^[0-9A-F]{24}$/, slurp_dir("$tempdir/backupxf/pg_xlog")),
+	'WAL files copied');
+$node->command_ok(
+	[ 'pg_basebackup', '-D', "$tempdir/backupxs", '-X', 'stream' ],
 	'pg_basebackup -X stream runs');
-ok(grep(/^[0-9A-F]{24}$/, slurp_dir("$tempdir/backupxf/pg_xlog")), 'WAL files copied');
+ok(grep(/^[0-9A-F]{24}$/, slurp_dir("$tempdir/backupxf/pg_xlog")),
+	'WAL files copied');
 
-$node->command_fails([ 'pg_basebackup', '-D', "$tempdir/fail", '-S', 'slot1' ],
+$node->command_fails(
+	[ 'pg_basebackup', '-D', "$tempdir/fail", '-S', 'slot1' ],
 	'pg_basebackup with replication slot fails without -X stream');
-$node->command_fails([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl_fail", '-X', 'stream', '-S', 'slot1' ],
+$node->command_fails(
+	[   'pg_basebackup',             '-D',
+		"$tempdir/backupxs_sl_fail", '-X',
+		'stream',                    '-S',
+		'slot1' ],
 	'pg_basebackup fails with nonexistent replication slot');
 
-$node->psql('postgres', q{SELECT * FROM pg_create_physical_replication_slot('slot1')});
-my $lsn = $node->psql('postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'});
+$node->psql('postgres',
+	q{SELECT * FROM pg_create_physical_replication_slot('slot1')});
+my $lsn = $node->psql('postgres',
+	q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'}
+);
 is($lsn, '', 'restart LSN of new slot is null');
-$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl", '-X', 'stream', '-S', 'slot1' ],
+$node->command_ok(
+	[   'pg_basebackup', '-D', "$tempdir/backupxs_sl", '-X',
+		'stream',        '-S', 'slot1' ],
 	'pg_basebackup -X stream with replication slot runs');
-$lsn = $node->psql('postgres', q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'});
+$lsn = $node->psql('postgres',
+	q{SELECT restart_lsn FROM pg_replication_slots WHERE slot_name = 'slot1'}
+);
 like($lsn, qr!^0/[0-9A-Z]{7,8}$!, 'restart LSN of slot has advanced');
 
-$node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxs_sl_R", '-X', 'stream', '-S', 'slot1', '-R' ],
+$node->command_ok(
+	[   'pg_basebackup', '-D', "$tempdir/backupxs_sl_R", '-X',
+		'stream',        '-S', 'slot1',                  '-R' ],
 	'pg_basebackup with replication slot and -R runs');
-like(slurp_file("$tempdir/backupxs_sl_R/recovery.conf"),
-	 qr/^primary_slot_name = 'slot1'$/m,
-	 'recovery.conf sets primary_slot_name');
+like(
+	slurp_file("$tempdir/backupxs_sl_R/recovery.conf"),
+	qr/^primary_slot_name = 'slot1'$/m,
+	'recovery.conf sets primary_slot_name');
diff --git a/src/bin/pg_ctl/t/001_start_stop.pl b/src/bin/pg_ctl/t/001_start_stop.pl
index a224ece..cbe99d7 100644
--- a/src/bin/pg_ctl/t/001_start_stop.pl
+++ b/src/bin/pg_ctl/t/001_start_stop.pl
@@ -18,13 +18,11 @@ command_exit_is([ 'pg_ctl', 'start', '-D', "$tempdir/nonexistent" ],
 
 command_ok([ 'pg_ctl', 'initdb', '-D', "$tempdir/data", '-o', '-N' ],
 	'pg_ctl initdb');
-command_ok(
-	[ $ENV{PG_REGRESS}, '--config-auth',
-		"$tempdir/data" ],
+command_ok([ $ENV{PG_REGRESS}, '--config-auth', "$tempdir/data" ],
 	'configure authentication');
 open CONF, ">>$tempdir/data/postgresql.conf";
 print CONF "fsync = off\n";
-if (! $windows_os)
+if (!$windows_os)
 {
 	print CONF "listen_addresses = ''\n";
 	print CONF "unix_socket_directories = '$tempdir_short'\n";
@@ -36,6 +34,7 @@ else
 close CONF;
 command_ok([ 'pg_ctl', 'start', '-D', "$tempdir/data", '-w' ],
 	'pg_ctl start -w');
+
 # sleep here is because Windows builds can't check postmaster.pid exactly,
 # so they may mistake a pre-existing postmaster.pid for one created by the
 # postmaster they start.  Waiting more than the 2 seconds slop time allowed
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index 55bbc9c..f2c0ca9 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -77,7 +77,7 @@ sub standby_psql
 	my $cmd = shift;
 
 	system_or_bail 'psql', '-q', '--no-psqlrc', '-d',
-      $node_standby->connstr('postgres'), '-c', "$cmd";
+	  $node_standby->connstr('postgres'), '-c', "$cmd";
 }
 
 # Run a query against the master, and check that the output matches what's
@@ -89,8 +89,9 @@ sub check_query
 
 	# we want just the output, no formatting
 	my $result = IPC::Run::run [
-		'psql',          '-q', '-A', '-t', '--no-psqlrc', '-d',
-		$node_master->connstr('postgres'), '-c', $query ],
+		'psql', '-q', '-A', '-t', '--no-psqlrc', '-d',
+		$node_master->connstr('postgres'),
+		'-c', $query ],
 	  '>', \$stdout, '2>', \$stderr;
 
 	# We don't use ok() for the exit code and stderr, because we want this
@@ -118,7 +119,8 @@ sub setup_cluster
 	$node_master->init;
 
 	# Custom parameters for master's postgresql.conf
-	$node_master->append_conf("postgresql.conf", qq(
+	$node_master->append_conf(
+		"postgresql.conf", qq(
 wal_level = hot_standby
 max_wal_senders = 2
 wal_keep_segments = 20
@@ -146,7 +148,8 @@ sub create_standby
 	$node_standby->init_from_backup($node_master, 'my_backup');
 	my $connstr_master = $node_master->connstr('postgres');
 
-	$node_standby->append_conf("recovery.conf", qq(
+	$node_standby->append_conf(
+		"recovery.conf", qq(
 primary_conninfo='$connstr_master application_name=rewind_standby'
 standby_mode=on
 recovery_target_timeline='latest'
@@ -174,7 +177,8 @@ sub promote_standby
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
 	system_or_bail('pg_ctl', '-w', '-D', $node_standby->data_dir, 'promote');
-	$node_standby->poll_query_until('postgres', "SELECT NOT pg_is_in_recovery()")
+	$node_standby->poll_query_until('postgres',
+		"SELECT NOT pg_is_in_recovery()")
 	  or die "Timed out while waiting for promotion of standby";
 
 	# Force a checkpoint after the promotion. pg_rewind looks at the control
@@ -188,11 +192,11 @@ sub promote_standby
 
 sub run_pg_rewind
 {
-	my $test_mode = shift;
-	my $master_pgdata = $node_master->data_dir;
-	my $standby_pgdata = $node_standby->data_dir;
+	my $test_mode       = shift;
+	my $master_pgdata   = $node_master->data_dir;
+	my $standby_pgdata  = $node_standby->data_dir;
 	my $standby_connstr = $node_standby->connstr('postgres');
-	my $tmp_folder = TestLib::tempdir;
+	my $tmp_folder      = TestLib::tempdir;
 
 	# Stop the master and be ready to perform the rewind
 	$node_master->stop;
@@ -204,8 +208,9 @@ sub run_pg_rewind
 
 	# Keep a temporary postgresql.conf for master node or it would be
 	# overwritten during the rewind.
-	copy("$master_pgdata/postgresql.conf",
-		 "$tmp_folder/master-postgresql.conf.tmp");
+	copy(
+		"$master_pgdata/postgresql.conf",
+		"$tmp_folder/master-postgresql.conf.tmp");
 
 	# Now run pg_rewind
 	if ($test_mode eq "local")
@@ -213,21 +218,21 @@ sub run_pg_rewind
 		# Do rewind using a local pgdata as source
 		# Stop the master and be ready to perform the rewind
 		$node_standby->stop;
-		command_ok(['pg_rewind',
-					"--debug",
-					"--source-pgdata=$standby_pgdata",
-					"--target-pgdata=$master_pgdata"],
-				   'pg_rewind local');
+		command_ok(
+			[   'pg_rewind',
+				"--debug",
+				"--source-pgdata=$standby_pgdata",
+				"--target-pgdata=$master_pgdata" ],
+			'pg_rewind local');
 	}
 	elsif ($test_mode eq "remote")
 	{
 		# Do rewind using a remote connection as source
-		command_ok(['pg_rewind',
-					"--debug",
-					"--source-server",
-					$standby_connstr,
-					"--target-pgdata=$master_pgdata"],
-				   'pg_rewind remote');
+		command_ok(
+			[   'pg_rewind',       "--debug",
+				"--source-server", $standby_connstr,
+				"--target-pgdata=$master_pgdata" ],
+			'pg_rewind remote');
 	}
 	else
 	{
@@ -236,12 +241,14 @@ sub run_pg_rewind
 	}
 
 	# Now move back postgresql.conf with old settings
-	move("$tmp_folder/master-postgresql.conf.tmp",
-		 "$master_pgdata/postgresql.conf");
+	move(
+		"$tmp_folder/master-postgresql.conf.tmp",
+		"$master_pgdata/postgresql.conf");
 
 	# Plug-in rewound node to the now-promoted standby node
 	my $port_standby = $node_standby->port;
-	$node_master->append_conf('recovery.conf', qq(
+	$node_master->append_conf(
+		'recovery.conf', qq(
 primary_conninfo='port=$port_standby'
 standby_mode=on
 recovery_target_timeline='latest'
@@ -256,7 +263,7 @@ recovery_target_timeline='latest'
 # Clean up after the test. Stop both servers, if they're still running.
 sub clean_rewind_test
 {
-	$node_master->teardown_node if defined $node_master;
+	$node_master->teardown_node  if defined $node_master;
 	$node_standby->teardown_node if defined $node_standby;
 }
 
diff --git a/src/bin/scripts/t/010_clusterdb.pl b/src/bin/scripts/t/010_clusterdb.pl
index bdb140f..5131b35 100644
--- a/src/bin/scripts/t/010_clusterdb.pl
+++ b/src/bin/scripts/t/010_clusterdb.pl
@@ -14,15 +14,16 @@ $node->init;
 $node->start;
 
 $node->issues_sql_like(
-	[ 'clusterdb' ],
+	['clusterdb'],
 	qr/statement: CLUSTER;/,
 	'SQL CLUSTER run');
 
 $node->command_fails([ 'clusterdb', '-t', 'nonexistent' ],
-			  'fails with nonexistent table');
+	'fails with nonexistent table');
 
 $node->psql('postgres',
-	'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x');
+'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a); CLUSTER test1 USING test1x'
+);
 $node->issues_sql_like(
 	[ 'clusterdb', '-t', 'test1' ],
 	qr/statement: CLUSTER test1;/,
diff --git a/src/bin/scripts/t/020_createdb.pl b/src/bin/scripts/t/020_createdb.pl
index 840c1b3..e0cf860 100644
--- a/src/bin/scripts/t/020_createdb.pl
+++ b/src/bin/scripts/t/020_createdb.pl
@@ -22,4 +22,5 @@ $node->issues_sql_like(
 	qr/statement: CREATE DATABASE foobar2 ENCODING 'LATIN1'/,
 	'create database with encoding');
 
-$node->command_fails([ 'createdb', 'foobar1' ], 'fails if database already exists');
+$node->command_fails([ 'createdb', 'foobar1' ],
+	'fails if database already exists');
diff --git a/src/bin/scripts/t/030_createlang.pl b/src/bin/scripts/t/030_createlang.pl
index 6d14c98..8ad391d 100644
--- a/src/bin/scripts/t/030_createlang.pl
+++ b/src/bin/scripts/t/030_createlang.pl
@@ -15,8 +15,7 @@ $node->start;
 
 $ENV{PGDATABASE} = 'postgres';
 
-$node->command_fails(
-	[ 'createlang', 'plpgsql' ],
+$node->command_fails([ 'createlang', 'plpgsql' ],
 	'fails if language already exists');
 
 $node->psql('postgres', 'DROP EXTENSION plpgsql');
@@ -25,5 +24,4 @@ $node->issues_sql_like(
 	qr/statement: CREATE EXTENSION "plpgsql"/,
 	'SQL CREATE EXTENSION run');
 
-$node->command_like([ 'createlang', '--list' ],
-	qr/plpgsql/, 'list output');
+$node->command_like([ 'createlang', '--list' ], qr/plpgsql/, 'list output');
diff --git a/src/bin/scripts/t/040_createuser.pl b/src/bin/scripts/t/040_createuser.pl
index ac08fde..fcada63 100644
--- a/src/bin/scripts/t/040_createuser.pl
+++ b/src/bin/scripts/t/040_createuser.pl
@@ -30,4 +30,5 @@ $node->issues_sql_like(
 qr/statement: CREATE ROLE user3 SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;/,
 	'create a superuser');
 
-$node->command_fails([ 'createuser', 'user1' ], 'fails if role already exists');
+$node->command_fails([ 'createuser', 'user1' ],
+	'fails if role already exists');
diff --git a/src/bin/scripts/t/050_dropdb.pl b/src/bin/scripts/t/050_dropdb.pl
index fb4feb8..2adc80a 100644
--- a/src/bin/scripts/t/050_dropdb.pl
+++ b/src/bin/scripts/t/050_dropdb.pl
@@ -19,4 +19,5 @@ $node->issues_sql_like(
 	qr/statement: DROP DATABASE foobar1/,
 	'SQL DROP DATABASE run');
 
-$node->command_fails([ 'dropdb', 'nonexistent' ], 'fails with nonexistent database');
+$node->command_fails([ 'dropdb', 'nonexistent' ],
+	'fails with nonexistent database');
diff --git a/src/bin/scripts/t/070_dropuser.pl b/src/bin/scripts/t/070_dropuser.pl
index 6e87f1e..0849f77 100644
--- a/src/bin/scripts/t/070_dropuser.pl
+++ b/src/bin/scripts/t/070_dropuser.pl
@@ -19,4 +19,5 @@ $node->issues_sql_like(
 	qr/statement: DROP ROLE foobar1/,
 	'SQL DROP ROLE run');
 
-$node->command_fails([ 'dropuser', 'nonexistent' ], 'fails with nonexistent user');
+$node->command_fails([ 'dropuser', 'nonexistent' ],
+	'fails with nonexistent user');
diff --git a/src/bin/scripts/t/090_reindexdb.pl b/src/bin/scripts/t/090_reindexdb.pl
index 68b2291..fd4eac3 100644
--- a/src/bin/scripts/t/090_reindexdb.pl
+++ b/src/bin/scripts/t/090_reindexdb.pl
@@ -21,7 +21,7 @@ $node->issues_sql_like(
 	'SQL REINDEX run');
 
 $node->psql('postgres',
-  'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a);');
+	'CREATE TABLE test1 (a int); CREATE INDEX test1x ON test1 (a);');
 $node->issues_sql_like(
 	[ 'reindexdb', '-t', 'test1', 'postgres' ],
 	qr/statement: REINDEX TABLE test1;/,
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index c182ce7..56628bf 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -176,7 +176,8 @@ sub init
 	my $pgdata = $self->data_dir;
 	my $host   = $self->host;
 
-	$params{hba_permit_replication} = 1 if (!defined($params{hba_permit_replication}));
+	$params{hba_permit_replication} = 1
+	  if (!defined($params{hba_permit_replication}));
 
 	mkdir $self->backup_dir;
 	mkdir $self->archive_dir;
@@ -243,7 +244,8 @@ sub init_from_backup
 	chmod(0700, $data_path);
 
 	# Base configuration for this node
-	$self->append_conf('postgresql.conf',
+	$self->append_conf(
+		'postgresql.conf',
 		qq(
 port = $port
 ));
@@ -380,8 +382,9 @@ sub psql
 	my ($stdout, $stderr);
 	print("# Running SQL command: $sql\n");
 
-	IPC::Run::run [ 'psql', '-XAtq', '-d', $self->connstr($dbname), '-f', '-' ],
-	  '<', \$sql, '>', \$stdout, '2>', \$stderr or die;
+	IPC::Run::run [ 'psql', '-XAtq', '-d', $self->connstr($dbname), '-f',
+		'-' ], '<', \$sql, '>', \$stdout, '2>', \$stderr
+	  or die;
 
 	if ($stderr ne "")
 	{
@@ -405,7 +408,8 @@ sub poll_query_until
 
 	while ($attempts < $max_attempts)
 	{
-		my $cmd = [ 'psql', '-At', '-c', $query, '-d', $self->connstr($dbname) ];
+		my $cmd =
+		  [ 'psql', '-At', '-c', $query, '-d', $self->connstr($dbname) ];
 		my $result = IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr;
 
 		chomp($stdout);
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index 4e58ad3..9362aa8 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -20,7 +20,7 @@ sub copypath
 	if (-f $srcpath)
 	{
 		copy($srcpath, $destpath)
-			or die "copy $srcpath -> $destpath failed: $!";
+		  or die "copy $srcpath -> $destpath failed: $!";
 		return 1;
 	}
 
@@ -33,7 +33,7 @@ sub copypath
 	{
 		next if ($entry eq '.' || $entry eq '..');
 		RecursiveCopy::copypath("$srcpath/$entry", "$destpath/$entry")
-			or die "copypath $srcpath/$entry -> $destpath/$entry failed";
+		  or die "copypath $srcpath/$entry -> $destpath/$entry failed";
 	}
 	closedir($directory);
 	return 1;
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 3a01b7a..b4f1080 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -214,7 +214,8 @@ sub program_help_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --help\n");
-	my $result = IPC::Run::run [ $cmd, '--help' ], '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--help' ], '>', \$stdout, '2>',
+	  \$stderr;
 	ok($result, "$cmd --help exit code 0");
 	isnt($stdout, '', "$cmd --help goes to stdout");
 	is($stderr, '', "$cmd --help nothing to stderr");
@@ -225,7 +226,8 @@ sub program_version_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --version\n");
-	my $result = IPC::Run::run [ $cmd, '--version' ], '>', \$stdout, '2>', \$stderr;
+	my $result = IPC::Run::run [ $cmd, '--version' ], '>', \$stdout, '2>',
+	  \$stderr;
 	ok($result, "$cmd --version exit code 0");
 	isnt($stdout, '', "$cmd --version goes to stdout");
 	is($stderr, '', "$cmd --version nothing to stderr");
@@ -236,7 +238,8 @@ sub program_options_handling_ok
 	my ($cmd) = @_;
 	my ($stdout, $stderr);
 	print("# Running: $cmd --not-a-valid-option\n");
-	my $result = IPC::Run::run [ $cmd, '--not-a-valid-option' ], '>', \$stdout,
+	my $result = IPC::Run::run [ $cmd, '--not-a-valid-option' ], '>',
+	  \$stdout,
 	  '2>', \$stderr;
 	ok(!$result, "$cmd with invalid option nonzero exit code");
 	isnt($stderr, '', "$cmd with invalid option prints error message");
diff --git a/src/test/ssl/t/001_ssltests.pl b/src/test/ssl/t/001_ssltests.pl
index dcc541d..92f16e4 100644
--- a/src/test/ssl/t/001_ssltests.pl
+++ b/src/test/ssl/t/001_ssltests.pl
@@ -77,6 +77,7 @@ chmod 0600, "ssl/client.key";
 diag "setting up data directory...";
 my $node = get_new_node();
 $node->init;
+
 # PGHOST is enforced here to set up the node, subsequent connections
 # will use a dedicated connection string.
 $ENV{PGHOST} = $node->host;
-- 
2.6.3

#93Noah Misch
noah@leadboat.com
In reply to: Alvaro Herrera (#88)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Tue, Dec 01, 2015 at 08:11:21PM -0300, Alvaro Herrera wrote:

Finally, I ran perltidy on all the files, which strangely changed stuff
that I didn't expect it to change. I wonder if this is related to the
perltidy version.

The last pgindent run (commit 807b9e0) used perltidy v20090616, and perltidy
behavior has changed slightly over time. Install that version to do your own
perltidy runs.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#94Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Noah Misch (#93)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Noah Misch wrote:

On Tue, Dec 01, 2015 at 08:11:21PM -0300, Alvaro Herrera wrote:

Finally, I ran perltidy on all the files, which strangely changed stuff
that I didn't expect it to change. I wonder if this is related to the
perltidy version.

The last pgindent run (commit 807b9e0) used perltidy v20090616, and perltidy
behavior has changed slightly over time. Install that version to do your own
perltidy runs.

I tried that version, but it seems to emit the same. How did you figure
that that was the version used, anyway?

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#95Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#92)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

Well, Alvaro has whispered me a more elegant method by using TestLib()
to only import a portion of the routines and avoid the redefinition
errors. Hence, patch 0001 attached creates equivalents of command_*
for PostgresNode and tests use it without setting PGPORT. Patch 0002
is a run of perltidy on the whole.

It seemed better to me to have the import list be empty, i.e. "use
TestLib ()" and then qualify the routine names inside PostgresNode,
instead of having to list the names of the routines to import, so I
pushed it that way after running the tests a few more times.

(Another option would be to have those routines be in EXPORT_OK instead
of EXPORT, but then every other user of TestLib would have to
explicitely declare that it wants those routines imported. Maybe this
is a good change; if anyone wants to push for that, patches welcome.)

I didn't push the changed for config_default you requested a few
messages upthread; it's not clear to me how setting it to undef affects
the whole thing. If setting it to undef makes the MSVC toolchain run
the tap tests in the default config, then I can do it; let's be clear
about what branch to backpatch this to. Also the "1;" at the end of
RewindTest.

Thanks for the patch.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#96Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#95)
2 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Dec 3, 2015 at 6:59 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

I didn't push the changes for config_default you requested a few
messages upthread; it's not clear to me how setting it to undef affects
the whole thing. If setting it to undef makes the MSVC toolchain run
the tap tests in the default config, then I can do it; let's be clear
about what branch to backpatch this to. Also the "1;" at the end of
RewindTest.

Setting it to undef will prevent the tests to run, per vcregress.pl:
sub tap_check
{
die "Tap tests not enabled in configuration"
unless $config->{tap_tests};
Also, setting it to undef will match the existing behavior on
platforms where ./configure is used because the switch
--enable-tap-tests needs to be used there. And I would believe that in
most cases Windows environments are not going to have IPC::Run
deployed.

I have also rebased the recovery test suite as the attached, using the
infrastructure that has been committed lately.
Regards,
--
Michael

Attachments:

0001-Fix-tap_test-configuration-in-MSVC-builds.patchtext/x-patch; charset=US-ASCII; name=0001-Fix-tap_test-configuration-in-MSVC-builds.patchDownload
From de2121eeb50c5ae49b29a2ac21a16579eae2de98 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Thu, 3 Dec 2015 13:48:48 +0900
Subject: [PATCH 1/2] Fix tap_test configuration in MSVC builds

---
 src/tools/msvc/config_default.pl | 27 ++++++++++++++-------------
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/src/tools/msvc/config_default.pl b/src/tools/msvc/config_default.pl
index b9f2ff4..e50be7e 100644
--- a/src/tools/msvc/config_default.pl
+++ b/src/tools/msvc/config_default.pl
@@ -3,7 +3,7 @@ use strict;
 use warnings;
 
 our $config = {
-	asserts => 0,    # --enable-cassert
+	asserts  => 0,    # --enable-cassert
 	  # integer_datetimes=>1,   # --enable-integer-datetimes - on is now default
 	  # float4byval=>1,         # --disable-float4-byval, on by default
 
@@ -13,18 +13,19 @@ our $config = {
 	# blocksize => 8,         # --with-blocksize, 8kB by default
 	# wal_blocksize => 8,     # --with-wal-blocksize, 8kB by default
 	# wal_segsize => 16,      # --with-wal-segsize, 16MB by default
-	ldap     => 1,        # --with-ldap
-	extraver => undef,    # --with-extra-version=<string>
-	nls      => undef,    # --enable-nls=<path>
-	tcl      => undef,    # --with-tls=<path>
-	perl     => undef,    # --with-perl
-	python   => undef,    # --with-python=<path>
-	openssl  => undef,    # --with-openssl=<path>
-	uuid     => undef,    # --with-ossp-uuid
-	xml      => undef,    # --with-libxml=<path>
-	xslt     => undef,    # --with-libxslt=<path>
-	iconv    => undef,    # (not in configure, path to iconv)
-	zlib     => undef     # --with-zlib=<path>
+	ldap      => 1,        # --with-ldap
+	extraver  => undef,    # --with-extra-version=<string>
+	nls       => undef,    # --enable-nls=<path>
+	tap_tests => undef,    # --enable-tap-tests
+	tcl       => undef,    # --with-tls=<path>
+	perl      => undef,    # --with-perl
+	python    => undef,    # --with-python=<path>
+	openssl   => undef,    # --with-openssl=<path>
+	uuid      => undef,    # --with-ossp-uuid
+	xml       => undef,    # --with-libxml=<path>
+	xslt      => undef,    # --with-libxslt=<path>
+	iconv     => undef,    # (not in configure, path to iconv)
+	zlib      => undef     # --with-zlib=<path>
 };
 
 1;
-- 
2.6.3

0002-Add-recovery-test-suite.patchtext/x-patch; charset=US-ASCII; name=0002-Add-recovery-test-suite.patchDownload
From 4ee2a99db2a414f85e961110279f4b97309cd927 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Thu, 3 Dec 2015 15:41:12 +0900
Subject: [PATCH 2/2] Add recovery test suite

This includes basic tests maipulating standbys, be they archiving or
streaming nodes, and some basic sanity checks around them.
---
 src/test/Makefile                           |   2 +-
 src/test/perl/RecoveryTest.pm               | 167 ++++++++++++++++++++++++++++
 src/test/recovery/.gitignore                |   3 +
 src/test/recovery/Makefile                  |  17 +++
 src/test/recovery/README                    |  19 ++++
 src/test/recovery/t/001_stream_rep.pl       |  58 ++++++++++
 src/test/recovery/t/002_archiving.pl        |  43 +++++++
 src/test/recovery/t/003_recovery_targets.pl | 123 ++++++++++++++++++++
 src/test/recovery/t/004_timeline_switch.pl  |  66 +++++++++++
 src/test/recovery/t/005_replay_delay.pl     |  41 +++++++
 10 files changed, 538 insertions(+), 1 deletion(-)
 create mode 100644 src/test/perl/RecoveryTest.pm
 create mode 100644 src/test/recovery/.gitignore
 create mode 100644 src/test/recovery/Makefile
 create mode 100644 src/test/recovery/README
 create mode 100644 src/test/recovery/t/001_stream_rep.pl
 create mode 100644 src/test/recovery/t/002_archiving.pl
 create mode 100644 src/test/recovery/t/003_recovery_targets.pl
 create mode 100644 src/test/recovery/t/004_timeline_switch.pl
 create mode 100644 src/test/recovery/t/005_replay_delay.pl

diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..7f7754f 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -12,7 +12,7 @@ subdir = src/test
 top_builddir = ../..
 include $(top_builddir)/src/Makefile.global
 
-SUBDIRS = regress isolation modules
+SUBDIRS = regress isolation modules recovery
 
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
diff --git a/src/test/perl/RecoveryTest.pm b/src/test/perl/RecoveryTest.pm
new file mode 100644
index 0000000..a947a2d
--- /dev/null
+++ b/src/test/perl/RecoveryTest.pm
@@ -0,0 +1,167 @@
+# RecoveryTest, Set of common routines for recovery tests
+#
+# This works as a layer on top of PostgresNode to manage nodes as archiving
+# or streaming standbys and master nodes.
+
+package RecoveryTest;
+
+use strict;
+use warnings;
+
+use Cwd;
+use Exporter 'import';
+use IPC::Run qw(run start);
+use PostgresNode;
+use RecursiveCopy;
+use TestLib;
+use Test::More;
+
+our @EXPORT = qw(
+	enable_archiving
+	enable_restoring
+	enable_streaming
+	make_master
+	make_archive_standby
+	make_stream_standby
+);
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $node_root = shift; # Instance to link to
+	my $node_standby = shift;
+	my $root_connstr = $node_root->connstr;
+	my $applname = $node_standby->applname;
+
+	$node_standby->append_conf('recovery.conf', qq(
+primary_conninfo='$root_connstr application_name=$applname'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $node_root = shift; # Instance to link to
+	my $node_standby = shift;
+	my $path = $node_root->archive_dir;
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"$path\\\\%f\" \"%p\"" :
+		"cp -i $path/%f %p";
+	$node_standby->append_conf('recovery.conf', qq(
+restore_command='$copy_command'
+standby_mode=on
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $node = shift;
+	my $path = $node->archive_dir;
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$path\\\\%f\"" :
+		"cp %p $path/%f";
+
+	# Enable archive_mode and archive_command on node
+	$node->append_conf('postgresql.conf', qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization.
+sub make_master
+{
+	my $node_master = get_new_node();
+	my $port_master = $node_master->port;
+	print "# Initializing master node wih port $port_master\n";
+	$node_master->init;
+	configure_base_node($node_master);
+	return $node_master;
+}
+
+sub configure_base_node
+{
+	my $node = shift;
+
+	$node->append_conf('postgresql.conf', qq(
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+));
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_stream_standby
+{
+	my $node_master = shift;
+	my $backup_name = shift;
+	my $node_standby = get_new_node();
+	my $master_port = $node_master->port;
+	my $standby_port = $node_standby->port;
+
+	print "# Initializing streaming mode for node $standby_port from node $master_port\n";
+	$node_standby->init_from_backup($node_master, $backup_name);
+	configure_base_node($node_standby);
+
+	# Start second node, streaming from first one
+	enable_streaming($node_master, $node_standby);
+	return $node_standby;
+}
+
+# Node getting WAL only from archives
+sub make_archive_standby
+{
+	my $node_master = shift;
+	my $backup_name = shift;
+	my $node_standby = get_new_node();
+	my $master_port = $node_master->port;
+	my $standby_port = $node_standby->port;
+
+	print "# Initializing archive mode for node $standby_port from node $master_port\n";
+	$node_standby->init_from_backup($node_master, $backup_name);
+	configure_base_node($node_standby);
+
+	# Start second node, restoring from first one
+	enable_restoring($node_master, $node_standby);
+	return $node_standby;
+}
+
+# Wait until a node is able to accept queries. Useful when putting a node
+# in recovery and wait for it to be able to work particularly on slow
+# machines.
+sub wait_for_node
+{
+	my $node         = shift;
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	while ($attempts < $max_attempts)
+	{
+		if (run_log(['pg_isready', '-p', $node->port]))
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+	return 0;
+}
+
+1;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..255559b
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,58 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+use RecoveryTest;
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->start;
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_master->backup($backup_name);
+
+# Create streaming standby linking to master
+my $node_standby_1 = make_stream_standby($node_master, $backup_name);
+$node_standby_1->start;
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+$node_standby_1->backup($backup_name);
+
+# Create second standby node linking to standby 1
+my $node_standby_2 = make_stream_standby($node_standby_1, $backup_name);
+$node_standby_2->start;
+$node_standby_2->backup($backup_name);
+
+# Create some content on master and check its presence in standby 1
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a");
+
+# Wait for standbys to catch up
+my $applname_1 = $node_standby_1->applname;
+my $applname_2 = $node_standby_2->applname;
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_1';";
+$node_master->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_2';";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = $node_standby_1->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result =  $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+$node_standby_1->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+$node_standby_2->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_2->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..09f3c71
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,43 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $node_master = make_master();
+my $backup_name = 'my_backup';
+enable_archiving($node_master);
+
+# Start it
+$node_master->start;
+
+# Take backup for slave
+$node_master->backup($backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $node_standby = make_archive_standby($node_master, $backup_name);
+$node_standby->append_conf('postgresql.conf', qq(
+wal_retrieve_retry_interval = '100ms'
+));
+$node_standby->start;
+
+# Create some content on master
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $current_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Force archiving of WAL file to make it present on master
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Add some more content, it should not be present on standby
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(1000), 'check content from archives');
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..428bf82
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,123 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $node_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $node_standby = make_archive_standby($node_master, 'my_backup');
+
+	foreach my $param_item (@$recovery_params)
+	{
+		$node_standby->append_conf('recovery.conf',
+					   qq($param_item
+));
+	}
+
+	$node_standby->start;
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	$node_standby->poll_query_until('postgres', $caughtup_query)
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	$node_standby->teardown_node;
+}
+
+# Initialize master node
+my $node_master = make_master();
+enable_archiving($node_master);
+
+# Start it
+$node_master->start;
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $lsn1 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Take backup from which all operations will be run
+$node_master->backup('my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+my $recovery_txid = $node_master->psql('postgres', "SELECT txid_current()");
+my $lsn2 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# More data, with recovery target timestamp
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(2001,3000))");
+my $recovery_time = $node_master->psql('postgres', "SELECT now()");
+my $lsn3 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Even more data, this time with a recovery target name
+$node_master->psql('postgres',
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))");
+my $recovery_name = "my_target";
+my $lsn4 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+$node_master->psql('postgres', "SELECT pg_create_restore_point('$recovery_name'");
+
+# Force archiving of WAL file
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $node_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..d32797c
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,66 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create two standbys linking to it
+my $node_standby_1 = make_stream_standby($node_master, $backup_name);
+$node_standby_1->start;
+my $node_standby_2 = make_stream_standby($node_master, $backup_name);
+$node_standby_2->start;
+
+# Create some content on master
+$node_master->psql('postgres',
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+$node_master->teardown_node;
+system_or_bail('pg_ctl', '-w', '-D', $node_standby_1->data_dir,
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree($node_standby_2->data_dir . '/recovery.conf');
+my $connstr_1 = $node_standby_1->connstr;
+$node_standby_2->append_conf('recovery.conf', qq(
+primary_conninfo='$connstr_1'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+$node_standby_2->restart;
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done. Standby 1 needs
+# to exit recovery first before moving on with the test.
+$node_standby_1->poll_query_until('postgres', "SELECT pg_is_in_recovery() <> true");
+$node_standby_1->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+$until_lsn = $node_standby_1->psql('postgres', "SELECT pg_current_xlog_location();");
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_2->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(2000), 'check content of standby 2');
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..0000c02
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,41 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->start;
+
+# And some content
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create streaming standby from backup
+my $node_standby = make_stream_standby($node_master, $backup_name);
+$node_standby->append_conf('recovery.conf', qq(
+recovery_min_apply_delay = '2s'
+));
+$node_standby->start;
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(11,20))");
+sleep 1;
+# Here we should have only 10 rows
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+$result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(20), 'check content with delay of 2s');
-- 
2.6.3

#97Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#95)
2 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Dec 3, 2015 at 6:59 AM, Alvaro Herrera wrote:

I didn't push the changed for config_default you requested a few
messages upthread; it's not clear to me how setting it to undef affects
the whole thing. If setting it to undef makes the MSVC toolchain run
the tap tests in the default config, then I can do it; let's be clear
about what branch to backpatch this to. Also the "1;" at the end of
RewindTest.

Setting it to undef will prevent the tests to run, per vcregress.pl:
die "Tap tests not enabled in configuration"
unless $config->{tap_tests};
Also, setting it to undef will match the existing behavior on
platforms where ./configure is used because the switch
--enable-tap-tests needs to be used there. And I would believe that in
most cases Windows environments are not going to have IPC::Run
deployed.

I have also rebased the recovery test suite as the attached, using the
infrastructure that has been committed lately.
Regards,
--
Michael

Attachments:

0001-Fix-tap_test-configuration-in-MSVC-builds.patchtext/x-patch; charset=US-ASCII; name=0001-Fix-tap_test-configuration-in-MSVC-builds.patchDownload
From de2121eeb50c5ae49b29a2ac21a16579eae2de98 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Thu, 3 Dec 2015 13:48:48 +0900
Subject: [PATCH 1/2] Fix tap_test configuration in MSVC builds

---
 src/tools/msvc/config_default.pl | 27 ++++++++++++++-------------
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/src/tools/msvc/config_default.pl b/src/tools/msvc/config_default.pl
index b9f2ff4..e50be7e 100644
--- a/src/tools/msvc/config_default.pl
+++ b/src/tools/msvc/config_default.pl
@@ -3,7 +3,7 @@ use strict;
 use warnings;
 
 our $config = {
-	asserts => 0,    # --enable-cassert
+	asserts  => 0,    # --enable-cassert
 	  # integer_datetimes=>1,   # --enable-integer-datetimes - on is now default
 	  # float4byval=>1,         # --disable-float4-byval, on by default
 
@@ -13,18 +13,19 @@ our $config = {
 	# blocksize => 8,         # --with-blocksize, 8kB by default
 	# wal_blocksize => 8,     # --with-wal-blocksize, 8kB by default
 	# wal_segsize => 16,      # --with-wal-segsize, 16MB by default
-	ldap     => 1,        # --with-ldap
-	extraver => undef,    # --with-extra-version=<string>
-	nls      => undef,    # --enable-nls=<path>
-	tcl      => undef,    # --with-tls=<path>
-	perl     => undef,    # --with-perl
-	python   => undef,    # --with-python=<path>
-	openssl  => undef,    # --with-openssl=<path>
-	uuid     => undef,    # --with-ossp-uuid
-	xml      => undef,    # --with-libxml=<path>
-	xslt     => undef,    # --with-libxslt=<path>
-	iconv    => undef,    # (not in configure, path to iconv)
-	zlib     => undef     # --with-zlib=<path>
+	ldap      => 1,        # --with-ldap
+	extraver  => undef,    # --with-extra-version=<string>
+	nls       => undef,    # --enable-nls=<path>
+	tap_tests => undef,    # --enable-tap-tests
+	tcl       => undef,    # --with-tls=<path>
+	perl      => undef,    # --with-perl
+	python    => undef,    # --with-python=<path>
+	openssl   => undef,    # --with-openssl=<path>
+	uuid      => undef,    # --with-ossp-uuid
+	xml       => undef,    # --with-libxml=<path>
+	xslt      => undef,    # --with-libxslt=<path>
+	iconv     => undef,    # (not in configure, path to iconv)
+	zlib      => undef     # --with-zlib=<path>
 };
 
 1;
-- 
2.6.3

0002-Add-recovery-test-suite.patchtext/x-patch; charset=US-ASCII; name=0002-Add-recovery-test-suite.patchDownload
From 4ee2a99db2a414f85e961110279f4b97309cd927 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Thu, 3 Dec 2015 15:41:12 +0900
Subject: [PATCH 2/2] Add recovery test suite

This includes basic tests maipulating standbys, be they archiving or
streaming nodes, and some basic sanity checks around them.
---
 src/test/Makefile                           |   2 +-
 src/test/perl/RecoveryTest.pm               | 167 ++++++++++++++++++++++++++++
 src/test/recovery/.gitignore                |   3 +
 src/test/recovery/Makefile                  |  17 +++
 src/test/recovery/README                    |  19 ++++
 src/test/recovery/t/001_stream_rep.pl       |  58 ++++++++++
 src/test/recovery/t/002_archiving.pl        |  43 +++++++
 src/test/recovery/t/003_recovery_targets.pl | 123 ++++++++++++++++++++
 src/test/recovery/t/004_timeline_switch.pl  |  66 +++++++++++
 src/test/recovery/t/005_replay_delay.pl     |  41 +++++++
 10 files changed, 538 insertions(+), 1 deletion(-)
 create mode 100644 src/test/perl/RecoveryTest.pm
 create mode 100644 src/test/recovery/.gitignore
 create mode 100644 src/test/recovery/Makefile
 create mode 100644 src/test/recovery/README
 create mode 100644 src/test/recovery/t/001_stream_rep.pl
 create mode 100644 src/test/recovery/t/002_archiving.pl
 create mode 100644 src/test/recovery/t/003_recovery_targets.pl
 create mode 100644 src/test/recovery/t/004_timeline_switch.pl
 create mode 100644 src/test/recovery/t/005_replay_delay.pl

diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..7f7754f 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -12,7 +12,7 @@ subdir = src/test
 top_builddir = ../..
 include $(top_builddir)/src/Makefile.global
 
-SUBDIRS = regress isolation modules
+SUBDIRS = regress isolation modules recovery
 
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
diff --git a/src/test/perl/RecoveryTest.pm b/src/test/perl/RecoveryTest.pm
new file mode 100644
index 0000000..a947a2d
--- /dev/null
+++ b/src/test/perl/RecoveryTest.pm
@@ -0,0 +1,167 @@
+# RecoveryTest, Set of common routines for recovery tests
+#
+# This works as a layer on top of PostgresNode to manage nodes as archiving
+# or streaming standbys and master nodes.
+
+package RecoveryTest;
+
+use strict;
+use warnings;
+
+use Cwd;
+use Exporter 'import';
+use IPC::Run qw(run start);
+use PostgresNode;
+use RecursiveCopy;
+use TestLib;
+use Test::More;
+
+our @EXPORT = qw(
+	enable_archiving
+	enable_restoring
+	enable_streaming
+	make_master
+	make_archive_standby
+	make_stream_standby
+);
+
+# Set of handy routines able to set up a node with different characteristics
+# Enable streaming replication
+sub enable_streaming
+{
+	my $node_root = shift; # Instance to link to
+	my $node_standby = shift;
+	my $root_connstr = $node_root->connstr;
+	my $applname = $node_standby->applname;
+
+	$node_standby->append_conf('recovery.conf', qq(
+primary_conninfo='$root_connstr application_name=$applname'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+}
+
+# Enable the use of restore_command from a node
+sub enable_restoring
+{
+	my $node_root = shift; # Instance to link to
+	my $node_standby = shift;
+	my $path = $node_root->archive_dir;
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"$path\\\\%f\" \"%p\"" :
+		"cp -i $path/%f %p";
+	$node_standby->append_conf('recovery.conf', qq(
+restore_command='$copy_command'
+standby_mode=on
+));
+}
+
+# Enable WAL archiving on a node
+sub enable_archiving
+{
+	my $node = shift;
+	my $path = $node->archive_dir;
+
+	# Switch path to use slashes on Windows
+	$path =~ tr#\\#/# if ($windows_os);
+	my $copy_command = $windows_os ?
+		"copy \"%p\" \"$path\\\\%f\"" :
+		"cp %p $path/%f";
+
+	# Enable archive_mode and archive_command on node
+	$node->append_conf('postgresql.conf', qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Master node initialization.
+sub make_master
+{
+	my $node_master = get_new_node();
+	my $port_master = $node_master->port;
+	print "# Initializing master node wih port $port_master\n";
+	$node_master->init;
+	configure_base_node($node_master);
+	return $node_master;
+}
+
+sub configure_base_node
+{
+	my $node = shift;
+
+	$node->append_conf('postgresql.conf', qq(
+wal_level = hot_standby
+max_wal_senders = 5
+wal_keep_segments = 20
+max_wal_size = 128MB
+shared_buffers = 1MB
+wal_log_hints = on
+hot_standby = on
+autovacuum = off
+));
+}
+
+# Standby node initializations
+# Node only streaming.
+sub make_stream_standby
+{
+	my $node_master = shift;
+	my $backup_name = shift;
+	my $node_standby = get_new_node();
+	my $master_port = $node_master->port;
+	my $standby_port = $node_standby->port;
+
+	print "# Initializing streaming mode for node $standby_port from node $master_port\n";
+	$node_standby->init_from_backup($node_master, $backup_name);
+	configure_base_node($node_standby);
+
+	# Start second node, streaming from first one
+	enable_streaming($node_master, $node_standby);
+	return $node_standby;
+}
+
+# Node getting WAL only from archives
+sub make_archive_standby
+{
+	my $node_master = shift;
+	my $backup_name = shift;
+	my $node_standby = get_new_node();
+	my $master_port = $node_master->port;
+	my $standby_port = $node_standby->port;
+
+	print "# Initializing archive mode for node $standby_port from node $master_port\n";
+	$node_standby->init_from_backup($node_master, $backup_name);
+	configure_base_node($node_standby);
+
+	# Start second node, restoring from first one
+	enable_restoring($node_master, $node_standby);
+	return $node_standby;
+}
+
+# Wait until a node is able to accept queries. Useful when putting a node
+# in recovery and wait for it to be able to work particularly on slow
+# machines.
+sub wait_for_node
+{
+	my $node         = shift;
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	while ($attempts < $max_attempts)
+	{
+		if (run_log(['pg_isready', '-p', $node->port]))
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+	return 0;
+}
+
+1;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..255559b
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,58 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 4;
+use RecoveryTest;
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->start;
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_master->backup($backup_name);
+
+# Create streaming standby linking to master
+my $node_standby_1 = make_stream_standby($node_master, $backup_name);
+$node_standby_1->start;
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+$node_standby_1->backup($backup_name);
+
+# Create second standby node linking to standby 1
+my $node_standby_2 = make_stream_standby($node_standby_1, $backup_name);
+$node_standby_2->start;
+$node_standby_2->backup($backup_name);
+
+# Create some content on master and check its presence in standby 1
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a");
+
+# Wait for standbys to catch up
+my $applname_1 = $node_standby_1->applname;
+my $applname_2 = $node_standby_2->applname;
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_1';";
+$node_master->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_2';";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = $node_standby_1->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result =  $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+$node_standby_1->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+$node_standby_2->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_2->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..09f3c71
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,43 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+use RecoveryTest;
+
+# Initialize master node, doing archives
+my $node_master = make_master();
+my $backup_name = 'my_backup';
+enable_archiving($node_master);
+
+# Start it
+$node_master->start;
+
+# Take backup for slave
+$node_master->backup($backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $node_standby = make_archive_standby($node_master, $backup_name);
+$node_standby->append_conf('postgresql.conf', qq(
+wal_retrieve_retry_interval = '100ms'
+));
+$node_standby->start;
+
+# Create some content on master
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $current_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Force archiving of WAL file to make it present on master
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Add some more content, it should not be present on standby
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(1000), 'check content from archives');
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..428bf82
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,123 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 7;
+
+use RecoveryTest;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $node_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $node_standby = make_archive_standby($node_master, 'my_backup');
+
+	foreach my $param_item (@$recovery_params)
+	{
+		$node_standby->append_conf('recovery.conf',
+					   qq($param_item
+));
+	}
+
+	$node_standby->start;
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	$node_standby->poll_query_until('postgres', $caughtup_query)
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	$node_standby->teardown_node;
+}
+
+# Initialize master node
+my $node_master = make_master();
+enable_archiving($node_master);
+
+# Start it
+$node_master->start;
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $lsn1 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Take backup from which all operations will be run
+$node_master->backup('my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+my $recovery_txid = $node_master->psql('postgres', "SELECT txid_current()");
+my $lsn2 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# More data, with recovery target timestamp
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(2001,3000))");
+my $recovery_time = $node_master->psql('postgres', "SELECT now()");
+my $lsn3 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Even more data, this time with a recovery target name
+$node_master->psql('postgres',
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))");
+my $recovery_name = "my_target";
+my $lsn4 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+$node_master->psql('postgres', "SELECT pg_create_restore_point('$recovery_name'");
+
+# Force archiving of WAL file
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $node_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..d32797c
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,66 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+
+use RecoveryTest;
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create two standbys linking to it
+my $node_standby_1 = make_stream_standby($node_master, $backup_name);
+$node_standby_1->start;
+my $node_standby_2 = make_stream_standby($node_master, $backup_name);
+$node_standby_2->start;
+
+# Create some content on master
+$node_master->psql('postgres',
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+$node_master->teardown_node;
+system_or_bail('pg_ctl', '-w', '-D', $node_standby_1->data_dir,
+			   'promote');
+print "# Promoted standby 1\n";
+
+# Switch standby 2 to replay from standby 1
+remove_tree($node_standby_2->data_dir . '/recovery.conf');
+my $connstr_1 = $node_standby_1->connstr;
+$node_standby_2->append_conf('recovery.conf', qq(
+primary_conninfo='$connstr_1'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+$node_standby_2->restart;
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done. Standby 1 needs
+# to exit recovery first before moving on with the test.
+$node_standby_1->poll_query_until('postgres', "SELECT pg_is_in_recovery() <> true");
+$node_standby_1->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+$until_lsn = $node_standby_1->psql('postgres', "SELECT pg_current_xlog_location();");
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_2->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(2000), 'check content of standby 2');
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..0000c02
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,41 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 2;
+
+use RecoveryTest;
+
+# Initialize master node
+my $node_master = make_master();
+$node_master->start;
+
+# And some content
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create streaming standby from backup
+my $node_standby = make_stream_standby($node_master, $backup_name);
+$node_standby->append_conf('recovery.conf', qq(
+recovery_min_apply_delay = '2s'
+));
+$node_standby->start;
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(11,20))");
+sleep 1;
+# Here we should have only 10 rows
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+$result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(20), 'check content with delay of 2s');
-- 
2.6.3

#98Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#97)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Dec 3, 2015 at 3:44 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Thu, Dec 3, 2015 at 6:59 AM, Alvaro Herrera wrote:

I didn't push the changed for config_default you requested a few
messages upthread; it's not clear to me how setting it to undef affects
the whole thing. If setting it to undef makes the MSVC toolchain run
the tap tests in the default config, then I can do it; let's be clear
about what branch to backpatch this to. Also the "1;" at the end of
RewindTest.

Setting it to undef will prevent the tests to run, per vcregress.pl:
die "Tap tests not enabled in configuration"
unless $config->{tap_tests};
Also, setting it to undef will match the existing behavior on
platforms where ./configure is used because the switch
--enable-tap-tests needs to be used there. And I would believe that in
most cases Windows environments are not going to have IPC::Run
deployed.

I have also rebased the recovery test suite as the attached, using the
infrastructure that has been committed lately.

By the way, if there are no objections, I am going to mark this patch
as committed in the CF app. Putting in the infrastructure is already a
good step forward, and I will add an entry in next CF to track the
patch to add tests for recovery itself. Alvaro, what do you think?
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#99Noah Misch
noah@leadboat.com
In reply to: Alvaro Herrera (#94)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Dec 02, 2015 at 04:33:50PM -0300, Alvaro Herrera wrote:

Noah Misch wrote:

On Tue, Dec 01, 2015 at 08:11:21PM -0300, Alvaro Herrera wrote:

Finally, I ran perltidy on all the files, which strangely changed stuff
that I didn't expect it to change. I wonder if this is related to the
perltidy version.

The last pgindent run (commit 807b9e0) used perltidy v20090616, and perltidy
behavior has changed slightly over time. Install that version to do your own
perltidy runs.

I tried that version, but it seems to emit the same.

git checkout 807b9e0
(find src -name \*.pl -o -name \*.pm ) | sort -u | xargs perltidy --profile=src/tools/pgindent/perltidyrc

perltidy v20090616 leaves the working directory clean, but perltidy v20150815
introduces diffs:

src/backend/catalog/genbki.pl | 15 ++++++++-------
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 2 +-
src/tools/msvc/Install.pm | 6 +++---
src/tools/msvc/Mkvcbuild.pm | 2 +-
src/tools/msvc/Project.pm | 2 +-
src/tools/msvc/Solution.pm | 5 ++---
src/tools/msvc/gendef.pl | 4 ++--
7 files changed, 18 insertions(+), 18 deletions(-)

You see a different result?

How did you figure
that that was the version used, anyway?

I asked Bruce at one point.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#100Michael Paquier
michael.paquier@gmail.com
In reply to: Noah Misch (#99)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Dec 4, 2015 at 2:43 PM, Noah Misch <noah@leadboat.com> wrote:

On Wed, Dec 02, 2015 at 04:33:50PM -0300, Alvaro Herrera wrote:

How did you figure
that that was the version used, anyway?

I asked Bruce at one point.

So we are trying to use the same version over the years to keep code
consistent across back-branches? Do you think we should try to use a
newer version instead with each pgindent run? That would induce a
rebasing cost when back-patching, but we cannot stay with the same
version of perltidy forever either...
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#101Noah Misch
noah@leadboat.com
In reply to: Michael Paquier (#100)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Dec 04, 2015 at 02:47:36PM +0900, Michael Paquier wrote:

On Fri, Dec 4, 2015 at 2:43 PM, Noah Misch <noah@leadboat.com> wrote:

On Wed, Dec 02, 2015 at 04:33:50PM -0300, Alvaro Herrera wrote:

How did you figure
that that was the version used, anyway?

I asked Bruce at one point.

So we are trying to use the same version over the years to keep code
consistent across back-branches?

No, I recall no policy discussion concerning this.

Do you think we should try to use a
newer version instead with each pgindent run? That would induce a
rebasing cost when back-patching, but we cannot stay with the same
version of perltidy forever either...

No. I expect we'll implicitly change perltidy versions when Bruce upgrades
his OS. Assuming future perltidy changes affect us not much more than past
changes (six years of perltidy development changed eighteen PostgreSQL source
lines), that's just fine.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#102Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Noah Misch (#99)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Noah Misch wrote:

git checkout 807b9e0
(find src -name \*.pl -o -name \*.pm ) | sort -u | xargs perltidy --profile=src/tools/pgindent/perltidyrc

perltidy v20090616 leaves the working directory clean, but perltidy v20150815
introduces diffs:

src/backend/catalog/genbki.pl | 15 ++++++++-------
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 2 +-
src/tools/msvc/Install.pm | 6 +++---
src/tools/msvc/Mkvcbuild.pm | 2 +-
src/tools/msvc/Project.pm | 2 +-
src/tools/msvc/Solution.pm | 5 ++---
src/tools/msvc/gendef.pl | 4 ++--
7 files changed, 18 insertions(+), 18 deletions(-)

You see a different result?

Oh, you're right -- on that commit, I get the same results as you with
v20090616 (no changes). I don't have v20150815; the version packaged by
Debian is v20140328, and Install.pm is not changed by that one (so I
only get 15 lines changed, not 18).

I think my confusion stems from code that was introduced after the last
pgindent run. I guess I could have tidied the original files prior to
patching.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#103Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#98)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

By the way, if there are no objections, I am going to mark this patch
as committed in the CF app. Putting in the infrastructure is already a
good step forward, and I will add an entry in next CF to track the
patch to add tests for recovery itself. Alvaro, what do you think?

Feel free to do that, but I'm likely to look into it before the next CF
anyway. The thread about this has been open since 2013, so that doesn't
seem unfair.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#104Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#103)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Dec 5, 2015 at 1:11 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

Michael Paquier wrote:

By the way, if there are no objections, I am going to mark this patch
as committed in the CF app. Putting in the infrastructure is already a
good step forward, and I will add an entry in next CF to track the
patch to add tests for recovery itself. Alvaro, what do you think?

Feel free to do that, but I'm likely to look into it before the next CF
anyway. The thread about this has been open since 2013, so that doesn't
seem unfair.

Let's see then. For the time being I have marked this patch as
committed, and created a new entry for the set of tests regarding
standbys:
https://commitfest.postgresql.org/8/438/
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#105Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andrew Dunstan (#84)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Andrew Dunstan <andrew@dunslane.net> writes:

On 11/29/2015 04:28 PM, Noah Misch wrote:

Never mutate the filesystem in a BEGIN block, because "perl -c" runs BEGIN
blocks. (Likewise for the BEGIN block this patch adds to TestLib.)

Yeah, those two lines might belong in an INIT block. "perldoc perlmod"
for details.

BTW, if we consider that gospel, why has PostgresNode.pm got this in its
BEGIN block?

# PGHOST is set once and for all through a single series of tests when
# this module is loaded.
$test_pghost =
$TestLib::windows_os ? "127.0.0.1" : TestLib::tempdir_short;
$ENV{PGHOST} = $test_pghost;

On non-Windows machines, the call of tempdir_short will result in
filesystem activity, ie creation of a directory. Offhand it looks like
all of the activity in this BEGIN block could go to an INIT block instead.

I'm also bemused by why there was any question about this being wrong:

# XXX: Should this use PG_VERSION_NUM?
$last_port_assigned = 90600 % 16384 + 49152;

If that's not a hard-coded PG version number then I don't know
what it is. Maybe it would be better to use random() instead,
but surely this isn't good as-is.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#106Noah Misch
noah@leadboat.com
In reply to: Alvaro Herrera (#95)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Dec 02, 2015 at 06:59:09PM -0300, Alvaro Herrera wrote:

It seemed better to me to have the import list be empty, i.e. "use
TestLib ()" and then qualify the routine names inside PostgresNode,
instead of having to list the names of the routines to import, so I
pushed it that way after running the tests a few more times.

I inspected commit 1caef31:

--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
-like($recovery_conf, qr/^standby_mode = 'on[']$/m, 'recovery.conf sets standby_mode');
-like($recovery_conf, qr/^primary_conninfo = '.*port=$ENV{PGPORT}.*'$/m, 'recovery.conf sets primary_conninfo');
-
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxf", '-X', 'fetch' ],
+like(
+	$recovery_conf,
+	qr/^standby_mode = 'on[']$/m,
+	'recovery.conf sets standby_mode');
+like(
+	$recovery_conf,
+	qr/^primary_conninfo = '.*port=$ENV{PGPORT}.*'$/m,
+	'recovery.conf sets primary_conninfo');

This now elicits a diagnostic:

Use of uninitialized value $ENV{"PGPORT"} in regexp compilation at t/010_pg_basebackup.pl line 175, <> line 1.

--- a/src/bin/pg_controldata/t/001_pg_controldata.pl
+++ b/src/bin/pg_controldata/t/001_pg_controldata.pl
-standard_initdb "$tempdir/data";
-command_like([ 'pg_controldata', "$tempdir/data" ],
+
+my $node = get_new_node();
+$node->init;
+$node->start;

Before the commit, this test did not start a server.

--- /dev/null
+++ b/src/test/perl/PostgresNode.pm
+sub _update_pid
+{
+	my $self = shift;
+
+	# If we can open the PID file, read its first line and that's the PID we
+	# want.  If the file cannot be opened, presumably the server is not
+	# running; don't be noisy in that case.
+	open my $pidfile, $self->data_dir . "/postmaster.pid";
+	if (not defined $pidfile)

$ grep -h 'at /.*line' src/bin/*/tmp_check/log/* |sort|uniq -c
1 cannot remove directory for /data/nmisch/src/pg/postgresql/src/bin/scripts/tmp_testNNCZ: Directory not empty at /home/nmisch/sw/cpan/lib/perl5/File/Temp.pm line 784
1 cannot remove directory for /data/nmisch/src/pg/postgresql/src/bin/scripts/tmp_testNNCZ/pgdata/base/13264: Directory not empty at /home/nmisch/sw/cpan/lib/perl5/File/Temp.pm line 784
1 cannot remove directory for /data/nmisch/src/pg/postgresql/src/bin/scripts/tmp_testNNCZ/pgdata/base: Directory not empty at /home/nmisch/sw/cpan/lib/perl5/File/Temp.pm line 784
1 cannot remove directory for /data/nmisch/src/pg/postgresql/src/bin/scripts/tmp_testNNCZ/pgdata: Directory not empty at /home/nmisch/sw/cpan/lib/perl5/File/Temp.pm line 784
28 readline() on closed filehandle $pidfile at /data/nmisch/src/pg/postgresql/src/bin/pg_rewind/../../../src/test/perl/PostgresNode.pm line 308.
28 Use of uninitialized value in concatenation (.) or string at /data/nmisch/src/pg/postgresql/src/bin/pg_rewind/../../../src/test/perl/PostgresNode.pm line 309.

$pidfile is always defined. Test the open() return value.

+	{
+		$self->{_pid} = undef;
+		print "# No postmaster PID\n";
+		return;
+	}
+
+	$self->{_pid} = <$pidfile>;

chomp() that value to remove its trailing newline.

+	print "# Postmaster PID is $self->{_pid}\n";
+	close $pidfile;
+}

+ my $devnull = $TestLib::windows_os ? "nul" : "/dev/null";

Unused variable.

+sub DESTROY
+{
+	my $self = shift;
+	return if not defined $self->{_pid};
+	print "# signalling QUIT to $self->{_pid}\n";
+	kill 'QUIT', $self->{_pid};

On Windows, that kill() is the wrong thing. I suspect "pg_ctl kill" will be
the right thing, but that warrants specific testing.

Postmaster log file names became less informative. Before the commit:

-rw-------. 1 nmisch nmisch 211265 2015-12-06 22:35:59.931114215 +0000 regress_log_001_basic
-rw-------. 1 nmisch nmisch 268887 2015-12-06 22:36:21.871165951 +0000 regress_log_002_databases
-rw-------. 1 nmisch nmisch 206808 2015-12-06 22:36:41.861213736 +0000 regress_log_003_extrafiles
-rw-------. 1 nmisch nmisch 7464 2015-12-06 22:37:00.371256795 +0000 master.log
-rw-------. 1 nmisch nmisch 6648 2015-12-06 22:37:01.381259211 +0000 standby.log
-rw-------. 1 nmisch nmisch 208561 2015-12-06 22:37:02.381261374 +0000 regress_log_004_pg_xlog_symlink

After:

-rw-------. 1 nmisch nmisch 219581 2015-12-06 23:00:50.504643175 +0000 regress_log_001_basic
-rw-------. 1 nmisch nmisch 276315 2015-12-06 23:01:12.924697085 +0000 regress_log_002_databases
-rw-------. 1 nmisch nmisch 213940 2015-12-06 23:01:33.574747195 +0000 regress_log_003_extrafiles
-rw-------. 1 nmisch nmisch 4114 2015-12-06 23:01:40.914764850 +0000 node_57834.log
-rw-------. 1 nmisch nmisch 6166 2015-12-06 23:01:43.184770615 +0000 node_57833.log
-rw-------. 1 nmisch nmisch 5550 2015-12-06 23:01:52.504792997 +0000 node_57835.log
-rw-------. 1 nmisch nmisch 9494 2015-12-06 23:01:53.514794802 +0000 node_57836.log
-rw-------. 1 nmisch nmisch 216212 2015-12-06 23:01:54.544797739 +0000 regress_log_004_pg_xlog_symlink

Should nodes get a name, so we instead see master_57834.log?

See also the things Tom Lane identified in <27550.1449342569@sss.pgh.pa.us>.

nm

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#107Michael Paquier
michael.paquier@gmail.com
In reply to: Noah Misch (#106)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Mon, Dec 7, 2015 at 12:06 PM, Noah Misch <noah@leadboat.com> wrote:

On Wed, Dec 02, 2015 at 06:59:09PM -0300, Alvaro Herrera wrote:

--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
-like($recovery_conf, qr/^standby_mode = 'on[']$/m, 'recovery.conf sets standby_mode');
-like($recovery_conf, qr/^primary_conninfo = '.*port=$ENV{PGPORT}.*'$/m, 'recovery.conf sets primary_conninfo');
-
-command_ok([ 'pg_basebackup', '-D', "$tempdir/backupxf", '-X', 'fetch' ],
+like(
+     $recovery_conf,
+     qr/^standby_mode = 'on[']$/m,
+     'recovery.conf sets standby_mode');
+like(
+     $recovery_conf,
+     qr/^primary_conninfo = '.*port=$ENV{PGPORT}.*'$/m,
+     'recovery.conf sets primary_conninfo');

This now elicits a diagnostic:

Use of uninitialized value $ENV{"PGPORT"} in regexp compilation at t/010_pg_basebackup.pl line 175, <> line 1.

Fixed.

--- a/src/bin/pg_controldata/t/001_pg_controldata.pl
+++ b/src/bin/pg_controldata/t/001_pg_controldata.pl
-standard_initdb "$tempdir/data";
-command_like([ 'pg_controldata', "$tempdir/data" ],
+
+my $node = get_new_node();
+$node->init;
+$node->start;

Before the commit, this test did not start a server.

Fixed.

--- /dev/null
+++ b/src/test/perl/PostgresNode.pm
+sub _update_pid
+{
+     my $self = shift;
+
+     # If we can open the PID file, read its first line and that's the PID we
+     # want.  If the file cannot be opened, presumably the server is not
+     # running; don't be noisy in that case.
+     open my $pidfile, $self->data_dir . "/postmaster.pid";
+     if (not defined $pidfile)

$ grep -h 'at /.*line' src/bin/*/tmp_check/log/* |sort|uniq -c
1 cannot remove directory for /data/nmisch/src/pg/postgresql/src/bin/scripts/tmp_testNNCZ: Directory not empty at /home/nmisch/sw/cpan/lib/perl5/File/Temp.pm line 784
1 cannot remove directory for /data/nmisch/src/pg/postgresql/src/bin/scripts/tmp_testNNCZ/pgdata/base/13264: Directory not empty at /home/nmisch/sw/cpan/lib/perl5/File/Temp.pm line 784
1 cannot remove directory for /data/nmisch/src/pg/postgresql/src/bin/scripts/tmp_testNNCZ/pgdata/base: Directory not empty at /home/nmisch/sw/cpan/lib/perl5/File/Temp.pm line 784
1 cannot remove directory for /data/nmisch/src/pg/postgresql/src/bin/scripts/tmp_testNNCZ/pgdata: Directory not empty at /home/nmisch/sw/cpan/lib/perl5/File/Temp.pm line 784
28 readline() on closed filehandle $pidfile at /data/nmisch/src/pg/postgresql/src/bin/pg_rewind/../../../src/test/perl/PostgresNode.pm line 308.
28 Use of uninitialized value in concatenation (.) or string at /data/nmisch/src/pg/postgresql/src/bin/pg_rewind/../../../src/test/perl/PostgresNode.pm line 309.

$pidfile is always defined. Test the open() return value.

That's something I have addressed here:
/messages/by-id/CAB7nPqTOP28Zxv_SXFo2axGJoesfvLLMO6syddAfV0DUvsFMDw@mail.gmail.com
I am including the fix as well here to do a group shot.

+     {
+             $self->{_pid} = undef;
+             print "# No postmaster PID\n";
+             return;
+     }
+
+     $self->{_pid} = <$pidfile>;

chomp() that value to remove its trailing newline.

Right.

+     print "# Postmaster PID is $self->{_pid}\n";
+     close $pidfile;
+}

+ my $devnull = $TestLib::windows_os ? "nul" : "/dev/null";

Unused variable.

Removed.

+sub DESTROY
+{
+     my $self = shift;
+     return if not defined $self->{_pid};
+     print "# signalling QUIT to $self->{_pid}\n";
+     kill 'QUIT', $self->{_pid};

On Windows, that kill() is the wrong thing. I suspect "pg_ctl kill" will be
the right thing, but that warrants specific testing.

I don't directly see any limitation with the use of kill on Windows..
http://perldoc.perl.org/functions/kill.html
But indeed using directly pg_ctl kill seems like a better fit for the
PG infrastructure.

Postmaster log file names became less informative. Before the commit:
Should nodes get a name, so we instead see master_57834.log?

I am not sure that this is necessary. There is definitely a limitation
regarding the fact that log files of nodes get overwritten after each
test, hence I would tend with just appending the test name in front of
the node_* files similarly to the other files.

See also the things Tom Lane identified in <27550.1449342569@sss.pgh.pa.us>.

Yep, I marked this email as something to address when it was sent.

On Sun, Dec 6, 2015 at 4:09 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

BTW, if we consider that gospel, why has PostgresNode.pm got this in its
BEGIN block?

# PGHOST is set once and for all through a single series of tests when
# this module is loaded.
$test_pghost =
$TestLib::windows_os ? "127.0.0.1" : TestLib::tempdir_short;
$ENV{PGHOST} = $test_pghost;

On non-Windows machines, the call of tempdir_short will result in
filesystem activity, ie creation of a directory. Offhand it looks like
all of the activity in this BEGIN block could go to an INIT block instead.

OK, the whole block is switched to INIT instead.

I'm also bemused by why there was any question about this being wrong:

# XXX: Should this use PG_VERSION_NUM?
$last_port_assigned = 90600 % 16384 + 49152;

If that's not a hard-coded PG version number then I don't know
what it is. Maybe it would be better to use random() instead,
but surely this isn't good as-is.

We would definitely want something within the ephemeral port range, so
we are up to that:
rand() * 16384 + 49152;

All those issues are addressed as per the patch attached.
Regards,
--
Michael

Attachments:

20151206_tap_fixes.patchtext/x-diff; charset=US-ASCII; name=20151206_tap_fixes.patchDownload
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 3e491a8..5991cf7 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -168,13 +168,14 @@ my $recovery_conf = slurp_file "$tempdir/backupR/recovery.conf";
 
 # using a character class for the final "'" here works around an apparent
 # bug in several version of the Msys DTK perl
+my $port = $node->port;
 like(
 	$recovery_conf,
 	qr/^standby_mode = 'on[']$/m,
 	'recovery.conf sets standby_mode');
 like(
 	$recovery_conf,
-	qr/^primary_conninfo = '.*port=$ENV{PGPORT}.*'$/m,
+	qr/^primary_conninfo = '.*port=$port.*'$/m,
 	'recovery.conf sets primary_conninfo');
 
 $node->command_ok(
diff --git a/src/bin/pg_controldata/t/001_pg_controldata.pl b/src/bin/pg_controldata/t/001_pg_controldata.pl
index ae45f41..073815a 100644
--- a/src/bin/pg_controldata/t/001_pg_controldata.pl
+++ b/src/bin/pg_controldata/t/001_pg_controldata.pl
@@ -13,7 +13,6 @@ command_fails([ 'pg_controldata', 'nonexistent' ],
 
 my $node = get_new_node();
 $node->init;
-$node->start;
 
 command_like([ 'pg_controldata', $node->data_dir ],
 	qr/checkpoint/, 'pg_controldata produces output');
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index aa7a00c..db7459f 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -27,7 +27,7 @@ our @EXPORT = qw(
 
 our ($test_pghost, $last_port_assigned, @all_nodes);
 
-BEGIN
+INIT
 {
 
 	# PGHOST is set once and for all through a single series of tests when
@@ -38,8 +38,7 @@ BEGIN
 	$ENV{PGDATABASE} = 'postgres';
 
 	# Tracking of last port value assigned to accelerate free port lookup.
-	# XXX: Should this use PG_VERSION_NUM?
-	$last_port_assigned = 90600 % 16384 + 49152;
+	$last_port_assigned = int(rand() * 16384) + 49152;
 
 	# Node tracking
 	@all_nodes = ();
@@ -50,12 +49,14 @@ sub new
 	my $class  = shift;
 	my $pghost = shift;
 	my $pgport = shift;
+	my $testname = basename($0);
+	$testname =~ s/\.[^.]+$//;
 	my $self   = {
 		_port     => $pgport,
 		_host     => $pghost,
 		_basedir  => TestLib::tempdir,
 		_applname => "node_$pgport",
-		_logfile  => "$TestLib::log_path/node_$pgport.log" };
+		_logfile  => $TestLib::log_path . '/' . $testname . '_node_' . $pgport . '.log' };
 
 	bless $self, $class;
 	$self->dump_info;
@@ -297,17 +298,17 @@ sub _update_pid
 	# If we can open the PID file, read its first line and that's the PID we
 	# want.  If the file cannot be opened, presumably the server is not
 	# running; don't be noisy in that case.
-	open my $pidfile, $self->data_dir . "/postmaster.pid";
-	if (not defined $pidfile)
+	if (open my $pidfile, $self->data_dir . "/postmaster.pid")
 	{
-		$self->{_pid} = undef;
-		print "# No postmaster PID\n";
+		my $pid = <$pidfile>;
+		$self->{_pid} = chomp($pid);
+		print "# Postmaster PID is $self->{_pid}\n";
+		close $pidfile;
 		return;
 	}
 
-	$self->{_pid} = <$pidfile>;
-	print "# Postmaster PID is $self->{_pid}\n";
-	close $pidfile;
+	$self->{_pid} = undef;
+	print "# No postmaster PID\n";
 }
 
 #
@@ -327,7 +328,6 @@ sub get_new_node
 	{
 		$port++;
 		print "# Checking for port $port\n";
-		my $devnull = $TestLib::windows_os ? "nul" : "/dev/null";
 		if (!TestLib::run_log([ 'pg_isready', '-p', $port ]))
 		{
 			$found = 1;
@@ -360,7 +360,7 @@ sub DESTROY
 	my $self = shift;
 	return if not defined $self->{_pid};
 	print "# signalling QUIT to $self->{_pid}\n";
-	kill 'QUIT', $self->{_pid};
+	TestLib::system_log('pg_ctl', 'kill', 'QUIT', $self->{_pid});
 }
 
 sub teardown_node
#108Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#107)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

If that's not a hard-coded PG version number then I don't know
what it is. Maybe it would be better to use random() instead,
but surely this isn't good as-is.

We would definitely want something within the ephemeral port range, so
we are up to that:
rand() * 16384 + 49152;

Yes, this seems to produce the correct range.

Thanks Noah and Tom for the review, and thanks Michael for the patch. I
pushed it. A slight fix was to change the chomp() call; it was always
returning 1 (number of elements chomped) so it tried to kill init.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#109Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#97)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

On Thu, Dec 3, 2015 at 6:59 AM, Alvaro Herrera wrote:

I didn't push the changed for config_default you requested a few
messages upthread; it's not clear to me how setting it to undef affects
the whole thing. If setting it to undef makes the MSVC toolchain run
the tap tests in the default config, then I can do it; let's be clear
about what branch to backpatch this to. Also the "1;" at the end of
RewindTest.

Setting it to undef will prevent the tests to run, per vcregress.pl:
die "Tap tests not enabled in configuration"
unless $config->{tap_tests};
Also, setting it to undef will match the existing behavior on
platforms where ./configure is used because the switch
--enable-tap-tests needs to be used there. And I would believe that in
most cases Windows environments are not going to have IPC::Run
deployed.

But if I don't set it to anything, then it will be "initialized" as
undef, so it has the same effect.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#110Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#108)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Alvaro Herrera <alvherre@2ndquadrant.com> writes:

Michael Paquier wrote:

We would definitely want something within the ephemeral port range, so
we are up to that:
rand() * 16384 + 49152;

Yes, this seems to produce the correct range.

Speaking of ephemeral port range ... if get_new_node() were to run
past port 65535, which is certainly possible with this new code,
it would fail altogether. Seems it needs to wrap around properly
within that port range.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#111Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Tom Lane (#110)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Tom Lane wrote:

Alvaro Herrera <alvherre@2ndquadrant.com> writes:

Michael Paquier wrote:

We would definitely want something within the ephemeral port range, so
we are up to that:
rand() * 16384 + 49152;

Yes, this seems to produce the correct range.

Speaking of ephemeral port range ... if get_new_node() were to run
past port 65535, which is certainly possible with this new code,
it would fail altogether. Seems it needs to wrap around properly
within that port range.

Oh, of course. Pushed fix.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#112Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#109)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Tue, Dec 8, 2015 at 7:46 AM, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

Michael Paquier wrote:

On Thu, Dec 3, 2015 at 6:59 AM, Alvaro Herrera wrote:

I didn't push the changed for config_default you requested a few
messages upthread; it's not clear to me how setting it to undef affects
the whole thing. If setting it to undef makes the MSVC toolchain run
the tap tests in the default config, then I can do it; let's be clear
about what branch to backpatch this to. Also the "1;" at the end of
RewindTest.

Setting it to undef will prevent the tests to run, per vcregress.pl:
die "Tap tests not enabled in configuration"
unless $config->{tap_tests};
Also, setting it to undef will match the existing behavior on
platforms where ./configure is used because the switch
--enable-tap-tests needs to be used there. And I would believe that in
most cases Windows environments are not going to have IPC::Run
deployed.

But if I don't set it to anything, then it will be "initialized" as
undef, so it has the same effect.

Yes, it will have the same effect. Still it is a problem to not list it in
default_config.pl as the other options, no? And that's as well the case
with GetFakeConfigure, which should list it I think for consistency with
the rest. See the attached for example.
--
Michael

Attachments:

20151208_tap_msvc_fix.patchbinary/octet-stream; name=20151208_tap_msvc_fix.patchDownload
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index 1564b72..712f2bb 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -635,6 +635,7 @@ sub GetFakeConfigure
 	$cfg .= ' --enable-integer-datetimes'
 	  if ($self->{options}->{integer_datetimes});
 	$cfg .= ' --enable-nls' if ($self->{options}->{nls});
+	$cfg .= ' --enable-tap-tests' if ($self->{options}->{tap_tests});
 	$cfg .= ' --with-ldap'  if ($self->{options}->{ldap});
 	$cfg .= ' --without-zlib' unless ($self->{options}->{zlib});
 	$cfg .= ' --with-extra-version' if ($self->{options}->{extraver});
diff --git a/src/tools/msvc/config_default.pl b/src/tools/msvc/config_default.pl
index b9f2ff4..e50be7e 100644
--- a/src/tools/msvc/config_default.pl
+++ b/src/tools/msvc/config_default.pl
@@ -3,7 +3,7 @@ use strict;
 use warnings;
 
 our $config = {
-	asserts => 0,    # --enable-cassert
+	asserts  => 0,    # --enable-cassert
 	  # integer_datetimes=>1,   # --enable-integer-datetimes - on is now default
 	  # float4byval=>1,         # --disable-float4-byval, on by default
 
@@ -13,18 +13,19 @@ our $config = {
 	# blocksize => 8,         # --with-blocksize, 8kB by default
 	# wal_blocksize => 8,     # --with-wal-blocksize, 8kB by default
 	# wal_segsize => 16,      # --with-wal-segsize, 16MB by default
-	ldap     => 1,        # --with-ldap
-	extraver => undef,    # --with-extra-version=<string>
-	nls      => undef,    # --enable-nls=<path>
-	tcl      => undef,    # --with-tls=<path>
-	perl     => undef,    # --with-perl
-	python   => undef,    # --with-python=<path>
-	openssl  => undef,    # --with-openssl=<path>
-	uuid     => undef,    # --with-ossp-uuid
-	xml      => undef,    # --with-libxml=<path>
-	xslt     => undef,    # --with-libxslt=<path>
-	iconv    => undef,    # (not in configure, path to iconv)
-	zlib     => undef     # --with-zlib=<path>
+	ldap      => 1,        # --with-ldap
+	extraver  => undef,    # --with-extra-version=<string>
+	nls       => undef,    # --enable-nls=<path>
+	tap_tests => undef,    # --enable-tap-tests
+	tcl       => undef,    # --with-tls=<path>
+	perl      => undef,    # --with-perl
+	python    => undef,    # --with-python=<path>
+	openssl   => undef,    # --with-openssl=<path>
+	uuid      => undef,    # --with-ossp-uuid
+	xml       => undef,    # --with-libxml=<path>
+	xslt      => undef,    # --with-libxslt=<path>
+	iconv     => undef,    # (not in configure, path to iconv)
+	zlib      => undef     # --with-zlib=<path>
 };
 
 1;
#113Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#97)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

I've been giving RecoveryTest.pm a look. I wonder if we really need that
as a separate package. My first thought was that we could have another
class that inherits from PostgresNode (say RecoveryNode). But later it
occured to me that we could have the new functions just be part of
PostgresNode itself directly; so we would have some new PostgresNode
methods:
$node->enable_streaming
$node->enable_restoring
$node->enable_archiving
$node->wait (your RecoveryTest::wait_for_node; better name for this?)

and some additional constructors:
make_master
make_stream_standby
make_archive_standby

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#114Noah Misch
noah@leadboat.com
In reply to: Michael Paquier (#107)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Mon, Dec 07, 2015 at 02:34:39PM +0900, Michael Paquier wrote:

On Mon, Dec 7, 2015 at 12:06 PM, Noah Misch <noah@leadboat.com> wrote:

On Wed, Dec 02, 2015 at 06:59:09PM -0300, Alvaro Herrera wrote:

+sub DESTROY
+{
+     my $self = shift;
+     return if not defined $self->{_pid};
+     print "# signalling QUIT to $self->{_pid}\n";
+     kill 'QUIT', $self->{_pid};

On Windows, that kill() is the wrong thing. I suspect "pg_ctl kill" will be
the right thing, but that warrants specific testing.

I don't directly see any limitation with the use of kill on Windows..
http://perldoc.perl.org/functions/kill.html
But indeed using directly pg_ctl kill seems like a better fit for the
PG infrastructure.

From http://perldoc.perl.org/perlwin32.html, "Using signals under this port
should currently be considered unsupported." Windows applications cannot
handle SIGQUIT: https://msdn.microsoft.com/en-us/library/xdkz3x12.aspx. The
PostgreSQL backend does not generate or expect Windows signals; see its
signal.c emulation facility.

Postmaster log file names became less informative. Before the commit:
Should nodes get a name, so we instead see master_57834.log?

I am not sure that this is necessary.

In general, you'd need to cross-reference the main log file to determine which
postmaster log corresponds to which action within the test. I did plenty of
"grep $PATTERN src/bin/pg_rewind/tmp_check/log/master.log" while debugging
that test. I'd like to be able to use /*master*.log, not rely on timestamps
or on scraping regress_log_002_databases to determine which logs are master
logs. Feel free to skip this point if I'm the only one minding, though.

There is definitely a limitation
regarding the fact that log files of nodes get overwritten after each
test, hence I would tend with just appending the test name in front of
the node_* files similarly to the other files.

They got appended, not overwritten. I like how you changed that to not
happen, but it doesn't address what I was reporting.

nm

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#115Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#113)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Dec 10, 2015 at 6:46 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

I've been giving RecoveryTest.pm a look. I wonder if we really need that
as a separate package. My first thought was that we could have another
class that inherits from PostgresNode (say RecoveryNode). But later it
occured to me that we could have the new functions just be part of
PostgresNode itself directly; so we would have some new PostgresNode
methods:
$node->enable_streaming
$node->enable_restoring
$node->enable_archiving

Sure.

$node->wait (your RecoveryTest::wait_for_node; better name for this?)

wait_for_access?

and some additional constructors:
make_master
make_stream_standby
make_archive_standby

I have done that a little bit differently. Those are completely
remove, then init() and init_from_backup() are extended with a new set
of parameters to enable archive, streaming or restore on a node.

Which gives the patch attached.
--
Michael

Attachments:

20151210_recovery_suite.patchbinary/octet-stream; name=20151210_recovery_suite.patchDownload
From 8c496a42c75eb696d2b930e9cffa883121060db1 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Thu, 10 Dec 2015 14:50:37 +0900
Subject: [PATCH] Add recovery test suite

This includes basic tests maipulating standbys, be they archiving or
streaming nodes, and some basic sanity checks around them. PostgresNode
is extended with a couple of routines allowing to set up WAL archiving,
WAL streaming or WAL restore on a node, as well as a commodity routine
to allow a promotion.
---
 src/bin/pg_rewind/RewindTest.pm             |   2 +-
 src/test/Makefile                           |   2 +-
 src/test/perl/PostgresNode.pm               | 127 +++++++++++++++++++++++++++-
 src/test/recovery/.gitignore                |   3 +
 src/test/recovery/Makefile                  |  17 ++++
 src/test/recovery/README                    |  19 +++++
 src/test/recovery/t/001_stream_rep.pl       |  62 ++++++++++++++
 src/test/recovery/t/002_archiving.pl        |  46 ++++++++++
 src/test/recovery/t/003_recovery_targets.pl | 124 +++++++++++++++++++++++++++
 src/test/recovery/t/004_timeline_switch.pl  |  67 +++++++++++++++
 src/test/recovery/t/005_replay_delay.pl     |  43 ++++++++++
 11 files changed, 506 insertions(+), 6 deletions(-)
 create mode 100644 src/test/recovery/.gitignore
 create mode 100644 src/test/recovery/Makefile
 create mode 100644 src/test/recovery/README
 create mode 100644 src/test/recovery/t/001_stream_rep.pl
 create mode 100644 src/test/recovery/t/002_archiving.pl
 create mode 100644 src/test/recovery/t/003_recovery_targets.pl
 create mode 100644 src/test/recovery/t/004_timeline_switch.pl
 create mode 100644 src/test/recovery/t/005_replay_delay.pl

diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index c1c7d1f..800b77a 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -177,7 +177,7 @@ sub promote_standby
 	# Now promote slave and insert some new data on master, this will put
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
-	system_or_bail('pg_ctl', '-w', '-D', $node_standby->data_dir, 'promote');
+	$node_standby->promote;
 	$node_standby->poll_query_until('postgres',
 		"SELECT NOT pg_is_in_recovery()")
 	  or die "Timed out while waiting for promotion of standby";
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..7f7754f 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -12,7 +12,7 @@ subdir = src/test
 top_builddir = ../..
 include $(top_builddir)/src/Makefile.global
 
-SUBDIRS = regress isolation modules
+SUBDIRS = regress isolation modules recovery
 
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 0632be2..6d37e14 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -170,6 +170,10 @@ sub init
 
 	$params{hba_permit_replication} = 1
 	  if (!defined($params{hba_permit_replication}));
+	$params{has_archiving} = 0
+	  if (!defined($params{has_archiving}));
+	$params{allows_streaming} = 0
+	  if (!defined($params{allows_streaming}));
 
 	mkdir $self->backup_dir;
 	mkdir $self->archive_dir;
@@ -182,6 +186,18 @@ sub init
 	print $conf "fsync = off\n";
 	print $conf "log_statement = all\n";
 	print $conf "port = $port\n";
+
+	if ($params{allows_streaming})
+	{
+		print $conf "wal_level = hot_standby\n";
+		print $conf "max_wal_senders = 5\n";
+		print $conf "wal_keep_segments = 20\n";
+		print $conf "max_wal_size = 128MB\n";
+		print $conf "shared_buffers = 1MB\n";
+		print $conf "wal_log_hints = on\n";
+		print $conf "hot_standby = on\n";
+	}
+
 	if ($TestLib::windows_os)
 	{
 		print $conf "listen_addresses = '$host'\n";
@@ -194,6 +210,7 @@ sub init
 	close $conf;
 
 	$self->set_replication_conf if ($params{hba_permit_replication});
+	$self->enable_archiving if ($params{has_archiving});
 }
 
 sub append_conf
@@ -218,11 +235,18 @@ sub backup
 
 sub init_from_backup
 {
-	my ($self, $root_node, $backup_name) = @_;
+	my ($self, $root_node, $backup_name, %params) = @_;
 	my $backup_path = $root_node->backup_dir . '/' . $backup_name;
 	my $port        = $self->port;
 	my $root_port   = $root_node->port;
 
+	$params{hba_permit_replication} = 1
+	   if (!defined($params{hba_permit_replication}));
+	$params{has_streaming} = 0
+	   if (!defined($params{has_streaming}));
+	$params{has_restoring} = 0
+	   if (!defined($params{has_restoring}));
+
 	print
 "Initializing node $port from backup \"$backup_name\" of node $root_port\n";
 	die "Backup $backup_path does not exist" unless -d $backup_path;
@@ -241,7 +265,10 @@ sub init_from_backup
 		qq(
 port = $port
 ));
-	$self->set_replication_conf;
+
+	$self->set_replication_conf if ($params{hba_permit_replication});
+	$self->enable_restoring($root_node) if ($params{has_restoring});
+	$self->enable_streaming($root_node) if ($params{has_streaming});
 }
 
 sub start
@@ -249,7 +276,7 @@ sub start
 	my ($self) = @_;
 	my $port   = $self->port;
 	my $pgdata = $self->data_dir;
-	print("### Starting test server in $pgdata\n");
+	print("### Starting test server in $pgdata with port $port\n");
 	my $ret = TestLib::system_log('pg_ctl', '-w', '-D', $self->data_dir, '-l',
 		$self->logfile, 'start');
 
@@ -261,7 +288,6 @@ sub start
 	}
 
 	$self->_update_pid;
-
 }
 
 sub stop
@@ -282,11 +308,104 @@ sub restart
 	my $port    = $self->port;
 	my $pgdata  = $self->data_dir;
 	my $logfile = $self->logfile;
+	print "### Restarting node in $pgdata with port $port\n";
 	TestLib::system_log('pg_ctl', '-D', $pgdata, '-w', '-l', $logfile,
 		'restart');
 	$self->_update_pid;
 }
 
+sub promote
+{
+	my ($self)  = @_;
+	my $port    = $self->port;
+	my $pgdata  = $self->data_dir;
+	my $logfile = $self->logfile;
+	print "### Promoting node in $pgdata with port $port\n";
+	TestLib::system_log('pg_ctl', '-D', $pgdata, '-w', '-l', $logfile,
+		'promote');
+}
+
+#
+# Set of routines for replication and recovery
+#
+sub enable_streaming
+{
+	my ($self, $root_node)  = @_;
+	my $root_connstr = $root_node->connstr;
+	my $applname = $self->applname;
+	my $pgdata  = $self->data_dir;
+	my $port    = $self->port;
+
+	print "### Enabling streaming replication for node in $pgdata with port $port\n";
+	$self->append_conf('recovery.conf', qq(
+primary_conninfo='$root_connstr application_name=$applname'
+standby_mode=on
+));
+}
+
+sub enable_restoring
+{
+	my ($self, $root_node)  = @_;
+	my $path = $root_node->archive_dir;
+	my $pgdata  = $self->data_dir;
+	my $port    = $self->port;
+
+	print "### Enabling restoring for node in $pgdata with port $port\n";
+
+	# Switch path to use slashes on Windows
+	my $copy_command = $TestLib::windows_os ?
+		qq{copy "$path/%f" "%p"} :
+		qq{cp $path/%f %p};
+
+	$self->append_conf('recovery.conf', qq(
+restore_command = '$copy_command'
+standby_mode = on
+));
+}
+
+sub enable_archiving
+{
+	my ($self) = @_;
+	my $path = $self->archive_dir;
+	my $pgdata  = $self->data_dir;
+	my $port    = $self->port;
+
+	print "### Enabling archiving for node in $pgdata with port $port\n";
+
+	# Switch path to use slashes on Windows
+	my $copy_command = $TestLib::windows_os ?
+		qq{copy "%p" "$path/%f"} :
+		qq{cp %p $path/%f};
+
+	# Enable archive_mode and archive_command on node
+	$self->append_conf('postgresql.conf', qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Wait until a node is able to accept queries. Useful when putting a node
+# in recovery and wait for it to be able to work particularly on slow
+# machines.
+sub wait_for_access
+{
+	my ($self) = @_;
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	while ($attempts < $max_attempts)
+	{
+		if (run_log(['pg_isready', '-d', $self->connstr('postgres')]))
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+	return 0;
+}
+
 sub _update_pid
 {
 	my $self = shift;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..f7642ce
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,62 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# Initialize master node
+my $node_master = get_new_node();
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_master->backup($backup_name);
+
+# Create streaming standby linking to master
+my $node_standby_1 = get_new_node();
+$node_standby_1->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_1->start;
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+$node_standby_1->backup($backup_name);
+
+# Create second standby node linking to standby 1
+my $node_standby_2 = get_new_node();
+$node_standby_2->init_from_backup($node_standby_1, $backup_name,
+								  has_streaming => 1);
+$node_standby_2->start;
+
+# Create some content on master and check its presence in standby 1
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a");
+
+# Wait for standbys to catch up
+my $applname_1 = $node_standby_1->applname;
+my $applname_2 = $node_standby_2->applname;
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_1';";
+$node_master->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_2';";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = $node_standby_1->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result =  $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+$node_standby_1->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+$node_standby_2->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_2->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..b2677d9
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,46 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+
+# Initialize master node, doing archives
+my $node_master = get_new_node();
+$node_master->init(has_archiving => 1,
+				   allows_streaming => 1);
+my $backup_name = 'my_backup';
+
+# Start it
+$node_master->start;
+
+# Take backup for slave
+$node_master->backup($backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $node_standby = get_new_node();
+$node_standby->init_from_backup($node_master, $backup_name,
+								has_restoring => 1);
+$node_standby->append_conf('postgresql.conf', qq(
+wal_retrieve_retry_interval = '100ms'
+));
+$node_standby->start;
+
+# Create some content on master
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $current_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Force archiving of WAL file to make it present on master
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Add some more content, it should not be present on standby
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(1000), 'check content from archives');
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..0badc93
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,124 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $node_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $node_standby = get_new_node();
+	$node_standby->init_from_backup($node_master, 'my_backup',
+									has_restoring => 1);
+
+	foreach my $param_item (@$recovery_params)
+	{
+		$node_standby->append_conf('recovery.conf',
+					   qq($param_item
+));
+	}
+
+	$node_standby->start;
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	$node_standby->poll_query_until('postgres', $caughtup_query)
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	$node_standby->teardown_node;
+}
+
+# Initialize master node
+my $node_master = get_new_node();
+$node_master->init(has_archiving => 1, allows_streaming => 1);
+
+# Start it
+$node_master->start;
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $lsn1 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Take backup from which all operations will be run
+$node_master->backup('my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+my $recovery_txid = $node_master->psql('postgres', "SELECT txid_current()");
+my $lsn2 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# More data, with recovery target timestamp
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(2001,3000))");
+my $recovery_time = $node_master->psql('postgres', "SELECT now()");
+my $lsn3 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Even more data, this time with a recovery target name
+$node_master->psql('postgres',
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))");
+my $recovery_name = "my_target";
+my $lsn4 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+$node_master->psql('postgres', "SELECT pg_create_restore_point('$recovery_name'");
+
+# Force archiving of WAL file
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', $node_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..79e80eb
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,67 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node
+my $node_master = get_new_node();
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create two standbys linking to it
+my $node_standby_1 = get_new_node();
+$node_standby_1->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_1->start;
+my $node_standby_2 = get_new_node();
+$node_standby_2->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_2->start;
+
+# Create some content on master
+$node_master->psql('postgres',
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+$node_master->teardown_node;
+$node_standby_1->promote;
+
+# Switch standby 2 to replay from standby 1
+remove_tree($node_standby_2->data_dir . '/recovery.conf');
+my $connstr_1 = $node_standby_1->connstr;
+$node_standby_2->append_conf('recovery.conf', qq(
+primary_conninfo='$connstr_1'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+$node_standby_2->restart;
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done. Standby 1 needs
+# to exit recovery first before moving on with the test.
+$node_standby_1->poll_query_until('postgres', "SELECT pg_is_in_recovery() <> true");
+$node_standby_1->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+$until_lsn = $node_standby_1->psql('postgres', "SELECT pg_current_xlog_location();");
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_2->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(2000), 'check content of standby 2');
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..14d9b29
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,43 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+# Initialize master node
+my $node_master = get_new_node();
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# And some content
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create streaming standby from backup
+my $node_standby = get_new_node();
+$node_standby->init_from_backup($node_master, $backup_name,
+								has_streaming => 1);
+$node_standby->append_conf('recovery.conf', qq(
+recovery_min_apply_delay = '2s'
+));
+$node_standby->start;
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(11,20))");
+sleep 1;
+# Here we should have only 10 rows
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+$result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(20), 'check content with delay of 2s');
-- 
2.6.3

#116Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Noah Misch (#114)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Noah Misch wrote:

On Mon, Dec 07, 2015 at 02:34:39PM +0900, Michael Paquier wrote:

I don't directly see any limitation with the use of kill on Windows..
http://perldoc.perl.org/functions/kill.html
But indeed using directly pg_ctl kill seems like a better fit for the
PG infrastructure.

From http://perldoc.perl.org/perlwin32.html, "Using signals under this port
should currently be considered unsupported." Windows applications cannot
handle SIGQUIT: https://msdn.microsoft.com/en-us/library/xdkz3x12.aspx. The
PostgreSQL backend does not generate or expect Windows signals; see its
signal.c emulation facility.

Makes sense. What we're doing now is what you suggested, so we should
be fine.

Postmaster log file names became less informative. Before the commit:
Should nodes get a name, so we instead see master_57834.log?

I am not sure that this is necessary.

In general, you'd need to cross-reference the main log file to determine which
postmaster log corresponds to which action within the test. I did plenty of
"grep $PATTERN src/bin/pg_rewind/tmp_check/log/master.log" while debugging
that test. I'd like to be able to use /*master*.log, not rely on timestamps
or on scraping regress_log_002_databases to determine which logs are master
logs. Feel free to skip this point if I'm the only one minding, though.

Since we now have the node name in the log file name, perhaps we no
longer need the port number in there. In fact, I find having the file
name change on every run (based on the port number) is slightly
annoying. I vote we change it back to using the node name without the
port number. (Also, some PostgresNode messages refer to the instance by
datadir and port number, which is unnecessary: it would be clearer to
use the name instead.)

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#117Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#116)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Dec 11, 2015 at 5:35 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Noah Misch wrote:

On Mon, Dec 07, 2015 at 02:34:39PM +0900, Michael Paquier wrote:

Postmaster log file names became less informative. Before the commit:
Should nodes get a name, so we instead see master_57834.log?

I am not sure that this is necessary.

In general, you'd need to cross-reference the main log file to determine which
postmaster log corresponds to which action within the test. I did plenty of
"grep $PATTERN src/bin/pg_rewind/tmp_check/log/master.log" while debugging
that test. I'd like to be able to use /*master*.log, not rely on timestamps
or on scraping regress_log_002_databases to determine which logs are master
logs. Feel free to skip this point if I'm the only one minding, though.

Since we now have the node name in the log file name, perhaps we no
longer need the port number in there

There is no node name in the log file name as of now, they are built
using the port number, and the information of a node is dumped into
the central log file when created (see dump_info).

In fact, I find having the file
name change on every run (based on the port number) is slightly
annoying. I vote we change it back to using the node name without the
port number. (Also, some PostgresNode messages refer to the instance by
datadir and port number, which is unnecessary: it would be clearer to
use the name instead.)

OK, so... What we have now as log file for a specific node is that:
${testname}_node_${port}.log
which is equivalent to that:
${testname}_${applname}.log

I guess that to complete your idea we could allow PostgresNode to get
a custom name for its log file through an optional parameter like
logfile => 'myname' or similar. And if nothing is defined, process
falls back to applname. So this would give the following:
${testname}_${logfile}.log

It seems that we had better keep the test name as a prefix of the log
file name though, to avoid an overlap with any other test in the same
series. Thoughts?
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#118Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#117)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

On Fri, Dec 11, 2015 at 5:35 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Since we now have the node name in the log file name, perhaps we no
longer need the port number in there

There is no node name in the log file name as of now, they are built
using the port number, and the information of a node is dumped into
the central log file when created (see dump_info).

Yeah, I realized this after posting. What I thought was the node name,
based on some of the files I had laying around, was actually the test
name.

I guess that to complete your idea we could allow PostgresNode to get
a custom name for its log file through an optional parameter like
logfile => 'myname' or similar. And if nothing is defined, process
falls back to applname. So this would give the following:
${testname}_${logfile}.log

Sure. I don't think we should the name only for the log file, though,
but also for things like the "## " informative messages we print here
and there. That would make the log file simpler to follow. Also, I'm
not sure about having it be optional. (TBH I'm not sure about applname
either; why do we keep that one?)

It seems that we had better keep the test name as a prefix of the log
file name though, to avoid an overlap with any other test in the same
series. Thoughts?

Yes, agreed on that.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#119Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#118)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Dec 11, 2015 at 8:48 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Michael Paquier wrote:

On Fri, Dec 11, 2015 at 5:35 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
I guess that to complete your idea we could allow PostgresNode to get
a custom name for its log file through an optional parameter like
logfile => 'myname' or similar. And if nothing is defined, process
falls back to applname. So this would give the following:
${testname}_${logfile}.log

Sure. I don't think we should the name only for the log file, though,
but also for things like the "## " informative messages we print here
and there. That would make the log file simpler to follow. Also, I'm
not sure about having it be optional. (TBH I'm not sure about applname
either; why do we keep that one?)

OK, so let's do this: the node name is a mandatory argument of
get_new_node, which is passed to "new PostgresNode" like the port and
the host, and it is then used in the log file name as well as in the
information messages you are mentioning. That's a patch simple enough.
Are you fine with this approach?

Regarding the application name, I still think it is useful to have it
though. pg_rewind should actually use it, and the other patch adding
the recovery routines will use it.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#120Noah Misch
noah@leadboat.com
In reply to: Michael Paquier (#119)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Dec 11, 2015 at 09:34:34PM +0900, Michael Paquier wrote:

On Fri, Dec 11, 2015 at 8:48 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Michael Paquier wrote:

On Fri, Dec 11, 2015 at 5:35 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
I guess that to complete your idea we could allow PostgresNode to get
a custom name for its log file through an optional parameter like
logfile => 'myname' or similar. And if nothing is defined, process
falls back to applname. So this would give the following:
${testname}_${logfile}.log

Sure. I don't think we should the name only for the log file, though,
but also for things like the "## " informative messages we print here
and there. That would make the log file simpler to follow. Also, I'm
not sure about having it be optional. (TBH I'm not sure about applname
either; why do we keep that one?)

OK, so let's do this: the node name is a mandatory argument of
get_new_node, which is passed to "new PostgresNode" like the port and
the host, and it is then used in the log file name as well as in the
information messages you are mentioning. That's a patch simple enough.
Are you fine with this approach?

Sounds reasonable so far.

Regarding the application name, I still think it is useful to have it
though. pg_rewind should actually use it, and the other patch adding
the recovery routines will use it.

Using the application_name connection parameter is fine, but I can't think of
a reason to set it to "node_".$node->port instead of $node->name. And I can't
think of a use for the $node->applname field once you have $node->name. What
use case would benefit?

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#121Michael Paquier
michael.paquier@gmail.com
In reply to: Noah Misch (#120)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Dec 12, 2015 at 11:37 AM, Noah Misch <noah@leadboat.com> wrote:

On Fri, Dec 11, 2015 at 09:34:34PM +0900, Michael Paquier wrote:

On Fri, Dec 11, 2015 at 8:48 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Michael Paquier wrote:

On Fri, Dec 11, 2015 at 5:35 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
I guess that to complete your idea we could allow PostgresNode to get
a custom name for its log file through an optional parameter like
logfile => 'myname' or similar. And if nothing is defined, process
falls back to applname. So this would give the following:
${testname}_${logfile}.log

Sure. I don't think we should the name only for the log file, though,
but also for things like the "## " informative messages we print here
and there. That would make the log file simpler to follow. Also, I'm
not sure about having it be optional. (TBH I'm not sure about applname
either; why do we keep that one?)

OK, so let's do this: the node name is a mandatory argument of
get_new_node, which is passed to "new PostgresNode" like the port and
the host, and it is then used in the log file name as well as in the
information messages you are mentioning. That's a patch simple enough.
Are you fine with this approach?

Sounds reasonable so far.

OK, done so.

Regarding the application name, I still think it is useful to have it
though. pg_rewind should actually use it, and the other patch adding
the recovery routines will use it.

Using the application_name connection parameter is fine, but I can't think of
a reason to set it to "node_".$node->port instead of $node->name. And I can't
think of a use for the $node->applname field once you have $node->name. What
use case would benefit?

I have the applname stuff, and updated the log messages to use the
node name for clarity.

The patch to address those points is attached.
Regards,
--
Michael

Attachments:

20151212_tap_node_name.patchbinary/octet-stream; name=20151212_tap_node_name.patchDownload
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 5991cf7..6a8d05f 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -12,7 +12,7 @@ program_options_handling_ok('pg_basebackup');
 
 my $tempdir = TestLib::tempdir;
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 
 # Initialize node without replication settings
 $node->init(hba_permit_replication => 0);
diff --git a/src/bin/pg_controldata/t/001_pg_controldata.pl b/src/bin/pg_controldata/t/001_pg_controldata.pl
index 073815a..88f8a13 100644
--- a/src/bin/pg_controldata/t/001_pg_controldata.pl
+++ b/src/bin/pg_controldata/t/001_pg_controldata.pl
@@ -11,7 +11,7 @@ command_fails(['pg_controldata'], 'pg_controldata without arguments fails');
 command_fails([ 'pg_controldata', 'nonexistent' ],
 	'pg_controldata with nonexistent directory fails');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 
 command_like([ 'pg_controldata', $node->data_dir ],
diff --git a/src/bin/pg_ctl/t/002_status.pl b/src/bin/pg_ctl/t/002_status.pl
index f1c131b..b94250a 100644
--- a/src/bin/pg_ctl/t/002_status.pl
+++ b/src/bin/pg_ctl/t/002_status.pl
@@ -11,7 +11,7 @@ my $tempdir_short = TestLib::tempdir_short;
 command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/nonexistent" ],
 	4, 'pg_ctl status with nonexistent directory');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 
 command_exit_is([ 'pg_ctl', 'status', '-D', $node->data_dir ],
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index c1c7d1f..3e43d39 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -116,7 +116,7 @@ sub setup_cluster
 {
 
 	# Initialize master, data checksums are mandatory
-	$node_master = get_new_node();
+	$node_master = get_new_node('master');
 	$node_master->init;
 
 	# Custom parameters for master's postgresql.conf
@@ -144,7 +144,7 @@ sub start_master
 
 sub create_standby
 {
-	$node_standby = get_new_node();
+	$node_standby = get_new_node('standby');
 	$node_master->backup('my_backup');
 	$node_standby->init_from_backup($node_master, 'my_backup');
 	my $connstr_master = $node_master->connstr('postgres');
diff --git a/src/bin/scripts/t/010_clusterdb.pl b/src/bin/scripts/t/010_clusterdb.pl
index 5131b35..7d78fce 100644
--- a/src/bin/scripts/t/010_clusterdb.pl
+++ b/src/bin/scripts/t/010_clusterdb.pl
@@ -9,7 +9,7 @@ program_help_ok('clusterdb');
 program_version_ok('clusterdb');
 program_options_handling_ok('clusterdb');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/011_clusterdb_all.pl b/src/bin/scripts/t/011_clusterdb_all.pl
index 15cd30c..32a81bf 100644
--- a/src/bin/scripts/t/011_clusterdb_all.pl
+++ b/src/bin/scripts/t/011_clusterdb_all.pl
@@ -5,7 +5,7 @@ use PostgresNode;
 use TestLib;
 use Test::More tests => 2;
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/020_createdb.pl b/src/bin/scripts/t/020_createdb.pl
index e0cf860..78f62ba 100644
--- a/src/bin/scripts/t/020_createdb.pl
+++ b/src/bin/scripts/t/020_createdb.pl
@@ -9,7 +9,7 @@ program_help_ok('createdb');
 program_version_ok('createdb');
 program_options_handling_ok('createdb');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/030_createlang.pl b/src/bin/scripts/t/030_createlang.pl
index 4097f03..c1701b0 100644
--- a/src/bin/scripts/t/030_createlang.pl
+++ b/src/bin/scripts/t/030_createlang.pl
@@ -9,7 +9,7 @@ program_help_ok('createlang');
 program_version_ok('createlang');
 program_options_handling_ok('createlang');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/040_createuser.pl b/src/bin/scripts/t/040_createuser.pl
index fcada63..93bad77 100644
--- a/src/bin/scripts/t/040_createuser.pl
+++ b/src/bin/scripts/t/040_createuser.pl
@@ -9,7 +9,7 @@ program_help_ok('createuser');
 program_version_ok('createuser');
 program_options_handling_ok('createuser');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/050_dropdb.pl b/src/bin/scripts/t/050_dropdb.pl
index 2adc80a..10fc9d9 100644
--- a/src/bin/scripts/t/050_dropdb.pl
+++ b/src/bin/scripts/t/050_dropdb.pl
@@ -9,7 +9,7 @@ program_help_ok('dropdb');
 program_version_ok('dropdb');
 program_options_handling_ok('dropdb');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/060_droplang.pl b/src/bin/scripts/t/060_droplang.pl
index 7228047..497ffd5 100644
--- a/src/bin/scripts/t/060_droplang.pl
+++ b/src/bin/scripts/t/060_droplang.pl
@@ -9,7 +9,7 @@ program_help_ok('droplang');
 program_version_ok('droplang');
 program_options_handling_ok('droplang');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/070_dropuser.pl b/src/bin/scripts/t/070_dropuser.pl
index 0849f77..61dd782 100644
--- a/src/bin/scripts/t/070_dropuser.pl
+++ b/src/bin/scripts/t/070_dropuser.pl
@@ -9,7 +9,7 @@ program_help_ok('dropuser');
 program_version_ok('dropuser');
 program_options_handling_ok('dropuser');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/080_pg_isready.pl b/src/bin/scripts/t/080_pg_isready.pl
index 8f3f25c..364efe3 100644
--- a/src/bin/scripts/t/080_pg_isready.pl
+++ b/src/bin/scripts/t/080_pg_isready.pl
@@ -11,7 +11,7 @@ program_options_handling_ok('pg_isready');
 
 command_fails(['pg_isready'], 'fails with no server running');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/090_reindexdb.pl b/src/bin/scripts/t/090_reindexdb.pl
index fd4eac3..e82ed77 100644
--- a/src/bin/scripts/t/090_reindexdb.pl
+++ b/src/bin/scripts/t/090_reindexdb.pl
@@ -9,7 +9,7 @@ program_help_ok('reindexdb');
 program_version_ok('reindexdb');
 program_options_handling_ok('reindexdb');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/091_reindexdb_all.pl b/src/bin/scripts/t/091_reindexdb_all.pl
index d47b18b..d067a7d 100644
--- a/src/bin/scripts/t/091_reindexdb_all.pl
+++ b/src/bin/scripts/t/091_reindexdb_all.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use Test::More tests => 2;
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 387d2b4..e248261 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -9,7 +9,7 @@ program_help_ok('vacuumdb');
 program_version_ok('vacuumdb');
 program_options_handling_ok('vacuumdb');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/101_vacuumdb_all.pl b/src/bin/scripts/t/101_vacuumdb_all.pl
index 8f1536f..4fa320b 100644
--- a/src/bin/scripts/t/101_vacuumdb_all.pl
+++ b/src/bin/scripts/t/101_vacuumdb_all.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use Test::More tests => 2;
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/102_vacuumdb_stages.pl b/src/bin/scripts/t/102_vacuumdb_stages.pl
index 4cb5b64..cd01c13 100644
--- a/src/bin/scripts/t/102_vacuumdb_stages.pl
+++ b/src/bin/scripts/t/102_vacuumdb_stages.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use Test::More tests => 4;
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 0632be2..60a2b81 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -43,6 +43,7 @@ INIT
 sub new
 {
 	my $class  = shift;
+	my $name   = shift;
 	my $pghost = shift;
 	my $pgport = shift;
 	my $testname = basename($0);
@@ -51,8 +52,8 @@ sub new
 		_port     => $pgport,
 		_host     => $pghost,
 		_basedir  => TestLib::tempdir,
-		_applname => "node_$pgport",
-		_logfile  => "$TestLib::log_path/${testname}_node_${pgport}.log" };
+		_name     => $name,
+		_logfile  => "$TestLib::log_path/${testname}_${name}.log" };
 
 	bless $self, $class;
 	$self->dump_info;
@@ -78,10 +79,10 @@ sub basedir
 	return $self->{_basedir};
 }
 
-sub applname
+sub name
 {
 	my ($self) = @_;
-	return $self->{_applname};
+	return $self->{_name};
 }
 
 sub logfile
@@ -127,11 +128,11 @@ sub backup_dir
 sub dump_info
 {
 	my ($self) = @_;
+	print "Name: " . $self->name . "\n";
 	print "Data directory: " . $self->data_dir . "\n";
 	print "Backup directory: " . $self->backup_dir . "\n";
 	print "Archive directory: " . $self->archive_dir . "\n";
 	print "Connection string: " . $self->connstr . "\n";
-	print "Application name: " . $self->applname . "\n";
 	print "Log file: " . $self->logfile . "\n";
 }
 
@@ -249,7 +250,8 @@ sub start
 	my ($self) = @_;
 	my $port   = $self->port;
 	my $pgdata = $self->data_dir;
-	print("### Starting test server in $pgdata\n");
+	my $name   = $self->name;
+	print("### Starting node \"$name\"\n");
 	my $ret = TestLib::system_log('pg_ctl', '-w', '-D', $self->data_dir, '-l',
 		$self->logfile, 'start');
 
@@ -269,8 +271,9 @@ sub stop
 	my ($self, $mode) = @_;
 	my $port   = $self->port;
 	my $pgdata = $self->data_dir;
+	my $name   = $self->name;
 	$mode = 'fast' if (!defined($mode));
-	print "### Stopping node in $pgdata with port $port using mode $mode\n";
+	print "### Stopping node \"$name\" using mode $mode\n";
 	TestLib::system_log('pg_ctl', '-D', $pgdata, '-m', $mode, 'stop');
 	$self->{_pid} = undef;
 	$self->_update_pid;
@@ -282,6 +285,8 @@ sub restart
 	my $port    = $self->port;
 	my $pgdata  = $self->data_dir;
 	my $logfile = $self->logfile;
+	my $name   = $self->name;
+	print "### Restarting node \"$name\"\n";
 	TestLib::system_log('pg_ctl', '-D', $pgdata, '-w', '-l', $logfile,
 		'restart');
 	$self->_update_pid;
@@ -316,6 +321,7 @@ sub _update_pid
 # for another node even when this one is not active.
 sub get_new_node
 {
+	my $name  = shift;
 	my $found = 0;
 	my $port  = $last_port_assigned;
 
@@ -340,7 +346,7 @@ sub get_new_node
 	print "# Found free port $port\n";
 
 	# Lock port number found by creating a new node
-	my $node = new PostgresNode($test_pghost, $port);
+	my $node = new PostgresNode($name, $test_pghost, $port);
 
 	# Add node to list of nodes
 	push(@all_nodes, $node);
@@ -354,8 +360,9 @@ sub get_new_node
 sub DESTROY
 {
 	my $self = shift;
+	my $name = $self->name;
 	return if not defined $self->{_pid};
-	print "# signalling QUIT to $self->{_pid}\n";
+	print "### Signalling QUIT for node \"$name\" to $self->{_pid}\n";
 	TestLib::system_log('pg_ctl', 'kill', 'QUIT', $self->{_pid});
 }
 
@@ -371,7 +378,8 @@ sub psql
 	my ($self, $dbname, $sql) = @_;
 
 	my ($stdout, $stderr);
-	print("# Running SQL command: $sql\n");
+	my $name = $self->name;
+	print("### Running SQL command on node \"$name\": $sql\n");
 
 	IPC::Run::run [ 'psql', '-XAtq', '-d', $self->connstr($dbname), '-f',
 		'-' ], '<', \$sql, '>', \$stdout, '2>', \$stderr
diff --git a/src/test/ssl/t/001_ssltests.pl b/src/test/ssl/t/001_ssltests.pl
index 92f16e4..d71f66b 100644
--- a/src/test/ssl/t/001_ssltests.pl
+++ b/src/test/ssl/t/001_ssltests.pl
@@ -75,7 +75,7 @@ chmod 0600, "ssl/client.key";
 #### Part 0. Set up the server.
 
 diag "setting up data directory...";
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 
 # PGHOST is enforced here to set up the node, subsequent connections
#122Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#121)
3 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Dec 12, 2015 at 8:29 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Sat, Dec 12, 2015 at 11:37 AM, Noah Misch <noah@leadboat.com> wrote:

On Fri, Dec 11, 2015 at 09:34:34PM +0900, Michael Paquier wrote:

On Fri, Dec 11, 2015 at 8:48 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Michael Paquier wrote:

On Fri, Dec 11, 2015 at 5:35 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
I guess that to complete your idea we could allow PostgresNode to get
a custom name for its log file through an optional parameter like
logfile => 'myname' or similar. And if nothing is defined, process
falls back to applname. So this would give the following:
${testname}_${logfile}.log

Sure. I don't think we should the name only for the log file, though,
but also for things like the "## " informative messages we print here
and there. That would make the log file simpler to follow. Also, I'm
not sure about having it be optional. (TBH I'm not sure about applname
either; why do we keep that one?)

OK, so let's do this: the node name is a mandatory argument of
get_new_node, which is passed to "new PostgresNode" like the port and
the host, and it is then used in the log file name as well as in the
information messages you are mentioning. That's a patch simple enough.
Are you fine with this approach?

Sounds reasonable so far.

OK, done so.

Regarding the application name, I still think it is useful to have it
though. pg_rewind should actually use it, and the other patch adding
the recovery routines will use it.

Using the application_name connection parameter is fine, but I can't think of
a reason to set it to "node_".$node->port instead of $node->name. And I can't
think of a use for the $node->applname field once you have $node->name. What
use case would benefit?

I have the applname stuff, and updated the log messages to use the
node name for clarity.

The patch to address those points is attached.

As this thread is stalling a bit, please find attached a series of
patch gathering all the pending issues for this thread:
- 0001, fix config_default.pl for MSVC builds to take into account TAP tests
- 0002, append a node name in get_new_node (per Noah's request)
- 0003, the actual recovery test suite
Hopefully this facilitates future reviews.
Regards,
--
Michael

Attachments:

0001-Fix-default-configuration-of-MSVC-builds-ignoring-TA.patchbinary/octet-stream; name=0001-Fix-default-configuration-of-MSVC-builds-ignoring-TA.patchDownload
From 7141977055cdfc1fb2f0e7a79bc9e82fb6bfbf53 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Mon, 21 Dec 2015 16:28:34 +0900
Subject: [PATCH 1/3] Fix default configuration of MSVC builds ignoring TAP
 tests

MSVC build scripts use a flag to track if TAP tests are supported or not
but this was not configured correctly. By default, like the other build
types using ./configure, this is disabled.
---
 src/tools/msvc/Solution.pm       |  1 +
 src/tools/msvc/config_default.pl | 27 ++++++++++++++-------------
 2 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index 1564b72..712f2bb 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -635,6 +635,7 @@ sub GetFakeConfigure
 	$cfg .= ' --enable-integer-datetimes'
 	  if ($self->{options}->{integer_datetimes});
 	$cfg .= ' --enable-nls' if ($self->{options}->{nls});
+	$cfg .= ' --enable-tap-tests' if ($self->{options}->{tap_tests});
 	$cfg .= ' --with-ldap'  if ($self->{options}->{ldap});
 	$cfg .= ' --without-zlib' unless ($self->{options}->{zlib});
 	$cfg .= ' --with-extra-version' if ($self->{options}->{extraver});
diff --git a/src/tools/msvc/config_default.pl b/src/tools/msvc/config_default.pl
index b9f2ff4..e50be7e 100644
--- a/src/tools/msvc/config_default.pl
+++ b/src/tools/msvc/config_default.pl
@@ -3,7 +3,7 @@ use strict;
 use warnings;
 
 our $config = {
-	asserts => 0,    # --enable-cassert
+	asserts  => 0,    # --enable-cassert
 	  # integer_datetimes=>1,   # --enable-integer-datetimes - on is now default
 	  # float4byval=>1,         # --disable-float4-byval, on by default
 
@@ -13,18 +13,19 @@ our $config = {
 	# blocksize => 8,         # --with-blocksize, 8kB by default
 	# wal_blocksize => 8,     # --with-wal-blocksize, 8kB by default
 	# wal_segsize => 16,      # --with-wal-segsize, 16MB by default
-	ldap     => 1,        # --with-ldap
-	extraver => undef,    # --with-extra-version=<string>
-	nls      => undef,    # --enable-nls=<path>
-	tcl      => undef,    # --with-tls=<path>
-	perl     => undef,    # --with-perl
-	python   => undef,    # --with-python=<path>
-	openssl  => undef,    # --with-openssl=<path>
-	uuid     => undef,    # --with-ossp-uuid
-	xml      => undef,    # --with-libxml=<path>
-	xslt     => undef,    # --with-libxslt=<path>
-	iconv    => undef,    # (not in configure, path to iconv)
-	zlib     => undef     # --with-zlib=<path>
+	ldap      => 1,        # --with-ldap
+	extraver  => undef,    # --with-extra-version=<string>
+	nls       => undef,    # --enable-nls=<path>
+	tap_tests => undef,    # --enable-tap-tests
+	tcl       => undef,    # --with-tls=<path>
+	perl      => undef,    # --with-perl
+	python    => undef,    # --with-python=<path>
+	openssl   => undef,    # --with-openssl=<path>
+	uuid      => undef,    # --with-ossp-uuid
+	xml       => undef,    # --with-libxml=<path>
+	xslt      => undef,    # --with-libxslt=<path>
+	iconv     => undef,    # (not in configure, path to iconv)
+	zlib      => undef     # --with-zlib=<path>
 };
 
 1;
-- 
2.6.4

0002-Assign-node-name-to-TAP-tests.patchbinary/octet-stream; name=0002-Assign-node-name-to-TAP-tests.patchDownload
From 7e725066de42f638ce7ef42589d27923e47449ca Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Mon, 21 Dec 2015 16:30:13 +0900
Subject: [PATCH 2/3] Assign node name to TAP tests

get_new_node gains a new mandatory parameter setting up a node name,
which is mainly used as an identifier in the log messages, and in the
log file name used per node, easing the visibility of those files for
users.

Per request from Noah Misch.
---
 src/bin/pg_basebackup/t/010_pg_basebackup.pl   |  2 +-
 src/bin/pg_controldata/t/001_pg_controldata.pl |  2 +-
 src/bin/pg_ctl/t/002_status.pl                 |  2 +-
 src/bin/pg_rewind/RewindTest.pm                |  4 ++--
 src/bin/scripts/t/010_clusterdb.pl             |  2 +-
 src/bin/scripts/t/011_clusterdb_all.pl         |  2 +-
 src/bin/scripts/t/020_createdb.pl              |  2 +-
 src/bin/scripts/t/030_createlang.pl            |  2 +-
 src/bin/scripts/t/040_createuser.pl            |  2 +-
 src/bin/scripts/t/050_dropdb.pl                |  2 +-
 src/bin/scripts/t/060_droplang.pl              |  2 +-
 src/bin/scripts/t/070_dropuser.pl              |  2 +-
 src/bin/scripts/t/080_pg_isready.pl            |  2 +-
 src/bin/scripts/t/090_reindexdb.pl             |  2 +-
 src/bin/scripts/t/091_reindexdb_all.pl         |  2 +-
 src/bin/scripts/t/100_vacuumdb.pl              |  2 +-
 src/bin/scripts/t/101_vacuumdb_all.pl          |  2 +-
 src/bin/scripts/t/102_vacuumdb_stages.pl       |  2 +-
 src/test/perl/PostgresNode.pm                  | 28 +++++++++++++++++---------
 src/test/ssl/t/001_ssltests.pl                 |  2 +-
 20 files changed, 38 insertions(+), 30 deletions(-)

diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 5991cf7..6a8d05f 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -12,7 +12,7 @@ program_options_handling_ok('pg_basebackup');
 
 my $tempdir = TestLib::tempdir;
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 
 # Initialize node without replication settings
 $node->init(hba_permit_replication => 0);
diff --git a/src/bin/pg_controldata/t/001_pg_controldata.pl b/src/bin/pg_controldata/t/001_pg_controldata.pl
index 073815a..88f8a13 100644
--- a/src/bin/pg_controldata/t/001_pg_controldata.pl
+++ b/src/bin/pg_controldata/t/001_pg_controldata.pl
@@ -11,7 +11,7 @@ command_fails(['pg_controldata'], 'pg_controldata without arguments fails');
 command_fails([ 'pg_controldata', 'nonexistent' ],
 	'pg_controldata with nonexistent directory fails');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 
 command_like([ 'pg_controldata', $node->data_dir ],
diff --git a/src/bin/pg_ctl/t/002_status.pl b/src/bin/pg_ctl/t/002_status.pl
index f1c131b..b94250a 100644
--- a/src/bin/pg_ctl/t/002_status.pl
+++ b/src/bin/pg_ctl/t/002_status.pl
@@ -11,7 +11,7 @@ my $tempdir_short = TestLib::tempdir_short;
 command_exit_is([ 'pg_ctl', 'status', '-D', "$tempdir/nonexistent" ],
 	4, 'pg_ctl status with nonexistent directory');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 
 command_exit_is([ 'pg_ctl', 'status', '-D', $node->data_dir ],
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index c1c7d1f..3e43d39 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -116,7 +116,7 @@ sub setup_cluster
 {
 
 	# Initialize master, data checksums are mandatory
-	$node_master = get_new_node();
+	$node_master = get_new_node('master');
 	$node_master->init;
 
 	# Custom parameters for master's postgresql.conf
@@ -144,7 +144,7 @@ sub start_master
 
 sub create_standby
 {
-	$node_standby = get_new_node();
+	$node_standby = get_new_node('standby');
 	$node_master->backup('my_backup');
 	$node_standby->init_from_backup($node_master, 'my_backup');
 	my $connstr_master = $node_master->connstr('postgres');
diff --git a/src/bin/scripts/t/010_clusterdb.pl b/src/bin/scripts/t/010_clusterdb.pl
index 5131b35..7d78fce 100644
--- a/src/bin/scripts/t/010_clusterdb.pl
+++ b/src/bin/scripts/t/010_clusterdb.pl
@@ -9,7 +9,7 @@ program_help_ok('clusterdb');
 program_version_ok('clusterdb');
 program_options_handling_ok('clusterdb');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/011_clusterdb_all.pl b/src/bin/scripts/t/011_clusterdb_all.pl
index 15cd30c..32a81bf 100644
--- a/src/bin/scripts/t/011_clusterdb_all.pl
+++ b/src/bin/scripts/t/011_clusterdb_all.pl
@@ -5,7 +5,7 @@ use PostgresNode;
 use TestLib;
 use Test::More tests => 2;
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/020_createdb.pl b/src/bin/scripts/t/020_createdb.pl
index e0cf860..78f62ba 100644
--- a/src/bin/scripts/t/020_createdb.pl
+++ b/src/bin/scripts/t/020_createdb.pl
@@ -9,7 +9,7 @@ program_help_ok('createdb');
 program_version_ok('createdb');
 program_options_handling_ok('createdb');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/030_createlang.pl b/src/bin/scripts/t/030_createlang.pl
index 4097f03..c1701b0 100644
--- a/src/bin/scripts/t/030_createlang.pl
+++ b/src/bin/scripts/t/030_createlang.pl
@@ -9,7 +9,7 @@ program_help_ok('createlang');
 program_version_ok('createlang');
 program_options_handling_ok('createlang');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/040_createuser.pl b/src/bin/scripts/t/040_createuser.pl
index fcada63..93bad77 100644
--- a/src/bin/scripts/t/040_createuser.pl
+++ b/src/bin/scripts/t/040_createuser.pl
@@ -9,7 +9,7 @@ program_help_ok('createuser');
 program_version_ok('createuser');
 program_options_handling_ok('createuser');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/050_dropdb.pl b/src/bin/scripts/t/050_dropdb.pl
index 2adc80a..10fc9d9 100644
--- a/src/bin/scripts/t/050_dropdb.pl
+++ b/src/bin/scripts/t/050_dropdb.pl
@@ -9,7 +9,7 @@ program_help_ok('dropdb');
 program_version_ok('dropdb');
 program_options_handling_ok('dropdb');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/060_droplang.pl b/src/bin/scripts/t/060_droplang.pl
index 7228047..497ffd5 100644
--- a/src/bin/scripts/t/060_droplang.pl
+++ b/src/bin/scripts/t/060_droplang.pl
@@ -9,7 +9,7 @@ program_help_ok('droplang');
 program_version_ok('droplang');
 program_options_handling_ok('droplang');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/070_dropuser.pl b/src/bin/scripts/t/070_dropuser.pl
index 0849f77..61dd782 100644
--- a/src/bin/scripts/t/070_dropuser.pl
+++ b/src/bin/scripts/t/070_dropuser.pl
@@ -9,7 +9,7 @@ program_help_ok('dropuser');
 program_version_ok('dropuser');
 program_options_handling_ok('dropuser');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/080_pg_isready.pl b/src/bin/scripts/t/080_pg_isready.pl
index 8f3f25c..364efe3 100644
--- a/src/bin/scripts/t/080_pg_isready.pl
+++ b/src/bin/scripts/t/080_pg_isready.pl
@@ -11,7 +11,7 @@ program_options_handling_ok('pg_isready');
 
 command_fails(['pg_isready'], 'fails with no server running');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/090_reindexdb.pl b/src/bin/scripts/t/090_reindexdb.pl
index fd4eac3..e82ed77 100644
--- a/src/bin/scripts/t/090_reindexdb.pl
+++ b/src/bin/scripts/t/090_reindexdb.pl
@@ -9,7 +9,7 @@ program_help_ok('reindexdb');
 program_version_ok('reindexdb');
 program_options_handling_ok('reindexdb');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/091_reindexdb_all.pl b/src/bin/scripts/t/091_reindexdb_all.pl
index d47b18b..d067a7d 100644
--- a/src/bin/scripts/t/091_reindexdb_all.pl
+++ b/src/bin/scripts/t/091_reindexdb_all.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use Test::More tests => 2;
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 387d2b4..e248261 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -9,7 +9,7 @@ program_help_ok('vacuumdb');
 program_version_ok('vacuumdb');
 program_options_handling_ok('vacuumdb');
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/101_vacuumdb_all.pl b/src/bin/scripts/t/101_vacuumdb_all.pl
index 8f1536f..4fa320b 100644
--- a/src/bin/scripts/t/101_vacuumdb_all.pl
+++ b/src/bin/scripts/t/101_vacuumdb_all.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use Test::More tests => 2;
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/bin/scripts/t/102_vacuumdb_stages.pl b/src/bin/scripts/t/102_vacuumdb_stages.pl
index 4cb5b64..cd01c13 100644
--- a/src/bin/scripts/t/102_vacuumdb_stages.pl
+++ b/src/bin/scripts/t/102_vacuumdb_stages.pl
@@ -4,7 +4,7 @@ use warnings;
 use PostgresNode;
 use Test::More tests => 4;
 
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 $node->start;
 
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 0632be2..60a2b81 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -43,6 +43,7 @@ INIT
 sub new
 {
 	my $class  = shift;
+	my $name   = shift;
 	my $pghost = shift;
 	my $pgport = shift;
 	my $testname = basename($0);
@@ -51,8 +52,8 @@ sub new
 		_port     => $pgport,
 		_host     => $pghost,
 		_basedir  => TestLib::tempdir,
-		_applname => "node_$pgport",
-		_logfile  => "$TestLib::log_path/${testname}_node_${pgport}.log" };
+		_name     => $name,
+		_logfile  => "$TestLib::log_path/${testname}_${name}.log" };
 
 	bless $self, $class;
 	$self->dump_info;
@@ -78,10 +79,10 @@ sub basedir
 	return $self->{_basedir};
 }
 
-sub applname
+sub name
 {
 	my ($self) = @_;
-	return $self->{_applname};
+	return $self->{_name};
 }
 
 sub logfile
@@ -127,11 +128,11 @@ sub backup_dir
 sub dump_info
 {
 	my ($self) = @_;
+	print "Name: " . $self->name . "\n";
 	print "Data directory: " . $self->data_dir . "\n";
 	print "Backup directory: " . $self->backup_dir . "\n";
 	print "Archive directory: " . $self->archive_dir . "\n";
 	print "Connection string: " . $self->connstr . "\n";
-	print "Application name: " . $self->applname . "\n";
 	print "Log file: " . $self->logfile . "\n";
 }
 
@@ -249,7 +250,8 @@ sub start
 	my ($self) = @_;
 	my $port   = $self->port;
 	my $pgdata = $self->data_dir;
-	print("### Starting test server in $pgdata\n");
+	my $name   = $self->name;
+	print("### Starting node \"$name\"\n");
 	my $ret = TestLib::system_log('pg_ctl', '-w', '-D', $self->data_dir, '-l',
 		$self->logfile, 'start');
 
@@ -269,8 +271,9 @@ sub stop
 	my ($self, $mode) = @_;
 	my $port   = $self->port;
 	my $pgdata = $self->data_dir;
+	my $name   = $self->name;
 	$mode = 'fast' if (!defined($mode));
-	print "### Stopping node in $pgdata with port $port using mode $mode\n";
+	print "### Stopping node \"$name\" using mode $mode\n";
 	TestLib::system_log('pg_ctl', '-D', $pgdata, '-m', $mode, 'stop');
 	$self->{_pid} = undef;
 	$self->_update_pid;
@@ -282,6 +285,8 @@ sub restart
 	my $port    = $self->port;
 	my $pgdata  = $self->data_dir;
 	my $logfile = $self->logfile;
+	my $name   = $self->name;
+	print "### Restarting node \"$name\"\n";
 	TestLib::system_log('pg_ctl', '-D', $pgdata, '-w', '-l', $logfile,
 		'restart');
 	$self->_update_pid;
@@ -316,6 +321,7 @@ sub _update_pid
 # for another node even when this one is not active.
 sub get_new_node
 {
+	my $name  = shift;
 	my $found = 0;
 	my $port  = $last_port_assigned;
 
@@ -340,7 +346,7 @@ sub get_new_node
 	print "# Found free port $port\n";
 
 	# Lock port number found by creating a new node
-	my $node = new PostgresNode($test_pghost, $port);
+	my $node = new PostgresNode($name, $test_pghost, $port);
 
 	# Add node to list of nodes
 	push(@all_nodes, $node);
@@ -354,8 +360,9 @@ sub get_new_node
 sub DESTROY
 {
 	my $self = shift;
+	my $name = $self->name;
 	return if not defined $self->{_pid};
-	print "# signalling QUIT to $self->{_pid}\n";
+	print "### Signalling QUIT for node \"$name\" to $self->{_pid}\n";
 	TestLib::system_log('pg_ctl', 'kill', 'QUIT', $self->{_pid});
 }
 
@@ -371,7 +378,8 @@ sub psql
 	my ($self, $dbname, $sql) = @_;
 
 	my ($stdout, $stderr);
-	print("# Running SQL command: $sql\n");
+	my $name = $self->name;
+	print("### Running SQL command on node \"$name\": $sql\n");
 
 	IPC::Run::run [ 'psql', '-XAtq', '-d', $self->connstr($dbname), '-f',
 		'-' ], '<', \$sql, '>', \$stdout, '2>', \$stderr
diff --git a/src/test/ssl/t/001_ssltests.pl b/src/test/ssl/t/001_ssltests.pl
index 92f16e4..d71f66b 100644
--- a/src/test/ssl/t/001_ssltests.pl
+++ b/src/test/ssl/t/001_ssltests.pl
@@ -75,7 +75,7 @@ chmod 0600, "ssl/client.key";
 #### Part 0. Set up the server.
 
 diag "setting up data directory...";
-my $node = get_new_node();
+my $node = get_new_node('master');
 $node->init;
 
 # PGHOST is enforced here to set up the node, subsequent connections
-- 
2.6.4

0003-Add-recovery-test-suite.patchbinary/octet-stream; name=0003-Add-recovery-test-suite.patchDownload
From 4a31143cb8dc348d4f8f915a76e5698b185bca14 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Mon, 21 Dec 2015 16:44:20 +0900
Subject: [PATCH 3/3] Add recovery test suite

This includes basic tests maipulating standbys, be they archiving or
streaming nodes, and some basic sanity checks around them. PostgresNode
is extended with a couple of routines allowing to set up WAL archiving,
WAL streaming or WAL restore on a node, as well as a commodity routine
to allow a promotion.
---
 src/bin/pg_rewind/RewindTest.pm             |   2 +-
 src/test/Makefile                           |   2 +-
 src/test/perl/PostgresNode.pm               | 125 +++++++++++++++++++++++++++-
 src/test/recovery/.gitignore                |   3 +
 src/test/recovery/Makefile                  |  17 ++++
 src/test/recovery/README                    |  19 +++++
 src/test/recovery/t/001_stream_rep.pl       |  62 ++++++++++++++
 src/test/recovery/t/002_archiving.pl        |  46 ++++++++++
 src/test/recovery/t/003_recovery_targets.pl | 125 ++++++++++++++++++++++++++++
 src/test/recovery/t/004_timeline_switch.pl  |  67 +++++++++++++++
 src/test/recovery/t/005_replay_delay.pl     |  43 ++++++++++
 11 files changed, 506 insertions(+), 5 deletions(-)
 create mode 100644 src/test/recovery/.gitignore
 create mode 100644 src/test/recovery/Makefile
 create mode 100644 src/test/recovery/README
 create mode 100644 src/test/recovery/t/001_stream_rep.pl
 create mode 100644 src/test/recovery/t/002_archiving.pl
 create mode 100644 src/test/recovery/t/003_recovery_targets.pl
 create mode 100644 src/test/recovery/t/004_timeline_switch.pl
 create mode 100644 src/test/recovery/t/005_replay_delay.pl

diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index 3e43d39..73ea203 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -177,7 +177,7 @@ sub promote_standby
 	# Now promote slave and insert some new data on master, this will put
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
-	system_or_bail('pg_ctl', '-w', '-D', $node_standby->data_dir, 'promote');
+	$node_standby->promote;
 	$node_standby->poll_query_until('postgres',
 		"SELECT NOT pg_is_in_recovery()")
 	  or die "Timed out while waiting for promotion of standby";
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..7f7754f 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -12,7 +12,7 @@ subdir = src/test
 top_builddir = ../..
 include $(top_builddir)/src/Makefile.global
 
-SUBDIRS = regress isolation modules
+SUBDIRS = regress isolation modules recovery
 
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 60a2b81..465b9f5 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -171,6 +171,10 @@ sub init
 
 	$params{hba_permit_replication} = 1
 	  if (!defined($params{hba_permit_replication}));
+	$params{has_archiving} = 0
+	  if (!defined($params{has_archiving}));
+	$params{allows_streaming} = 0
+	  if (!defined($params{allows_streaming}));
 
 	mkdir $self->backup_dir;
 	mkdir $self->archive_dir;
@@ -183,6 +187,18 @@ sub init
 	print $conf "fsync = off\n";
 	print $conf "log_statement = all\n";
 	print $conf "port = $port\n";
+
+	if ($params{allows_streaming})
+	{
+		print $conf "wal_level = hot_standby\n";
+		print $conf "max_wal_senders = 5\n";
+		print $conf "wal_keep_segments = 20\n";
+		print $conf "max_wal_size = 128MB\n";
+		print $conf "shared_buffers = 1MB\n";
+		print $conf "wal_log_hints = on\n";
+		print $conf "hot_standby = on\n";
+	}
+
 	if ($TestLib::windows_os)
 	{
 		print $conf "listen_addresses = '$host'\n";
@@ -195,6 +211,7 @@ sub init
 	close $conf;
 
 	$self->set_replication_conf if ($params{hba_permit_replication});
+	$self->enable_archiving if ($params{has_archiving});
 }
 
 sub append_conf
@@ -219,11 +236,18 @@ sub backup
 
 sub init_from_backup
 {
-	my ($self, $root_node, $backup_name) = @_;
+	my ($self, $root_node, $backup_name, %params) = @_;
 	my $backup_path = $root_node->backup_dir . '/' . $backup_name;
 	my $port        = $self->port;
 	my $root_port   = $root_node->port;
 
+	$params{hba_permit_replication} = 1
+	   if (!defined($params{hba_permit_replication}));
+	$params{has_streaming} = 0
+	   if (!defined($params{has_streaming}));
+	$params{has_restoring} = 0
+	   if (!defined($params{has_restoring}));
+
 	print
 "Initializing node $port from backup \"$backup_name\" of node $root_port\n";
 	die "Backup $backup_path does not exist" unless -d $backup_path;
@@ -242,7 +266,10 @@ sub init_from_backup
 		qq(
 port = $port
 ));
-	$self->set_replication_conf;
+
+	$self->set_replication_conf if ($params{hba_permit_replication});
+	$self->enable_restoring($root_node) if ($params{has_restoring});
+	$self->enable_streaming($root_node) if ($params{has_streaming});
 }
 
 sub start
@@ -263,7 +290,6 @@ sub start
 	}
 
 	$self->_update_pid;
-
 }
 
 sub stop
@@ -292,6 +318,99 @@ sub restart
 	$self->_update_pid;
 }
 
+sub promote
+{
+	my ($self)  = @_;
+	my $port    = $self->port;
+	my $pgdata  = $self->data_dir;
+	my $logfile = $self->logfile;
+	my $name    = $self->name;
+	print "### Promoting node \"$name\"\n";
+	TestLib::system_log('pg_ctl', '-D', $pgdata, '-w', '-l', $logfile,
+		'promote');
+}
+
+#
+# Set of routines for replication and recovery
+#
+sub enable_streaming
+{
+	my ($self, $root_node)  = @_;
+	my $root_connstr = $root_node->connstr;
+	my $name = $self->name;
+	my $pgdata  = $self->data_dir;
+	my $port    = $self->port;
+
+	print "### Enabling streaming replication for node in $pgdata with port $port\n";
+	$self->append_conf('recovery.conf', qq(
+primary_conninfo='$root_connstr application_name=$name'
+standby_mode=on
+));
+}
+
+sub enable_restoring
+{
+	my ($self, $root_node)  = @_;
+	my $path = $root_node->archive_dir;
+	my $pgdata  = $self->data_dir;
+	my $port    = $self->port;
+
+	print "### Enabling restoring for node in $pgdata with port $port\n";
+
+	# Switch path to use slashes on Windows
+	my $copy_command = $TestLib::windows_os ?
+		qq{copy "$path/%f" "%p"} :
+		qq{cp $path/%f %p};
+
+	$self->append_conf('recovery.conf', qq(
+restore_command = '$copy_command'
+standby_mode = on
+));
+}
+
+sub enable_archiving
+{
+	my ($self) = @_;
+	my $path = $self->archive_dir;
+	my $pgdata  = $self->data_dir;
+	my $port    = $self->port;
+
+	print "### Enabling archiving for node in $pgdata with port $port\n";
+
+	# Switch path to use slashes on Windows
+	my $copy_command = $TestLib::windows_os ?
+		qq{copy "%p" "$path/%f"} :
+		qq{cp %p $path/%f};
+
+	# Enable archive_mode and archive_command on node
+	$self->append_conf('postgresql.conf', qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Wait until a node is able to accept queries. Useful when putting a node
+# in recovery and wait for it to be able to work particularly on slow
+# machines.
+sub wait_for_access
+{
+	my ($self) = @_;
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	while ($attempts < $max_attempts)
+	{
+		if (run_log(['pg_isready', '-d', $self->connstr('postgres')]))
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+	return 0;
+}
+
 sub _update_pid
 {
 	my $self = shift;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..3ed9be3
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,62 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_master->backup($backup_name);
+
+# Create streaming standby linking to master
+my $node_standby_1 = get_new_node('standby_1');
+$node_standby_1->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_1->start;
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+$node_standby_1->backup($backup_name);
+
+# Create second standby node linking to standby 1
+my $node_standby_2 = get_new_node('standby_2');
+$node_standby_2->init_from_backup($node_standby_1, $backup_name,
+								  has_streaming => 1);
+$node_standby_2->start;
+
+# Create some content on master and check its presence in standby 1
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a");
+
+# Wait for standbys to catch up
+my $applname_1 = $node_standby_1->name;
+my $applname_2 = $node_standby_2->name;
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_1';";
+$node_master->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_2';";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = $node_standby_1->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result =  $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+$node_standby_1->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+$node_standby_2->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_2->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..930125c
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,46 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+
+# Initialize master node, doing archives
+my $node_master = get_new_node('master');
+$node_master->init(has_archiving => 1,
+				   allows_streaming => 1);
+my $backup_name = 'my_backup';
+
+# Start it
+$node_master->start;
+
+# Take backup for slave
+$node_master->backup($backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $node_standby = get_new_node('standby');
+$node_standby->init_from_backup($node_master, $backup_name,
+								has_restoring => 1);
+$node_standby->append_conf('postgresql.conf', qq(
+wal_retrieve_retry_interval = '100ms'
+));
+$node_standby->start;
+
+# Create some content on master
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $current_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Force archiving of WAL file to make it present on master
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Add some more content, it should not be present on standby
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(1000), 'check content from archives');
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..293603a
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,125 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $node_name = shift;
+	my $node_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $node_standby = get_new_node($node_name);
+	$node_standby->init_from_backup($node_master, 'my_backup',
+									has_restoring => 1);
+
+	foreach my $param_item (@$recovery_params)
+	{
+		$node_standby->append_conf('recovery.conf',
+					   qq($param_item
+));
+	}
+
+	$node_standby->start;
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	$node_standby->poll_query_until('postgres', $caughtup_query)
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	$node_standby->teardown_node;
+}
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(has_archiving => 1, allows_streaming => 1);
+
+# Start it
+$node_master->start;
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $lsn1 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Take backup from which all operations will be run
+$node_master->backup('my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+my $recovery_txid = $node_master->psql('postgres', "SELECT txid_current()");
+my $lsn2 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# More data, with recovery target timestamp
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(2001,3000))");
+my $recovery_time = $node_master->psql('postgres', "SELECT now()");
+my $lsn3 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Even more data, this time with a recovery target name
+$node_master->psql('postgres',
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))");
+my $recovery_name = "my_target";
+my $lsn4 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+$node_master->psql('postgres', "SELECT pg_create_restore_point('$recovery_name'");
+
+# Force archiving of WAL file
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', 'standby_1', $node_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', 'standby_2', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', 'standby_3', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', 'standby_4', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', 'standby_5', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', 'standby_6', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', 'standby_7', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..c58c602
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,67 @@
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create two standbys linking to it
+my $node_standby_1 = get_new_node('standby_1');
+$node_standby_1->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_1->start;
+my $node_standby_2 = get_new_node('standby_2');
+$node_standby_2->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_2->start;
+
+# Create some content on master
+$node_master->psql('postgres',
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+$node_master->teardown_node;
+$node_standby_1->promote;
+
+# Switch standby 2 to replay from standby 1
+remove_tree($node_standby_2->data_dir . '/recovery.conf');
+my $connstr_1 = $node_standby_1->connstr;
+$node_standby_2->append_conf('recovery.conf', qq(
+primary_conninfo='$connstr_1'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+$node_standby_2->restart;
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done. Standby 1 needs
+# to exit recovery first before moving on with the test.
+$node_standby_1->poll_query_until('postgres', "SELECT pg_is_in_recovery() <> true");
+$node_standby_1->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+$until_lsn = $node_standby_1->psql('postgres', "SELECT pg_current_xlog_location();");
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_2->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(2000), 'check content of standby 2');
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..14d9b29
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,43 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+# Initialize master node
+my $node_master = get_new_node();
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# And some content
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create streaming standby from backup
+my $node_standby = get_new_node();
+$node_standby->init_from_backup($node_master, $backup_name,
+								has_streaming => 1);
+$node_standby->append_conf('recovery.conf', qq(
+recovery_min_apply_delay = '2s'
+));
+$node_standby->start;
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(11,20))");
+sleep 1;
+# Here we should have only 10 rows
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+$result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(20), 'check content with delay of 2s');
-- 
2.6.4

#123Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#122)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Mon, Dec 21, 2015 at 4:45 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

As this thread is stalling a bit, please find attached a series of
patch gathering all the pending issues for this thread:
- 0001, fix config_default.pl for MSVC builds to take into account TAP tests
- 0002, append a node name in get_new_node (per Noah's request)
- 0003, the actual recovery test suite
Hopefully this facilitates future reviews.

Patch 2 has been pushed as c8642d9 (thanks Alvaro). The remaining two
patches still apply and pass cleanly.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#124Stas Kelvich
s.kelvich@postgrespro.ru
In reply to: Michael Paquier (#123)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Hi.

I’ve looked over proposed patch and migrated my shell tests scripts that i’ve used for testing twophase commits on master/slave to this test framework. Everything looks mature, and I didn’t encountered any problems with writing new tests using this infrastructure.

From my point of view I don’t see any problems to commit this patches in their current state.

Also some things that came into mind about test suite:

0) There are several routines that does actual checking, like is/command_ok/command_fails. I think it will be very handy to have wrappers psql_ok/psql_fails that calls psql through the command_ok/fails.

1) Better to raise more meaningful error when IPC::Run is absend.

2) --enable-tap-tests deserves mention in test/recovery/README and more obvious error message when one trying to run make check in test/recovery without --enable-tap-tests.

3) Is it hard to give ability to run TAP tests in extensions?

4) It will be handy if make check will write path to log files in case of failed test.

5) psql() accepts database name as a first argument, but everywhere in tests it is ‘postgres’. Isn’t it simpler to store dbname in connstr, and have separate function to change database?

6) Clean logs on prove restart? Clean up tmp installations?

7) Make check sets PGPORT PG_REGRESS for prove. Is it necessary?

On 22 Jan 2016, at 09:17, Michael Paquier <michael.paquier@gmail.com> wrote:

On Mon, Dec 21, 2015 at 4:45 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

As this thread is stalling a bit, please find attached a series of
patch gathering all the pending issues for this thread:
- 0001, fix config_default.pl for MSVC builds to take into account TAP tests
- 0002, append a node name in get_new_node (per Noah's request)
- 0003, the actual recovery test suite
Hopefully this facilitates future reviews.

Patch 2 has been pushed as c8642d9 (thanks Alvaro). The remaining two
patches still apply and pass cleanly.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Stas Kelvich
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#125Kyotaro HORIGUCHI
horiguchi.kyotaro@lab.ntt.co.jp
In reply to: Michael Paquier (#123)
1 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Hello, I'm studying this.

Two hunks in 0003 needed a fix but the other part applied cleanly
on master.

At Fri, 22 Jan 2016 15:17:51 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqTTAtVCEXAoyMtF4Xu9g=mXY4cjnP=+hy7jgYfnFzM=JA@mail.gmail.com>

On Mon, Dec 21, 2015 at 4:45 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

As this thread is stalling a bit, please find attached a series of
patch gathering all the pending issues for this thread:
- 0001, fix config_default.pl for MSVC builds to take into account TAP tests
- 0002, append a node name in get_new_node (per Noah's request)
- 0003, the actual recovery test suite
Hopefully this facilitates future reviews.

Patch 2 has been pushed as c8642d9 (thanks Alvaro). The remaining two
patches still apply and pass cleanly.

The TAP test framework doesn't remove existing temporary
directories when a test script suite (or a prove?) starts, and it
in turn removes all temprorary directories it has made even if
ended with fairures. It would be sometimes inconvenient to find
the cause of the failures and inconsistent with the behavior of
the ordinary(?) make check, as far as my understanding goes.

tmp_check is left remained but it would be ok to preserve logs,
which is located in tmp_check differently than the ordinary
regressions.

One annoyance is the name of data directories is totally
meaningless. We cannot investigate them even if it is left
behind.

Addition to them, maybe it is useful that a test script can get
stderr content from PostgresNode->psql(). Setting
client_min_messages lower can give a plenty of useful information
about how server is working.

So, I'd like to propose four (or five) changes to this harness.

- prove_check to remove all in tmp_check

- TestLib to preserve temporary directories/files if the current
test fails.

- PostgresNode::get_new_node to create data directory with
meaningful basenames.

- PostgresNode::psql to return a list of ($stdout, $stderr) if
requested. (The previous behavior is not changed)

- (recovery/t/00x_* gives test number to node name)

As a POC, the attached diff will appliy on the 0001 and (fixed)
0003 patches.

It might be good to give test number to the name of temp dirs by
any automated way, but it is not included in it.

Opinions? Thoughts?

regards,

--
Kyotaro Horiguchi
NTT Open Source Software Center

Attachments:

20160204_prove_recovery_kh.difftext/x-patch; charset=us-asciiDownload
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 4602e3e..75ddcaf 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -51,7 +51,7 @@ sub new
 	my $self   = {
 		_port     => $pgport,
 		_host     => $pghost,
-		_basedir  => TestLib::tempdir,
+		_basedir  => TestLib::tempdir($name),
 		_name     => $name,
 		_logfile  => "$TestLib::log_path/${testname}_${name}.log" };
 
@@ -513,10 +513,16 @@ sub psql
 		print "#### Begin standard error\n";
 		print $stderr;
 		print "#### End standard error\n";
+		if (wantarray)
+		{
+			chomp $stderr;
+			$stderr =~ s/\r//g if $Config{osname} eq 'msys';
+		}
 	}
 	chomp $stdout;
 	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-	return $stdout;
+
+	return wantarray ? ($stdout, $stderr) : $stdout;
 }
 
 # Run a query once a second, until it returns 't' (i.e. SQL boolean true).
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 3d11cbb..841dc06 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -107,13 +107,24 @@ INIT
 	autoflush TESTLOG 1;
 }
 
+END
+{
+	# Preserve temporary directory for this test if failure
+	if (!Test::More->builder->is_passing)
+	{
+		$File::Temp::KEEP_ALL = 1;
+	}
+}
+
 #
 # Helper functions
 #
 sub tempdir
 {
+	my ($prefix) = @_;
+	$prefix = "tmp_test" if (!$prefix);
 	return File::Temp::tempdir(
-		'tmp_testXXXX',
+		$prefix.'_XXXX',
 		DIR => $tmp_check,
 		CLEANUP => 1);
 }
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index 3ed9be3..8a00d00 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -6,16 +6,18 @@ use TestLib;
 use Test::More tests => 4;
 
 # Initialize master node
-my $node_master = get_new_node('master');
+my $node_master = get_new_node('001_master');
 $node_master->init(allows_streaming => 1);
 $node_master->start;
 my $backup_name = 'my_backup';
 
+$File::Temp::KEEP_ALL = 0;
+
 # Take backup
 $node_master->backup($backup_name);
 
 # Create streaming standby linking to master
-my $node_standby_1 = get_new_node('standby_1');
+my $node_standby_1 = get_new_node('001_standby_1');
 $node_standby_1->init_from_backup($node_master, $backup_name,
 								  has_streaming => 1);
 $node_standby_1->start;
@@ -25,7 +27,7 @@ $node_standby_1->start;
 $node_standby_1->backup($backup_name);
 
 # Create second standby node linking to standby 1
-my $node_standby_2 = get_new_node('standby_2');
+my $node_standby_2 = get_new_node('001_standby_2');
 $node_standby_2->init_from_backup($node_standby_1, $backup_name,
 								  has_streaming => 1);
 $node_standby_2->start;
@@ -43,11 +45,11 @@ $caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM p
 $node_standby_1->poll_query_until('postgres', $caughtup_query)
 	or die "Timed out while waiting for standby 2 to catch up";
 
-my $result = $node_standby_1->psql('postgres', "SELECT count(*) FROM tab_int");
+my $result = $node_standby_1->psql('postgres', "set client_min_messages to debug5;SELECT count(*) FROM tab_int");
 print "standby 1: $result\n";
-is($result, qq(1002), 'check streamed content on standby 1');
+isnt($result[0], qq(1002), 'check streamed content on standby 1');
 
-$result =  $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+my $result =  $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
 print "standby 2: $result\n";
 is($result, qq(1002), 'check streamed content on standby 2');
 
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
index 930125c..bb620c6 100644
--- a/src/test/recovery/t/002_archiving.pl
+++ b/src/test/recovery/t/002_archiving.pl
@@ -7,7 +7,7 @@ use Test::More tests => 1;
 use File::Copy;
 
 # Initialize master node, doing archives
-my $node_master = get_new_node('master');
+my $node_master = get_new_node('002_master');
 $node_master->init(has_archiving => 1,
 				   allows_streaming => 1);
 my $backup_name = 'my_backup';
@@ -19,7 +19,7 @@ $node_master->start;
 $node_master->backup($backup_name);
 
 # Initialize standby node from backup, fetching WAL from archives
-my $node_standby = get_new_node('standby');
+my $node_standby = get_new_node('002_standby');
 $node_standby->init_from_backup($node_master, $backup_name,
 								has_restoring => 1);
 $node_standby->append_conf('postgresql.conf', qq(
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
index 293603a..4d59277 100644
--- a/src/test/recovery/t/003_recovery_targets.pl
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -16,7 +16,7 @@ sub test_recovery_standby
 	my $num_rows = shift;
 	my $until_lsn = shift;
 
-	my $node_standby = get_new_node($node_name);
+	my $node_standby = get_new_node("003_".$node_name);
 	$node_standby->init_from_backup($node_master, 'my_backup',
 									has_restoring => 1);
 
@@ -43,7 +43,7 @@ sub test_recovery_standby
 }
 
 # Initialize master node
-my $node_master = get_new_node('master');
+my $node_master = get_new_node('003_master');
 $node_master->init(has_archiving => 1, allows_streaming => 1);
 
 # Start it
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
index c58c602..0a2362b 100644
--- a/src/test/recovery/t/004_timeline_switch.pl
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -11,7 +11,7 @@ use Test::More tests => 1;
 $ENV{PGDATABASE} = 'postgres';
 
 # Initialize master node
-my $node_master = get_new_node('master');
+my $node_master = get_new_node('004_master');
 $node_master->init(allows_streaming => 1);
 $node_master->start;
 
@@ -20,11 +20,11 @@ my $backup_name = 'my_backup';
 $node_master->backup($backup_name);
 
 # Create two standbys linking to it
-my $node_standby_1 = get_new_node('standby_1');
+my $node_standby_1 = get_new_node('004_standby_1');
 $node_standby_1->init_from_backup($node_master, $backup_name,
 								  has_streaming => 1);
 $node_standby_1->start;
-my $node_standby_2 = get_new_node('standby_2');
+my $node_standby_2 = get_new_node('004_standby_2');
 $node_standby_2->init_from_backup($node_master, $backup_name,
 								  has_streaming => 1);
 $node_standby_2->start;
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
index 14d9b29..ae209e4 100644
--- a/src/test/recovery/t/005_replay_delay.pl
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -6,7 +6,7 @@ use TestLib;
 use Test::More tests => 2;
 
 # Initialize master node
-my $node_master = get_new_node();
+my $node_master = get_new_node('005_master');
 $node_master->init(allows_streaming => 1);
 $node_master->start;
 
@@ -18,7 +18,7 @@ my $backup_name = 'my_backup';
 $node_master->backup($backup_name);
 
 # Create streaming standby from backup
-my $node_standby = get_new_node();
+my $node_standby = get_new_node('005_standby');
 $node_standby->init_from_backup($node_master, $backup_name,
 								has_streaming => 1);
 $node_standby->append_conf('recovery.conf', qq(
#126Michael Paquier
michael.paquier@gmail.com
In reply to: Stas Kelvich (#124)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Feb 4, 2016 at 5:23 PM, Stas Kelvich <s.kelvich@postgrespro.ru> wrote:

(Please do not top-post, this breaks the thread flow.)

I’ve looked over proposed patch and migrated my shell tests scripts that i’ve used for testing twophase commits on master/slave to this test framework. Everything looks mature, and I didn’t encountered any problems with writing new tests using this infrastructure.
From my point of view I don’t see any problems to commit this patches in their current state.

Thanks for the review!

0) There are several routines that does actual checking, like is/command_ok/command_fails. I think it will be very handy to have wrappers psql_ok/psql_fails that calls psql through the command_ok/fails.

Do you have a test case in mind for it?

1) Better to raise more meaningful error when IPC::Run is absent.

This has been discussed before, and as far as I recall the current
behavior has been concluded as being fine. That's where
--enable-tap-tests becomes meaningful.

2) --enable-tap-tests deserves mention in test/recovery/README and more obvious error message when one trying to run make check in test/recovery without --enable-tap-tests.

When running without --enable-tap-tests from src/test/recovery you
would get the following error per how prove_check is defined:
"TAP tests not enabled"

3) Is it hard to give ability to run TAP tests in extensions?

Not really. You would need to enforce a check rule or similar. For the
recovery test suite I have mapped the check rule with prove_check.

4) It will be handy if make check will write path to log files in case of failed test.

Hm, perhaps. The log files are hardcoded in log/, so it is not like we
don't know it. That's an argument for the main TAP suite though, not
really this series of patch.

5) psql() accepts database name as a first argument, but everywhere in tests it is ‘postgres’. Isn’t it simpler to store dbname in connstr, and have separate function to change database?
6) Clean logs on prove restart? Clean up tmp installations?

Those are issues proper to the main TAP infrastructure, though I agree
that we could improve things here, particularly for temporary
installations that get automatically... Hm... Cleaned up should a test
failure happen?

7) Make check sets PGPORT PG_REGRESS for prove. Is it necessary?

No, that's not needed (I think I noticed that at some point) and
that's a bug. We could live without setting it.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#127Victor Wagner
vitus@wagner.pp.ru
In reply to: Michael Paquier (#126)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, 4 Feb 2016 12:59:03 +0300
Michael Paquier <michael.paquier@gmail.com> wrote:

for it?

1) Better to raise more meaningful error when IPC::Run is absent.

This has been discussed before, and as far as I recall the current
behavior has been concluded as being fine. That's where
--enable-tap-tests becomes meaningful.

Really, it is not so hard to add configure checks for perl modules.
And we need to test not only for IPC::Run, but for Test::More too,
because some Linux distributions put modules which come with perl into
separate package.

The only problem that most m4 files with tests for perl modules, which
can be found in the Internet, have GPL license, so someone have either
to write his own and publish under PostgreSQL license or contact
author of one of them and ask to publish it under PostgreSQL license.

First seems to be much easier.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#128Michael Paquier
michael.paquier@gmail.com
In reply to: Victor Wagner (#127)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Feb 4, 2016 at 4:43 PM, Victor Wagner <vitus@wagner.pp.ru> wrote:

On Thu, 4 Feb 2016 12:59:03 +0300
Michael Paquier <michael.paquier@gmail.com> wrote:

1) Better to raise more meaningful error when IPC::Run is absent.

This has been discussed before, and as far as I recall the current
behavior has been concluded as being fine. That's where
--enable-tap-tests becomes meaningful.

Really, it is not so hard to add configure checks for perl modules.
And we need to test not only for IPC::Run, but for Test::More too,
because some Linux distributions put modules which come with perl into
separate package.

The last time we discussed about that on this list we concluded that
it was not really necessary to have such checks, for one it makes the
code more simple, and because this is leveraged by the presence of
--enable-tap-tests, tests which can get actually costly with
check-world. But this is digressing the subject of this thread, which
deals with the fact of having recovery tests integrated in core...
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#129Victor Wagner
vitus@wagner.pp.ru
In reply to: Michael Paquier (#128)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, 4 Feb 2016 18:33:27 +0300
Michael Paquier <michael.paquier@gmail.com> wrote:

Really, it is not so hard to add configure checks for perl modules.
And we need to test not only for IPC::Run, but for Test::More too,
because some Linux distributions put modules which come with perl
into separate package.

The last time we discussed about that on this list we concluded that
it was not really necessary to have such checks, for one it makes the
code more simple, and because this is leveraged by the presence of
--enable-tap-tests, tests which can get actually costly with
check-world. But this is digressing the subject of this thread, which
deals with the fact of having recovery tests integrated in core...

Of course, such configure tests should be run only if
--enable-tap-tests is passed to the configure script

It would look like

if test "$enable_tap_tests" = "yes"; then
AX_PROG_PERL_MODULES( Test::More, , AC_MSG_ERROR([Test::More is
necessary to run TAP Tests])
AX_PROG_PERL_MODULES( IPC::Run, , AC_MSG_ERROR([IPC::Run is
necessary to run TAP Tests])
fi

in the configure.in

May be it is not strictly necessary, but it is really useful to see
such problems as clear error message during configure stage, rather
than successfully configure, compile, run tests and only then find out,
that something is forgotten.

I don't see why having such tests in the configure.in, makes code more
complex. It just prevents configure to finish successfully if
--enable-tap-tests is specified and required modules are not available.

--
Victor Wagner <vitus@wagner.pp.ru>

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#130Victor Wagner
vitus@wagner.pp.ru
In reply to: Michael Paquier (#123)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

This patch adds a long-awaited functionality to the PostgreSQL test
suite - testing of cluster configuration.

It contains bare minimum of replication and recovery test, but it should
be a good starting point for other people.

Really, adding a much more tests for replication and recovery
is problematic, because these tests are resource-hungry, and take big
enough time even on relatively powerful machines, but it seems to be
necessary, because they need to create several temporary installation.

So, set of tests, included into this patch is reasonably good choice.

I think that readability of tests can be improved a bit, because these
tests would serve as an example for all tap test writers.

It's quite good that patch sets standard of using 'use strict; use
warnings;' in the test script.

It is bad, that Postgres-specific perl modules do not have embedded
documentation. It would be nice to see POD documentation in the
TestLib.pm and PostgresNode.pm instead of just comments. It would be
much easier to test writers to read documentation using perldoc utility,
rather than browse through the code.

I'll second Stas' suggestion about psql_ok/psql_fail functions.

1. psql_ok instead of just psql would provide visual feedback for the
reader of code. One would see 'here condition is tested, here is
something ended with _ok/_fail'.

It would be nice that seeing say "use Test::More tests => 4"
one can immediately see "Yes, there is three _ok's and one _fail in the
script'

2. I have use case for psql_fail code. In my libpq failover patch there
is number of cases, where it should be tested that connection is not
established,

But this is rather about further evolution of the tap test library, not
about this set of tests.

I think that this patch should be commited as soon as possible in its
current form (short of already reported reject in the PostgresNode.pm
init function).

--
Victor Wagner <vitus@wagner.pp.ru>

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#131Michael Paquier
michael.paquier@gmail.com
In reply to: Victor Wagner (#130)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Feb 4, 2016 at 9:18 PM, Victor Wagner wrote:

It's quite good that patch sets standard of using 'use strict; use
warnings;' in the test script.

FWIW, this is decided as being a standard rule for any modules/script
added in the main tree.

It is bad, that Postgres-specific perl modules do not have embedded
documentation. It would be nice to see POD documentation in the
TestLib.pm and PostgresNode.pm instead of just comments. It would be
much easier to test writers to read documentation using perldoc utility,
rather than browse through the code.

Why not. I am no perlist but those prove to be helpful, however those
Postgres modules are not dedicated to a large audience, so we could
live without for now.

I think that this patch should be commited as soon as possible in its
current form (short of already reported reject in the PostgresNode.pm
init function).

Thanks for your enthusiasm. Now, to do an auto-critic of my patch:

+       if ($params{allows_streaming})
+       {
+               print $conf "wal_level = hot_standby\n";
+               print $conf "max_wal_senders = 5\n";
+               print $conf "wal_keep_segments = 20\n";
+               print $conf "max_wal_size = 128MB\n";
+               print $conf "shared_buffers = 1MB\n";
+               print $conf "wal_log_hints = on\n";
+               print $conf "hot_standby = on\n";
+       }
This could have more thoughts, particularly for wal_log_hints which is
not used all the time, I think that we'd actually want to complete
that with an optional hash of parameter/values that get appended at
the end of the configuration file, then pass wal_log_hints in the
tests where it is needed. The default set of parameter is maybe fine
if done this way, still wal_keep_segments could be removed.
+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
Two typos in two lines.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#132Noah Misch
noah@leadboat.com
In reply to: Victor Wagner (#129)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Feb 04, 2016 at 09:13:46PM +0300, Victor Wagner wrote:

On Thu, 4 Feb 2016 18:33:27 +0300 Michael Paquier <michael.paquier@gmail.com> wrote:

Really, it is not so hard to add configure checks for perl modules.
And we need to test not only for IPC::Run, but for Test::More too,
because some Linux distributions put modules which come with perl
into separate package.

The last time we discussed about that on this list we concluded that
it was not really necessary to have such checks, for one it makes the
code more simple, and because this is leveraged by the presence of
--enable-tap-tests, tests which can get actually costly with
check-world. But this is digressing the subject of this thread, which
deals with the fact of having recovery tests integrated in core...

Of course, such configure tests should be run only if
--enable-tap-tests is passed to the configure script

It would look like

if test "$enable_tap_tests" = "yes"; then
AX_PROG_PERL_MODULES( Test::More, , AC_MSG_ERROR([Test::More is
necessary to run TAP Tests])
AX_PROG_PERL_MODULES( IPC::Run, , AC_MSG_ERROR([IPC::Run is
necessary to run TAP Tests])
fi

in the configure.in

May be it is not strictly necessary, but it is really useful to see
such problems as clear error message during configure stage, rather
than successfully configure, compile, run tests and only then find out,
that something is forgotten.

I don't see why having such tests in the configure.in, makes code more
complex. It just prevents configure to finish successfully if
--enable-tap-tests is specified and required modules are not available.

Even if detecting missing modules at "configure" time is the right thing, it
belongs in a distinct patch, discussed on a distinct thread. The absence of
IPC::Run affects the proposed replication tests in the same way it affects
current TAP suites, so this thread has no business revisiting it.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#133Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#91)
Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Thu, Feb 4, 2016 at 11:58 PM, Stas Kelvich <s.kelvich@postgrespro.ru> wrote:

On 04 Feb 2016, at 12:59, Michael Paquier <michael.paquier@gmail.com> wrote:

0) There are several routines that does actual checking, like is/command_ok/command_fails. I think it will be very handy to have wrappers psql_ok/psql_fails that calls psql through the command_ok/fails.

Do you have a test case in mind for it?

Yes, I’ve used that to test prepare/commit while recovery (script attached, it’s in WIP state, i’ll submit that later along with other twophase stuff).

Oh, OK, now I see. Well it seems to make sense for your case, though
it does not seem to be directly linked to the patch here. We could
incrementally add something on top of the existing infrastructure that
gets into the code tree once the 2PC patch gets in a more advanced
shape.

2) --enable-tap-tests deserves mention in test/recovery/README and more obvious error message when one trying to run make check in test/recovery without --enable-tap-tests.

When running without --enable-tap-tests from src/test/recovery you
would get the following error per how prove_check is defined:
"TAP tests not enabled"

Yes, but that message doesn’t mention --enable-tap-tests and README also silent about that too. I didn’t know about that flag and had to search in makefiles for this error message to see what conditions leads to it. I think we can save planet from one more stackoverflow question if the error message will mention that flag.

Well, that works for the whole TAP test infrastructure and not really
this patch only. Let's not forget that the goal of this thread is to
provide a basic set of tests and routines to help people building test
cases for more advanced clustering scenarios, so I'd rather not
complicate the code with side things and remain focused on the core
problem.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#134Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#131)
3 attachment(s)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Feb 5, 2016 at 4:17 AM, Michael Paquier wrote:

Thanks for your enthusiasm. Now, to do an auto-critic of my patch:
+       if ($params{allows_streaming})
+       {
+               print $conf "wal_level = hot_standby\n";
+               print $conf "max_wal_senders = 5\n";
+               print $conf "wal_keep_segments = 20\n";
+               print $conf "max_wal_size = 128MB\n";
+               print $conf "shared_buffers = 1MB\n";
+               print $conf "wal_log_hints = on\n";
+               print $conf "hot_standby = on\n";
+       }
This could have more thoughts, particularly for wal_log_hints which is
not used all the time, I think that we'd actually want to complete
that with an optional hash of parameter/values that get appended at
the end of the configuration file, then pass wal_log_hints in the
tests where it is needed. The default set of parameter is maybe fine
if done this way, still wal_keep_segments could be removed.

At the end I have refrained from doing that, and refactoring
setup_cluster@RewindTest.pm to use the new option allows_streaming,
simplifying a bit the code. The introduction of allows_streaming could
be done in a separate patch, though it did not seem worth the
complication when hacking at that. The split is simple, though.

+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
Two typos in two lines.

Fixed.

I also found an issue with the use of application_name causing test
001 to fail, bug squashed on the way. The generation of paths for
archive_command and restore_command was incorrect on Windows. Those
need to use two backslashes (to detect correctly files) and need to be
double-quoted (to avoid errors with command copy should a space be
included in the path). I have added as well a new subcommand in
vcregress.pl called recoverycheck where one can run the recovery test
suite on Windows using MSVC.

Attached are rebased patches, split into 3 parts doing the following:
- 0001, fix default configuration of MSVC builds ignoring TAP tests
- 0002, add a promote routine in PostgresNode.pm. pg_rewind's tests
can make immediate use of that.
- 0003, the actual test suite.
This is registered in CF 2016-03 as well for further consideration.
--
Michael

Attachments:

0001-Fix-default-configuration-of-MSVC-builds-ignoring-TA.patchtext/x-patch; charset=US-ASCII; name=0001-Fix-default-configuration-of-MSVC-builds-ignoring-TA.patchDownload
From 0231866094b793ccd7c6941a19d5565c3d5811b6 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Mon, 21 Dec 2015 16:28:34 +0900
Subject: [PATCH 1/3] Fix default configuration of MSVC builds ignoring TAP
 tests

MSVC build scripts use a flag to track if TAP tests are supported or not
but this was not configured correctly. By default, like the other build
types using ./configure, this is disabled.
---
 src/tools/msvc/Solution.pm       |  1 +
 src/tools/msvc/config_default.pl | 27 ++++++++++++++-------------
 2 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index ac116b7..c5a43f9 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -643,6 +643,7 @@ sub GetFakeConfigure
 	$cfg .= ' --enable-integer-datetimes'
 	  if ($self->{options}->{integer_datetimes});
 	$cfg .= ' --enable-nls' if ($self->{options}->{nls});
+	$cfg .= ' --enable-tap-tests' if ($self->{options}->{tap_tests});
 	$cfg .= ' --with-ldap'  if ($self->{options}->{ldap});
 	$cfg .= ' --without-zlib' unless ($self->{options}->{zlib});
 	$cfg .= ' --with-extra-version' if ($self->{options}->{extraver});
diff --git a/src/tools/msvc/config_default.pl b/src/tools/msvc/config_default.pl
index b9f2ff4..e50be7e 100644
--- a/src/tools/msvc/config_default.pl
+++ b/src/tools/msvc/config_default.pl
@@ -3,7 +3,7 @@ use strict;
 use warnings;
 
 our $config = {
-	asserts => 0,    # --enable-cassert
+	asserts  => 0,    # --enable-cassert
 	  # integer_datetimes=>1,   # --enable-integer-datetimes - on is now default
 	  # float4byval=>1,         # --disable-float4-byval, on by default
 
@@ -13,18 +13,19 @@ our $config = {
 	# blocksize => 8,         # --with-blocksize, 8kB by default
 	# wal_blocksize => 8,     # --with-wal-blocksize, 8kB by default
 	# wal_segsize => 16,      # --with-wal-segsize, 16MB by default
-	ldap     => 1,        # --with-ldap
-	extraver => undef,    # --with-extra-version=<string>
-	nls      => undef,    # --enable-nls=<path>
-	tcl      => undef,    # --with-tls=<path>
-	perl     => undef,    # --with-perl
-	python   => undef,    # --with-python=<path>
-	openssl  => undef,    # --with-openssl=<path>
-	uuid     => undef,    # --with-ossp-uuid
-	xml      => undef,    # --with-libxml=<path>
-	xslt     => undef,    # --with-libxslt=<path>
-	iconv    => undef,    # (not in configure, path to iconv)
-	zlib     => undef     # --with-zlib=<path>
+	ldap      => 1,        # --with-ldap
+	extraver  => undef,    # --with-extra-version=<string>
+	nls       => undef,    # --enable-nls=<path>
+	tap_tests => undef,    # --enable-tap-tests
+	tcl       => undef,    # --with-tls=<path>
+	perl      => undef,    # --with-perl
+	python    => undef,    # --with-python=<path>
+	openssl   => undef,    # --with-openssl=<path>
+	uuid      => undef,    # --with-ossp-uuid
+	xml       => undef,    # --with-libxml=<path>
+	xslt      => undef,    # --with-libxslt=<path>
+	iconv     => undef,    # (not in configure, path to iconv)
+	zlib      => undef     # --with-zlib=<path>
 };
 
 1;
-- 
2.7.1

0002-Add-routine-for-node-promotion-in-PostgresNode.patchtext/x-patch; charset=US-ASCII; name=0002-Add-routine-for-node-promotion-in-PostgresNode.patchDownload
From f866f5a211ba1fc3c34b4272c9b752ced90684ae Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Wed, 17 Feb 2016 15:21:32 +0900
Subject: [PATCH 2/3] Add routine for node promotion in PostgresNode

This is useful for tests to trigger a promotion on a node via pg_ctl,
and pg_rewind can make immediate use of it.
---
 src/bin/pg_rewind/RewindTest.pm |  2 +-
 src/test/perl/PostgresNode.pm   | 12 ++++++++++++
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index 68834cd..e204830 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -177,7 +177,7 @@ sub promote_standby
 	# Now promote slave and insert some new data on master, this will put
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
-	system_or_bail('pg_ctl', '-w', '-D', $node_standby->data_dir, 'promote');
+	$node_standby->promote;
 	$node_standby->poll_query_until('postgres',
 		"SELECT NOT pg_is_in_recovery()")
 	  or die "Timed out while waiting for promotion of standby";
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 2ab9aee..5a57e31 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -294,6 +294,18 @@ sub restart
 	$self->_update_pid;
 }
 
+sub promote
+{
+	my ($self)  = @_;
+	my $port    = $self->port;
+	my $pgdata  = $self->data_dir;
+	my $logfile = $self->logfile;
+	my $name    = $self->name;
+	print "### Promoting node \"$name\"\n";
+	TestLib::system_log('pg_ctl', '-D', $pgdata, '-w', '-l', $logfile,
+						'promote');
+}
+
 sub _update_pid
 {
 	my $self = shift;
-- 
2.7.1

0003-Add-recovery-test-suite.patchtext/x-patch; charset=US-ASCII; name=0003-Add-recovery-test-suite.patchDownload
From 5959031af49436262788fede65c3fcb60fb343b7 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Wed, 17 Feb 2016 21:50:19 +0900
Subject: [PATCH 3/3] Add recovery test suite

This includes basic tests maipulating standbys, be they archiving or
streaming nodes, and some basic sanity checks around them. PostgresNode
is extended with a couple of routines allowing to set up WAL archiving,
WAL streaming or WAL restore on a node, as well as a commodity routine
to allow a promotion.
---
 doc/src/sgml/install-windows.sgml           |   4 +-
 src/bin/pg_rewind/RewindTest.pm             |  17 +---
 src/test/Makefile                           |   2 +-
 src/test/perl/PostgresNode.pm               | 126 +++++++++++++++++++++++++++-
 src/test/recovery/.gitignore                |   3 +
 src/test/recovery/Makefile                  |  17 ++++
 src/test/recovery/README                    |  19 +++++
 src/test/recovery/t/001_stream_rep.pl       |  62 ++++++++++++++
 src/test/recovery/t/002_archiving.pl        |  46 ++++++++++
 src/test/recovery/t/003_recovery_targets.pl | 125 +++++++++++++++++++++++++++
 src/test/recovery/t/004_timeline_switch.pl  |  67 +++++++++++++++
 src/test/recovery/t/005_replay_delay.pl     |  43 ++++++++++
 src/tools/msvc/config_default.pl            |   2 +-
 src/tools/msvc/vcregress.pl                 |  13 ++-
 14 files changed, 524 insertions(+), 22 deletions(-)
 create mode 100644 src/test/recovery/.gitignore
 create mode 100644 src/test/recovery/Makefile
 create mode 100644 src/test/recovery/README
 create mode 100644 src/test/recovery/t/001_stream_rep.pl
 create mode 100644 src/test/recovery/t/002_archiving.pl
 create mode 100644 src/test/recovery/t/003_recovery_targets.pl
 create mode 100644 src/test/recovery/t/004_timeline_switch.pl
 create mode 100644 src/test/recovery/t/005_replay_delay.pl

diff --git a/doc/src/sgml/install-windows.sgml b/doc/src/sgml/install-windows.sgml
index ba60a6b..8e87b70 100644
--- a/doc/src/sgml/install-windows.sgml
+++ b/doc/src/sgml/install-windows.sgml
@@ -440,6 +440,7 @@ $ENV{CONFIG}="Debug";
 <userinput>vcregress ecpgcheck</userinput>
 <userinput>vcregress isolationcheck</userinput>
 <userinput>vcregress bincheck</userinput>
+<userinput>vcregress recoverycheck</userinput>
 <userinput>vcregress upgradecheck</userinput>
 </screen>
 
@@ -455,7 +456,8 @@ $ENV{CONFIG}="Debug";
 
   <para>
    Running the regression tests on client programs, with "vcregress bincheck",
-   requires an additional Perl module to be installed:
+   or on recovery tests, with "vceregress recoverycheck" requires an additional
+   Perl module to be installed:
    <variablelist>
     <varlistentry>
      <term><productname>IPC::Run</productname></term>
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index e204830..bda0516 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -114,24 +114,9 @@ sub check_query
 
 sub setup_cluster
 {
-
 	# Initialize master, data checksums are mandatory
 	$node_master = get_new_node('master');
-	$node_master->init;
-
-	# Custom parameters for master's postgresql.conf
-	$node_master->append_conf(
-		"postgresql.conf", qq(
-wal_level = hot_standby
-max_wal_senders = 2
-wal_keep_segments = 20
-max_wal_size = 200MB
-shared_buffers = 1MB
-wal_log_hints = on
-hot_standby = on
-autovacuum = off
-max_connections = 10
-));
+	$node_master->init(allows_streaming => 1);
 }
 
 sub start_master
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..7f7754f 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -12,7 +12,7 @@ subdir = src/test
 top_builddir = ../..
 include $(top_builddir)/src/Makefile.global
 
-SUBDIRS = regress isolation modules
+SUBDIRS = regress isolation modules recovery
 
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 5a57e31..82a0d30 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -171,6 +171,10 @@ sub init
 
 	$params{hba_permit_replication} = 1
 	  if (!defined($params{hba_permit_replication}));
+	$params{has_archiving} = 0
+	  if (!defined($params{has_archiving}));
+	$params{allows_streaming} = 0
+	  if (!defined($params{allows_streaming}));
 
 	mkdir $self->backup_dir;
 	mkdir $self->archive_dir;
@@ -183,6 +187,20 @@ sub init
 	print $conf "fsync = off\n";
 	print $conf "log_statement = all\n";
 	print $conf "port = $port\n";
+
+	if ($params{allows_streaming})
+	{
+		print $conf "wal_level = hot_standby\n";
+		print $conf "max_wal_senders = 5\n";
+		print $conf "wal_keep_segments = 20\n";
+		print $conf "max_wal_size = 128MB\n";
+		print $conf "shared_buffers = 1MB\n";
+		print $conf "autovacuum = off\n";
+		print $conf "wal_log_hints = on\n";
+		print $conf "hot_standby = on\n";
+		print $conf "max_connections = 10\n";
+	}
+
 	if ($TestLib::windows_os)
 	{
 		print $conf "listen_addresses = '$host'\n";
@@ -195,6 +213,7 @@ sub init
 	close $conf;
 
 	$self->set_replication_conf if ($params{hba_permit_replication});
+	$self->enable_archiving if ($params{has_archiving});
 }
 
 sub append_conf
@@ -220,12 +239,19 @@ sub backup
 
 sub init_from_backup
 {
-	my ($self, $root_node, $backup_name) = @_;
+	my ($self, $root_node, $backup_name, %params) = @_;
 	my $backup_path = $root_node->backup_dir . '/' . $backup_name;
 	my $port        = $self->port;
 	my $node_name   = $self->name;
 	my $root_name   = $root_node->name;
 
+	$params{hba_permit_replication} = 1
+	   if (!defined($params{hba_permit_replication}));
+	$params{has_streaming} = 0
+	   if (!defined($params{has_streaming}));
+	$params{has_restoring} = 0
+	   if (!defined($params{has_restoring}));
+
 	print
 "# Initializing node \"$node_name\" from backup \"$backup_name\" of node \"$root_name\"\n";
 	die "Backup \"$backup_name\" does not exist at $backup_path"
@@ -245,7 +271,10 @@ sub init_from_backup
 		qq(
 port = $port
 ));
-	$self->set_replication_conf;
+
+	$self->set_replication_conf if ($params{hba_permit_replication});
+	$self->enable_restoring($root_node) if ($params{has_restoring});
+	$self->enable_streaming($root_node) if ($params{has_streaming});
 }
 
 sub start
@@ -306,6 +335,99 @@ sub promote
 						'promote');
 }
 
+#
+# Set of routines for replication and recovery
+#
+sub enable_streaming
+{
+	my ($self, $root_node)  = @_;
+	my $root_connstr = $root_node->connstr;
+	my $name    = $self->name;
+	my $pgdata  = $self->data_dir;
+	my $port    = $self->port;
+
+	print "### Enabling streaming replication for node in $pgdata with port $port\n";
+	$self->append_conf('recovery.conf', qq(
+primary_conninfo='$root_connstr application_name=$name'
+standby_mode=on
+));
+}
+
+sub enable_restoring
+{
+	my ($self, $root_node)  = @_;
+	my $path = $root_node->archive_dir;
+	my $pgdata  = $self->data_dir;
+	my $port    = $self->port;
+
+	print "### Enabling restoring for node in $pgdata with port $port\n";
+
+	# On Windows, the path specified in the restore command needs to use
+	# double back-slashes to work properly and to be able to detect properly
+	# the file targetted by the copy command, so the directory value used
+	# in this routine, using only one back-slash, need to be properly changed
+	# first. Paths also need to be double-quoted to prevent failures where
+	# the path contains spaces.
+	$path =~ s{\\}{\\\\}g if ($TestLib::windows_os);
+	my $copy_command = $TestLib::windows_os ?
+		qq{copy "$path\\\\%f" "%p"} :
+		qq{cp $path/%f %p};
+
+	$self->append_conf('recovery.conf', qq(
+restore_command = '$copy_command'
+standby_mode = on
+));
+}
+
+sub enable_archiving
+{
+	my ($self) = @_;
+	my $path = $self->archive_dir;
+	my $pgdata  = $self->data_dir;
+	my $port    = $self->port;
+
+	print "### Enabling archiving for node in $pgdata with port $port\n";
+
+	# On Windows, the path specified in the restore command needs to use
+	# double back-slashes to work properly and to be able to detect properly
+	# the file targetted by the copy command, so the directory value used
+	# in this routine, using only one back-slash, need to be properly changed
+	# first. Paths also need to be double-quoted to prevent failures where
+	# the path contains spaces.
+	$path =~ s{\\}{\\\\}g if ($TestLib::windows_os);
+	my $copy_command = $TestLib::windows_os ?
+		qq{copy "%p" "$path\\\\%f"} :
+		qq{cp %p $path/%f};
+
+	# Enable archive_mode and archive_command on node
+	$self->append_conf('postgresql.conf', qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
+# Wait until a node is able to accept queries. Useful when putting a node
+# in recovery and wait for it to be able to work particularly on slow
+# machines.
+sub wait_for_access
+{
+	my ($self) = @_;
+	my $max_attempts = 30;
+	my $attempts     = 0;
+	while ($attempts < $max_attempts)
+	{
+		if (run_log(['pg_isready', '-d', $self->connstr('postgres')]))
+		{
+			return 1;
+		}
+
+		# Wait a second before retrying.
+		sleep 1;
+		$attempts++;
+	}
+	return 0;
+}
+
 sub _update_pid
 {
 	my $self = shift;
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..3ed9be3
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,62 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_master->backup($backup_name);
+
+# Create streaming standby linking to master
+my $node_standby_1 = get_new_node('standby_1');
+$node_standby_1->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_1->start;
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+$node_standby_1->backup($backup_name);
+
+# Create second standby node linking to standby 1
+my $node_standby_2 = get_new_node('standby_2');
+$node_standby_2->init_from_backup($node_standby_1, $backup_name,
+								  has_streaming => 1);
+$node_standby_2->start;
+
+# Create some content on master and check its presence in standby 1
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a");
+
+# Wait for standbys to catch up
+my $applname_1 = $node_standby_1->name;
+my $applname_2 = $node_standby_2->name;
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_1';";
+$node_master->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_2';";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = $node_standby_1->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result =  $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+$node_standby_1->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+$node_standby_2->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_2->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..930125c
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,46 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+
+# Initialize master node, doing archives
+my $node_master = get_new_node('master');
+$node_master->init(has_archiving => 1,
+				   allows_streaming => 1);
+my $backup_name = 'my_backup';
+
+# Start it
+$node_master->start;
+
+# Take backup for slave
+$node_master->backup($backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $node_standby = get_new_node('standby');
+$node_standby->init_from_backup($node_master, $backup_name,
+								has_restoring => 1);
+$node_standby->append_conf('postgresql.conf', qq(
+wal_retrieve_retry_interval = '100ms'
+));
+$node_standby->start;
+
+# Create some content on master
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $current_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Force archiving of WAL file to make it present on master
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Add some more content, it should not be present on standby
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(1000), 'check content from archives');
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..293603a
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,125 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $node_name = shift;
+	my $node_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $node_standby = get_new_node($node_name);
+	$node_standby->init_from_backup($node_master, 'my_backup',
+									has_restoring => 1);
+
+	foreach my $param_item (@$recovery_params)
+	{
+		$node_standby->append_conf('recovery.conf',
+					   qq($param_item
+));
+	}
+
+	$node_standby->start;
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	$node_standby->poll_query_until('postgres', $caughtup_query)
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	$node_standby->teardown_node;
+}
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(has_archiving => 1, allows_streaming => 1);
+
+# Start it
+$node_master->start;
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $lsn1 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Take backup from which all operations will be run
+$node_master->backup('my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+my $recovery_txid = $node_master->psql('postgres', "SELECT txid_current()");
+my $lsn2 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# More data, with recovery target timestamp
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(2001,3000))");
+my $recovery_time = $node_master->psql('postgres', "SELECT now()");
+my $lsn3 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Even more data, this time with a recovery target name
+$node_master->psql('postgres',
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))");
+my $recovery_name = "my_target";
+my $lsn4 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+$node_master->psql('postgres', "SELECT pg_create_restore_point('$recovery_name'");
+
+# Force archiving of WAL file
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', 'standby_1', $node_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', 'standby_2', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', 'standby_3', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', 'standby_4', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', 'standby_5', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', 'standby_6', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', 'standby_7', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..dc08ec1
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,67 @@
+# Test for timeline switch
+# Ensure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create two standbys linking to it
+my $node_standby_1 = get_new_node('standby_1');
+$node_standby_1->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_1->start;
+my $node_standby_2 = get_new_node('standby_2');
+$node_standby_2->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_2->start;
+
+# Create some content on master
+$node_master->psql('postgres',
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+$node_master->teardown_node;
+$node_standby_1->promote;
+
+# Switch standby 2 to replay from standby 1
+remove_tree($node_standby_2->data_dir . '/recovery.conf');
+my $connstr_1 = $node_standby_1->connstr;
+$node_standby_2->append_conf('recovery.conf', qq(
+primary_conninfo='$connstr_1'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+$node_standby_2->restart;
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done. Standby 1 needs
+# to exit recovery first before moving on with the test.
+$node_standby_1->poll_query_until('postgres', "SELECT pg_is_in_recovery() <> true");
+$node_standby_1->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+$until_lsn = $node_standby_1->psql('postgres', "SELECT pg_current_xlog_location();");
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_2->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(2000), 'check content of standby 2');
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..2d8c690
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,43 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# And some content
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create streaming standby from backup
+my $node_standby = get_new_node('standby');
+$node_standby->init_from_backup($node_master, $backup_name,
+								has_streaming => 1);
+$node_standby->append_conf('recovery.conf', qq(
+recovery_min_apply_delay = '2s'
+));
+$node_standby->start;
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(11,20))");
+sleep 1;
+# Here we should have only 10 rows
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+$result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(20), 'check content with delay of 2s');
diff --git a/src/tools/msvc/config_default.pl b/src/tools/msvc/config_default.pl
index e50be7e..636beea 100644
--- a/src/tools/msvc/config_default.pl
+++ b/src/tools/msvc/config_default.pl
@@ -16,7 +16,7 @@ our $config = {
 	ldap      => 1,        # --with-ldap
 	extraver  => undef,    # --with-extra-version=<string>
 	nls       => undef,    # --enable-nls=<path>
-	tap_tests => undef,    # --enable-tap-tests
+	tap_tests => 1,    # --enable-tap-tests
 	tcl       => undef,    # --with-tls=<path>
 	perl      => undef,    # --with-perl
 	python    => undef,    # --with-python=<path>
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index df1348b..3d14544 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -34,7 +34,7 @@ if (-e "src/tools/msvc/buildenv.pl")
 
 my $what = shift || "";
 if ($what =~
-/^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck)$/i
+/^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck|recoverycheck)$/i
   )
 {
 	$what = uc $what;
@@ -89,6 +89,7 @@ my %command = (
 	MODULESCHECK   => \&modulescheck,
 	ISOLATIONCHECK => \&isolationcheck,
 	BINCHECK       => \&bincheck,
+	RECOVERYCHECK  => \&recoverycheck,
 	UPGRADECHECK   => \&upgradecheck,);
 
 my $proc = $command{$what};
@@ -360,6 +361,16 @@ sub modulescheck
 	exit $mstat if $mstat;
 }
 
+sub recoverycheck
+{
+	InstallTemp();
+
+	my $mstat = 0;
+	my $dir = "$topdir/src/test/recovery";
+	my $status = tap_check($dir);
+	exit $status if $status;
+}
+
 # Run "initdb", then reconfigure authentication.
 sub standard_initdb
 {
-- 
2.7.1

#135Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#134)
3 attachment(s)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On Wed, Feb 17, 2016 at 9:52 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Fri, Feb 5, 2016 at 4:17 AM, Michael Paquier wrote:

Thanks for your enthusiasm. Now, to do an auto-critic of my patch:
+       if ($params{allows_streaming})
+       {
+               print $conf "wal_level = hot_standby\n";
+               print $conf "max_wal_senders = 5\n";
+               print $conf "wal_keep_segments = 20\n";
+               print $conf "max_wal_size = 128MB\n";
+               print $conf "shared_buffers = 1MB\n";
+               print $conf "wal_log_hints = on\n";
+               print $conf "hot_standby = on\n";
+       }
This could have more thoughts, particularly for wal_log_hints which is
not used all the time, I think that we'd actually want to complete
that with an optional hash of parameter/values that get appended at
the end of the configuration file, then pass wal_log_hints in the
tests where it is needed. The default set of parameter is maybe fine
if done this way, still wal_keep_segments could be removed.

At the end I have refrained from doing that, and refactoring
setup_cluster@RewindTest.pm to use the new option allows_streaming,
simplifying a bit the code. The introduction of allows_streaming could
be done in a separate patch, though it did not seem worth the
complication when hacking at that. The split is simple, though.

+# Tets for timeline switch
+# Encure that a standby is able to follow a newly-promoted standby
Two typos in two lines.

Fixed.

I also found an issue with the use of application_name causing test
001 to fail, bug squashed on the way. The generation of paths for
archive_command and restore_command was incorrect on Windows. Those
need to use two backslashes (to detect correctly files) and need to be
double-quoted (to avoid errors with command copy should a space be
included in the path). I have added as well a new subcommand in
vcregress.pl called recoverycheck where one can run the recovery test
suite on Windows using MSVC.

Attached are rebased patches, split into 3 parts doing the following:
- 0001, fix default configuration of MSVC builds ignoring TAP tests
- 0002, add a promote routine in PostgresNode.pm. pg_rewind's tests
can make immediate use of that.
- 0003, the actual test suite.
This is registered in CF 2016-03 as well for further consideration.

Here is a rebased set after the conflicts created by e640093, with the
following changes:
- In 0002, added perldoc for new promote routine
- In 0003, added perldoc documentation for the new options introduced
in init and init_from_backup, and fixed some log entries not using the
node name to identify the node involved when enabling archive,
streaming or recovery.
- Craig has pinged me regarding tap_tests being incorrectly updated in
config_default.pl in 0003.
I just re-ran the tests on OSX and Windows (MSVC 2010 with Win7) to be
sure that nothing broke, and nothing has been reported as broken.
--
Michael

Attachments:

0001-Fix-default-configuration-of-MSVC-builds-ignoring-TA.patchapplication/x-patch; name=0001-Fix-default-configuration-of-MSVC-builds-ignoring-TA.patchDownload
From 258dc4978b682f4fed953a5857fc3f50aacc8342 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Mon, 21 Dec 2015 16:28:34 +0900
Subject: [PATCH 1/3] Fix default configuration of MSVC builds ignoring TAP
 tests

MSVC build scripts use a flag to track if TAP tests are supported or not
but this was not configured correctly. By default, like the other build
types using ./configure, this is disabled.
---
 src/tools/msvc/Solution.pm       |  1 +
 src/tools/msvc/config_default.pl | 27 ++++++++++++++-------------
 2 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index ac116b7..c5a43f9 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -643,6 +643,7 @@ sub GetFakeConfigure
 	$cfg .= ' --enable-integer-datetimes'
 	  if ($self->{options}->{integer_datetimes});
 	$cfg .= ' --enable-nls' if ($self->{options}->{nls});
+	$cfg .= ' --enable-tap-tests' if ($self->{options}->{tap_tests});
 	$cfg .= ' --with-ldap'  if ($self->{options}->{ldap});
 	$cfg .= ' --without-zlib' unless ($self->{options}->{zlib});
 	$cfg .= ' --with-extra-version' if ($self->{options}->{extraver});
diff --git a/src/tools/msvc/config_default.pl b/src/tools/msvc/config_default.pl
index b9f2ff4..e50be7e 100644
--- a/src/tools/msvc/config_default.pl
+++ b/src/tools/msvc/config_default.pl
@@ -3,7 +3,7 @@ use strict;
 use warnings;
 
 our $config = {
-	asserts => 0,    # --enable-cassert
+	asserts  => 0,    # --enable-cassert
 	  # integer_datetimes=>1,   # --enable-integer-datetimes - on is now default
 	  # float4byval=>1,         # --disable-float4-byval, on by default
 
@@ -13,18 +13,19 @@ our $config = {
 	# blocksize => 8,         # --with-blocksize, 8kB by default
 	# wal_blocksize => 8,     # --with-wal-blocksize, 8kB by default
 	# wal_segsize => 16,      # --with-wal-segsize, 16MB by default
-	ldap     => 1,        # --with-ldap
-	extraver => undef,    # --with-extra-version=<string>
-	nls      => undef,    # --enable-nls=<path>
-	tcl      => undef,    # --with-tls=<path>
-	perl     => undef,    # --with-perl
-	python   => undef,    # --with-python=<path>
-	openssl  => undef,    # --with-openssl=<path>
-	uuid     => undef,    # --with-ossp-uuid
-	xml      => undef,    # --with-libxml=<path>
-	xslt     => undef,    # --with-libxslt=<path>
-	iconv    => undef,    # (not in configure, path to iconv)
-	zlib     => undef     # --with-zlib=<path>
+	ldap      => 1,        # --with-ldap
+	extraver  => undef,    # --with-extra-version=<string>
+	nls       => undef,    # --enable-nls=<path>
+	tap_tests => undef,    # --enable-tap-tests
+	tcl       => undef,    # --with-tls=<path>
+	perl      => undef,    # --with-perl
+	python    => undef,    # --with-python=<path>
+	openssl   => undef,    # --with-openssl=<path>
+	uuid      => undef,    # --with-ossp-uuid
+	xml       => undef,    # --with-libxml=<path>
+	xslt      => undef,    # --with-libxslt=<path>
+	iconv     => undef,    # (not in configure, path to iconv)
+	zlib      => undef     # --with-zlib=<path>
 };
 
 1;
-- 
2.7.2

0002-Add-routine-for-node-promotion-in-PostgresNode.patchapplication/x-patch; name=0002-Add-routine-for-node-promotion-in-PostgresNode.patchDownload
From 3873746ccbfbba7b7e711fde10e60b96caa1605b Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Wed, 17 Feb 2016 15:21:32 +0900
Subject: [PATCH 2/3] Add routine for node promotion in PostgresNode

This is useful for tests to trigger a promotion on a node via pg_ctl,
and pg_rewind can make immediate use of it.
---
 src/bin/pg_rewind/RewindTest.pm |  2 +-
 src/test/perl/PostgresNode.pm   | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index 68834cd..e204830 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -177,7 +177,7 @@ sub promote_standby
 	# Now promote slave and insert some new data on master, this will put
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
-	system_or_bail('pg_ctl', '-w', '-D', $node_standby->data_dir, 'promote');
+	$node_standby->promote;
 	$node_standby->poll_query_until('postgres',
 		"SELECT NOT pg_is_in_recovery()")
 	  or die "Timed out while waiting for promotion of standby";
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index aec3b9a..7efbf5f 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -553,6 +553,25 @@ sub restart
 	$self->_update_pid;
 }
 
+=pod
+
+=item $node->promote()
+
+wrapper for pg_ctl promote
+
+=cut
+
+sub promote
+{
+	my ($self)  = @_;
+	my $port    = $self->port;
+	my $pgdata  = $self->data_dir;
+	my $logfile = $self->logfile;
+	my $name    = $self->name;
+	print "### Promoting node \"$name\"\n";
+	TestLib::system_log('pg_ctl', '-D', $pgdata, '-l', $logfile,
+						'promote');
+}
 
 # Internal method
 sub _update_pid
-- 
2.7.2

0003-Add-recovery-test-suite.patchapplication/x-patch; name=0003-Add-recovery-test-suite.patchDownload
From 805d2e533a46599e4538df2a81d0ca7868236436 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Wed, 17 Feb 2016 21:50:19 +0900
Subject: [PATCH 3/3] Add recovery test suite

This includes basic tests maipulating standbys, be they archiving or
streaming nodes, and some basic sanity checks around them. PostgresNode
is extended with a couple of routines allowing to set up WAL archiving,
WAL streaming or WAL restore on a node, as well as a commodity routine
to allow a promotion.

Windows support is provided, MSVC-based installation can use the new
subcommand recoverycheck in vcregress.pl to trigger the test run.
---
 doc/src/sgml/install-windows.sgml           |   4 +-
 src/bin/pg_rewind/RewindTest.pm             |  17 +---
 src/test/Makefile                           |   2 +-
 src/test/perl/PostgresNode.pm               | 116 +++++++++++++++++++++++++-
 src/test/recovery/.gitignore                |   3 +
 src/test/recovery/Makefile                  |  17 ++++
 src/test/recovery/README                    |  19 +++++
 src/test/recovery/t/001_stream_rep.pl       |  62 ++++++++++++++
 src/test/recovery/t/002_archiving.pl        |  46 ++++++++++
 src/test/recovery/t/003_recovery_targets.pl | 125 ++++++++++++++++++++++++++++
 src/test/recovery/t/004_timeline_switch.pl  |  67 +++++++++++++++
 src/test/recovery/t/005_replay_delay.pl     |  43 ++++++++++
 src/tools/msvc/vcregress.pl                 |  13 ++-
 13 files changed, 512 insertions(+), 22 deletions(-)
 create mode 100644 src/test/recovery/.gitignore
 create mode 100644 src/test/recovery/Makefile
 create mode 100644 src/test/recovery/README
 create mode 100644 src/test/recovery/t/001_stream_rep.pl
 create mode 100644 src/test/recovery/t/002_archiving.pl
 create mode 100644 src/test/recovery/t/003_recovery_targets.pl
 create mode 100644 src/test/recovery/t/004_timeline_switch.pl
 create mode 100644 src/test/recovery/t/005_replay_delay.pl

diff --git a/doc/src/sgml/install-windows.sgml b/doc/src/sgml/install-windows.sgml
index ba60a6b..8e87b70 100644
--- a/doc/src/sgml/install-windows.sgml
+++ b/doc/src/sgml/install-windows.sgml
@@ -440,6 +440,7 @@ $ENV{CONFIG}="Debug";
 <userinput>vcregress ecpgcheck</userinput>
 <userinput>vcregress isolationcheck</userinput>
 <userinput>vcregress bincheck</userinput>
+<userinput>vcregress recoverycheck</userinput>
 <userinput>vcregress upgradecheck</userinput>
 </screen>
 
@@ -455,7 +456,8 @@ $ENV{CONFIG}="Debug";
 
   <para>
    Running the regression tests on client programs, with "vcregress bincheck",
-   requires an additional Perl module to be installed:
+   or on recovery tests, with "vceregress recoverycheck" requires an additional
+   Perl module to be installed:
    <variablelist>
     <varlistentry>
      <term><productname>IPC::Run</productname></term>
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index e204830..bda0516 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -114,24 +114,9 @@ sub check_query
 
 sub setup_cluster
 {
-
 	# Initialize master, data checksums are mandatory
 	$node_master = get_new_node('master');
-	$node_master->init;
-
-	# Custom parameters for master's postgresql.conf
-	$node_master->append_conf(
-		"postgresql.conf", qq(
-wal_level = hot_standby
-max_wal_senders = 2
-wal_keep_segments = 20
-max_wal_size = 200MB
-shared_buffers = 1MB
-wal_log_hints = on
-hot_standby = on
-autovacuum = off
-max_connections = 10
-));
+	$node_master->init(allows_streaming => 1);
 }
 
 sub start_master
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..7f7754f 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -12,7 +12,7 @@ subdir = src/test
 top_builddir = ../..
 include $(top_builddir)/src/Makefile.global
 
-SUBDIRS = regress isolation modules
+SUBDIRS = regress isolation modules recovery
 
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 7efbf5f..c2799e4 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -347,6 +347,12 @@ On Windows, we use SSPI authentication to ensure the same (by pg_regress
 pg_hba.conf is configured to allow replication connections. Pass the keyword
 parameter hba_permit_replication => 0 to disable this.
 
+WAL archiving can be enabled on this node by passing the keyword parameter
+has_archiving => 1. This is disabled by default.
+
+postgresql.conf can be set up for replication by passing the keyword
+parameter allows_streaming => 1. This is disabled by default.
+
 The new node is set up in a fast but unsafe configuration where fsync is
 disabled.
 
@@ -361,6 +367,10 @@ sub init
 
 	$params{hba_permit_replication} = 1
 	  if (!defined($params{hba_permit_replication}));
+	$params{has_archiving} = 0
+	  if (!defined($params{has_archiving}));
+	$params{allows_streaming} = 0
+	  if (!defined($params{allows_streaming}));
 
 	mkdir $self->backup_dir;
 	mkdir $self->archive_dir;
@@ -373,6 +383,20 @@ sub init
 	print $conf "fsync = off\n";
 	print $conf "log_statement = all\n";
 	print $conf "port = $port\n";
+
+	if ($params{allows_streaming})
+	{
+		print $conf "wal_level = hot_standby\n";
+		print $conf "max_wal_senders = 5\n";
+		print $conf "wal_keep_segments = 20\n";
+		print $conf "max_wal_size = 128MB\n";
+		print $conf "shared_buffers = 1MB\n";
+		print $conf "autovacuum = off\n";
+		print $conf "wal_log_hints = on\n";
+		print $conf "hot_standby = on\n";
+		print $conf "max_connections = 10\n";
+	}
+
 	if ($TestLib::windows_os)
 	{
 		print $conf "listen_addresses = '$host'\n";
@@ -385,6 +409,7 @@ sub init
 	close $conf;
 
 	$self->set_replication_conf if ($params{hba_permit_replication});
+	$self->enable_archiving if ($params{has_archiving});
 }
 
 =pod
@@ -444,7 +469,15 @@ of a backup previously created on that node with $node->backup.
 
 Does not start the node after init.
 
-A recovery.conf is not created.
+pg_hba.conf is configured to allow replication connections. Pass the keyword
+parameter hba_permit_replication => 0 to disable this.
+
+Streaming replication can be enabled on this node by passing the keyword
+parameter has_streaming => 1. This is disabled by default.
+
+Restoring WAL segments from archives using restore_command can be enabled
+by passiong the keyword parameter has_restoring => 1. This is disabled by
+default.
 
 The backup is copied, leaving the original unmodified. pg_hba.conf is
 unconditionally set to enable replication connections.
@@ -453,12 +486,19 @@ unconditionally set to enable replication connections.
 
 sub init_from_backup
 {
-	my ($self, $root_node, $backup_name) = @_;
+	my ($self, $root_node, $backup_name, %params) = @_;
 	my $backup_path = $root_node->backup_dir . '/' . $backup_name;
 	my $port        = $self->port;
 	my $node_name   = $self->name;
 	my $root_name   = $root_node->name;
 
+	$params{hba_permit_replication} = 1
+	   if (!defined($params{hba_permit_replication}));
+	$params{has_streaming} = 0
+	   if (!defined($params{has_streaming}));
+	$params{has_restoring} = 0
+	   if (!defined($params{has_restoring}));
+
 	print
 "# Initializing node \"$node_name\" from backup \"$backup_name\" of node \"$root_name\"\n";
 	die "Backup \"$backup_name\" does not exist at $backup_path"
@@ -478,7 +518,10 @@ sub init_from_backup
 		qq(
 port = $port
 ));
-	$self->set_replication_conf;
+
+	$self->set_replication_conf if ($params{hba_permit_replication});
+	$self->enable_restoring($root_node) if ($params{has_restoring});
+	$self->enable_streaming($root_node) if ($params{has_streaming});
 }
 
 =pod
@@ -573,6 +616,73 @@ sub promote
 						'promote');
 }
 
+# Internal routine to enable streaming replication on a standby node.
+sub enable_streaming
+{
+	my ($self, $root_node)  = @_;
+	my $root_connstr = $root_node->connstr;
+	my $name    = $self->name;
+
+	print "### Enabling streaming replication for node \"$name\"\n";
+	$self->append_conf('recovery.conf', qq(
+primary_conninfo='$root_connstr application_name=$name'
+standby_mode=on
+));
+}
+
+# Internal routine to enable archive recovery command on a standby node
+sub enable_restoring
+{
+	my ($self, $root_node)  = @_;
+	my $path = $root_node->archive_dir;
+	my $name = $self->name;
+
+	print "### Enabling restoring for node \"$name\"\n";
+
+	# On Windows, the path specified in the restore command needs to use
+	# double back-slashes to work properly and to be able to detect properly
+	# the file targetted by the copy command, so the directory value used
+	# in this routine, using only one back-slash, need to be properly changed
+	# first. Paths also need to be double-quoted to prevent failures where
+	# the path contains spaces.
+	$path =~ s{\\}{\\\\}g if ($TestLib::windows_os);
+	my $copy_command = $TestLib::windows_os ?
+		qq{copy "$path\\\\%f" "%p"} :
+		qq{cp $path/%f %p};
+
+	$self->append_conf('recovery.conf', qq(
+restore_command = '$copy_command'
+standby_mode = on
+));
+}
+
+# Internal routine to enable archiving
+sub enable_archiving
+{
+	my ($self) = @_;
+	my $path   = $self->archive_dir;
+	my $name   = $self->name;
+
+	print "### Enabling WAL archiving for node \"$name\"\n";
+
+	# On Windows, the path specified in the restore command needs to use
+	# double back-slashes to work properly and to be able to detect properly
+	# the file targetted by the copy command, so the directory value used
+	# in this routine, using only one back-slash, need to be properly changed
+	# first. Paths also need to be double-quoted to prevent failures where
+	# the path contains spaces.
+	$path =~ s{\\}{\\\\}g if ($TestLib::windows_os);
+	my $copy_command = $TestLib::windows_os ?
+		qq{copy "%p" "$path\\\\%f"} :
+		qq{cp %p $path/%f};
+
+	# Enable archive_mode and archive_command on node
+	$self->append_conf('postgresql.conf', qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
 # Internal method
 sub _update_pid
 {
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..3ed9be3
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,62 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_master->backup($backup_name);
+
+# Create streaming standby linking to master
+my $node_standby_1 = get_new_node('standby_1');
+$node_standby_1->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_1->start;
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+$node_standby_1->backup($backup_name);
+
+# Create second standby node linking to standby 1
+my $node_standby_2 = get_new_node('standby_2');
+$node_standby_2->init_from_backup($node_standby_1, $backup_name,
+								  has_streaming => 1);
+$node_standby_2->start;
+
+# Create some content on master and check its presence in standby 1
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a");
+
+# Wait for standbys to catch up
+my $applname_1 = $node_standby_1->name;
+my $applname_2 = $node_standby_2->name;
+my $caughtup_query = "SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_1';";
+$node_master->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_2';";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = $node_standby_1->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result =  $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+$node_standby_1->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+$node_standby_2->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_2->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..930125c
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,46 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+
+# Initialize master node, doing archives
+my $node_master = get_new_node('master');
+$node_master->init(has_archiving => 1,
+				   allows_streaming => 1);
+my $backup_name = 'my_backup';
+
+# Start it
+$node_master->start;
+
+# Take backup for slave
+$node_master->backup($backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $node_standby = get_new_node('standby');
+$node_standby->init_from_backup($node_master, $backup_name,
+								has_restoring => 1);
+$node_standby->append_conf('postgresql.conf', qq(
+wal_retrieve_retry_interval = '100ms'
+));
+$node_standby->start;
+
+# Create some content on master
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $current_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Force archiving of WAL file to make it present on master
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Add some more content, it should not be present on standby
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(1000), 'check content from archives');
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..293603a
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,125 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $node_name = shift;
+	my $node_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $node_standby = get_new_node($node_name);
+	$node_standby->init_from_backup($node_master, 'my_backup',
+									has_restoring => 1);
+
+	foreach my $param_item (@$recovery_params)
+	{
+		$node_standby->append_conf('recovery.conf',
+					   qq($param_item
+));
+	}
+
+	$node_standby->start;
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	$node_standby->poll_query_until('postgres', $caughtup_query)
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	$node_standby->teardown_node;
+}
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(has_archiving => 1, allows_streaming => 1);
+
+# Start it
+$node_master->start;
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $lsn1 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Take backup from which all operations will be run
+$node_master->backup('my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+my $recovery_txid = $node_master->psql('postgres', "SELECT txid_current()");
+my $lsn2 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# More data, with recovery target timestamp
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(2001,3000))");
+my $recovery_time = $node_master->psql('postgres', "SELECT now()");
+my $lsn3 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Even more data, this time with a recovery target name
+$node_master->psql('postgres',
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))");
+my $recovery_name = "my_target";
+my $lsn4 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+$node_master->psql('postgres', "SELECT pg_create_restore_point('$recovery_name'");
+
+# Force archiving of WAL file
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', 'standby_1', $node_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', 'standby_2', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', 'standby_3', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', 'standby_4', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', 'standby_5', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', 'standby_6', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', 'standby_7', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..dc08ec1
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,67 @@
+# Test for timeline switch
+# Ensure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create two standbys linking to it
+my $node_standby_1 = get_new_node('standby_1');
+$node_standby_1->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_1->start;
+my $node_standby_2 = get_new_node('standby_2');
+$node_standby_2->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_2->start;
+
+# Create some content on master
+$node_master->psql('postgres',
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+$node_master->teardown_node;
+$node_standby_1->promote;
+
+# Switch standby 2 to replay from standby 1
+remove_tree($node_standby_2->data_dir . '/recovery.conf');
+my $connstr_1 = $node_standby_1->connstr;
+$node_standby_2->append_conf('recovery.conf', qq(
+primary_conninfo='$connstr_1'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+$node_standby_2->restart;
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done. Standby 1 needs
+# to exit recovery first before moving on with the test.
+$node_standby_1->poll_query_until('postgres', "SELECT pg_is_in_recovery() <> true");
+$node_standby_1->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+$until_lsn = $node_standby_1->psql('postgres', "SELECT pg_current_xlog_location();");
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_2->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(2000), 'check content of standby 2');
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..2d8c690
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,43 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# And some content
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create streaming standby from backup
+my $node_standby = get_new_node('standby');
+$node_standby->init_from_backup($node_master, $backup_name,
+								has_streaming => 1);
+$node_standby->append_conf('recovery.conf', qq(
+recovery_min_apply_delay = '2s'
+));
+$node_standby->start;
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(11,20))");
+sleep 1;
+# Here we should have only 10 rows
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+$result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(20), 'check content with delay of 2s');
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index df1348b..3d14544 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -34,7 +34,7 @@ if (-e "src/tools/msvc/buildenv.pl")
 
 my $what = shift || "";
 if ($what =~
-/^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck)$/i
+/^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck|recoverycheck)$/i
   )
 {
 	$what = uc $what;
@@ -89,6 +89,7 @@ my %command = (
 	MODULESCHECK   => \&modulescheck,
 	ISOLATIONCHECK => \&isolationcheck,
 	BINCHECK       => \&bincheck,
+	RECOVERYCHECK  => \&recoverycheck,
 	UPGRADECHECK   => \&upgradecheck,);
 
 my $proc = $command{$what};
@@ -360,6 +361,16 @@ sub modulescheck
 	exit $mstat if $mstat;
 }
 
+sub recoverycheck
+{
+	InstallTemp();
+
+	my $mstat = 0;
+	my $dir = "$topdir/src/test/recovery";
+	my $status = tap_check($dir);
+	exit $status if $status;
+}
+
 # Run "initdb", then reconfigure authentication.
 sub standard_initdb
 {
-- 
2.7.2

#136Craig Ringer
craig@2ndquadrant.com
In reply to: Michael Paquier (#135)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On 26 February 2016 at 10:58, Michael Paquier <michael.paquier@gmail.com>
wrote:

Here is a rebased set after the conflicts created by e640093, with the
following changes:

Thanks for rebasing on top of that. Not totally fair when your patch came
first, but I guess it was simpler to merge the other one first.

- In 0002, added perldoc for new promote routine
- In 0003, added perldoc documentation for the new options introduced
in init and init_from_backup, and fixed some log entries not using the
node name to identify the node involved when enabling archive,
streaming or recovery.

Very much appreciated.

- Craig has pinged me regarding tap_tests being incorrectly updated in
config_default.pl in 0003.
I just re-ran the tests on OSX and Windows (MSVC 2010 with Win7) to be
sure that nothing broke, and nothing has been reported as broken.

I've looked over the tests. I see that you've updated the docs for the
Windows tests to reflect the changes, which is good, thanks.

I like the patch and would love to see it committed soon.

I do have one major disagreement, which is that you turn autovacuum off if
streaming is enabled. This is IMO completely wrong and must be removed.
It's making the tests ignore a major and important part of real-world use.

If you did it to make it easier to test replay catchup etc, just use
pg_xlog_location_diff instead of an equality test. Instead of:

my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <=
pg_last_xlog_replay_location()";

use

my $caughtup_query = "SELECT pg_xlog_location_diff('$current_lsn',
pg_last_xlog_replay_location()) <= 0";

so it doesn't care if we replay past the expected LSN on the master due to
autovacuum activity. That's what's done in the real world and what should
be covered by the tests IMO.

The patch sets

tap_tests => 1,

in config_default.pl. Was that on purpose? I'd have no problem with running
the TAP tests by default if they worked by default, but the docs say that
at least with ActiveState's Perl you have to jump through some hoops to get
IPC::Run.

Typo in PostgresNode.pm: passiong should be 'passing' .

Otherwise looks _really_ good and I'd love to see this committed very soon.

I'd like a way to append parameters in a way that won't clobber settings
made implicitly by the module through things like enable_streaming but I
can add that in a followup patch. It doesn't need to complicate this one.
I'm thinking of having the tests append an include_dir directive when they
create a node, maintain a hash of all parameters and rewrite a
postgresql.conf.taptests file in the include_dir when params are updated.
Also exposing a 'reload' call.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

#137Michael Paquier
michael.paquier@gmail.com
In reply to: Craig Ringer (#136)
3 attachment(s)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On Fri, Feb 26, 2016 at 1:47 PM, Craig Ringer <craig@2ndquadrant.com> wrote:

On 26 February 2016 at 10:58, Michael Paquier <michael.paquier@gmail.com>
wrote:

Here is a rebased set after the conflicts created by e640093, with the
following changes:

Thanks for rebasing on top of that. Not totally fair when your patch came
first, but I guess it was simpler to merge the other one first.

At this point the final result is the same. It does not matter what
gets in first.

I do have one major disagreement, which is that you turn autovacuum off if
streaming is enabled. This is IMO completely wrong and must be removed. It's
making the tests ignore a major and important part of real-world use.

This has been chosen for consistency with what is in pg_rewind tests,
the idea being to keep the runs more stable with a WAL output under
control to allow predictable results. Though I do not see any direct
reason to not remove it actually now that I think about it.

If you did it to make it easier to test replay catchup etc, just use
pg_xlog_location_diff instead of an equality test. Instead of:
my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <=
pg_last_xlog_replay_location()";
use
my $caughtup_query = "SELECT pg_xlog_location_diff('$current_lsn',
pg_last_xlog_replay_location()) <= 0";
so it doesn't care if we replay past the expected LSN on the master due to
autovacuum activity. That's what's done in the real world and what should be
covered by the tests IMO.

Those two statements have the same meaning. pg_xlog_location_diff does
exactly the same thing as the pg_lsn data type in terms of LSN
comparisons. Choosing one or the other is really a matter of taste.
Though I see that 001 is the only test that uses an equality, this
should not be the case I agree.

The patch sets

tap_tests => 1,

in config_default.pl. Was that on purpose? I'd have no problem with running
the TAP tests by default if they worked by default, but the docs say that at
least with ActiveState's Perl you have to jump through some hoops to get
IPC::Run.

No, this was an error in the previous version of the patch 0003. Those
tests should be disabled by default, to match what ./configure does,
and also because installing IPC::Run requires some extra operations,
but that's easily doable with a bit of black magic.

Typo in PostgresNode.pm: passiong should be 'passing'.

Oops.

I'd like a way to append parameters in a way that won't clobber settings
made implicitly by the module through things like enable_streaming but I can
add that in a followup patch. It doesn't need to complicate this one.

This is something that I have been thinking about for some time while
hacking this thing, but I finished with the current version to not
complicate the patch more than it needs to be, and because the current
version is enough for the needs of all the tests present. Surely this
can be extended further more. One idea that I had was for example to
pass as parameter to init() and init_from_backup() a set of key/values
that would be appended to postgresql.conf.

I'm thinking of having the tests append an include_dir directive when they
create a node, maintain a hash of all parameters and rewrite a
postgresql.conf.taptests file in the include_dir when params are updated.
Also exposing a 'reload' call.

The reload wrapper would make sense to have. That has not proved to be
necessary yet.
--
Michael

Attachments:

0001-Fix-default-configuration-of-MSVC-builds-ignoring-TA.patchapplication/x-patch; name=0001-Fix-default-configuration-of-MSVC-builds-ignoring-TA.patchDownload
From 258dc4978b682f4fed953a5857fc3f50aacc8342 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Mon, 21 Dec 2015 16:28:34 +0900
Subject: [PATCH 1/3] Fix default configuration of MSVC builds ignoring TAP
 tests

MSVC build scripts use a flag to track if TAP tests are supported or not
but this was not configured correctly. By default, like the other build
types using ./configure, this is disabled.
---
 src/tools/msvc/Solution.pm       |  1 +
 src/tools/msvc/config_default.pl | 27 ++++++++++++++-------------
 2 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index ac116b7..c5a43f9 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -643,6 +643,7 @@ sub GetFakeConfigure
 	$cfg .= ' --enable-integer-datetimes'
 	  if ($self->{options}->{integer_datetimes});
 	$cfg .= ' --enable-nls' if ($self->{options}->{nls});
+	$cfg .= ' --enable-tap-tests' if ($self->{options}->{tap_tests});
 	$cfg .= ' --with-ldap'  if ($self->{options}->{ldap});
 	$cfg .= ' --without-zlib' unless ($self->{options}->{zlib});
 	$cfg .= ' --with-extra-version' if ($self->{options}->{extraver});
diff --git a/src/tools/msvc/config_default.pl b/src/tools/msvc/config_default.pl
index b9f2ff4..e50be7e 100644
--- a/src/tools/msvc/config_default.pl
+++ b/src/tools/msvc/config_default.pl
@@ -3,7 +3,7 @@ use strict;
 use warnings;
 
 our $config = {
-	asserts => 0,    # --enable-cassert
+	asserts  => 0,    # --enable-cassert
 	  # integer_datetimes=>1,   # --enable-integer-datetimes - on is now default
 	  # float4byval=>1,         # --disable-float4-byval, on by default
 
@@ -13,18 +13,19 @@ our $config = {
 	# blocksize => 8,         # --with-blocksize, 8kB by default
 	# wal_blocksize => 8,     # --with-wal-blocksize, 8kB by default
 	# wal_segsize => 16,      # --with-wal-segsize, 16MB by default
-	ldap     => 1,        # --with-ldap
-	extraver => undef,    # --with-extra-version=<string>
-	nls      => undef,    # --enable-nls=<path>
-	tcl      => undef,    # --with-tls=<path>
-	perl     => undef,    # --with-perl
-	python   => undef,    # --with-python=<path>
-	openssl  => undef,    # --with-openssl=<path>
-	uuid     => undef,    # --with-ossp-uuid
-	xml      => undef,    # --with-libxml=<path>
-	xslt     => undef,    # --with-libxslt=<path>
-	iconv    => undef,    # (not in configure, path to iconv)
-	zlib     => undef     # --with-zlib=<path>
+	ldap      => 1,        # --with-ldap
+	extraver  => undef,    # --with-extra-version=<string>
+	nls       => undef,    # --enable-nls=<path>
+	tap_tests => undef,    # --enable-tap-tests
+	tcl       => undef,    # --with-tls=<path>
+	perl      => undef,    # --with-perl
+	python    => undef,    # --with-python=<path>
+	openssl   => undef,    # --with-openssl=<path>
+	uuid      => undef,    # --with-ossp-uuid
+	xml       => undef,    # --with-libxml=<path>
+	xslt      => undef,    # --with-libxslt=<path>
+	iconv     => undef,    # (not in configure, path to iconv)
+	zlib      => undef     # --with-zlib=<path>
 };
 
 1;
-- 
2.7.2

0002-Add-routine-for-node-promotion-in-PostgresNode.patchapplication/x-patch; name=0002-Add-routine-for-node-promotion-in-PostgresNode.patchDownload
From 3873746ccbfbba7b7e711fde10e60b96caa1605b Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Wed, 17 Feb 2016 15:21:32 +0900
Subject: [PATCH 2/3] Add routine for node promotion in PostgresNode

This is useful for tests to trigger a promotion on a node via pg_ctl,
and pg_rewind can make immediate use of it.
---
 src/bin/pg_rewind/RewindTest.pm |  2 +-
 src/test/perl/PostgresNode.pm   | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index 68834cd..e204830 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -177,7 +177,7 @@ sub promote_standby
 	# Now promote slave and insert some new data on master, this will put
 	# the master out-of-sync with the standby. Wait until the standby is
 	# out of recovery mode, and is ready to accept read-write connections.
-	system_or_bail('pg_ctl', '-w', '-D', $node_standby->data_dir, 'promote');
+	$node_standby->promote;
 	$node_standby->poll_query_until('postgres',
 		"SELECT NOT pg_is_in_recovery()")
 	  or die "Timed out while waiting for promotion of standby";
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index aec3b9a..7efbf5f 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -553,6 +553,25 @@ sub restart
 	$self->_update_pid;
 }
 
+=pod
+
+=item $node->promote()
+
+wrapper for pg_ctl promote
+
+=cut
+
+sub promote
+{
+	my ($self)  = @_;
+	my $port    = $self->port;
+	my $pgdata  = $self->data_dir;
+	my $logfile = $self->logfile;
+	my $name    = $self->name;
+	print "### Promoting node \"$name\"\n";
+	TestLib::system_log('pg_ctl', '-D', $pgdata, '-l', $logfile,
+						'promote');
+}
 
 # Internal method
 sub _update_pid
-- 
2.7.2

0003-Add-recovery-test-suite.patchapplication/x-patch; name=0003-Add-recovery-test-suite.patchDownload
From f09e88e648f17f90ac77d310833bf41ce77e3093 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Fri, 26 Feb 2016 14:40:52 +0900
Subject: [PATCH 3/3] Add recovery test suite

This includes basic tests maipulating standbys, be they archiving or
streaming nodes, and some basic sanity checks around them. PostgresNode
is extended with a couple of routines allowing to set up WAL archiving,
WAL streaming or WAL restore on a node, as well as a commodity routine
to allow a promotion.

Windows support is provided, MSVC-based installation can use the new
subcommand recoverycheck in vcregress.pl to trigger the test run.
---
 doc/src/sgml/install-windows.sgml           |   4 +-
 src/bin/pg_rewind/RewindTest.pm             |  17 +---
 src/test/Makefile                           |   2 +-
 src/test/perl/PostgresNode.pm               | 115 ++++++++++++++++++++++++-
 src/test/recovery/.gitignore                |   3 +
 src/test/recovery/Makefile                  |  17 ++++
 src/test/recovery/README                    |  19 +++++
 src/test/recovery/t/001_stream_rep.pl       |  62 ++++++++++++++
 src/test/recovery/t/002_archiving.pl        |  46 ++++++++++
 src/test/recovery/t/003_recovery_targets.pl | 125 ++++++++++++++++++++++++++++
 src/test/recovery/t/004_timeline_switch.pl  |  67 +++++++++++++++
 src/test/recovery/t/005_replay_delay.pl     |  43 ++++++++++
 src/tools/msvc/vcregress.pl                 |  13 ++-
 13 files changed, 511 insertions(+), 22 deletions(-)
 create mode 100644 src/test/recovery/.gitignore
 create mode 100644 src/test/recovery/Makefile
 create mode 100644 src/test/recovery/README
 create mode 100644 src/test/recovery/t/001_stream_rep.pl
 create mode 100644 src/test/recovery/t/002_archiving.pl
 create mode 100644 src/test/recovery/t/003_recovery_targets.pl
 create mode 100644 src/test/recovery/t/004_timeline_switch.pl
 create mode 100644 src/test/recovery/t/005_replay_delay.pl

diff --git a/doc/src/sgml/install-windows.sgml b/doc/src/sgml/install-windows.sgml
index ba60a6b..8e87b70 100644
--- a/doc/src/sgml/install-windows.sgml
+++ b/doc/src/sgml/install-windows.sgml
@@ -440,6 +440,7 @@ $ENV{CONFIG}="Debug";
 <userinput>vcregress ecpgcheck</userinput>
 <userinput>vcregress isolationcheck</userinput>
 <userinput>vcregress bincheck</userinput>
+<userinput>vcregress recoverycheck</userinput>
 <userinput>vcregress upgradecheck</userinput>
 </screen>
 
@@ -455,7 +456,8 @@ $ENV{CONFIG}="Debug";
 
   <para>
    Running the regression tests on client programs, with "vcregress bincheck",
-   requires an additional Perl module to be installed:
+   or on recovery tests, with "vceregress recoverycheck" requires an additional
+   Perl module to be installed:
    <variablelist>
     <varlistentry>
      <term><productname>IPC::Run</productname></term>
diff --git a/src/bin/pg_rewind/RewindTest.pm b/src/bin/pg_rewind/RewindTest.pm
index e204830..bda0516 100644
--- a/src/bin/pg_rewind/RewindTest.pm
+++ b/src/bin/pg_rewind/RewindTest.pm
@@ -114,24 +114,9 @@ sub check_query
 
 sub setup_cluster
 {
-
 	# Initialize master, data checksums are mandatory
 	$node_master = get_new_node('master');
-	$node_master->init;
-
-	# Custom parameters for master's postgresql.conf
-	$node_master->append_conf(
-		"postgresql.conf", qq(
-wal_level = hot_standby
-max_wal_senders = 2
-wal_keep_segments = 20
-max_wal_size = 200MB
-shared_buffers = 1MB
-wal_log_hints = on
-hot_standby = on
-autovacuum = off
-max_connections = 10
-));
+	$node_master->init(allows_streaming => 1);
 }
 
 sub start_master
diff --git a/src/test/Makefile b/src/test/Makefile
index b713c2c..7f7754f 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -12,7 +12,7 @@ subdir = src/test
 top_builddir = ../..
 include $(top_builddir)/src/Makefile.global
 
-SUBDIRS = regress isolation modules
+SUBDIRS = regress isolation modules recovery
 
 # We don't build or execute examples/, locale/, or thread/ by default,
 # but we do want "make clean" etc to recurse into them.  Likewise for ssl/,
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 7efbf5f..837205d 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -347,6 +347,12 @@ On Windows, we use SSPI authentication to ensure the same (by pg_regress
 pg_hba.conf is configured to allow replication connections. Pass the keyword
 parameter hba_permit_replication => 0 to disable this.
 
+WAL archiving can be enabled on this node by passing the keyword parameter
+has_archiving => 1. This is disabled by default.
+
+postgresql.conf can be set up for replication by passing the keyword
+parameter allows_streaming => 1. This is disabled by default.
+
 The new node is set up in a fast but unsafe configuration where fsync is
 disabled.
 
@@ -361,6 +367,10 @@ sub init
 
 	$params{hba_permit_replication} = 1
 	  if (!defined($params{hba_permit_replication}));
+	$params{has_archiving} = 0
+	  if (!defined($params{has_archiving}));
+	$params{allows_streaming} = 0
+	  if (!defined($params{allows_streaming}));
 
 	mkdir $self->backup_dir;
 	mkdir $self->archive_dir;
@@ -373,6 +383,19 @@ sub init
 	print $conf "fsync = off\n";
 	print $conf "log_statement = all\n";
 	print $conf "port = $port\n";
+
+	if ($params{allows_streaming})
+	{
+		print $conf "wal_level = hot_standby\n";
+		print $conf "max_wal_senders = 5\n";
+		print $conf "wal_keep_segments = 20\n";
+		print $conf "max_wal_size = 128MB\n";
+		print $conf "shared_buffers = 1MB\n";
+		print $conf "wal_log_hints = on\n";
+		print $conf "hot_standby = on\n";
+		print $conf "max_connections = 10\n";
+	}
+
 	if ($TestLib::windows_os)
 	{
 		print $conf "listen_addresses = '$host'\n";
@@ -385,6 +408,7 @@ sub init
 	close $conf;
 
 	$self->set_replication_conf if ($params{hba_permit_replication});
+	$self->enable_archiving if ($params{has_archiving});
 }
 
 =pod
@@ -444,7 +468,15 @@ of a backup previously created on that node with $node->backup.
 
 Does not start the node after init.
 
-A recovery.conf is not created.
+pg_hba.conf is configured to allow replication connections. Pass the keyword
+parameter hba_permit_replication => 0 to disable this.
+
+Streaming replication can be enabled on this node by passing the keyword
+parameter has_streaming => 1. This is disabled by default.
+
+Restoring WAL segments from archives using restore_command can be enabled
+by passing the keyword parameter has_restoring => 1. This is disabled by
+default.
 
 The backup is copied, leaving the original unmodified. pg_hba.conf is
 unconditionally set to enable replication connections.
@@ -453,12 +485,19 @@ unconditionally set to enable replication connections.
 
 sub init_from_backup
 {
-	my ($self, $root_node, $backup_name) = @_;
+	my ($self, $root_node, $backup_name, %params) = @_;
 	my $backup_path = $root_node->backup_dir . '/' . $backup_name;
 	my $port        = $self->port;
 	my $node_name   = $self->name;
 	my $root_name   = $root_node->name;
 
+	$params{hba_permit_replication} = 1
+	   if (!defined($params{hba_permit_replication}));
+	$params{has_streaming} = 0
+	   if (!defined($params{has_streaming}));
+	$params{has_restoring} = 0
+	   if (!defined($params{has_restoring}));
+
 	print
 "# Initializing node \"$node_name\" from backup \"$backup_name\" of node \"$root_name\"\n";
 	die "Backup \"$backup_name\" does not exist at $backup_path"
@@ -478,7 +517,10 @@ sub init_from_backup
 		qq(
 port = $port
 ));
-	$self->set_replication_conf;
+
+	$self->set_replication_conf if ($params{hba_permit_replication});
+	$self->enable_restoring($root_node) if ($params{has_restoring});
+	$self->enable_streaming($root_node) if ($params{has_streaming});
 }
 
 =pod
@@ -573,6 +615,73 @@ sub promote
 						'promote');
 }
 
+# Internal routine to enable streaming replication on a standby node.
+sub enable_streaming
+{
+	my ($self, $root_node)  = @_;
+	my $root_connstr = $root_node->connstr;
+	my $name    = $self->name;
+
+	print "### Enabling streaming replication for node \"$name\"\n";
+	$self->append_conf('recovery.conf', qq(
+primary_conninfo='$root_connstr application_name=$name'
+standby_mode=on
+));
+}
+
+# Internal routine to enable archive recovery command on a standby node
+sub enable_restoring
+{
+	my ($self, $root_node)  = @_;
+	my $path = $root_node->archive_dir;
+	my $name = $self->name;
+
+	print "### Enabling restoring for node \"$name\"\n";
+
+	# On Windows, the path specified in the restore command needs to use
+	# double back-slashes to work properly and to be able to detect properly
+	# the file targetted by the copy command, so the directory value used
+	# in this routine, using only one back-slash, need to be properly changed
+	# first. Paths also need to be double-quoted to prevent failures where
+	# the path contains spaces.
+	$path =~ s{\\}{\\\\}g if ($TestLib::windows_os);
+	my $copy_command = $TestLib::windows_os ?
+		qq{copy "$path\\\\%f" "%p"} :
+		qq{cp $path/%f %p};
+
+	$self->append_conf('recovery.conf', qq(
+restore_command = '$copy_command'
+standby_mode = on
+));
+}
+
+# Internal routine to enable archiving
+sub enable_archiving
+{
+	my ($self) = @_;
+	my $path   = $self->archive_dir;
+	my $name   = $self->name;
+
+	print "### Enabling WAL archiving for node \"$name\"\n";
+
+	# On Windows, the path specified in the restore command needs to use
+	# double back-slashes to work properly and to be able to detect properly
+	# the file targetted by the copy command, so the directory value used
+	# in this routine, using only one back-slash, need to be properly changed
+	# first. Paths also need to be double-quoted to prevent failures where
+	# the path contains spaces.
+	$path =~ s{\\}{\\\\}g if ($TestLib::windows_os);
+	my $copy_command = $TestLib::windows_os ?
+		qq{copy "%p" "$path\\\\%f"} :
+		qq{cp %p $path/%f};
+
+	# Enable archive_mode and archive_command on node
+	$self->append_conf('postgresql.conf', qq(
+archive_mode = on
+archive_command = '$copy_command'
+));
+}
+
 # Internal method
 sub _update_pid
 {
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
new file mode 100644
index 0000000..499fa7d
--- /dev/null
+++ b/src/test/recovery/.gitignore
@@ -0,0 +1,3 @@
+# Generated by test suite
+/regress_log/
+/tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
new file mode 100644
index 0000000..16c063a
--- /dev/null
+++ b/src/test/recovery/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/recovery
+#
+# Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/recovery/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+check:
+	$(prove_check)
diff --git a/src/test/recovery/README b/src/test/recovery/README
new file mode 100644
index 0000000..20b98e0
--- /dev/null
+++ b/src/test/recovery/README
@@ -0,0 +1,19 @@
+src/test/recovery/README
+
+Regression tests for recovery and replication
+=============================================
+
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
+
+Running the tests
+=================
+
+    make check
+
+NOTE: This creates a temporary installation, and some tests may
+create one or multiple nodes, be they master or standby(s) for the
+purpose of the tests.
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
new file mode 100644
index 0000000..3ca4c84
--- /dev/null
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -0,0 +1,62 @@
+# Minimal test testing streaming replication
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_master->backup($backup_name);
+
+# Create streaming standby linking to master
+my $node_standby_1 = get_new_node('standby_1');
+$node_standby_1->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_1->start;
+
+# Take backup of standby 1 (not mandatory, but useful to check if
+# pg_basebackup works on a standby).
+$node_standby_1->backup($backup_name);
+
+# Create second standby node linking to standby 1
+my $node_standby_2 = get_new_node('standby_2');
+$node_standby_2->init_from_backup($node_standby_1, $backup_name,
+								  has_streaming => 1);
+$node_standby_2->start;
+
+# Create some content on master and check its presence in standby 1
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a");
+
+# Wait for standbys to catch up
+my $applname_1 = $node_standby_1->name;
+my $applname_2 = $node_standby_2->name;
+my $caughtup_query = "SELECT pg_current_xlog_location() <= write_location FROM pg_stat_replication WHERE application_name = '$applname_1';";
+$node_master->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 1 to catch up";
+$caughtup_query = "SELECT pg_last_xlog_replay_location() <= write_location FROM pg_stat_replication WHERE application_name = '$applname_2';";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby 2 to catch up";
+
+my $result = $node_standby_1->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+$result =  $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 2: $result\n";
+is($result, qq(1002), 'check streamed content on standby 2');
+
+# Check that only READ-only queries can run on standbys
+$node_standby_1->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_1->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 1');
+$node_standby_2->command_fails(['psql', '-A', '-t',  '--no-psqlrc',
+	'-d', $node_standby_2->connstr, '-c',
+    "INSERT INTO tab_int VALUES (1)"],
+	'Read-only queries on standby 2');
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
new file mode 100644
index 0000000..930125c
--- /dev/null
+++ b/src/test/recovery/t/002_archiving.pl
@@ -0,0 +1,46 @@
+# test for archiving with warm standby
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+use File::Copy;
+
+# Initialize master node, doing archives
+my $node_master = get_new_node('master');
+$node_master->init(has_archiving => 1,
+				   allows_streaming => 1);
+my $backup_name = 'my_backup';
+
+# Start it
+$node_master->start;
+
+# Take backup for slave
+$node_master->backup($backup_name);
+
+# Initialize standby node from backup, fetching WAL from archives
+my $node_standby = get_new_node('standby');
+$node_standby->init_from_backup($node_master, $backup_name,
+								has_restoring => 1);
+$node_standby->append_conf('postgresql.conf', qq(
+wal_retrieve_retry_interval = '100ms'
+));
+$node_standby->start;
+
+# Create some content on master
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $current_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Force archiving of WAL file to make it present on master
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Add some more content, it should not be present on standby
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+
+# Wait until necessary replay has been done on standby
+my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(1000), 'check content from archives');
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
new file mode 100644
index 0000000..293603a
--- /dev/null
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -0,0 +1,125 @@
+# Test for recovery targets: name, timestamp, XID
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+
+# Create and test a standby from given backup, with a certain
+# recovery target.
+sub test_recovery_standby
+{
+	my $test_name = shift;
+	my $node_name = shift;
+	my $node_master = shift;
+	my $recovery_params = shift;
+	my $num_rows = shift;
+	my $until_lsn = shift;
+
+	my $node_standby = get_new_node($node_name);
+	$node_standby->init_from_backup($node_master, 'my_backup',
+									has_restoring => 1);
+
+	foreach my $param_item (@$recovery_params)
+	{
+		$node_standby->append_conf('recovery.conf',
+					   qq($param_item
+));
+	}
+
+	$node_standby->start;
+
+	# Wait until standby has replayed enough data
+	my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+	$node_standby->poll_query_until('postgres', $caughtup_query)
+		or die "Timed out while waiting for standby to catch up";
+
+	# Create some content on master and check its presence in standby
+	my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+	is($result, qq($num_rows), "check standby content for $test_name");
+
+	# Stop standby node
+	$node_standby->teardown_node;
+}
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(has_archiving => 1, allows_streaming => 1);
+
+# Start it
+$node_master->start;
+
+# Create data before taking the backup, aimed at testing
+# recovery_target = 'immediate'
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $lsn1 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Take backup from which all operations will be run
+$node_master->backup('my_backup');
+
+# Insert some data with used as a replay reference, with a recovery
+# target TXID.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+my $recovery_txid = $node_master->psql('postgres', "SELECT txid_current()");
+my $lsn2 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# More data, with recovery target timestamp
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(2001,3000))");
+my $recovery_time = $node_master->psql('postgres', "SELECT now()");
+my $lsn3 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Even more data, this time with a recovery target name
+$node_master->psql('postgres',
+	"INSERT INTO tab_int VALUES (generate_series(3001,4000))");
+my $recovery_name = "my_target";
+my $lsn4 = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+$node_master->psql('postgres', "SELECT pg_create_restore_point('$recovery_name'");
+
+# Force archiving of WAL file
+$node_master->psql('postgres', "SELECT pg_switch_xlog()");
+
+# Test recovery targets
+my @recovery_params = ( "recovery_target = 'immediate'" );
+test_recovery_standby('immediate target', 'standby_1', $node_master,
+					  \@recovery_params,
+					  "1000", $lsn1);
+@recovery_params = ( "recovery_target_xid = '$recovery_txid'" );
+test_recovery_standby('XID', 'standby_2', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = ( "recovery_target_time = '$recovery_time'" );
+test_recovery_standby('Time', 'standby_3', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = ( "recovery_target_name = '$recovery_name'" );
+test_recovery_standby('Name', 'standby_4', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
+
+# Multiple targets
+# Last entry has priority (note that an array respects the order of items
+# not hashes).
+@recovery_params = (
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'"
+);
+test_recovery_standby('Name + XID + Time', 'standby_5', $node_master,
+					  \@recovery_params,
+					  "3000", $lsn3);
+@recovery_params = (
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'",
+	"recovery_target_xid  = '$recovery_txid'"
+);
+test_recovery_standby('Time + Name + XID', 'standby_6', $node_master,
+					  \@recovery_params,
+					  "2000", $lsn2);
+@recovery_params = (
+	"recovery_target_xid  = '$recovery_txid'",
+	"recovery_target_time = '$recovery_time'",
+	"recovery_target_name = '$recovery_name'"
+);
+test_recovery_standby('XID + Time + Name', 'standby_7', $node_master,
+					  \@recovery_params,
+					  "4000", $lsn4);
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
new file mode 100644
index 0000000..dc08ec1
--- /dev/null
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -0,0 +1,67 @@
+# Test for timeline switch
+# Ensure that a standby is able to follow a newly-promoted standby
+# on a new timeline.
+use strict;
+use warnings;
+use File::Path qw(remove_tree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 1;
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create two standbys linking to it
+my $node_standby_1 = get_new_node('standby_1');
+$node_standby_1->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_1->start;
+my $node_standby_2 = get_new_node('standby_2');
+$node_standby_2->init_from_backup($node_master, $backup_name,
+								  has_streaming => 1);
+$node_standby_2->start;
+
+# Create some content on master
+$node_master->psql('postgres',
+	"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+
+# Wait until standby has replayed enough data on standby 1
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_1->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+# Stop and remove master, and promote standby 1, switching it to a new timeline
+$node_master->teardown_node;
+$node_standby_1->promote;
+
+# Switch standby 2 to replay from standby 1
+remove_tree($node_standby_2->data_dir . '/recovery.conf');
+my $connstr_1 = $node_standby_1->connstr;
+$node_standby_2->append_conf('recovery.conf', qq(
+primary_conninfo='$connstr_1'
+standby_mode=on
+recovery_target_timeline='latest'
+));
+$node_standby_2->restart;
+
+# Insert some data in standby 1 and check its presence in standby 2
+# to ensure that the timeline switch has been done. Standby 1 needs
+# to exit recovery first before moving on with the test.
+$node_standby_1->poll_query_until('postgres', "SELECT pg_is_in_recovery() <> true");
+$node_standby_1->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(1001,2000))");
+$until_lsn = $node_standby_1->psql('postgres', "SELECT pg_current_xlog_location();");
+$caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby_2->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+
+my $result = $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(2000), 'check content of standby 2');
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
new file mode 100644
index 0000000..2d8c690
--- /dev/null
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -0,0 +1,43 @@
+# Checks for recovery_min_apply_delay
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+# Initialize master node
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# And some content
+$node_master->psql('postgres', "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create streaming standby from backup
+my $node_standby = get_new_node('standby');
+$node_standby->init_from_backup($node_master, $backup_name,
+								has_streaming => 1);
+$node_standby->append_conf('recovery.conf', qq(
+recovery_min_apply_delay = '2s'
+));
+$node_standby->start;
+
+# Make new content on master and check its presence in standby
+# depending on the delay of 2s applied above.
+$node_master->psql('postgres', "INSERT INTO tab_int VALUES (generate_series(11,20))");
+sleep 1;
+# Here we should have only 10 rows
+my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(10), 'check content with delay of 1s');
+
+# Now wait for replay to complete on standby
+my $until_lsn = $node_master->psql('postgres', "SELECT pg_current_xlog_location();");
+my $caughtup_query = "SELECT '$until_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
+$node_standby->poll_query_until('postgres', $caughtup_query)
+	or die "Timed out while waiting for standby to catch up";
+$result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
+is($result, qq(20), 'check content with delay of 2s');
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index df1348b..3d14544 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -34,7 +34,7 @@ if (-e "src/tools/msvc/buildenv.pl")
 
 my $what = shift || "";
 if ($what =~
-/^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck)$/i
+/^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck|recoverycheck)$/i
   )
 {
 	$what = uc $what;
@@ -89,6 +89,7 @@ my %command = (
 	MODULESCHECK   => \&modulescheck,
 	ISOLATIONCHECK => \&isolationcheck,
 	BINCHECK       => \&bincheck,
+	RECOVERYCHECK  => \&recoverycheck,
 	UPGRADECHECK   => \&upgradecheck,);
 
 my $proc = $command{$what};
@@ -360,6 +361,16 @@ sub modulescheck
 	exit $mstat if $mstat;
 }
 
+sub recoverycheck
+{
+	InstallTemp();
+
+	my $mstat = 0;
+	my $dir = "$topdir/src/test/recovery";
+	my $status = tap_check($dir);
+	exit $status if $status;
+}
+
 # Run "initdb", then reconfigure authentication.
 sub standard_initdb
 {
-- 
2.7.2

#138Craig Ringer
craig@2ndquadrant.com
In reply to: Michael Paquier (#137)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On 26 February 2016 at 13:43, Michael Paquier <michael.paquier@gmail.com>
wrote:

my $caughtup_query = "SELECT '$current_lsn'::pg_lsn <=
pg_last_xlog_replay_location()";
use
my $caughtup_query = "SELECT pg_xlog_location_diff('$current_lsn',
pg_last_xlog_replay_location()) <= 0";
so it doesn't care if we replay past the expected LSN on the master due

to

autovacuum activity. That's what's done in the real world and what

should be

covered by the tests IMO.

Those two statements have the same meaning. pg_xlog_location_diff does
exactly the same thing as the pg_lsn data type in terms of LSN
comparisons.

Ah. Whoops. I meant to write '=' in the first, to reflect what the code
does.

You're quite right that casting to pg_lsn has the same effect and looks
cleaner.

I think this looks good as of the last version. I'm not keen on disabling
autovacuum but that can be addressed in a followup that makes it easier to
configure params, as discussed. I definitely don't want to complicate this
patch with it.

Should be committed ASAP IMO.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

#139Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Victor Wagner (#130)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

Victor Wagner wrote:

I'll second Stas' suggestion about psql_ok/psql_fail functions.

1. psql_ok instead of just psql would provide visual feedback for the
reader of code. One would see 'here condition is tested, here is
something ended with _ok/_fail'.

It would be nice that seeing say "use Test::More tests => 4"
one can immediately see "Yes, there is three _ok's and one _fail in the
script'

2. I have use case for psql_fail code. In my libpq failover patch there
is number of cases, where it should be tested that connection is not
established,

But this is rather about further evolution of the tap test library, not
about this set of tests.

This makes sense to me. Please submit a patch for this.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#140Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#134)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

Attached are rebased patches, split into 3 parts doing the following:
- 0001, fix default configuration of MSVC builds ignoring TAP tests

BTW you keep submitting this one and I keep ignoring it. I think you
should start a separate thread for this one, so that some
Windows-enabled committer can look at it and maybe push it. I still
don't understand why this matters.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#141Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Kyotaro HORIGUCHI (#125)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Kyotaro HORIGUCHI wrote:

So, I'd like to propose four (or five) changes to this harness.

- prove_check to remove all in tmp_check

- TestLib to preserve temporary directories/files if the current
test fails.

- PostgresNode::get_new_node to create data directory with
meaningful basenames.

- PostgresNode::psql to return a list of ($stdout, $stderr) if
requested. (The previous behavior is not changed)

- (recovery/t/00x_* gives test number to node name)

As a POC, the attached diff will appliy on the 0001 and (fixed)
0003 patches.

These changes all seem very reasonable to me. I'm not so sure about the
last one. Perhaps the framework ought to generate an appropriate subdir
name using the test file name plus the node name, so that instead of
tmp_XXXX it becomes tmp_001_master_XXXX or something like that? Having
be a coding convention doesn't look real nice to me.

I didn't try to apply your patch but I'm fairly certain it would
conflict with what's here now; can you please rebase and resend?

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#142Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Craig Ringer (#138)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

Craig Ringer wrote:

Should be committed ASAP IMO.

Finally pushed it. Let's see how it does in the buildfarm. Now let's
get going and add more tests, I know there's no shortage of people with
test scripts waiting for this.

Thanks, Michael, for the persistency, and thanks to all reviewers. (I'm
sorry we seem to have lost Amir Rohan in the process. He was doing
a great job.)

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#143Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#142)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Feb 27, 2016 at 4:30 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Craig Ringer wrote:

Should be committed ASAP IMO.

Finally pushed it. Let's see how it does in the buildfarm. Now let's
get going and add more tests, I know there's no shortage of people with
test scripts waiting for this.

Thanks, Michael, for the persistency, and thanks to all reviewers. (I'm
sorry we seem to have lost Amir Rohan in the process. He was doing
a great job.)

Date of first message of this thread: Mon, 2 Dec 2013 15:40:41 +0900
Date of this message: Fri, 26 Feb 2016 16:30:08 -0300
This has been a long trip. Thanks a lot to all involved. Many people
have reviewed and helped out with this patch.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#144Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#143)
1 attachment(s)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Feb 27, 2016 at 7:37 AM, Michael Paquier wrote:

Date of first message of this thread: Mon, 2 Dec 2013 15:40:41 +0900
Date of this message: Fri, 26 Feb 2016 16:30:08 -0300
This has been a long trip. Thanks a lot to all involved. Many people
have reviewed and helped out with this patch.

I just had a closer look at what has been committed, and I found a
couple of minor issues, addressed via the patch attached:
1) install-windows.sgml should use the markup command when mentioning
bincheck and recoverycheck
2) src/test/recovery/.gitignore is listing /regress_log/ but that's
not needed (this is a remnant of a previous version of the patch
posted on this thread).
3) Header of 002_archiving.pl mentions that the tests are running on a
warm standby, but that's a hot standby (found by Craig and reported on
github on my account)
4) src/test/recovery/Makefile is missing a clean target, to remove tmp_check/.
5) src/tools/msvc/clean.bat is missing the same cleanup command for
the same thing after running the tests.
6) Header of 004_timeline_switch.pl should perhaps mention that a
cascading standby is used (idea of Craig, a good addition IMO)

Regards,
--
Michael

Attachments:

test-recovery-fixes.patchbinary/octet-stream; name=test-recovery-fixes.patchDownload
diff --git a/doc/src/sgml/install-windows.sgml b/doc/src/sgml/install-windows.sgml
index bfe840c..f08cca7 100644
--- a/doc/src/sgml/install-windows.sgml
+++ b/doc/src/sgml/install-windows.sgml
@@ -455,9 +455,10 @@ $ENV{CONFIG}="Debug";
   </para>
 
   <para>
-   Running the regression tests on client programs, with "vcregress bincheck",
-   or on recovery tests, with "vcregress recoverycheck" requires an additional
-   Perl module to be installed:
+   Running the regression tests on client programs, with
+   <command>vcregress bincheck</>, or on recovery tests, with
+   <command>vcregress recoverycheck</> requires an additional Perl module
+   to be installed:
    <variablelist>
     <varlistentry>
      <term><productname>IPC::Run</productname></term>
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
index 499fa7d..871e943 100644
--- a/src/test/recovery/.gitignore
+++ b/src/test/recovery/.gitignore
@@ -1,3 +1,2 @@
 # Generated by test suite
-/regress_log/
 /tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
index 330ab2b..9290719 100644
--- a/src/test/recovery/Makefile
+++ b/src/test/recovery/Makefile
@@ -15,3 +15,6 @@ include $(top_builddir)/src/Makefile.global
 
 check:
 	$(prove_check)
+
+clean distclean maintainer-clean:
+	rm -rf tmp_check
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
index 06c3ff4..67bb7df 100644
--- a/src/test/recovery/t/002_archiving.pl
+++ b/src/test/recovery/t/002_archiving.pl
@@ -1,4 +1,4 @@
-# test for archiving with warm standby
+# test for archiving with hot standby
 use strict;
 use warnings;
 use PostgresNode;
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
index 573f40a..58bf580 100644
--- a/src/test/recovery/t/004_timeline_switch.pl
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -1,5 +1,5 @@
 # Test for timeline switch
-# Ensure that a standby is able to follow a newly-promoted standby
+# Ensure that a cascading standby is able to follow a newly-promoted standby
 # on a new timeline.
 use strict;
 use warnings;
diff --git a/src/tools/msvc/clean.bat b/src/tools/msvc/clean.bat
index 58357f8..3491344 100755
--- a/src/tools/msvc/clean.bat
+++ b/src/tools/msvc/clean.bat
@@ -97,6 +97,7 @@ if exist src\bin\pg_ctl\tmp_check rd /s /q src\bin\pg_ctl\tmp_check
 if exist src\bin\pg_rewind\tmp_check rd /s /q src\bin\pg_rewind\tmp_check
 if exist src\bin\pgbench\tmp_check rd /s /q src\bin\pgbench\tmp_check
 if exist src\bin\scripts\tmp_check rd /s /q src\bin\scripts\tmp_check
+if exist src\test\recovery\tmp_check rd /s /q src\test\recovery\tmp_check
 
 REM Clean up datafiles built with contrib
 REM cd contrib
#145Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#144)
1 attachment(s)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sun, Feb 28, 2016 at 10:41 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Sat, Feb 27, 2016 at 7:37 AM, Michael Paquier wrote:

Date of first message of this thread: Mon, 2 Dec 2013 15:40:41 +0900
Date of this message: Fri, 26 Feb 2016 16:30:08 -0300
This has been a long trip. Thanks a lot to all involved. Many people
have reviewed and helped out with this patch.

I just had a closer look at what has been committed, and I found a
couple of minor issues, addressed via the patch attached:
1) install-windows.sgml should use the markup command when mentioning
bincheck and recoverycheck
2) src/test/recovery/.gitignore is listing /regress_log/ but that's
not needed (this is a remnant of a previous version of the patch
posted on this thread).
3) Header of 002_archiving.pl mentions that the tests are running on a
warm standby, but that's a hot standby (found by Craig and reported on
github on my account)
4) src/test/recovery/Makefile is missing a clean target, to remove tmp_check/.
5) src/tools/msvc/clean.bat is missing the same cleanup command for
the same thing after running the tests.
6) Header of 004_timeline_switch.pl should perhaps mention that a
cascading standby is used (idea of Craig, a good addition IMO)

7) src/test/README is not describing recovery/
8) This description in src/test/recovery/README is not exact, it
mentions a set of routines that are now part of PostgresNode.pm:
+This directory contains a test suite for recovery and replication,
+testing mainly the interactions of recovery.conf with cluster
+instances by providing a simple set of routines that can be used
+to define a custom cluster for a test, including backup, archiving,
+and streaming configuration.
I would suggest removing the last 4 lines and simplify the paragraph.
9) I have no logical explanation to explain why I am seeing all those
things now.
v2 is attached.
-- 
Michael

Attachments:

test-recovery-fixes-v2.patchbinary/octet-stream; name=test-recovery-fixes-v2.patchDownload
diff --git a/doc/src/sgml/install-windows.sgml b/doc/src/sgml/install-windows.sgml
index bfe840c..f08cca7 100644
--- a/doc/src/sgml/install-windows.sgml
+++ b/doc/src/sgml/install-windows.sgml
@@ -455,9 +455,10 @@ $ENV{CONFIG}="Debug";
   </para>
 
   <para>
-   Running the regression tests on client programs, with "vcregress bincheck",
-   or on recovery tests, with "vcregress recoverycheck" requires an additional
-   Perl module to be installed:
+   Running the regression tests on client programs, with
+   <command>vcregress bincheck</>, or on recovery tests, with
+   <command>vcregress recoverycheck</> requires an additional Perl module
+   to be installed:
    <variablelist>
     <varlistentry>
      <term><productname>IPC::Run</productname></term>
diff --git a/src/test/README b/src/test/README
index 5587422..62395e7 100644
--- a/src/test/README
+++ b/src/test/README
@@ -28,6 +28,9 @@ modules/
 perl/
   Infrastructure for Perl-based TAP tests
 
+recovery/
+  Test suite for recovery and replication
+
 regress/
   PostgreSQL's main regression test suite, pg_regress
 
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
index 499fa7d..871e943 100644
--- a/src/test/recovery/.gitignore
+++ b/src/test/recovery/.gitignore
@@ -1,3 +1,2 @@
 # Generated by test suite
-/regress_log/
 /tmp_check/
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
index 330ab2b..9290719 100644
--- a/src/test/recovery/Makefile
+++ b/src/test/recovery/Makefile
@@ -15,3 +15,6 @@ include $(top_builddir)/src/Makefile.global
 
 check:
 	$(prove_check)
+
+clean distclean maintainer-clean:
+	rm -rf tmp_check
diff --git a/src/test/recovery/README b/src/test/recovery/README
index a9ee865..3cafb9d 100644
--- a/src/test/recovery/README
+++ b/src/test/recovery/README
@@ -3,11 +3,7 @@ src/test/recovery/README
 Regression tests for recovery and replication
 =============================================
 
-This directory contains a test suite for recovery and replication,
-testing mainly the interactions of recovery.conf with cluster
-instances by providing a simple set of routines that can be used
-to define a custom cluster for a test, including backup, archiving,
-and streaming configuration.
+This directory contains a test suite for recovery and replication.
 
 Running the tests
 =================
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
index 06c3ff4..67bb7df 100644
--- a/src/test/recovery/t/002_archiving.pl
+++ b/src/test/recovery/t/002_archiving.pl
@@ -1,4 +1,4 @@
-# test for archiving with warm standby
+# test for archiving with hot standby
 use strict;
 use warnings;
 use PostgresNode;
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
index 573f40a..58bf580 100644
--- a/src/test/recovery/t/004_timeline_switch.pl
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -1,5 +1,5 @@
 # Test for timeline switch
-# Ensure that a standby is able to follow a newly-promoted standby
+# Ensure that a cascading standby is able to follow a newly-promoted standby
 # on a new timeline.
 use strict;
 use warnings;
diff --git a/src/tools/msvc/clean.bat b/src/tools/msvc/clean.bat
index 58357f8..3491344 100755
--- a/src/tools/msvc/clean.bat
+++ b/src/tools/msvc/clean.bat
@@ -97,6 +97,7 @@ if exist src\bin\pg_ctl\tmp_check rd /s /q src\bin\pg_ctl\tmp_check
 if exist src\bin\pg_rewind\tmp_check rd /s /q src\bin\pg_rewind\tmp_check
 if exist src\bin\pgbench\tmp_check rd /s /q src\bin\pgbench\tmp_check
 if exist src\bin\scripts\tmp_check rd /s /q src\bin\scripts\tmp_check
+if exist src\test\recovery\tmp_check rd /s /q src\test\recovery\tmp_check
 
 REM Clean up datafiles built with contrib
 REM cd contrib
#146Craig Ringer
craig@2ndquadrant.com
In reply to: Michael Paquier (#143)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On 27 February 2016 at 06:37, Michael Paquier <michael.paquier@gmail.com>
wrote:

On Sat, Feb 27, 2016 at 4:30 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Craig Ringer wrote:

Should be committed ASAP IMO.

Finally pushed it. Let's see how it does in the buildfarm. Now let's
get going and add more tests, I know there's no shortage of people with
test scripts waiting for this.

Thanks, Michael, for the persistency, and thanks to all reviewers. (I'm
sorry we seem to have lost Amir Rohan in the process. He was doing
a great job.)

Date of first message of this thread: Mon, 2 Dec 2013 15:40:41 +0900
Date of this message: Fri, 26 Feb 2016 16:30:08 -0300
This has been a long trip. Thanks a lot to all involved. Many people
have reviewed and helped out with this patch.

Congratulations and thanks.

I don't see any new buildfarm failures. The BinInstallCheck failure on
Windows predates this and the isolationtest failure on OpenBSD is unrelated.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

#147Michael Paquier
michael.paquier@gmail.com
In reply to: Craig Ringer (#146)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sun, Feb 28, 2016 at 11:22 PM, Craig Ringer <craig@2ndquadrant.com> wrote:

On 27 February 2016 at 06:37, Michael Paquier <michael.paquier@gmail.com>
wrote:

On Sat, Feb 27, 2016 at 4:30 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Craig Ringer wrote:

Should be committed ASAP IMO.

Finally pushed it. Let's see how it does in the buildfarm. Now let's
get going and add more tests, I know there's no shortage of people with
test scripts waiting for this.

Thanks, Michael, for the persistency, and thanks to all reviewers. (I'm
sorry we seem to have lost Amir Rohan in the process. He was doing
a great job.)

Date of first message of this thread: Mon, 2 Dec 2013 15:40:41 +0900
Date of this message: Fri, 26 Feb 2016 16:30:08 -0300
This has been a long trip. Thanks a lot to all involved. Many people
have reviewed and helped out with this patch.

Congratulations and thanks.

I don't see any new buildfarm failures. The BinInstallCheck failure on
Windows predates this and the isolationtest failure on OpenBSD is unrelated.

The buildfarm does not have infrastructure to test that yet.. I need
to craft a patch for the client-side code and send it to Andrew. Will
try to do so today.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#148Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#147)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On Mon, Feb 29, 2016 at 7:40 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

The buildfarm does not have infrastructure to test that yet.. I need
to craft a patch for the client-side code and send it to Andrew. Will
try to do so today.

For those interested, here is where things are going to happen:
https://github.com/PGBuildFarm/client-code/pull/7
(This patch could be more refactored, I am not sure how much Andrew
would like things to be less duplicated, so I went to the most simple
solution).
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#149Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#145)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

9) I have no logical explanation to explain why I am seeing all those
things now.

Happens all the time ...

Pushed, thanks.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#150Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#149)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On Tue, Mar 1, 2016 at 6:25 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

Michael Paquier wrote:

9) I have no logical explanation to explain why I am seeing all those
things now.

Happens all the time ...

Pushed, thanks.

Thanks. I am going to patch by buildfarm scripts to run those tests on
hamster. Let's see what happens for this machine.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#151Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#150)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On Tue, Mar 1, 2016 at 11:28 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Tue, Mar 1, 2016 at 6:25 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

Michael Paquier wrote:

9) I have no logical explanation to explain why I am seeing all those
things now.

Happens all the time ...

Pushed, thanks.

Thanks. I am going to patch by buildfarm scripts to run those tests on
hamster. Let's see what happens for this machine.

It looks like it works fine (see step recovery-check):
http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hamster&amp;dt=2016-03-01%2002%3A34%3A26
http://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=hamster&amp;dt=2016-03-01%2002%3A34%3A26&amp;stg=recovery-check
Let's that run for a couple of days..
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#152Kyotaro HORIGUCHI
horiguchi.kyotaro@lab.ntt.co.jp
In reply to: Alvaro Herrera (#141)
2 attachment(s)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Hello,

At Fri, 26 Feb 2016 15:43:14 -0300, Alvaro Herrera <alvherre@2ndquadrant.com> wrote in <20160226184314.GA205945@alvherre.pgsql>

Kyotaro HORIGUCHI wrote:

So, I'd like to propose four (or five) changes to this harness.

- prove_check to remove all in tmp_check

- TestLib to preserve temporary directories/files if the current
test fails.

- PostgresNode::get_new_node to create data directory with
meaningful basenames.

- PostgresNode::psql to return a list of ($stdout, $stderr) if
requested. (The previous behavior is not changed)

- (recovery/t/00x_* gives test number to node name)

As a POC, the attached diff will appliy on the 0001 and (fixed)
0003 patches.

These changes all seem very reasonable to me. I'm not so sure about the
last one. Perhaps the framework ought to generate an appropriate subdir
name using the test file name plus the node name, so that instead of
tmp_XXXX it becomes tmp_001_master_XXXX or something like that? Having
be a coding convention doesn't look real nice to me.

Thank you for mentioning this.

Sorry, the last one accidentally contained garbage code to
intentionally raise an error. The last one names the nodes as
such like '001_master_24f8'. Maybe prefixing "tmp_" would be
better.

I didn't try to apply your patch but I'm fairly certain it would
conflict with what's here now; can you please rebase and resend?

This was a very small patch made on the old, uncommited
patches. I remade the patch and split it into two parts.

0001-Change-behavior-...

Changes of PostmasterNode.pm and TestLib.pm to add some
features and change a behavior.

0002-Prefix-test-numbers-to-node-

This is rather a example usage of 0001- patch (except for
stderr stuff). 00n_xxx test leaves temporary directories with
the names of 00n_(master|standby)_XXXX on failure. If this is
considered reasonable, I'll make same patches for the other
/t/nnn_*.pl tests.

regards,

--
Kyotaro Horiguchi
NTT Open Source Software Center

Attachments:

0001-Change-behavior-of-PostgresNode.pm-and-TestLib.pm.patchtext/x-patch; charset=us-asciiDownload
From 4e2c5123e249ef3278da8d5260bbe1e4799988b2 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horiguchi.kyotaro@lab.ntt.co.jp>
Date: Tue, 1 Mar 2016 16:09:25 +0900
Subject: [PATCH 1/2] Change behavior of PostgresNode.pm and TestLib.pm

Modify PostgresNode.new to use a temporary directory with the name
after the node name instead of tmp_dirXXXX. Modify PostgresNode.psql
to allow callers to get stderr output. Make TestLib to preserve the
temporary directory if a test has been failed. Modify TestLib.tempdir
to allow callers to prefix the tempdir name with an arbitrary prefix.
---
 src/test/perl/PostgresNode.pm |  9 +++++++--
 src/test/perl/TestLib.pm      | 10 +++++++++-
 2 files changed, 16 insertions(+), 3 deletions(-)

diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index a8e6f0c..ad27361 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -115,7 +115,7 @@ sub new
 	my $self   = {
 		_port     => $pgport,
 		_host     => $pghost,
-		_basedir  => TestLib::tempdir,
+		_basedir  => TestLib::tempdir($name),
 		_name     => $name,
 		_logfile  => "$TestLib::log_path/${testname}_${name}.log" };
 
@@ -805,10 +805,15 @@ sub psql
 		print "#### Begin standard error\n";
 		print $stderr;
 		print "#### End standard error\n";
+		if (wantarray)
+		{
+			chomp $stderr;
+			$stderr =~ s/\r//g if $Config{osname} eq 'msys';
+		}
 	}
 	chomp $stdout;
 	$stdout =~ s/\r//g if $Config{osname} eq 'msys';
-	return $stdout;
+	return wantarray ? ($stdout, $stderr) : $stdout;
 }
 
 =pod
diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 3d11cbb..da0e617 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -107,13 +107,21 @@ INIT
 	autoflush TESTLOG 1;
 }
 
+END
+{
+	# Preserve temporary directory for this test if failure
+	$File::Temp::KEEP_ALL = 1 unless Test::More->builder->is_passing;
+}
+
 #
 # Helper functions
 #
 sub tempdir
 {
+	my ($prefix) = @_;
+	$prefix = "tmp_test" if (!$prefix);
 	return File::Temp::tempdir(
-		'tmp_testXXXX',
+		$prefix.'_XXXX',
 		DIR => $tmp_check,
 		CLEANUP => 1);
 }
-- 
1.8.3.1

0002-Prefix-test-numbers-to-node-directories.patchtext/x-patch; charset=us-asciiDownload
From fab2f28637a4392ef7e0bfbdf9bcac51f5fb5fd7 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horiguchi.kyotaro@lab.ntt.co.jp>
Date: Tue, 1 Mar 2016 16:50:24 +0900
Subject: [PATCH 2/2] Prefix test numbers to node directories

Add prefix numbers to node names to make the replication tests leave
directories with reasonable names on failure.
---
 src/test/recovery/t/001_stream_rep.pl       | 6 +++---
 src/test/recovery/t/002_archiving.pl        | 4 ++--
 src/test/recovery/t/003_recovery_targets.pl | 4 ++--
 src/test/recovery/t/004_timeline_switch.pl  | 6 +++---
 src/test/recovery/t/005_replay_delay.pl     | 4 ++--
 5 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index 7dcca65..6f2d13b 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -6,7 +6,7 @@ use TestLib;
 use Test::More tests => 4;
 
 # Initialize master node
-my $node_master = get_new_node('master');
+my $node_master = get_new_node('001_master');
 $node_master->init(allows_streaming => 1);
 $node_master->start;
 my $backup_name = 'my_backup';
@@ -15,7 +15,7 @@ my $backup_name = 'my_backup';
 $node_master->backup($backup_name);
 
 # Create streaming standby linking to master
-my $node_standby_1 = get_new_node('standby_1');
+my $node_standby_1 = get_new_node('001_standby_1');
 $node_standby_1->init_from_backup($node_master, $backup_name,
 	has_streaming => 1);
 $node_standby_1->start;
@@ -25,7 +25,7 @@ $node_standby_1->start;
 $node_standby_1->backup($backup_name);
 
 # Create second standby node linking to standby 1
-my $node_standby_2 = get_new_node('standby_2');
+my $node_standby_2 = get_new_node('001_standby_2');
 $node_standby_2->init_from_backup($node_standby_1, $backup_name,
 	has_streaming => 1);
 $node_standby_2->start;
diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl
index 67bb7df..5dafcf2 100644
--- a/src/test/recovery/t/002_archiving.pl
+++ b/src/test/recovery/t/002_archiving.pl
@@ -7,7 +7,7 @@ use Test::More tests => 1;
 use File::Copy;
 
 # Initialize master node, doing archives
-my $node_master = get_new_node('master');
+my $node_master = get_new_node('002_master');
 $node_master->init(
 	has_archiving    => 1,
 	allows_streaming => 1);
@@ -20,7 +20,7 @@ $node_master->start;
 $node_master->backup($backup_name);
 
 # Initialize standby node from backup, fetching WAL from archives
-my $node_standby = get_new_node('standby');
+my $node_standby = get_new_node('002_standby');
 $node_standby->init_from_backup($node_master, $backup_name,
 	has_restoring => 1);
 $node_standby->append_conf(
diff --git a/src/test/recovery/t/003_recovery_targets.pl b/src/test/recovery/t/003_recovery_targets.pl
index 8b581cc..8552e5b 100644
--- a/src/test/recovery/t/003_recovery_targets.pl
+++ b/src/test/recovery/t/003_recovery_targets.pl
@@ -16,7 +16,7 @@ sub test_recovery_standby
 	my $num_rows        = shift;
 	my $until_lsn       = shift;
 
-	my $node_standby = get_new_node($node_name);
+	my $node_standby = get_new_node("003_".$node_name);
 	$node_standby->init_from_backup($node_master, 'my_backup',
 		has_restoring => 1);
 
@@ -46,7 +46,7 @@ sub test_recovery_standby
 }
 
 # Initialize master node
-my $node_master = get_new_node('master');
+my $node_master = get_new_node('003_master');
 $node_master->init(has_archiving => 1, allows_streaming => 1);
 
 # Start it
diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
index 58bf580..fc5399d 100644
--- a/src/test/recovery/t/004_timeline_switch.pl
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -11,7 +11,7 @@ use Test::More tests => 1;
 $ENV{PGDATABASE} = 'postgres';
 
 # Initialize master node
-my $node_master = get_new_node('master');
+my $node_master = get_new_node('004_master');
 $node_master->init(allows_streaming => 1);
 $node_master->start;
 
@@ -20,11 +20,11 @@ my $backup_name = 'my_backup';
 $node_master->backup($backup_name);
 
 # Create two standbys linking to it
-my $node_standby_1 = get_new_node('standby_1');
+my $node_standby_1 = get_new_node('004_standby_1');
 $node_standby_1->init_from_backup($node_master, $backup_name,
 	has_streaming => 1);
 $node_standby_1->start;
-my $node_standby_2 = get_new_node('standby_2');
+my $node_standby_2 = get_new_node('004_standby_2');
 $node_standby_2->init_from_backup($node_master, $backup_name,
 	has_streaming => 1);
 $node_standby_2->start;
diff --git a/src/test/recovery/t/005_replay_delay.pl b/src/test/recovery/t/005_replay_delay.pl
index 401d17b..b04fbdb 100644
--- a/src/test/recovery/t/005_replay_delay.pl
+++ b/src/test/recovery/t/005_replay_delay.pl
@@ -6,7 +6,7 @@ use TestLib;
 use Test::More tests => 2;
 
 # Initialize master node
-my $node_master = get_new_node('master');
+my $node_master = get_new_node('005_master');
 $node_master->init(allows_streaming => 1);
 $node_master->start;
 
@@ -19,7 +19,7 @@ my $backup_name = 'my_backup';
 $node_master->backup($backup_name);
 
 # Create streaming standby from backup
-my $node_standby = get_new_node('standby');
+my $node_standby = get_new_node('005_standby');
 $node_standby->init_from_backup($node_master, $backup_name,
 	has_streaming => 1);
 $node_standby->append_conf(
-- 
1.8.3.1

#153Michael Paquier
michael.paquier@gmail.com
In reply to: Kyotaro HORIGUCHI (#152)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On Tue, Mar 1, 2016 at 5:13 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:

0001-Change-behavior-...

Changes of PostmasterNode.pm and TestLib.pm to add some
features and change a behavior.

+   # Preserve temporary directory for this test if failure
+   $File::Temp::KEEP_ALL = 1 unless Test::More->builder->is_passing;
+1. Having the data folders being removed even in case of a failure is
really annoying.
+   # Preserve temporary directory for this test if failure
+   $File::Temp::KEEP_ALL = 1 unless Test::More->builder->is_passing;
s/if failure/in the event of a failure/?

+ return wantarray ? ($stdout, $stderr) : $stdout;
So you are willing to extend that so as you could perform conparison
tests on the error strings returned. Why no, it looks useful, though
now there is no test in need of it I think. So without a proper need I
think that we could live without.

0002-Prefix-test-numbers-to-node-

This is rather a example usage of 0001- patch (except for
stderr stuff). 00n_xxx test leaves temporary directories with
the names of 00n_(master|standby)_XXXX on failure. If this is
considered reasonable, I'll make same patches for the other
/t/nnn_*.pl tests.

-my $node_master = get_new_node('master');
+my $node_master = get_new_node('001_master');
I am not a fan of appending the test number in the node name. For one,
this complicates the log file name associated with a node by
duplicating the test number in its name. Also, it is possible to
easily get the name of the data folder for a node by looking at the
logs.
-- 
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#154Michael Paquier
michael.paquier@gmail.com
In reply to: Alvaro Herrera (#140)
Re: [REVIEW] In-core regression tests for replication, cascading, archiving, PITR, etc.

On Sat, Feb 27, 2016 at 1:02 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

Michael Paquier wrote:

Attached are rebased patches, split into 3 parts doing the following:
- 0001, fix default configuration of MSVC builds ignoring TAP tests

BTW you keep submitting this one and I keep ignoring it. I think you
should start a separate thread for this one, so that some
Windows-enabled committer can look at it and maybe push it. I still
don't understand why this matters.

OK. I will create a separate thread on hackers. I think that this is
still a bug.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#155Craig Ringer
craig@2ndquadrant.com
In reply to: Michael Paquier (#153)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 1 March 2016 at 20:45, Michael Paquier <michael.paquier@gmail.com> wrote:

On Tue, Mar 1, 2016 at 5:13 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:

0001-Change-behavior-...

Changes of PostmasterNode.pm and TestLib.pm to add some
features and change a behavior.

+   # Preserve temporary directory for this test if failure
+   $File::Temp::KEEP_ALL = 1 unless Test::More->builder->is_passing;
+1. Having the data folders being removed even in case of a failure is
really annoying.

I agree on all points re your review. Keeping tempdirs is really needed,
the tempdir name change is good, the the rest I'm not keen on.

I've addressed the need to get stderr from psql in a patch I'll submit
separately, which provides a thinner wrapper around IPC::Run for more
complex needs, then uses that for the existing 'psql' function. It also
provides a 'psql_check' that dies on error.

I'll incorporate the wanted changes into that patch.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

#156Craig Ringer
craig@2ndquadrant.com
In reply to: Craig Ringer (#155)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

On 1 March 2016 at 21:05, Craig Ringer <craig@2ndquadrant.com> wrote:

On 1 March 2016 at 20:45, Michael Paquier <michael.paquier@gmail.com>
wrote:

On Tue, Mar 1, 2016 at 5:13 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:

0001-Change-behavior-...

Changes of PostmasterNode.pm and TestLib.pm to add some
features and change a behavior.

+   # Preserve temporary directory for this test if failure
+   $File::Temp::KEEP_ALL = 1 unless Test::More->builder->is_passing;
+1. Having the data folders being removed even in case of a failure is
really annoying.

I agree on all points re your review. Keeping tempdirs is really needed,
the tempdir name change is good, the the rest I'm not keen on.

I've addressed the need to get stderr from psql in a patch I'll submit
separately, which provides a thinner wrapper around IPC::Run for more
complex needs, then uses that for the existing 'psql' function. It also
provides a 'psql_check' that dies on error.

I'll incorporate the wanted changes into that patch.

OK, done.

https://commitfest.postgresql.org/9/569/#

Michael, I'd value your thoughts on the patches. I think the psql one is
important, I found the existing 'psql' function too limiting and I think
defaulting to ignoring errors is sub-optimal.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

#157Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#153)
Re: Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

Michael Paquier wrote:

On Tue, Mar 1, 2016 at 5:13 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:

+ return wantarray ? ($stdout, $stderr) : $stdout;
So you are willing to extend that so as you could perform conparison
tests on the error strings returned. Why no, it looks useful, though
now there is no test in need of it I think. So without a proper need I
think that we could live without.

Does this change let us implement psql_ok and psql_fail? I think I've
seen a few places already, both in committed code and in submitted
patches, that test for some kind of failure from psql.

0002-Prefix-test-numbers-to-node-

This is rather a example usage of 0001- patch (except for
stderr stuff). 00n_xxx test leaves temporary directories with
the names of 00n_(master|standby)_XXXX on failure. If this is
considered reasonable, I'll make same patches for the other
/t/nnn_*.pl tests.

-my $node_master = get_new_node('master');
+my $node_master = get_new_node('001_master');
I am not a fan of appending the test number in the node name. For one,
this complicates the log file name associated with a node by
duplicating the test number in its name. Also, it is possible to
easily get the name of the data folder for a node by looking at the
logs.

Why don't we use something similar to what's in $test_logfile in
TestLib?

Also, it is possible to easily get the name of the data folder for a
node by looking at the logs.

No disagreement on it being possible, but "easily" seems a bad
description for that.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers