In-core regression tests for replication, cascading, archiving, PITR, etc.
Hi all,
The data replication bug causing data corruption on hot slaves found
lately (http://wiki.postgresql.org/wiki/Nov2013ReplicationIssue) is
causing a certain amount of damage among the users of Postgres, either
companies or individuals, and impacts a lot of people. So perhaps it
would be a good time to start thinking about adding some dedicated
regression tests for replication, archiving, PITR, data integrity,
parameter reloading, etc. The possible number of things or use-cases
that could be applied to that is very vast, but let's say that it
involves the manipulation of multiple Postgres nodes and structures
(like a WAL archive) defining a Postgres cluster.
The main purpose of those test would be really simple: using a given
GIT repository, a buildfarm or developer should be able to run a
single "make check" command that runs those tests on a local machine
in a fashion similar to isolation or regression tests and validate
builds or patches.
I imagine that there would be roughly two ways to implement such a facility:
1) Use of a smart set of bash scripts. This would be easy to implement
but reduces pluggability of custom scripts (I am sure that each
user/company has already its own set of scenarios.). pg_upgrade uses
something similar with its test.sh.
2) Use a scripting language, in a way similar to how isolation tests
are done. This would make custom test more customizable.
Here is for example an approach that has been presented at the
unconference of PGcon 2013 (would be something different though as
this proposal does not include node manipulation *during* the tests
like promotion):
https://wiki.postgresql.org/images/1/14/Pg_testframework.pptx
3) Import (and improve) solutions that other projects based on
Postgres technology use for those things.
In all cases, here are the common primary actions that could be run for a test:
- Define and perform actions on a node: init, start, stop, promote,
create_archive, base_backup. So it is a sort of improved wrapper of
pg_ctl.
- Pass parameters to a configuration file, either postgresql.conf,
recovery.conf, or anything.
- Launch SQL commands to a node.
Things like the creation of folders for WAL archiving should be simply
harcoded to simplify the life of developer... As well, the facility
should be smart enough to be allow the use of custom commands that are
the combination of primary actions above, like for example the
possibility to define a sync slave, linked to a root node, is simply
1) create a base backup from a node, 2) pass parameters to
postgresql.conf and recovery.conf, 3) start the node.
Let me know your thoughts.
Regards,
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 12/02/2013 08:40 AM, Michael Paquier wrote:
The data replication bug causing data corruption on hot slaves found
lately (http://wiki.postgresql.org/wiki/Nov2013ReplicationIssue) is
causing a certain amount of damage among the users of Postgres, either
companies or individuals, and impacts a lot of people. So perhaps it
would be a good time to start thinking about adding some dedicated
regression tests for replication, archiving, PITR, data integrity,
parameter reloading, etc. The possible number of things or use-cases
that could be applied to that is very vast, but let's say that it
involves the manipulation of multiple Postgres nodes and structures
(like a WAL archive) defining a Postgres cluster.The main purpose of those test would be really simple: using a given
GIT repository, a buildfarm or developer should be able to run a
single "make check" command that runs those tests on a local machine
in a fashion similar to isolation or regression tests and validate
builds or patches.
+1. The need for such a test suite has been mentioned every single time
that a bug or new feature related to replication, PITR or hot standby
has come up. So yes please! The only thing missing is someone to
actually write the thing. So if you have the time and energy, that'd be
great!
- Heikki
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Dec 2, 2013 at 6:24 PM, Heikki Linnakangas
<hlinnakangas@vmware.com> wrote:
+1. The need for such a test suite has been mentioned every single time that
a bug or new feature related to replication, PITR or hot standby has come
up. So yes please! The only thing missing is someone to actually write the
thing. So if you have the time and energy, that'd be great!
I am sure you know who we need to convince in this case :)
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2013-12-02 18:45:37 +0900, Michael Paquier wrote:
On Mon, Dec 2, 2013 at 6:24 PM, Heikki Linnakangas
<hlinnakangas@vmware.com> wrote:+1. The need for such a test suite has been mentioned every single time that
a bug or new feature related to replication, PITR or hot standby has come
up. So yes please! The only thing missing is someone to actually write the
thing. So if you have the time and energy, that'd be great!
I am sure you know who we need to convince in this case :)
If you're alluding to Tom, I'd guess he doesn't need to be convinced of
such a facility in general. I seem to remember him complaining about the
lack of testing that as well.
Maybe that it shouldn't be part of the main regression schedule...
+many from me as well. I think the big battle will be how to do it, not
if in general.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Dec 2, 2013 at 7:07 PM, Andres Freund <andres@2ndquadrant.com> wrote:
Maybe that it shouldn't be part of the main regression schedule...
Yes, like isolation tests, it don't see those new tests in the main
flow as well.
+many from me as well. I think the big battle will be how to do it, not
if in general.
Yeah, that's why we should gather first feedback about the methods
that other projects (Slony, Londiste, Pgpool) are using as well before
heading to a solution or another. Having smth, whatever small for 9.4
would also be of great help.
I am however sure that getting a small prototype integrated with some
of my in-house scripts would not take that much time though...
Regards,
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund <andres@2ndquadrant.com> writes:
Maybe that it shouldn't be part of the main regression schedule...
It *can't* be part of the main regression tests; those are supposed to
be runnable against an already-installed server, and fooling with that
server's configuration is off-limits too. But I agree that some
other facility to simplify running tests like this would be handy.
At the same time, I'm pretty skeptical that any simple regression-test
type facility would have caught the bugs we've fixed lately ...
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2013-12-02 09:41:39 -0500, Tom Lane wrote:
At the same time, I'm pretty skeptical that any simple regression-test
type facility would have caught the bugs we've fixed lately ...
Agreed, but it would make reorganizing stuff to be more robust more
realistic. At the moment for everything you change you have to hand-test
everyting possibly affected which takes ages.
I think we also needs support for testing xid/multixid wraparound. It
currently isn't realistically testable because of the timeframes
involved.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund <andres@2ndquadrant.com> writes:
I think we also needs support for testing xid/multixid wraparound. It
currently isn't realistically testable because of the timeframes
involved.
When I've wanted to do that in the past, I've used pg_resetxlog to
adjust a cluster's counters. It still requires some manual hacking
though because pg_resetxlog isn't bright enough to create the new
pg_clog files needed when you move the xid counter a long way.
We could fix that, or we could make the backend more forgiving of
not finding the initial clog segment present at startup ...
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Tom Lane escribi�:
When I've wanted to do that in the past, I've used pg_resetxlog to
adjust a cluster's counters. It still requires some manual hacking
though because pg_resetxlog isn't bright enough to create the new
pg_clog files needed when you move the xid counter a long way.
We could fix that, or we could make the backend more forgiving of
not finding the initial clog segment present at startup ...
FWIW we already have some new code that creates segments when not found.
It's currently used in multixact, and the submitted "commit timestamp"
module uses it too.
--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2013-12-02 09:59:12 -0500, Tom Lane wrote:
Andres Freund <andres@2ndquadrant.com> writes:
I think we also needs support for testing xid/multixid wraparound. It
currently isn't realistically testable because of the timeframes
involved.When I've wanted to do that in the past, I've used pg_resetxlog to
adjust a cluster's counters.
I've done that as well, but it's painful and not neccessarily testing
the right thing. E.g. I am far from sure we handle setting the
anti-wraparound limits correctly when promoting a standby - a restart to
adapt pg_control changes things and it might get rolled back because of
a already logged checkpoints.
What I'd love is a function that gives me the opportunity to
*efficiently* move forward pg_clog, pg_multixact/offset,members by large
chunks. So e.g. I could run a normal pgbench alongside another pgbench
moving clog forward in 500k chunks, but so it creates the necessary
files I could possibly need to access.
If you do it naivly you get into quite some fun with hot standby btw. I
can tell you that from experience :P
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Dec 2, 2013 at 11:41 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Andres Freund <andres@2ndquadrant.com> writes:
At the same time, I'm pretty skeptical that any simple regression-test
type facility would have caught the bugs we've fixed lately ...
The replication bug would have been reproducible at least, Heikki
produced a simple test case able to reproduce it. For the MultiXact
stuff... well some more infrastructure in core might be needed before
having a wrapper calling test scripts aimed to manipulate cluster of
nodes.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Dec 2, 2013 at 7:07 PM, Andres Freund <andres@2ndquadrant.com> wrote:
On 2013-12-02 18:45:37 +0900, Michael Paquier wrote:
On Mon, Dec 2, 2013 at 6:24 PM, Heikki Linnakangas
<hlinnakangas@vmware.com> wrote:+1. The need for such a test suite has been mentioned every single time that
a bug or new feature related to replication, PITR or hot standby has come
up. So yes please! The only thing missing is someone to actually write the
thing. So if you have the time and energy, that'd be great!I am sure you know who we need to convince in this case :)
If you're alluding to Tom, I'd guess he doesn't need to be convinced of
such a facility in general. I seem to remember him complaining about the
lack of testing that as well.
Maybe that it shouldn't be part of the main regression schedule...+many from me as well. I think the big battle will be how to do it, not
if in general.
(Reviving an old thread)
So I am planning to seriously focus soon on this stuff, basically
using the TAP tests as base infrastructure for this regression test
suite. First, does using the TAP tests sound fine?
On the top of my mind I got the following items that should be tested:
- WAL replay: from archive, from stream
- hot standby and read-only queries
- node promotion
- recovery targets and their interferences when multiple targets are
specified (XID, name, timestamp, immediate)
- timelines
- recovery_target_action
- recovery_min_apply_delay (check that WAL is fetch from a source at
some correct interval, can use a special restore_command for that)
- archive_cleanup_command (check that command is kicked at each restart point)
- recovery_end_command (check that command is kicked at the end of recovery)
- timeline jump of a standby after reconnecting to a promoted node
Regards,
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 3/8/15 6:19 AM, Michael Paquier wrote:
On Mon, Dec 2, 2013 at 7:07 PM, Andres Freund <andres@2ndquadrant.com> wrote:
On 2013-12-02 18:45:37 +0900, Michael Paquier wrote:
On Mon, Dec 2, 2013 at 6:24 PM, Heikki Linnakangas
<hlinnakangas@vmware.com> wrote:+1. The need for such a test suite has been mentioned every single time that
a bug or new feature related to replication, PITR or hot standby has come
up. So yes please! The only thing missing is someone to actually write the
thing. So if you have the time and energy, that'd be great!I am sure you know who we need to convince in this case :)
If you're alluding to Tom, I'd guess he doesn't need to be convinced of
such a facility in general. I seem to remember him complaining about the
lack of testing that as well.
Maybe that it shouldn't be part of the main regression schedule...+many from me as well. I think the big battle will be how to do it, not
if in general.(Reviving an old thread)
So I am planning to seriously focus soon on this stuff, basically
using the TAP tests as base infrastructure for this regression test
suite. First, does using the TAP tests sound fine?On the top of my mind I got the following items that should be tested:
- WAL replay: from archive, from stream
- hot standby and read-only queries
- node promotion
- recovery targets and their interferences when multiple targets are
specified (XID, name, timestamp, immediate)
- timelines
- recovery_target_action
- recovery_min_apply_delay (check that WAL is fetch from a source at
some correct interval, can use a special restore_command for that)
- archive_cleanup_command (check that command is kicked at each restart point)
- recovery_end_command (check that command is kicked at the end of recovery)
- timeline jump of a standby after reconnecting to a promoted node
If we're keeping a list, there's also hot_standby_feedback,
max_standby_archive_delay and max_standby_streaming_delay.
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sun, Mar 08, 2015 at 08:19:39PM +0900, Michael Paquier wrote:
So I am planning to seriously focus soon on this stuff, basically
using the TAP tests as base infrastructure for this regression test
suite. First, does using the TAP tests sound fine?
Yes.
On the top of my mind I got the following items that should be tested:
- WAL replay: from archive, from stream
- hot standby and read-only queries
- node promotion
- recovery targets and their interferences when multiple targets are
specified (XID, name, timestamp, immediate)
- timelines
- recovery_target_action
- recovery_min_apply_delay (check that WAL is fetch from a source at
some correct interval, can use a special restore_command for that)
- archive_cleanup_command (check that command is kicked at each restart point)
- recovery_end_command (check that command is kicked at the end of recovery)
- timeline jump of a standby after reconnecting to a promoted node
Those sound good. The TAP suites still lack support for any Windows target.
If you're inclined to fix that, it would be a great contribution. The more we
accrue tests before doing that, the harder it will be to dig out.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, Mar 11, 2015 at 2:47 PM, Noah Misch <noah@leadboat.com> wrote:
On Sun, Mar 08, 2015 at 08:19:39PM +0900, Michael Paquier wrote:
So I am planning to seriously focus soon on this stuff, basically
using the TAP tests as base infrastructure for this regression test
suite. First, does using the TAP tests sound fine?Yes.
Check.
On the top of my mind I got the following items that should be tested:
- WAL replay: from archive, from stream
- hot standby and read-only queries
- node promotion
- recovery targets and their interferences when multiple targets are
specified (XID, name, timestamp, immediate)
- timelines
- recovery_target_action
- recovery_min_apply_delay (check that WAL is fetch from a source at
some correct interval, can use a special restore_command for that)
- archive_cleanup_command (check that command is kicked at each restart point)
- recovery_end_command (check that command is kicked at the end of recovery)
- timeline jump of a standby after reconnecting to a promoted nodeThose sound good. The TAP suites still lack support for any Windows target.
If you're inclined to fix that, it would be a great contribution. The more we
accrue tests before doing that, the harder it will be to dig out.
Yeah, that's already on my TODO list.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, Mar 11, 2015 at 3:04 PM, Michael Paquier wrote:
On Wed, Mar 11, 2015 at 2:47 PM, Noah Misch <noah@leadboat.com> wrote:
On Sun, Mar 08, 2015 at 08:19:39PM +0900, Michael Paquier wrote:
So I am planning to seriously focus soon on this stuff, basically
using the TAP tests as base infrastructure for this regression test
suite. First, does using the TAP tests sound fine?Yes.
Check.
On the top of my mind I got the following items that should be tested:
- WAL replay: from archive, from stream
- hot standby and read-only queries
- node promotion
- recovery targets and their interferences when multiple targets are
specified (XID, name, timestamp, immediate)
- timelines
- recovery_target_action
- recovery_min_apply_delay (check that WAL is fetch from a source at
some correct interval, can use a special restore_command for that)
- archive_cleanup_command (check that command is kicked at each restart point)
- recovery_end_command (check that command is kicked at the end of recovery)
- timeline jump of a standby after reconnecting to a promoted node
So, as long as I had a clear picture of what I wanted to do regarding
this stuff (even if this is a busy commit fest, sorry), I have been
toying around with perl and I have finished with the patch attached,
adding some base structure for a new test suite covering recovery.
This patch includes basic tests for the following items:
- node promotion, test of archiving, streaming, replication cascading
- recovery targets XID, name, timestamp, immediate and PITR
- Timeline jump of a standby when reconnecting to a newly-promoted standby
- Replay delay
Tests are located in src/test/recovery, and are not part of the main
test suite, similarly to the ssl stuff.
I have dropped recovery_target_action for the time being as long as
the matter on the other thread is not set
(/messages/by-id/20150315132707.GB19792@alap3.anarazel.de),
and I don't think that it would be complicated to create tests for
that btw.
The most important part of this patch is not the tests themselves, but
the base set of routines allowing to simply create nodes, take
backups, create standbys from backups, and set up nodes to do stuff
like streaming, archiving, or restoring from archives. There are many
configurations possible of course in recovery.conf, but the set of
routines that this patch present are made to be *simple* to not
overcomplicate the way tests can be written.
Feedback is of course welcome, but note that I am not seriously
expecting any until we get into 9.6 development cycle and I am adding
this patch to the next CF.
Regards,
--
Michael
Attachments:
20150318_recovery_regressions.patchtext/x-patch; charset=US-ASCII; name=20150318_recovery_regressions.patchDownload+759-1
On Wed, Mar 18, 2015 at 1:59 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
Feedback is of course welcome, but note that I am not seriously
expecting any until we get into 9.6 development cycle and I am adding
this patch to the next CF.
I have moved this patch to CF 2015-09, as I have enough patches to
take care of for now... Let's focus on Windows support and improvement
of logging for TAP in the first round. That will be already a good
step forward.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Jun 29, 2015 at 10:11 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
On Wed, Mar 18, 2015 at 1:59 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:Feedback is of course welcome, but note that I am not seriously
expecting any until we get into 9.6 development cycle and I am adding
this patch to the next CF.I have moved this patch to CF 2015-09, as I have enough patches to
take care of for now... Let's focus on Windows support and improvement
of logging for TAP in the first round. That will be already a good
step forward.
OK, attached is a new version of this patch, that I have largely
reworked to have more user-friendly routines for the tests. The number
of tests is still limited still it shows what this facility can do:
that's on purpose as it does not make much sense to code a complete
and complicated set of tests as long as the core routines are not
stable, hence let's focus on that first.
I have not done yet tests on Windows, I am expecting some tricks
needed for the archive and recovery commands generated for the tests.
Regards,
--
Michael
Attachments:
20150814_recovery_regressions.patchtext/x-diff; charset=US-ASCII; name=20150814_recovery_regressions.patchDownload+748-34
On Fri, Aug 14, 2015 at 12:54 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:
On Mon, Jun 29, 2015 at 10:11 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:On Wed, Mar 18, 2015 at 1:59 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:Feedback is of course welcome, but note that I am not seriously
expecting any until we get into 9.6 development cycle and I am adding
this patch to the next CF.I have moved this patch to CF 2015-09, as I have enough patches to
take care of for now... Let's focus on Windows support and improvement
of logging for TAP in the first round. That will be already a good
step forward.OK, attached is a new version of this patch, that I have largely
reworked to have more user-friendly routines for the tests. The number
of tests is still limited still it shows what this facility can do:
that's on purpose as it does not make much sense to code a complete
and complicated set of tests as long as the core routines are not
stable, hence let's focus on that first.
I have not done yet tests on Windows, I am expecting some tricks
needed for the archive and recovery commands generated for the tests.
Attached is v3. I have tested and fixed the tests such as they can run
on Windows. archive_command and restore_command are using Windows'
copy when needed. There was also a bug with the use of a hot standby
instead of a warm one, causing test 002 to fail.
I am rather happy with the shape of this patch now, so feel free to review it...
Regards,
--
Michael
Attachments:
20150814_recovery_regressions_v3.patchtext/x-diff; charset=US-ASCII; name=20150814_recovery_regressions_v3.patchDownload+755-34
On 08/14/2015 06:32 AM, Michael Paquier wrote:
On Fri, Aug 14, 2015 at 12:54 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:On Mon, Jun 29, 2015 at 10:11 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:On Wed, Mar 18, 2015 at 1:59 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:Feedback is of course welcome, but note that I am not seriously
expecting any until we get into 9.6 development cycle and I am adding
this patch to the next CF.I have moved this patch to CF 2015-09, as I have enough patches to
take care of for now... Let's focus on Windows support and improvement
of logging for TAP in the first round. That will be already a good
step forward.OK, attached is a new version of this patch, that I have largely
reworked to have more user-friendly routines for the tests. The number
of tests is still limited still it shows what this facility can do:
that's on purpose as it does not make much sense to code a complete
and complicated set of tests as long as the core routines are not
stable, hence let's focus on that first.
I have not done yet tests on Windows, I am expecting some tricks
needed for the archive and recovery commands generated for the tests.Attached is v3. I have tested and fixed the tests such as they can run
on Windows. archive_command and restore_command are using Windows'
copy when needed. There was also a bug with the use of a hot standby
instead of a warm one, causing test 002 to fail.
I am rather happy with the shape of this patch now, so feel free to review it...
Regards,
Michael, I've ran these and it worked fine for me.
See attached patch with a couple of minor fixes.
Amir
