Test code is worth the space
We've too often criticized patches for carrying many lines/bytes of test case
additions. Let's continue to demand debuggable, secure tests that fail only
when something is wrong, but let's stop pushing for test minimalism. Such
objections may improve the individual patch, but that doesn't make up for the
chilling effect on test contributions. I remember clearly the first time I
submitted thorough test coverage with a feature. Multiple committers attacked
it in the name of brevity. PostgreSQL would be better off with 10x its
current test bulk, even if the average line of test code were considerably
less important than we expect today.
Thanks,
nm
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sat, Aug 8, 2015 at 9:47 AM, Noah Misch <noah@leadboat.com> wrote:
We've too often criticized patches for carrying many lines/bytes of test case
additions. Let's continue to demand debuggable, secure tests that fail only
when something is wrong, but let's stop pushing for test minimalism. Such
objections may improve the individual patch, but that doesn't make up for the
chilling effect on test contributions. I remember clearly the first time I
submitted thorough test coverage with a feature. Multiple committers attacked
it in the name of brevity. PostgreSQL would be better off with 10x its
current test bulk, even if the average line of test code were considerably
less important than we expect today.
I strongly agree. I am fond of the example of external sorting, which
at present has exactly zero test coverage, even though in production
those code paths are exercised all the time.
I think that there needs to be a way of running an extended set of
regression tests. I could definitely respect the desire for minimalism
when it comes to adding tests to the regression tests proper if there
was an extended set of tests that could be run during development less
frequently. I thought about doing the extended set as a satellite
project, but that may not be workable.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 08/08/2015 12:24 PM, Peter Geoghegan wrote:
I think that there needs to be a way of running an extended set of
regression tests. I could definitely respect the desire for minimalism
when it comes to adding tests to the regression tests proper if there
was an extended set of tests that could be run during development less
frequently. I thought about doing the extended set as a satellite
project, but that may not be workable.
There already is, isn't there? All of those named sets of regression
tests which aren't run by default.
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Import Notes
Reply to msg id not found: WM88d7f86472b7b1a2da32b56c95702cad2a03e2f4be8de2d07833c4e3434c5adf3f7ad144d1c3215660e52d22bd9f4e60@asav-3.01.com
On 08/08/2015 05:09 PM, Josh Berkus wrote:
On 08/08/2015 12:24 PM, Peter Geoghegan wrote:
I think that there needs to be a way of running an extended set of
regression tests. I could definitely respect the desire for minimalism
when it comes to adding tests to the regression tests proper if there
was an extended set of tests that could be run during development less
frequently. I thought about doing the extended set as a satellite
project, but that may not be workable.There already is, isn't there? All of those named sets of regression
tests which aren't run by default.
Exactly. And there is nothing to stop us expanding those. For example,
it might be useful to have a suite that provably touched every code
path, or something close to it.
Incidentally, making the buildfarm client run extra sets of tests in the
main tree is almost completely trivial. Making it run tests from
somewhere else, such as a different git repo, is only slightly harder.
cheers
andrew
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sat, Aug 8, 2015 at 8:24 PM, Peter Geoghegan <pg@heroku.com> wrote:
I think that there needs to be a way of running an extended set of
regression tests. I could definitely respect the desire for minimalism
The larger expense in having extensive test suites is the cost to
maintain them. With our current test framework tests have to be fairly
carefully written to produce output that isn't very fragile and
sensitive to minor changes in the code. In practice this is what
really drives the push towards minimal tests. If we tried testing
every code path right now -- which I tried to do once for the TOAST
code -- what you'll find is that what you're really testing is not
that the code is correct but that it does exactly what it does today.
At that point the test failures become entirely noise meaning
"something changed" rather than signal meaning "something's broken".
A better test framework can go a long way towards fixing this. It
would be much less of a problem if we had a unit test framework rather
than only black box integration tests. That would allow us to test
that every function in tuplestore.c meets its contract, not just that
some SQL query produces output that's correct and we think happened to
go through some code path we were interested in. There are many code
paths that will be hard to arrange to reach from SQL and impossible to
have any confidence will continue to be reached in the future when the
behaviour is intentionally changed.
That said, I don't think anyone would object to adding some regression
tests that at least test basic correctness of the sorting code. That's
a pretty embarrassing gap and iirc the only reason for it is that it
would make the regression tests sensitive to collation locale
settings.
--
greg
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 8 August 2015 at 17:47, Noah Misch <noah@leadboat.com> wrote:
We've too often criticized patches for carrying many lines/bytes of test
case
additions. Let's continue to demand debuggable, secure tests that fail
only
when something is wrong, but let's stop pushing for test minimalism. Such
objections may improve the individual patch, but that doesn't make up for
the
chilling effect on test contributions. I remember clearly the first time I
submitted thorough test coverage with a feature. Multiple committers
attacked
it in the name of brevity. PostgreSQL would be better off with 10x its
current test bulk, even if the average line of test code were considerably
less important than we expect today.
Almost every patch I review has either zero or insufficient tests.
If we care about robustness, then we must discuss tests. Here are my two
recent experiences:
I agree we could do with x10 as many tests, but that doesn't mean all tests
make sense, so there needs to be some limiting principles to avoid adding
pointless test cases. I recently objected to adding a test case to
Isolation tester for the latest ALTER TABLE patch because in that case
there is no concurrent behavior to test. There is already a regression test
that tests lock levels for those statements, so in my view it is more
sensible to modify the existing test than to add a whole new isolation test
that does nothing more than demonstrate the lock levels work as described,
which we already knew.
On another review I suggested we add a function to core to allow it to be
used in regression tests. A long debate ensued, deciding that we must be
consistent and put diagnostic functions in contrib. My understanding is
that we are not allowed to write regression tests that use contrib modules,
yet the consistent place to put diagnostic functions is contrib -
therefore, we are never allowed to write tests utilizing diagnostic
functions. We are allowed to put modules for testing in another directory,
but these are not distributed to users so cannot be used for production
diagnosis. Systemic fail, advice needed.
--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/>
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Mon, Aug 10, 2015 at 2:02 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
Almost every patch I review has either zero or insufficient tests.
If we care about robustness, then we must discuss tests. Here are my two
recent experiences:I agree we could do with x10 as many tests, but that doesn't mean all tests
make sense, so there needs to be some limiting principles to avoid adding
pointless test cases. I recently objected to adding a test case to Isolation
tester for the latest ALTER TABLE patch because in that case there is no
concurrent behavior to test. There is already a regression test that tests
lock levels for those statements, so in my view it is more sensible to
modify the existing test than to add a whole new isolation test that does
nothing more than demonstrate the lock levels work as described, which we
already knew.On another review I suggested we add a function to core to allow it to be
used in regression tests. A long debate ensued, deciding that we must be
consistent and put diagnostic functions in contrib. My understanding is that
we are not allowed to write regression tests that use contrib modules, yet
the consistent place to put diagnostic functions is contrib - therefore, we
are never allowed to write tests utilizing diagnostic functions. We are
allowed to put modules for testing in another directory, but these are not
distributed to users so cannot be used for production diagnosis. Systemic
fail, advice needed.
There are actually two ways to do this.
One model is the dummy_seclabel module. The build system arranges for
that to be available when running the core regression tests, so they
can use it. And the dummy_seclabel test does. There are actually a
couple of other loadable modules that are used by the core regression
tests in this kind of way (refint and autoinc, from contrib/spi).
The other model is what I'll call the test_decoding model. Since the
core code can't be tested without a module, we have a module. But
then the tests are in the module's directory
(contrib/test_decoding/{sql,expected}) not the core regression tests.
In general, I have a mild preference for the second model. It seems
more scalable, and keeps the core tests quick to run, which is
appropriate for more obscure tests that are unlikely to break very
often. But the first model can also be done, as show by the fact that
we have in fact done it several times.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 10 August 2015 at 13:55, Robert Haas <robertmhaas@gmail.com> wrote:
On Mon, Aug 10, 2015 at 2:02 AM, Simon Riggs <simon@2ndquadrant.com>
wrote:On another review I suggested we add a function to core to allow it to be
used in regression tests. A long debate ensued, deciding that we must be
consistent and put diagnostic functions in contrib. My understanding isthat
we are not allowed to write regression tests that use contrib modules,
yet
the consistent place to put diagnostic functions is contrib - therefore,
we
are never allowed to write tests utilizing diagnostic functions. We are
allowed to put modules for testing in another directory, but these arenot
distributed to users so cannot be used for production diagnosis. Systemic
fail, advice needed.There are actually two ways to do this.
One model is the dummy_seclabel module. The build system arranges for
that to be available when running the core regression tests, so they
can use it. And the dummy_seclabel test does. There are actually a
couple of other loadable modules that are used by the core regression
tests in this kind of way (refint and autoinc, from contrib/spi).The other model is what I'll call the test_decoding model. Since the
core code can't be tested without a module, we have a module. But
then the tests are in the module's directory
(contrib/test_decoding/{sql,expected}) not the core regression tests.In general, I have a mild preference for the second model. It seems
more scalable, and keeps the core tests quick to run, which is
appropriate for more obscure tests that are unlikely to break very
often. But the first model can also be done, as show by the fact that
we have in fact done it several times.
Neither of those uses a diagnostic function that also has value in
production environments - for logical decoding we added something to core
specifically to allow this pg_recvlogical.
Anyway, resolving this isn't important anymore because I wish to pursue a
different mechanism for freezing, but its possible I hit the same issue.
--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/>
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Mon, Aug 10, 2015 at 07:02:17AM +0100, Simon Riggs wrote:
Almost every patch I review has either zero or insufficient tests.
If we care about robustness, then we must discuss tests. Here are my two
recent experiences:I agree we could do with x10 as many tests, but that doesn't mean all tests
make sense, so there needs to be some limiting principles to avoid adding
pointless test cases. I recently objected to adding a test case to
Isolation tester for the latest ALTER TABLE patch because in that case
there is no concurrent behavior to test. There is already a regression test
that tests lock levels for those statements, so in my view it is more
sensible to modify the existing test than to add a whole new isolation test
that does nothing more than demonstrate the lock levels work as described,
which we already knew.
Committers press authors to delete tests more often than we press them to
resubmit with more tests. No wonder so many patches have insufficient tests;
we treat those patches more favorably, on average. I have no objective
principles for determining whether a test is pointlessly redundant, but I
think the principles should become roughly 10x more permissive than the
(unspecified) ones we've been using.
On another review I suggested we add a function to core to allow it to be
used in regression tests. A long debate ensued, deciding that we must be
consistent and put diagnostic functions in contrib. My understanding is
that we are not allowed to write regression tests that use contrib modules,
yet the consistent place to put diagnostic functions is contrib -
therefore, we are never allowed to write tests utilizing diagnostic
functions. We are allowed to put modules for testing in another directory,
but these are not distributed to users so cannot be used for production
diagnosis. Systemic fail, advice needed.
If you want a user-visible function for production diagnosis, set aside the
fact that you wish to use it in a test, and find the best place for it. Then,
place the tests with the function. (That is, if the function belongs in
contrib for other reasons, put tests calling it in the contrib module itself.)
Place non-user-visible test support functions in regress.c, or use one of the
options Robert described.
Thanks,
nm
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 12 August 2015 at 03:10, Noah Misch <noah@leadboat.com> wrote:
On another review I suggested we add a function to core to allow it to be
used in regression tests. A long debate ensued, deciding that we must be
consistent and put diagnostic functions in contrib. My understanding is
that we are not allowed to write regression tests that use contribmodules,
yet the consistent place to put diagnostic functions is contrib -
therefore, we are never allowed to write tests utilizing diagnostic
functions. We are allowed to put modules for testing in anotherdirectory,
but these are not distributed to users so cannot be used for production
diagnosis. Systemic fail, advice needed.If you want a user-visible function for production diagnosis, set aside the
fact that you wish to use it in a test, and find the best place for it.
Then,
place the tests with the function. (That is, if the function belongs in
contrib for other reasons, put tests calling it in the contrib module
itself.)Place non-user-visible test support functions in regress.c, or use one of
the
options Robert described.
That helps, thanks.
--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/>
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Wed, Aug 12, 2015 at 3:10 AM, Noah Misch <noah@leadboat.com> wrote:
Committers press authors to delete tests more often than we press them to
resubmit with more tests. No wonder so many patches have insufficient tests;
we treat those patches more favorably, on average. I have no objective
principles for determining whether a test is pointlessly redundant, but I
think the principles should become roughly 10x more permissive than the
(unspecified) ones we've been using.
I would suggest the metric should be "if this test fails is it more
likely to be noise due to an intentional change in behaviour or more
likely to be a bug?"
The only time I've seen pushback against tests is when the test author
made valiant efforts to test every codepath and the expected output
embeds the precise behaviour of the current code as "correct". Even
when patches have extensive tests I don't recall seeing much pushback
(though I've been having trouble keeping up with the list in recent
months) if the tests are written in a way that they will only fail if
there's a bug, even if behaviour changes in unrelated ways.
--
greg
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, Aug 12, 2015 at 10:46 AM, Greg Stark <stark@mit.edu> wrote:
The only time I've seen pushback against tests is when the test author
made valiant efforts to test every codepath and the expected output
embeds the precise behaviour of the current code as "correct". Even
when patches have extensive tests I don't recall seeing much pushback
(though I've been having trouble keeping up with the list in recent
months) if the tests are written in a way that they will only fail if
there's a bug, even if behaviour changes in unrelated ways.
Really? I think Noah's description of how less testing is in effect
incentivized by committers is totally accurate. No patch author is
going to dig their heals in over the objections of a committer when
the complaint is about brevity of tests.
This resistance to adding tests seems quite short sighted to me,
especially when the concern is about queries that will each typically
take less than 1ms to execute. Like Noah, I think that it would be
very helpful to simply be more inclusive of additional tests that
don't increase test coverage by as much as each query in a minimal
subset. I am not at all convinced by arguments about the cost of
maintaining tests when a simple behavioral change occurs.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
One trouble I face when adding tests is that sometimes they require
hooks in the code, to test for race conditions. In BRIN I cannot test
some code paths without resorting to adding breakpoints in GDB, for
instance. If there's no support for such in the core code, it's
essentially impossible to add meaningful test for certain code paths. I
toyed with the notion of adding hook points (such that a test module can
put the backend to sleep until the other backend acknowledges hitting
the race condition window); I decided not to because it seems a more
general discussion is necessary first about the desirability of such.
As I recall we needed this in SKIP LOCKED and of course for multixacts
also.
I agree with the general idea that it's worthwhile to add lots more
tests than we currently have, both for the current code and for future
patches. Surely we don't need to reach the point where every single
nuance of every single user interface is verified to the point that
tests are such a large burden to maintain as is being suggested.
--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, Aug 12, 2015 at 2:04 PM, Peter Geoghegan <pg@heroku.com> wrote:
This resistance to adding tests seems quite short sighted to me,
especially when the concern is about queries that will each typically
take less than 1ms to execute. Like Noah, I think that it would be
very helpful to simply be more inclusive of additional tests that
don't increase test coverage by as much as each query in a minimal
subset. I am not at all convinced by arguments about the cost of
maintaining tests when a simple behavioral change occurs.
I've removed tests from patches that in my opinion were unlikely to
fail either (a) for any reason or (b) for any reason other than an
intentional change, and I think that's a reasonable thing to do.
However, I still think it's a good idea, and useful, to try to expand
the code coverage we get from 'make check'. However, the bigger
issue, IMHO, is the stuff that can't be tested via pg_regress, e.g.
because it needs hooks, like what Alvaro is talking about, or because
it needs a custom testing framework. Recovery, for example, really
needs a lot more testing, as we talked about at PGCon. If we just
expand what 'make check' covers and don't deal with those kinds of
things, we will be improving our test coverage but maybe not all that
much.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
* Robert Haas (robertmhaas@gmail.com) wrote:
On Wed, Aug 12, 2015 at 2:04 PM, Peter Geoghegan <pg@heroku.com> wrote:
This resistance to adding tests seems quite short sighted to me,
especially when the concern is about queries that will each typically
take less than 1ms to execute. Like Noah, I think that it would be
very helpful to simply be more inclusive of additional tests that
don't increase test coverage by as much as each query in a minimal
subset. I am not at all convinced by arguments about the cost of
maintaining tests when a simple behavioral change occurs.I've removed tests from patches that in my opinion were unlikely to
fail either (a) for any reason or (b) for any reason other than an
intentional change, and I think that's a reasonable thing to do.
However, I still think it's a good idea, and useful, to try to expand
the code coverage we get from 'make check'. However, the bigger
issue, IMHO, is the stuff that can't be tested via pg_regress, e.g.
because it needs hooks, like what Alvaro is talking about, or because
it needs a custom testing framework. Recovery, for example, really
needs a lot more testing, as we talked about at PGCon. If we just
expand what 'make check' covers and don't deal with those kinds of
things, we will be improving our test coverage but maybe not all that
much.
Agreed, and this is something which I said we'd work to help with and
which we have some things to show now, for those interested.
The regression tests included in pgBackRest (available here:
https://github.com/pgmasters/backrest) go through a number of different
recovery tests. There's vagrant configs for a few different VMs too
(CentOS 6, CentOS 7, Ubuntu 12.04 and Ubuntu 14.04) which is what we've
been testing.
We're working to continue expanding those tests and will also be adding
tests for replication and promotion in the future. Eventually, we plan
to write a buildfarm module for pgBackRest, to allow it to be run in the
same manner as the regular buildfarm animals with the results posted.
David, feel free to correct me if I'm misconstrued anything above.
Thanks!
Stephen
Robert Haas <robertmhaas@gmail.com> writes:
On Wed, Aug 12, 2015 at 2:04 PM, Peter Geoghegan <pg@heroku.com> wrote:
This resistance to adding tests seems quite short sighted to me,
especially when the concern is about queries that will each typically
take less than 1ms to execute. Like Noah, I think that it would be
very helpful to simply be more inclusive of additional tests that
don't increase test coverage by as much as each query in a minimal
subset. I am not at all convinced by arguments about the cost of
maintaining tests when a simple behavioral change occurs.
I've removed tests from patches that in my opinion were unlikely to
fail either (a) for any reason or (b) for any reason other than an
intentional change, and I think that's a reasonable thing to do.
FWIW, I've objected in the past to tests that would significantly
increase the runtime of "make check", unless I thought they were
especially valuable (which enumerating every minor behavior of a
feature patch generally isn't IMO). I still think that that's an
important consideration: every second you add to "make check" is
multiplied many times over when you consider how many developers
run that how many times a day.
We've talked about having some sort of second rank of tests that
people wouldn't necessarily run before committing, and that would
be allowed to eat more time than the core regression tests would.
I think that might be a valuable direction to pursue if people start
submitting very bulky tests.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, Aug 12, 2015 at 7:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
FWIW, I've objected in the past to tests that would significantly
increase the runtime of "make check", unless I thought they were
especially valuable (which enumerating every minor behavior of a
feature patch generally isn't IMO). I still think that that's an
important consideration: every second you add to "make check" is
multiplied many times over when you consider how many developers
run that how many times a day.We've talked about having some sort of second rank of tests that
people wouldn't necessarily run before committing, and that would
be allowed to eat more time than the core regression tests would.
I think that might be a valuable direction to pursue if people start
submitting very bulky tests.
Maybe. Adding a whole new test suite is significantly more
administratively complex, because the BF client has to get updated to
run it. And if expected outputs in that test suite change very often
at all, then committers will have to run it before committing anyway.
The value of a core regression suite that takes less time to run has
to be weighed against the possibility that a better core regression
suite might cause us to find more bugs before committing. That could
easily be worth the price in runtime.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, Aug 13, 2015 at 5:54 AM, Stephen Frost wrote:
The regression tests included in pgBackRest (available here:
https://github.com/pgmasters/backrest) go through a number of different
recovery tests. There's vagrant configs for a few different VMs too
(CentOS 6, CentOS 7, Ubuntu 12.04 and Ubuntu 14.04) which is what we've
been testing.
We're working to continue expanding those tests and will also be adding
tests for replication and promotion in the future. Eventually, we plan
to write a buildfarm module for pgBackRest, to allow it to be run in the
same manner as the regular buildfarm animals with the results posted.
Interesting. Do you mind if I pick up from it some ideas for the
in-core replication test suite based on TAP stuff? That's still in the
works for the next CF.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
* Michael Paquier (michael.paquier@gmail.com) wrote:
On Thu, Aug 13, 2015 at 5:54 AM, Stephen Frost wrote:
The regression tests included in pgBackRest (available here:
https://github.com/pgmasters/backrest) go through a number of different
recovery tests. There's vagrant configs for a few different VMs too
(CentOS 6, CentOS 7, Ubuntu 12.04 and Ubuntu 14.04) which is what we've
been testing.
We're working to continue expanding those tests and will also be adding
tests for replication and promotion in the future. Eventually, we plan
to write a buildfarm module for pgBackRest, to allow it to be run in the
same manner as the regular buildfarm animals with the results posted.Interesting. Do you mind if I pick up from it some ideas for the
in-core replication test suite based on TAP stuff? That's still in the
works for the next CF.
Certainly don't mind at all, entirely open source under the MIT
license.
Thanks!
Stephen
On Wed, Aug 12, 2015 at 9:24 PM, Stephen Frost <sfrost@snowman.net> wrote:
* Michael Paquier (michael.paquier@gmail.com) wrote:
On Thu, Aug 13, 2015 at 5:54 AM, Stephen Frost wrote:
The regression tests included in pgBackRest (available here:
https://github.com/pgmasters/backrest) go through a number of different
recovery tests. There's vagrant configs for a few different VMs too
(CentOS 6, CentOS 7, Ubuntu 12.04 and Ubuntu 14.04) which is what we've
been testing.
We're working to continue expanding those tests and will also be adding
tests for replication and promotion in the future. Eventually, we plan
to write a buildfarm module for pgBackRest, to allow it to be run in the
same manner as the regular buildfarm animals with the results posted.Interesting. Do you mind if I pick up from it some ideas for the
in-core replication test suite based on TAP stuff? That's still in the
works for the next CF.Certainly don't mind at all, entirely open source under the MIT
license.
Why not the PG license? It would be nicer if we didn't have to worry
about license contamination here.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers