multi-platform, multi-locale regression tests
I'm looking for some ideas on how to deal with the regression tests for
the per-column collation feature. These are the issues:
* The feature only works on some platforms (tentatively: Linux,
Windows).
-> Possible solution: like xml test
* The locale names are platform dependent, so there would need to be
different test files per locale.
* The test files need to use some non-ASCII characters. So far, I have
encoded the test file in UTF-8 and run the tests with make check
MULTIBYTE=UTF8.
* Also, the allowed collations depend on the server encoding, so any
solution for the previous point that results in the server encoding of
the test database being variable will make the setup of the regression
test SQL file more interesting.
* Of course the actual sort orders could also be different on different
platforms, but that problem can likely be contained.
One possible way out is not to include these tests in the main test set
and instead require manual invocation.
Better ideas?
On Nov 9, 2010, at 12:18 PM, Peter Eisentraut wrote:
One possible way out is not to include these tests in the main test set
and instead require manual invocation.Better ideas?
I've been talking to Nasby and Dunstan about adding a Test::More/pgTAP-based test suite to the core. It wouldn't run with the usual core suite used by developers, which would continue to use pg_regress. But they could run it if they wanted (and had the prerequisites), and the build farm animals would run them regularly.
The nice thing about using a TAP-based framework is that you can skip tests that don't meet platform requirements, and compare values within the tests, right where you write them, rather than in a separate file. You can also dynamically change how you compare things depending on the environment, such as the locales that vary on different platforms.
Thoughts?
Best,
David
2010/11/9 David E. Wheeler <david@kineticode.com>:
On Nov 9, 2010, at 12:18 PM, Peter Eisentraut wrote:
One possible way out is not to include these tests in the main test set
and instead require manual invocation.Better ideas?
I've been talking to Nasby and Dunstan about adding a Test::More/pgTAP-based test suite to the core. It wouldn't run with the usual core suite used by developers, which would continue to use pg_regress. But they could run it if they wanted (and had the prerequisites), and the build farm animals would run them regularly.
The nice thing about using a TAP-based framework is that you can skip tests that don't meet platform requirements, and compare values within the tests, right where you write them, rather than in a separate file. You can also dynamically change how you compare things depending on the environment, such as the locales that vary on different platforms.
Thoughts?
Are you thinking of a contrib module 'pgtap' that we can use to
accomplish the optionnal regression tests ?
--
Cédric Villemain 2ndQuadrant
http://2ndQuadrant.fr/ PostgreSQL : Expertise, Formation et Support
On Nov 9, 2010, at 2:42 PM, Cédric Villemain wrote:
Are you thinking of a contrib module 'pgtap' that we can use to
accomplish the optionnal regression tests ?
Oh, if the project wants it in contrib, sure. Otherwise I'd probably just have the test stuff include it somehow.
David
2010/11/9 David E. Wheeler <david@kineticode.com>:
On Nov 9, 2010, at 2:42 PM, Cédric Villemain wrote:
Are you thinking of a contrib module 'pgtap' that we can use to
accomplish the optionnal regression tests ?Oh, if the project wants it in contrib, sure. Otherwise I'd probably just have the test stuff include it somehow.
Adding a unit test layer shipped with postgresql sounds good to me.
And pgTAP can claim to be platform agnostic.
--
Cédric Villemain 2ndQuadrant
http://2ndQuadrant.fr/ PostgreSQL : Expertise, Formation et Support
On tis, 2010-11-09 at 14:00 -0800, David E. Wheeler wrote:
I've been talking to Nasby and Dunstan about adding a Test::More/pgTAP-based test suite to the core. It wouldn't run with the usual core suite used by developers, which would continue to use pg_regress. But they could run it if they wanted (and had the prerequisites), and the build farm animals would run them regularly.
I'd welcome something like that, but I'm not sure that that's the best
overall solution to this particular problem in the short run. But it
would be great to have anyway.
Peter Eisentraut wrote:
On tis, 2010-11-09 at 14:00 -0800, David E. Wheeler wrote:I've been talking to Nasby and Dunstan about adding a
Test::More/pgTAP-based test suite to the core. It wouldn't run
with the usual core suite used by developers, which would continue
to use pg_regress. But they could run it if they wanted (and had
the prerequisites), and the build farm animals would run them
regularly.I'd welcome something like that, but I'm not sure that that's the
best overall solution to this particular problem in the short run.
But it would be great to have anyway.
For the Serializable Snapshot Isolation (SSI) patch I needed a test
suite which would handle concurrent sessions which interleaved
statements in predictable ways. I was told pgTAP wasn't a good
choice for that and went with Markus Wanner's dtester package. The
SSI patch adds a "dcheck" build target which is not included in any
others to run the dtester tests.
I don't know if dtester meets the other needs people have, or whether
this is a complementary approach, but it seemed worth mentioning.
-Kevin
Import Notes
Resolved by subject fallback
On Nov 10, 2010, at 5:31 AM, Kevin Grittner wrote:
For the Serializable Snapshot Isolation (SSI) patch I needed a test
suite which would handle concurrent sessions which interleaved
statements in predictable ways. I was told pgTAP wasn't a good
choice for that and went with Markus Wanner's dtester package. The
SSI patch adds a "dcheck" build target which is not included in any
others to run the dtester tests.
Right. pgTAP doesn't run tests, it's just a collection of assertion functions written in SQL and PL/pgSQL. It could have been used via a forking Perl script that would connect to the proper boxes, run the tests, collect the results, etc. But it clearly would have been a PITA, and the path of least resistance is often the best solution when hacking. Going with dcheck, which already did what you wanted, was clearly the right choice.
Hopefully we can have the build farm animals run the dcheck target once SSI is committed.
Best,
David
On Wed, Nov 10, 2010 at 08:33:13AM -0800, David Wheeler wrote:
On Nov 10, 2010, at 5:31 AM, Kevin Grittner wrote:
For the Serializable Snapshot Isolation (SSI) patch I needed a
test suite which would handle concurrent sessions which
interleaved statements in predictable ways. I was told pgTAP
wasn't a good choice for that and went with Markus Wanner's
dtester package. The SSI patch adds a "dcheck" build target which
is not included in any others to run the dtester tests.Right. pgTAP doesn't run tests, it's just a collection of assertion
functions written in SQL and PL/pgSQL. It could have been used via
a forking Perl script that would connect to the proper boxes, run
the tests, collect the results, etc. But it clearly would have been
a PITA, and the path of least resistance is often the best solution
when hacking. Going with dcheck, which already did what you wanted,
was clearly the right choice.Hopefully we can have the build farm animals run the dcheck target
once SSI is committed.
Does Perl have some kind of concurrency-controlled test framework?
Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david.fetter@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
On 11/10/2010 08:31 AM, Kevin Grittner wrote:
I don't know if dtester meets the other needs people have, or whether
this is a complementary approach, but it seemed worth mentioning.
Where is this available? Is it self-contained? And what does it require?
cheers
andrew
On Nov 10, 2010, at 9:48 AM, Andrew Dunstan wrote:
I don't know if dtester meets the other needs people have, or whether
this is a complementary approach, but it seemed worth mentioning.Where is this available? Is it self-contained? And what does it require?
Python.
http://www.bluegap.ch/projects/dtester/
Best,
David
"David E. Wheeler" <david@kineticode.com> wrote:
On Nov 10, 2010, at 9:48 AM, Andrew Dunstan wrote:
Where is this available? Is it self-contained? And what does it
require?Python.
And some optional python packages, like twisted.
It looks like I may have raised the issue at a particularly
inopportune time -- it looks like maybe Markus is reloading his git
repo based on the new "official" git repo for PostgreSQL.
-Kevin
On Wed, Nov 10, 2010 at 15:31, Kevin Grittner
<Kevin.Grittner@wicourts.gov> wrote:
For the Serializable Snapshot Isolation (SSI) patch I needed a test
suite which would handle concurrent sessions which interleaved
statements in predictable ways. I was told pgTAP wasn't a good
choice for that and went with Markus Wanner's dtester package.
Sounds like you could use pgTAP with dblink to do the same? :)
Regards,
Marti
Hi,
On 11/10/2010 07:28 PM, Kevin Grittner wrote:
It looks like I may have raised the issue at a particularly
inopportune time -- it looks like maybe Markus is reloading his git
repo based on the new "official" git repo for PostgreSQL.
Thanks for noticing me. The dtester repository should be there again.
Sorry for the inconvenience.
Regards
Markus
On ons, 2010-11-10 at 07:31 -0600, Kevin Grittner wrote:
I don't know if dtester meets the other needs people have, or whether
this is a complementary approach, but it seemed worth mentioning.
The right tool for the right job, I'd say.
One thing to aim for, perhaps, would be to make all tools in use produce
a common output format, at least optionally, so that creating a common
test run dashboard or something like that is more easily possible. TAP
and xUnit come to mind.
Marti Raudsepp <marti@juffo.org> wrote:
Sounds like you could use pgTAP with dblink to do the same? :)
I had never read through the docs for dblink until you posted this.
In fact, it appears that some testing of proper SSI behavior can be
added to standard regression tests with dblink (without needing
pgTAP) if there is some way to allow a contrib module like that to
be used. Would I have to add the SSI tests to the dblink regression
tests, or is there some more graceful way that might be made to
work?
I don't think this would be a sane way to *replace* the dcheck
tests, but it might be a way to work *some* testing of SSI into a
more frequently run test set.
-Kevin
On 11/10/2010 05:06 PM, Kevin Grittner wrote:
Marti Raudsepp<marti@juffo.org> wrote:
Sounds like you could use pgTAP with dblink to do the same? :)
I had never read through the docs for dblink until you posted this.
In fact, it appears that some testing of proper SSI behavior can be
added to standard regression tests with dblink (without needing
pgTAP) if there is some way to allow a contrib module like that to
be used. Would I have to add the SSI tests to the dblink regression
tests, or is there some more graceful way that might be made to
work?I don't think this would be a sane way to *replace* the dcheck
tests, but it might be a way to work *some* testing of SSI into a
more frequently run test set.
We already use some contrib stuff in the regression tests. (It really is
time we stopped calling it contrib.)
cheers
andrew
On Nov 10, 2010, at 2:15 PM, Andrew Dunstan wrote:
We already use some contrib stuff in the regression tests. (It really is time we stopped calling it contrib.)
Call them "core extensions". Works well considering Dimitri's work, which explicitly makes them extensions. So maybe change the directory name to "extensions" or "ext"?
Best,
David
"David E. Wheeler" <david@kineticode.com> writes:
On Nov 10, 2010, at 2:15 PM, Andrew Dunstan wrote:
We already use some contrib stuff in the regression tests. (It really is time we stopped calling it contrib.)
Call them "core extensions". Works well considering Dimitri's work, which explicitly makes them extensions. So maybe change the directory name to "extensions" or "ext"?
We've been calling it "contrib" for a dozen years, so that name is
pretty well baked in by now. IMO renaming it is pointless and will
accomplish little beyond creating confusion and making back-patches
harder. (And no, don't you dare breathe a word about git making that
all automagically better. I have enough back-patching experience with
git by now to be unimpressed; in fact, I notice that its rename-tracking
feature falls over entirely when trying to back-patch further than 8.3.
Apparently there's some hardwired limit on the number of files it can
cope with.)
regards, tom lane
On Nov 10, 2010, at 3:17 PM, Tom Lane wrote:
We've been calling it "contrib" for a dozen years, so that name is
pretty well baked in by now. IMO renaming it is pointless and will
accomplish little beyond creating confusion and making back-patches
harder.
*Shrug*. Just change the name in the docs, then. It's currently "Additional Supplied Modules". Maybe just change that to "Additional Supplied Extensions" or, even better, "Core Extensions"?
Best,
David
(And no, don't you dare breathe a word about git making that
all automagically better. I have enough back-patching experience with
git by now to be unimpressed; in fact, I notice that its rename-tracking
feature falls over entirely when trying to back-patch further than 8.3.
Apparently there's some hardwired limit on the number of files it can
cope with.)
How often do you have to back-patch contrib, anyway?
David
