pg_upgrade automatic testing

On Thu, Apr 7, 2011 at 4:02 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

Seeing that 9.1-to-9.1 pg_upgrade has apparently been broken for months,
it would probably be good to have some kind of automatic testing for it.
Attached is something I hacked together that at least exposes the
current problems, easily available by typing "make check" and waiting.
It does not yet fully implement the full testing procedure in the
TESTING file, in particular the diffing of the dumps (well, because you
can't get there yet).

Is that something that people are interested in refining?

(I think it would even be possible under this setup to create special
regression test cases that are only run under the pg_upgrade test run,
to exercise particularly tricky upgrade cases.)

I think it's a worthwhile thing to do, but right now I think it would
be more helpful if you could help fix the breakage your patch created,
rather than working on new stuff.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On tor, 2011-04-07 at 17:03 -0400, Robert Haas wrote:

I think it's a worthwhile thing to do, but right now I think it would
be more helpful if you could help fix the breakage your patch created,
rather than working on new stuff.

This is part of that.

On Thu, Apr 7, 2011 at 5:20 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

On tor, 2011-04-07 at 17:03 -0400, Robert Haas wrote:

I think it's a worthwhile thing to do, but right now I think it would
be more helpful if you could help fix the breakage your patch created,
rather than working on new stuff.

This is part of that.

It's related. But we can release beta1 without improving the
regression testing framework for pg_upgrade. We cannot release beta1
with pg_upgrade broken.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On tor, 2011-04-07 at 23:02 +0300, Peter Eisentraut wrote:

Seeing that 9.1-to-9.1 pg_upgrade has apparently been broken for months,
it would probably be good to have some kind of automatic testing for it.
Attached is something I hacked together that at least exposes the
current problems, easily available by typing "make check" and waiting.
It does not yet fully implement the full testing procedure in the
TESTING file, in particular the diffing of the dumps (well, because you
can't get there yet).

Is that something that people are interested in refining?

(I think it would even be possible under this setup to create special
regression test cases that are only run under the pg_upgrade test run,
to exercise particularly tricky upgrade cases.)

Now that this is bound to be fixed, here is an updated script that runs
the entire test procedure including diffing the dumps at the end.

On Wed, Apr 27, 2011 at 09:32:16PM +0300, Peter Eisentraut wrote:

On tor, 2011-04-07 at 23:02 +0300, Peter Eisentraut wrote:

Seeing that 9.1-to-9.1 pg_upgrade has apparently been broken for months,
it would probably be good to have some kind of automatic testing for it.
Attached is something I hacked together that at least exposes the
current problems, easily available by typing "make check" and waiting.
It does not yet fully implement the full testing procedure in the
TESTING file, in particular the diffing of the dumps (well, because you
can't get there yet).

Is that something that people are interested in refining?

(I think it would even be possible under this setup to create special
regression test cases that are only run under the pg_upgrade test run,
to exercise particularly tricky upgrade cases.)

Now that this is bound to be fixed, here is an updated script that runs
the entire test procedure including diffing the dumps at the end.

Enthusiastic +1 for this concept. There's at least one rough edge: it fails if
you have another postmaster running on port 5432.

Thanks,
nm

On ons, 2011-04-27 at 18:14 -0400, Noah Misch wrote:

Enthusiastic +1 for this concept. There's at least one rough edge: it fails if
you have another postmaster running on port 5432.

This has now been addressed: pg_upgrade accepts PGPORT settings.
Attached is a slightly updated patch runs the test suite with a port of
65432, which you can override by setting PGPORT yourself.

Anyway, is this something that people want in the repository? It's not
as polished as the pg_regress business, but it is definitely helpful.

On Wed, May 25, 2011 at 3:07 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

On ons, 2011-04-27 at 18:14 -0400, Noah Misch wrote:

Enthusiastic +1 for this concept.  There's at least one rough edge: it fails if
you have another postmaster running on port 5432.

This has now been addressed: pg_upgrade accepts PGPORT settings.
Attached is a slightly updated patch runs the test suite with a port of
65432, which you can override by setting PGPORT yourself.

Anyway, is this something that people want in the repository?  It's not
as polished as the pg_regress business, but it is definitely helpful.

Is this going to result in using the built binaries with the installed
libraries, a la Tom's recent complaint about the isolation tests?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, May 25, 2011 at 10:07:45PM +0300, Peter Eisentraut wrote:

On ons, 2011-04-27 at 18:14 -0400, Noah Misch wrote:

Enthusiastic +1 for this concept. There's at least one rough edge: it fails if
you have another postmaster running on port 5432.

This has now been addressed: pg_upgrade accepts PGPORT settings.
Attached is a slightly updated patch runs the test suite with a port of
65432, which you can override by setting PGPORT yourself.

Anyway, is this something that people want in the repository? It's not
as polished as the pg_regress business, but it is definitely helpful.

I'd like it. We've had bugs sit for months that would have been found
immediately by a buildfarm member running this test. Having it in the
repository at least opens up that possibility.

Noah Misch wrote:

On Wed, May 25, 2011 at 10:07:45PM +0300, Peter Eisentraut wrote:

On ons, 2011-04-27 at 18:14 -0400, Noah Misch wrote:

Enthusiastic +1 for this concept. There's at least one rough edge: it fails if
you have another postmaster running on port 5432.

This has now been addressed: pg_upgrade accepts PGPORT settings.
Attached is a slightly updated patch runs the test suite with a port of
65432, which you can override by setting PGPORT yourself.

Anyway, is this something that people want in the repository? It's not
as polished as the pg_regress business, but it is definitely helpful.

I'd like it. We've had bugs sit for months that would have been found
immediately by a buildfarm member running this test. Having it in the
repository at least opens up that possibility.

Yep, looks fine to me.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 05/25/2011 03:07 PM, Peter Eisentraut wrote:

On ons, 2011-04-27 at 18:14 -0400, Noah Misch wrote:

Enthusiastic +1 for this concept. There's at least one rough edge: it fails if
you have another postmaster running on port 5432.

This has now been addressed: pg_upgrade accepts PGPORT settings.
Attached is a slightly updated patch runs the test suite with a port of
65432, which you can override by setting PGPORT yourself.

Anyway, is this something that people want in the repository? It's not
as polished as the pg_regress business, but it is definitely helpful.

As is, this will probably break on a bunch of platforms. I suspect you
will need the equivalent of this snippet from pg_regress.c:

add_to_path("LD_LIBRARY_PATH", ':', libdir);
add_to_path("DYLD_LIBRARY_PATH", ':', libdir);
add_to_path("LIBPATH", ':', libdir);
#if defined(WIN32)
add_to_path("PATH", ';', libdir);
#elif defined(__CYGWIN__)
add_to_path("PATH", ':', libdir);
#endif

For extra credit, you could create a subroutine in vcregress.pl to run
the check for MSVC builds.

cheers

andrew

Here is an updated version of the pg_upgrade test script I posted a
while ago. I've cleaned it up so that it offers moderately
user-friendly feedback, it supports check and installcheck mode, and
should use all the things from the right directories in either case.

Peter Eisentraut <peter_e@gmx.net> writes:

+# contrib/pg_upgrade/test.sh
+#
+# Test driver for pg_upgrade.  Initializes a new database cluster,
+# runs the regression tests (to put in some data), runs pg_dumpall,
+# runs pg_upgrade, runs pg_dumpall again, compares the dumps.

Hm .. my experience is that that doesn't work at all, because the
regression tests set up assorted C functions whose implementations are
in pg_regress.so, and it creates them with absolute path references
to pg_regress.so. When you try to load that into another installation
that's a different version of PG, it quite properly fails. So I think
that as given, this script is only useful for testing pg_upgrade of
$currentversion to $currentversion. Which is surely better than no test
at all, but it would not for example have caught the 8.3 incompatibility
that was just reported.

How can we improve things here? I've toyed with the idea of installing
pg_regress.so so that we can refer to it relative to $libdir, but that
might be a bit invasive, especially if we were to try to back-patch it
as far as 8.3.

regards, tom lane

On Tue, Aug 30, 2011 at 22:25, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

+# contrib/pg_upgrade/test.sh
+#
+# Test driver for pg_upgrade.  Initializes a new database cluster,
+# runs the regression tests (to put in some data), runs pg_dumpall,
+# runs pg_upgrade, runs pg_dumpall again, compares the dumps.

Hm .. my experience is that that doesn't work at all, because the
regression tests set up assorted C functions whose implementations are
in pg_regress.so, and it creates them with absolute path references
to pg_regress.so.  When you try to load that into another installation
that's a different version of PG, it quite properly fails.  So I think
that as given, this script is only useful for testing pg_upgrade of
$currentversion to $currentversion.  Which is surely better than no test
at all, but it would not for example have caught the 8.3 incompatibility
that was just reported.

How can we improve things here?  I've toyed with the idea of installing
pg_regress.so so that we can refer to it relative to $libdir, but that
might be a bit invasive, especially if we were to try to back-patch it
as far as 8.3.

Would turning pg_regress.so into an extension and using that way fix
it? That won't help for the 9.0->9.1 stage, but it would for
9.1->9.2...

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus@hagander.net> writes:

On Tue, Aug 30, 2011 at 22:25, Tom Lane <tgl@sss.pgh.pa.us> wrote:

How can we improve things here? �I've toyed with the idea of installing
pg_regress.so so that we can refer to it relative to $libdir, but that
might be a bit invasive, especially if we were to try to back-patch it
as far as 8.3.

Would turning pg_regress.so into an extension and using that way fix
it? That won't help for the 9.0->9.1 stage, but it would for
9.1->9.2...

Not really excited about that. The contrib regression tests already
exercise the extension functionality, so making pg_regress.so into
another one would just reduce the number of code paths being covered.

In any case, if we don't find a way to allow automated testing of
pg_upgrade from the pre-9.1 versions, we have not fixed the problem.

regards, tom lane

On tis, 2011-08-30 at 16:25 -0400, Tom Lane wrote:

So I think that as given, this script is only useful for testing
pg_upgrade of $currentversion to $currentversion. Which is surely
better than no test at all, but it would not for example have caught
the 8.3 incompatibility that was just reported.

Well, the goal was always current to current version. Cross-version
testing is obviously important, but will be quite a bit harder.

How can we improve things here? I've toyed with the idea of
installing pg_regress.so so that we can refer to it relative to
$libdir, but that might be a bit invasive, especially if we were to
try to back-patch it as far as 8.3.

Aside from hesitations to backpatch those sorts of changes, it would
effectively prevent us from ever removing anything from the C libraries
used in the regression tests, because we need to keep the symbols around
so that the schema dump can load successfully into the new instance.

I think a solution would have to be one of:

1) pg_upgrade needs a mode to cope with these situations. It can tell
the user, I upgraded your installation, but some dynamic modules appear
to be missing, you need to sort that out before you can put this back
into use.

2) Design a different test schema to load into the database before
running pg_upgrade. This would then be a one-line change in the script.

Tom Lane wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

+# contrib/pg_upgrade/test.sh
+#
+# Test driver for pg_upgrade.  Initializes a new database cluster,
+# runs the regression tests (to put in some data), runs pg_dumpall,
+# runs pg_upgrade, runs pg_dumpall again, compares the dumps.

Hm .. my experience is that that doesn't work at all, because the
regression tests set up assorted C functions whose implementations are
in pg_regress.so, and it creates them with absolute path references
to pg_regress.so. When you try to load that into another installation
that's a different version of PG, it quite properly fails. So I think
that as given, this script is only useful for testing pg_upgrade of
$currentversion to $currentversion. Which is surely better than no test

Reminder --- you can't use pg_upgrade to go from the same catalog
version to the same catalog version because the catalog version is
embedded in the tablespace directory name.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Peter Eisentraut wrote:

On tis, 2011-08-30 at 16:25 -0400, Tom Lane wrote:

So I think that as given, this script is only useful for testing
pg_upgrade of $currentversion to $currentversion. Which is surely
better than no test at all, but it would not for example have caught
the 8.3 incompatibility that was just reported.

Well, the goal was always current to current version. Cross-version
testing is obviously important, but will be quite a bit harder.

How can we improve things here? I've toyed with the idea of
installing pg_regress.so so that we can refer to it relative to
$libdir, but that might be a bit invasive, especially if we were to
try to back-patch it as far as 8.3.

Aside from hesitations to backpatch those sorts of changes, it would
effectively prevent us from ever removing anything from the C libraries
used in the regression tests, because we need to keep the symbols around
so that the schema dump can load successfully into the new instance.

I think a solution would have to be one of:

1) pg_upgrade needs a mode to cope with these situations. It can tell
the user, I upgraded your installation, but some dynamic modules appear
to be missing, you need to sort that out before you can put this back
into use.

2) Design a different test schema to load into the database before
running pg_upgrade. This would then be a one-line change in the script.

Here are the scripts I use for testing:

http://momjian.us/expire/pg_upgrade_test.tgz

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On tor, 2011-09-01 at 18:55 -0400, Bruce Momjian wrote:

Tom Lane wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

+# contrib/pg_upgrade/test.sh
+#
+# Test driver for pg_upgrade.  Initializes a new database cluster,
+# runs the regression tests (to put in some data), runs pg_dumpall,
+# runs pg_upgrade, runs pg_dumpall again, compares the dumps.

Hm .. my experience is that that doesn't work at all, because the
regression tests set up assorted C functions whose implementations are
in pg_regress.so, and it creates them with absolute path references
to pg_regress.so. When you try to load that into another installation
that's a different version of PG, it quite properly fails. So I think
that as given, this script is only useful for testing pg_upgrade of
$currentversion to $currentversion. Which is surely better than no test

Reminder --- you can't use pg_upgrade to go from the same catalog
version to the same catalog version because the catalog version is
embedded in the tablespace directory name.

Well, it does work, but only because the regression tests don't keep a
tablespace around at the end. Would pg_upgrade complain otherwise?

On 09/02/2011 10:36 AM, Peter Eisentraut wrote:

On tor, 2011-09-01 at 18:55 -0400, Bruce Momjian wrote:

Tom Lane wrote:

Peter Eisentraut<peter_e@gmx.net> writes:

+# contrib/pg_upgrade/test.sh
+#
+# Test driver for pg_upgrade.  Initializes a new database cluster,
+# runs the regression tests (to put in some data), runs pg_dumpall,
+# runs pg_upgrade, runs pg_dumpall again, compares the dumps.

Hm .. my experience is that that doesn't work at all, because the
regression tests set up assorted C functions whose implementations are
in pg_regress.so, and it creates them with absolute path references
to pg_regress.so. When you try to load that into another installation
that's a different version of PG, it quite properly fails. So I think
that as given, this script is only useful for testing pg_upgrade of
$currentversion to $currentversion. Which is surely better than no test

Reminder --- you can't use pg_upgrade to go from the same catalog
version to the same catalog version because the catalog version is
embedded in the tablespace directory name.

Well, it does work, but only because the regression tests don't keep a
tablespace around at the end. Would pg_upgrade complain otherwise?

In any case, it would be good to get rid of the limitation if possible.
Then we could look at creating an automated test that we could use in
the buildfarm.

cheers

andrew