Testing Extension Upgrade Paths

On Tue, Dec 12, 2017 at 2:55 AM, Jeremy Finzel <finzelj@gmail.com> wrote:

I understand how to setup a regression suite for a postgres extension, but
what I'm not clear on from the docs is if there is a pattern that exists for
testing not only the latest version of an extension, but also an upgraded
previous version.

Is there any straightforward way to do this that doesn't involve manually
copying tests?

It is perfectly possible to do tests on a specific version and test
upgrade paths using CREATE EXTENSION and ALTER EXTENSION which support
versioning in a SQL regression script, say:
CREATE EXTENSION foo VERSION "0.1";
-- Do stuff
ALTER EXTENSION foo UPDATE TO "0.2";
-- Do other stuff

P.S. is this the right list for extension dev questions ???

You are sure to get answers here, most folks on this list have likely
the experience of working on extensions.
--
Michael

On 12 December 2017 at 07:25, Michael Paquier <michael.paquier@gmail.com>
wrote:

On Tue, Dec 12, 2017 at 2:55 AM, Jeremy Finzel <finzelj@gmail.com> wrote:

I understand how to setup a regression suite for a postgres extension,

but

what I'm not clear on from the docs is if there is a pattern that exists

for

testing not only the latest version of an extension, but also an upgraded
previous version.

Is there any straightforward way to do this that doesn't involve manually
copying tests?

It is perfectly possible to do tests on a specific version and test
upgrade paths using CREATE EXTENSION and ALTER EXTENSION which support
versioning in a SQL regression script, say:
CREATE EXTENSION foo VERSION "0.1";
-- Do stuff
ALTER EXTENSION foo UPDATE TO "0.2";
-- Do other stuff

This works well when you want to test 1.1 binaries with a 1.0 SQL extension
definition.

It's not effective if you need to test 1.0 binaries+extension, then an
upgrade to 1.1 binaries and extension. You have no way to install and run
the 1.0 binaries without going outside pg_regress / TAP and using an
external test harness/framework/script. I've been bitten by this multiple
times before. For example, if 1.1 always assigns a value to some column
that was nullable in 1.0, you're never going to exercise 1.1's handling of
nulls in that column.

It also doesn't lend its self to complex multi-path upgrades, where for
example you could upgrade 1.0.0 -> 1.0.1 -> 1.1.0 or upgrade directly from
1.0.0 -> 1.1.0. This rapidly becomes an issue when you release 1.0.0,
release 1.1.0, then release a maintenance 1.0.1 version. Now you have
toadd the 1.0.1 -> 1.1.0 upgrade path, but you still have the 1.0.0 ->
1.1.0 path.

You can handle that with TAP if you're willing to write something to do the
various combinations of steps and exercise them. It's not practical with
pg_regress.

More thorough testing benefits from an external harness.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Mon, Dec 11, 2017 at 7:55 PM, Craig Ringer <craig@2ndquadrant.com> wrote:

On 12 December 2017 at 07:25, Michael Paquier <michael.paquier@gmail.com>
wrote:

On Tue, Dec 12, 2017 at 2:55 AM, Jeremy Finzel <finzelj@gmail.com> wrote:

I understand how to setup a regression suite for a postgres extension,

but

what I'm not clear on from the docs is if there is a pattern that

exists for

testing not only the latest version of an extension, but also an

upgraded

previous version.

Is there any straightforward way to do this that doesn't involve

manually

copying tests?

It is perfectly possible to do tests on a specific version and test
upgrade paths using CREATE EXTENSION and ALTER EXTENSION which support
versioning in a SQL regression script, say:
CREATE EXTENSION foo VERSION "0.1";
-- Do stuff
ALTER EXTENSION foo UPDATE TO "0.2";
-- Do other stuff

This works well when you want to test 1.1 binaries with a 1.0 SQL
extension definition.

It's not effective if you need to test 1.0 binaries+extension, then an
upgrade to 1.1 binaries and extension. You have no way to install and run
the 1.0 binaries without going outside pg_regress / TAP and using an
external test harness/framework/script. I've been bitten by this multiple
times before. For example, if 1.1 always assigns a value to some column
that was nullable in 1.0, you're never going to exercise 1.1's handling of
nulls in that column.

It also doesn't lend its self to complex multi-path upgrades, where for
example you could upgrade 1.0.0 -> 1.0.1 -> 1.1.0 or upgrade directly from
1.0.0 -> 1.1.0. This rapidly becomes an issue when you release 1.0.0,
release 1.1.0, then release a maintenance 1.0.1 version. Now you have
toadd the 1.0.1 -> 1.1.0 upgrade path, but you still have the 1.0.0 ->
1.1.0 path.

You can handle that with TAP if you're willing to write something to do
the various combinations of steps and exercise them. It's not practical
with pg_regress.

More thorough testing benefits from an external harness.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Thanks both of you for the feedback.

I understood that I can do CREATE then ALTER EXTENSION UPDATE, but my goal
was to actually be able to run this workflow multiple ways without having
to hardcode that or duplicate steps. To clarify, my intention was to only
test the binary for 1.1, but to then run the same test steps (without
having to duplicate things) under 2 separate scenarios:

1. Create the extension right away at version 1.1, run the suite
2. Create the extension at 1.0, populate extension config tables as if I
have actually been using it, then upgrade to 1.1, and run the suite.

Here is how I ended up implementing this, and I am very open to feedback:

The first file is like this:

-- Allow running regression suite with upgrade paths
\set v `echo ${FROMVERSION:-1.1}`
SET client_min_messages = warning;
CREATE EXTENSION pglogical;
CREATE EXTENSION pgl_ddl_deploy VERSION :'v';

So if you run the suite without any environment variable, it will start at
1.1. But if you add FROMVERSION=1.0, it will start at 1.0.

Then in test steps 2 and 3, I assume test results are identical between 1.0
and 1.1. But in step 4, I run ALTER EXTENSION UPDATE, which is either a
no-op, or actually upgrades from 1.0. The remaining tests in the whole
suite are designed to yield identical results in either path.

I am fairly happy with this because what I really wanted to test is
upgrading from 1.0 to 1.1 as opposed to a bare 1.1 install.

With yet later versions, I would of course need to modify this as
necessary, and I would want to test then yet more of the upgrade paths as
it is feasible.

Thanks,
Jeremy