Direction for test frameworks: Perl TAP vs. Python/pytest
Hi,
We now have several pieces of test-infrastructure work in flight that
point in
different directions, and I'd like the community to settle the direction
before
more effort is expended on directions we might not want to follow.
Several things exist today:
1. My PostgreSQL::Test::Session work, which gives the Perl TAP framework
in-process libpq sessions instead of forking psql per query. Part of the
motivation is that BackgroundPsql is fragile and has a habit of
swallowing
errors; a real session object sidesteps that. It also means far less
forking, so things run a good deal more efficiently -- especially on
Windows, where process creation is expensive. The Perl framework is
still
gaining features. It currently relies on FFI::Platypus, and it's not
clear
that's available everywhere we build Postgres -- if not, we'd need
to fall
back on an XS wrapper. ("Pq Session object plus TAP test improvements",
CF #6834 [1]https://commitfest.postgresql.org/patch/6834/; origin thread "Using LibPq in TAP tests via FFI" [2]/messages/by-id/997f67ac-cc02-4076-9fda-db465e89c622@dunslane.net.)
2. Jelte Fennema-Nio's proposed minimal Python/pytest framework
(src/test/pytest), aimed mainly at new tests. It already carries an
in-process ctypes libpq layer of its own -- and note that on the Python
side this needs no FFI module or XS wrapper at all, since ctypes is
in the
standard library and can bind to libpq natively. ("RFC: adding
pytest as a
supported test framework", CF #6045 [3]https://commitfest.postgresql.org/patch/6045/.)
3. Two full ports of the existing TAP suite (~300 tests plus the
framework) to
that pytest shape, both validated against the existing suite. Greg
Burd's [4]https://github.com/gburd/postgres/pull/24
is a fully additive port: it leaves the existing Perl suite in place
and runs
a Python twin of each test beside it. Queries go through psql by default
(matching safe_psql), with an opt-in in-process ctypes libpq layer for
protocol-level tests. Mine [5]https://github.com/adunstan/pgdev/pull/5 builds on Jelte's framework rather than
standing apart from it: I've modularised and extended its libpq
layer into a
full Session along the lines of Test::Session, expanded the server and
fixture helpers to cover replication, logical decoding, backups, and
log and
connection/auth checks, and added harness support for the SSL, LDAP,
Kerberos and OAuth suites along with a pg_regress runner. I did this
partly
to show that we have an adequate basis for porting even if we never
declare
a flag day, and partly because, for what it's worth, I think the
Test::Session model is a much more solid foundation for a port than the
current framework is.
So the work supports more than one strategy. The question is which we want:
(A) Wholesale port with a flag day -- Python becomes *the* framework,
migrate the existing tests, stop accepting and eventually remove the
Perl one.
One framework to learn and maintain, but a large disruptive change with real
dependency questions (which Python, which libpq binding, buildfarm
minimums).
(B) Minimal Python framework for new tests, Perl preserved alongside.
Less disruptive, but commits us to two frameworks for some time.
(C) Something in between -- adopt the Python framework and port over
just the
most fragile tests (the ones leaning on BackgroundPsql, say), leaving
the rest
in Perl for now. This gets the worst pain points onto a sounder footing
without committing to a full migration up front.
(D) Keep both, but put each on a sounder footing: land the Test::Session
work
so the Perl side stops depending on BackgroundPsql, and adopt the Python
framework alongside it for new tests. This addresses the fragility
within Perl
rather than by porting, and doesn't force a migration either way.
If we land on (B), (C) or (D), I think we have to answer these questions:
- Do we implement new infrastructure features (e.g. the Session work) twice,
or let the two frameworks drift?
- When an existing Perl test needs substantial new coverage, do we extend it
in Perl or port it? "Port on touch" front-loads cost onto bug fixers.
Having shown a full port is feasible, my inclination is that the maintenance
argument favours converging on one framework -- but I'm wary of the
disruption,
so I could see (D) as a sane and fairly safe option.
So how would people like to proceed?
cheers
andrew
[1]: https://commitfest.postgresql.org/patch/6834/
[2]: /messages/by-id/997f67ac-cc02-4076-9fda-db465e89c622@dunslane.net
/messages/by-id/997f67ac-cc02-4076-9fda-db465e89c622@dunslane.net
[3]: https://commitfest.postgresql.org/patch/6045/
[4]: https://github.com/gburd/postgres/pull/24
[5]: https://github.com/adunstan/pgdev/pull/5
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
On 16/06/2026 17:23, Andrew Dunstan wrote:
(A) Wholesale port with a flag day -- Python becomes *the*
framework, migrate the existing tests, stop accepting and eventually
remove the Perl one. One framework to learn and maintain, but a
large disruptive change with real dependency questions (which
Python, which libpq binding, buildfarm minimums).
+1 on this
I don't mind keeping the existing Perl tests for a while, but I'd like
all new tests to be in Python, and have an agreement to migrate all
existing tests over time. I haven't really looked at these efforts in
detail so I don't know how long a transition period we need. Months? Years?
- Heikki
Hi,
(A) Wholesale port with a flag day -- Python becomes *the*
framework, migrate the existing tests, stop accepting and eventually
remove the Perl one. One framework to learn and maintain, but a
large disruptive change with real dependency questions (which
Python, which libpq binding, buildfarm minimums).+1 on this
I don't mind keeping the existing Perl tests for a while, but I'd like
all new tests to be in Python, and have an agreement to migrate all
existing tests over time. I haven't really looked at these efforts in
detail so I don't know how long a transition period we need. Months? Years?
+1
I don't think we necessarily need a strict deadline on migrating
completely to Python. We will eventually and it doesn't look like Perl
tests require that much effort in terms of maintenance.
--
Best regards,
Aleksander Alekseev
Heikki Linnakangas <hlinnaka@iki.fi> writes:
On 16/06/2026 17:23, Andrew Dunstan wrote:
(A) Wholesale port with a flag day -- Python becomes *the*
framework, migrate the existing tests, stop accepting and eventually
remove the Perl one. One framework to learn and maintain, but a
large disruptive change with real dependency questions (which
Python, which libpq binding, buildfarm minimums).
+1 on this
I don't mind keeping the existing Perl tests for a while, but I'd like
all new tests to be in Python, and have an agreement to migrate all
existing tests over time. I haven't really looked at these efforts in
detail so I don't know how long a transition period we need. Months? Years?
I agree that "one framework" is a good long-term goal, but I think the
key word there has to be "long". In particular, what are we going to
do when we need to back-patch a new test?
I'm not super attracted to the idea of back-patching a whole new test
framework into stable branches. But the alternatives are not pretty
either --- eg, who will want to write a python test and then translate
it to perl for the back branches?
Discuss.
regards, tom lane
Hi Tom,
I'm not super attracted to the idea of back-patching a whole new test
framework into stable branches. But the alternatives are not pretty
either --- eg, who will want to write a python test and then translate
it to perl for the back branches?
That's a fair concern. It seems to me that back-patching TAP tests
doesn't happen too often. Also modern LLM agents should be quite good
at it.
--
Best regards,
Aleksander Alekseev
On 16 Jun 2026, at 17:15, Aleksander Alekseev <aleksander@tigerdata.com> wrote:
Hi Tom,
I'm not super attracted to the idea of back-patching a whole new test
framework into stable branches. But the alternatives are not pretty
either --- eg, who will want to write a python test and then translate
it to perl for the back branches?
Even if we backpatch a new framework, we'd still need to port all the tests of
that release for it to be useful which seems like a pretty big hammer to swing.
That's a fair concern. It seems to me that back-patching TAP tests
doesn't happen too often.
A quick look at REL_14 shows that there have been 450+ commits touching
src/test in some fashion since 14.0 was stamped. It's not the same as
backpatching of TAP tests but it's a data point.
Also modern LLM agents should be quite good at it.
I don't disagree, and it will no doubt be helpful, but I also don't think
process should be based on availability of tooling outside of our control.
--
Daniel Gustafsson
On Tue, Jun 16, 2026 at 11:16 AM Aleksander Alekseev
<aleksander@tigerdata.com> wrote:
I'm not super attracted to the idea of back-patching a whole new test
framework into stable branches. But the alternatives are not pretty
either --- eg, who will want to write a python test and then translate
it to perl for the back branches?That's a fair concern. It seems to me that back-patching TAP tests
doesn't happen too often. Also modern LLM agents should be quite good
at it.
I've written a few backpatches, and actually writing a TAP test for
those is pretty common because you are fixing a bug and often one that
is hard to reproduce (or it would get caught earlier) and only
testable in TAP. However, I always had to write variants for each
branch that were different enough for different reasons that I
wouldn't have minded writing it in a different framework -- especially
when the initial draft version could be written by an LLM.
I think doing this is far preferable to backporting the test framework
-- which will likely be a quite active development area and thus
itself require lots of backpatches. That sounds like a good way to
make everyone hate the new test framework.
Now, granted, I have only averaged a couple backpatches a year, so
those more prolific authors may feel different.
- Melanie
Aleksander Alekseev <aleksander@tigerdata.com> writes:
I'm not super attracted to the idea of back-patching a whole new test
framework into stable branches. But the alternatives are not pretty
either --- eg, who will want to write a python test and then translate
it to perl for the back branches?
That's a fair concern. It seems to me that back-patching TAP tests
doesn't happen too often. Also modern LLM agents should be quite good
at it.
I think you're underestimating the frequency with which we back-patch
test changes. Looking just at the v17 branch back to 1 January,
I see four commits that added entirely new *.pl test files, and
another 16 that modified one, plus four more that touched test
infrastructure (*.pm files). So that's nearly 10% of the total
commit traffic in that branch (262 commits). We're going to need
to deal with this on a very regular basis. Maybe we can wave it
all away with "AI will save us", but I've got doubts.
regards, tom lane
Daniel Gustafsson <daniel@yesql.se> writes:
Even if we backpatch a new framework, we'd still need to port all the tests of
that release for it to be useful which seems like a pretty big hammer to swing.
Agreed. But based on the quick count I just did, we are also going to
need to deal with a lot of cases of modifying an existing test. If
the "same" test is written completely differently in master, that's
going to be super painful.
I'm starting to think that we'd thank ourselves in the long run
if we did do the conversion in all branches at once.
regards, tom lane
On 16 Jun 2026, at 17:40, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Daniel Gustafsson <daniel@yesql.se> writes:
Even if we backpatch a new framework, we'd still need to port all the tests of
that release for it to be useful which seems like a pretty big hammer to swing.Agreed. But based on the quick count I just did, we are also going to
need to deal with a lot of cases of modifying an existing test. If
the "same" test is written completely differently in master, that's
going to be super painful.
I'm afraid that you're right, and I 100% agree that any decision we make should
take ease-of-backpatching into strong consideration.
--
Daniel Gustafsson
On Tue, Jun 16, 2026 at 8:41 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
I'm starting to think that we'd thank ourselves in the long run
if we did do the conversion in all branches at once.
Avoiding the "big bang" port was one of the reasons I proposed adding
a Python suite that only does things we don't currently do in Perl (a
pared-down option B, plus Test::Session). That way, we shake out the
Python deployment/maintenance questions and begin to develop a shared
vocabulary *before* the entirety of the test suite starts depending on
it.
That wouldn't preclude an eventual flag day, if we decided to. But if
Python can land and be developed while other people are continuing to
hack on features and backport tests, I think that's more likely to
succeed. I agree with Melanie; I really don't want people to hate
this.
--Jacob
Jacob Champion <jacob.champion@enterprisedb.com> writes:
On Tue, Jun 16, 2026 at 8:41 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
I'm starting to think that we'd thank ourselves in the long run
if we did do the conversion in all branches at once.
Avoiding the "big bang" port was one of the reasons I proposed adding
a Python suite that only does things we don't currently do in Perl (a
pared-down option B, plus Test::Session). That way, we shake out the
Python deployment/maintenance questions and begin to develop a shared
vocabulary *before* the entirety of the test suite starts depending on
it.
There's definitely a lot to be said for having an initial phase where
only all-new test scripts get written in Python, and we shake out the
test infrastructure's teething problems in master only. Once we feel
that we have that pretty much down, we could think about converting
the existing tests. Based on this conversation, I think that step
will probably need to touch the back branches too.
regards, tom lane
On Tue, Jun 16, 2026 at 9:02 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Once we feel
that we have that pretty much down, we could think about converting
the existing tests. Based on this conversation, I think that step
will probably need to touch the back branches too.
It would probably be pretty easy to do that in slices. Backport the
Python core, then develop the replacement for a segment of related
tests as a reviewable and backportable chunk. That lets us refactor
the testing APIs as we go, and if there are failures after backport,
at least it's just that one area of the tests and not "the whole
thing".
--Jacob
Jacob Champion <jacob.champion@enterprisedb.com> writes:
It would probably be pretty easy to do that in slices. Backport the
Python core, then develop the replacement for a segment of related
tests as a reviewable and backportable chunk. That lets us refactor
the testing APIs as we go, and if there are failures after backport,
at least it's just that one area of the tests and not "the whole
thing".
Makes sense.
regards, tom lane
On Tue, Jun 16, 2026 at 12:20:12PM -0400, Tom Lane wrote:
Jacob Champion <jacob.champion@enterprisedb.com> writes:
It would probably be pretty easy to do that in slices. Backport the
Python core, then develop the replacement for a segment of related
tests as a reviewable and backportable chunk. That lets us refactor
the testing APIs as we go, and if there are failures after backport,
at least it's just that one area of the tests and not "the whole
thing".Makes sense.
Backporting the core Python infrastructure sounds like a first
mandatory step here. Under time-pressure with a release timeline,
complicated/sensitive fixes may require tests across all the stable
branches. Not having to translate the same test across two
infrastructures would save a lot of brain resources.
Backporting changes of the tests into the new Python infrastructure
would make backpatching less complicated, indeed. It is much more
common for me to add tests to existing scripts than create new ones.
For a routine fix, I'd be OK to switch an existing test from perl to
Python before doing a backpatch and expanding a script. Now, I don't
think that we should close the door to the possibility of having to
add some perl code in an existing perl script if we are under a time
constraint (imagine in the security area).
The only thing I'd argue to avoid is having to maintain two versions
of the same test across *two* infrastructures.
--
Michael
On Tue, Jun 16, 2026 at 4:02 PM Michael Paquier <michael@paquier.xyz> wrote:
The only thing I'd argue to avoid is having to maintain two versions
of the same test across *two* infrastructures.
I think that's a reasonable requirement.
--Jacob
On 2026-06-16 Tu 7:02 PM, Michael Paquier wrote:
On Tue, Jun 16, 2026 at 12:20:12PM -0400, Tom Lane wrote:
Jacob Champion <jacob.champion@enterprisedb.com> writes:
It would probably be pretty easy to do that in slices. Backport the
Python core, then develop the replacement for a segment of related
tests as a reviewable and backportable chunk. That lets us refactor
the testing APIs as we go, and if there are failures after backport,
at least it's just that one area of the tests and not "the whole
thing".Makes sense.
Backporting the core Python infrastructure sounds like a first
mandatory step here. Under time-pressure with a release timeline,
complicated/sensitive fixes may require tests across all the stable
branches. Not having to translate the same test across two
infrastructures would save a lot of brain resources.Backporting changes of the tests into the new Python infrastructure
would make backpatching less complicated, indeed. It is much more
common for me to add tests to existing scripts than create new ones.
For a routine fix, I'd be OK to switch an existing test from perl to
Python before doing a backpatch and expanding a script. Now, I don't
think that we should close the door to the possibility of having to
add some perl code in an existing perl script if we are under a time
constraint (imagine in the security area).The only thing I'd argue to avoid is having to maintain two versions
of the same test across *two* infrastructures.
OK, so the consensus I'm sensing is that we should add a pytest
framework, backpatch it into all the live branches, and initially port
some relatively contained set of tests and backpatch that too. If I had
to pick a candidate suite it would be the recovery tests (and my patch
set conveniently contains a patch specifically for those). It's the
suite that has had the most churn, and has also been somewhat fragile
historically. Then we can convert tests in manageable chunks. I guess
that means the Test::Session work gets left on the cutting room floor -
if we run into issues with perl tests the answer will be to convert them
to pytest.
This is going to make Python a hard requirement for non-psql based
testing. There are currently 7 buildfarm animals building master without
enabling python, although 4 of those are building with meson so
presumably python is available to them.
It's also going to mean that some of us (including me) will need to
vastly improve our python fluency.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
On Wed, Jun 17, 2026 at 6:39 AM Andrew Dunstan <andrew@dunslane.net> wrote:
OK, so the consensus I'm sensing is that we should add a pytest
framework, backpatch it into all the live branches, and initially port
some relatively contained set of tests and backpatch that too.
As for "new tests that can't easily be written today": I've already
written several protocol-level tests for OAuth that implement broken
client and server behavior (they're in the pg-pytest-suite thread).
Those could be ported for review on top of the new framework. I could
look at some of the more recent ad-hoc Perl socket tests at the same
time.
If I had
to pick a candidate suite it would be the recovery tests (and my patch
set conveniently contains a patch specifically for those).
No strong opinions here on the candidates for initial backport -- but
I will note that src/test/recovery doesn't look like a small slice, at
13k lines and more than half of the Test::Cluster API used.
I guess
that means the Test::Session work gets left on the cutting room floor -
if we run into issues with perl tests the answer will be to convert them
to pytest.
If you think Test::Session is good to go, I see no reason not to
improve the Perl tests (since there seems to be a growing consensus
that the conversion won't happen overnight).
--Jacob
On 2026-06-17 We 12:01 PM, Jacob Champion wrote:
On Wed, Jun 17, 2026 at 6:39 AM Andrew Dunstan <andrew@dunslane.net> wrote:
OK, so the consensus I'm sensing is that we should add a pytest
framework, backpatch it into all the live branches, and initially port
some relatively contained set of tests and backpatch that too.As for "new tests that can't easily be written today": I've already
written several protocol-level tests for OAuth that implement broken
client and server behavior (they're in the pg-pytest-suite thread).
Those could be ported for review on top of the new framework. I could
look at some of the more recent ad-hoc Perl socket tests at the same
time.If I had
to pick a candidate suite it would be the recovery tests (and my patch
set conveniently contains a patch specifically for those).No strong opinions here on the candidates for initial backport -- but
I will note that src/test/recovery doesn't look like a small slice, at
13k lines and more than half of the Test::Cluster API used.I guess
that means the Test::Session work gets left on the cutting room floor -
if we run into issues with perl tests the answer will be to convert them
to pytest.If you think Test::Session is good to go, I see no reason not to
improve the Perl tests (since there seems to be a growing consensus
that the conversion won't happen overnight).
There's a fairly high amount of friction here, though. If we didn't plan
to replace the perl tests it would be worth it, but I'm not sure it's
worth it if we are. To keep things in sync, which people have expressed
concerns about, we'd need to backport it. The whole set is not humungous
but it's not tiny either:
59 files changed, 3317 insertions(+), 1113 deletions(-)
Then there's the matter of needing either libffi + FFI::Platypus or
needing to create/maintain an XS wrapper.
With pytest I feel much more confident about backpatching it because all
we need is python itself + pytest and ideally pexpect;
So I'm prepared to put the work in if people think it's a good idea, but
I want a fairly solid indication of that.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
On Wed, Jun 17, 2026 at 2:05 PM Andrew Dunstan <andrew@dunslane.net> wrote:
If you think Test::Session is good to go, I see no reason not to
improve the Perl tests (since there seems to be a growing consensus
that the conversion won't happen overnight).There's a fairly high amount of friction here, though. If we didn't plan
to replace the perl tests it would be worth it, but I'm not sure it's
worth it if we are.
Ah, I didn't realize that. In that case you're probably right
(especially if the opportunity cost is time helping with the Python
port).
--Jacob