Python/pytest test framework take two
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
Based on various discussions there seems to be enough desire for writing
tests using pytest. The previous mailinglist discussion[1]/messages/by-id/CAOYmi+kThkM9Z87u=R_Wi7fCor2i+UZKAyq0UCyprzCwTQvqgA@mail.gmail.com became quite
long and discussed various different aspects. So I'm starting a new
thread with my most recent attempt at a "nice to use" framework. It has
many changes since the last one.
The changes resulted from my experience converting ~half of the Perl
tests[2]https://github.com/JelteF/postgres/pull/36 using an LLM, which revealed gaps and awkward usage patterns.
The last couple of patches in this patchset are a few of those
conversions. These are NOT meant to be committed, at least not until
this thread[3]/messages/by-id/cdaaf722-4529-435b-9340-cedf1a3a277f@dunslane.net reaches a conclusion about conversions. They are only
meant to showcase the framework for now.
The patches in the patchset have detailed commit messages. But here's a
short summary per patch:
0001: Small patch to make meson CI output better in general by not
truncating error logs as much
0002: Small meson refactoring, moving a magic number into a variable
0003: Installing 32-bit python, can be removed after pg-vm-images PR is
merged
0004: Basic pytest support for the build system and CI.
0005: Introduction of the actual framework that can manage Postgres
servers and send queries
0006-0007: Two short converted tests that run much faster using this
framework
0008: Shows off the background_sql functionality, which is much easier
to use and more robust than Perl its background_psql
0009: Conversion of the 001_aio test, which is the most unreadable Perl
test I could find (sorry Andres). It becomes, less buggy, more
readable (imo), shorter and faster when written in Python.
0010-0012: Some more tests that are faster and/or easier to read+write
in Python.
[1]: /messages/by-id/CAOYmi+kThkM9Z87u=R_Wi7fCor2i+UZKAyq0UCyprzCwTQvqgA@mail.gmail.com
[2]: https://github.com/JelteF/postgres/pull/36
[3]: /messages/by-id/cdaaf722-4529-435b-9340-cedf1a3a277f@dunslane.net
P.S. A full re-review of the framework code is needed, but to the main
things that changed since the first attempt:
1. Each test file now gets its own server, instead of every file sharing
a single one for the whole session.
2. The reloading()/restarting() context managers became plain
append_conf()/adjust_conf() methods.
3. sql() now sends queries using the extended protocol so it can take
query parameters. It also gained batch, oneshot and background variants.
4. pg.sql() reuses a cached connection per test, so the separate conn
fixture is gone.
5. NOTICE/WARNING arrive as Python warnings so they can be asserted on
and so they are logged.
6. PostgresServer grew from 15 to ~55 methods based on missing
functionality. New things that are supported are: backups, streaming,
catchup, logs, injection points, WAL.
7. Installed programs are callables that can be pointed at a server,
e.g. psql("-c", "SELECT 1", server=pg) or
pg_controldata.capture(pg.datadir).
8. PG_TEST_TIMEOUT_DEFAULT now bounds each individual wait or poll
instead of the test as a whole (some tests took longer than the
default for their full runtime.
9. Ports are chosen the same way that Perl does, so the co-operative
locking is shared between the two frameworks to avoid port conflicts.
10. Nearly the whole framework is type-annotated now (155 of 169
function signatures, up from 13 of 139), which editors and pyright
can check.
11. installcheck works now
12. pgxs.mk support
13. Works with the new GitHub Actions CI now.
Attachments:
v1-0001-ci-show-more-than-100-lines-of-a-failing-test-s-l.patchtext/x-patch; charset=utf-8; name=v1-0001-ci-show-more-than-100-lines-of-a-failing-test-s-l.patchDownload+1-2
v1-0002-meson-Deduplicate-test-timeout-into-a-variable.patchtext/x-patch; charset=utf-8; name=v1-0002-meson-Deduplicate-test-timeout-into-a-variable.patchDownload+8-5
v1-0003-ci-temporarily-install-a-32-bit-Python-in-the-32-.patchtext/x-patch; charset=utf-8; name=v1-0003-ci-temporarily-install-a-32-bit-Python-in-the-32-.patchDownload+36-1
v1-0004-Add-support-for-pytest-test-suites.patchtext/x-patch; charset=utf-8; name=v1-0004-Add-support-for-pytest-test-suites.patchDownload+707-10
v1-0005-Add-pytest-infrastructure-to-interact-with-Postgr.patchtext/x-patch; charset=utf-8; name=v1-0005-Add-pytest-infrastructure-to-interact-with-Postgr.patchDownload+3977-13
v1-0006-POC-amcheck-port-TAP-test-001_verify_heapam-to-py.patchtext/x-patch; charset=utf-8; name=v1-0006-POC-amcheck-port-TAP-test-001_verify_heapam-to-py.patchDownload+203-275
v1-0007-POC-test_json_parser-port-TAP-test-002_inline-to-.patchtext/x-patch; charset=utf-8; name=v1-0007-POC-test_json_parser-port-TAP-test-002_inline-to-.patchDownload+169-170
v1-0008-POC-recovery-port-TAP-test-049_wait_for_lsn-to-py.patchtext/x-patch; charset=utf-8; name=v1-0008-POC-recovery-port-TAP-test-049_wait_for_lsn-to-py.patchDownload+808-1141
v1-0009-POC-test_aio-port-TAP-test-001_aio-to-pytest.patchtext/x-patch; charset=utf-8; name=v1-0009-POC-test_aio-port-TAP-test-001_aio-to-pytest.patchDownload+1144-1904
v1-0010-POC-recovery-port-TAP-test-029_stats_restart-to-p.patchtext/x-patch; charset=utf-8; name=v1-0010-POC-recovery-port-TAP-test-029_stats_restart-to-p.patchDownload+306-378
v1-0011-POC-recovery-port-TAP-test-031_recovery_conflict-.patchtext/x-patch; charset=utf-8; name=v1-0011-POC-recovery-port-TAP-test-031_recovery_conflict-.patchDownload+309-335
v1-0012-POC-subscription-port-TAP-test-031_column_list-to.patchtext/x-patch; charset=utf-8; name=v1-0012-POC-subscription-port-TAP-test-031_column_list-to.patchDownload+973-1310
On 19 Aug 2026, at 10:59, Jelte Fennema-Nio <me@jeltef.nl> wrote:
0009: Conversion of the 001_aio test, which is the most unreadable Perl
test I could find (sorry Andres). It becomes, less buggy, more
readable (imo), shorter and faster when written in Python.
Are these bugs on the list already? If there are bugs in the current test it
would be helpful to submit them separately to make sure they are fixed in a
timely manner and not relying on the outcome of this thread.
--
Daniel Gustafsson
On Wed, 19 Aug 2026 at 11:47, Daniel Gustafsson <daniel@yesql.se> wrote:
On 19 Aug 2026, at 10:59, Jelte Fennema-Nio <me@jeltef.nl> wrote:
0009: Conversion of the 001_aio test, which is the most unreadable Perl
test I could find (sorry Andres). It becomes, less buggy, more
readable (imo), shorter and faster when written in Python.Are these bugs on the list already? If there are bugs in the current test it
would be helpful to submit them separately to make sure they are fixed in a
timely manner and not relying on the outcome of this thread.
They weren't yet, but I created a thread now:
/messages/by-id/DKSU6GI1YLG5.3VF6M4IRKQ7XE@jeltef.nl
On 19 Aug 2026, at 12:10, Jelte Fennema-Nio <me@jeltef.nl> wrote:
On Wed, 19 Aug 2026 at 11:47, Daniel Gustafsson <daniel@yesql.se> wrote:
On 19 Aug 2026, at 10:59, Jelte Fennema-Nio <me@jeltef.nl> wrote:
0009: Conversion of the 001_aio test, which is the most unreadable Perl
test I could find (sorry Andres). It becomes, less buggy, more
readable (imo), shorter and faster when written in Python.Are these bugs on the list already? If there are bugs in the current test it
would be helpful to submit them separately to make sure they are fixed in a
timely manner and not relying on the outcome of this thread.They weren't yet, but I created a thread now:
/messages/by-id/DKSU6GI1YLG5.3VF6M4IRKQ7XE@jeltef.nl
Great, thanks!
--
Daniel Gustafsson
Hi,
On 2026-08-19 10:59:16 +0200, Jelte Fennema-Nio wrote:
Based on various discussions there seems to be enough desire for writing
tests using pytest. The previous mailinglist discussion[1] became quite
long and discussed various different aspects. So I'm starting a new
thread with my most recent attempt at a "nice to use" framework. It has
many changes since the last one.The changes resulted from my experience converting ~half of the Perl
tests[2] using an LLM, which revealed gaps and awkward usage patterns.
The last couple of patches in this patchset are a few of those
conversions. These are NOT meant to be committed, at least not until
this thread[3] reaches a conclusion about conversions. They are only
meant to showcase the framework for now.The patches in the patchset have detailed commit messages. But here's a
short summary per patch:
0001: Small patch to make meson CI output better in general by not
truncating error logs as much
-1. I don't see why we would want to do that. Printing 10k lines onto the fake
terminal in CI makes no sense whatsoever. If the pytest errors are unreadably
verbose, we need to fix that, not display more unusable output.
0002: Small meson refactoring, moving a magic number into a variable
WFM.
0003: Installing 32-bit python, can be removed after pg-vm-images PR is
merged
-1. As commented on that PR just now, this is a maintenance burden going
forward that likely Bilal or I will have to pay.
0004: Basic pytest support for the build system and CI.
0005: Introduction of the actual framework that can manage Postgres
servers and send queries
Phew, that's quite a bit of code to review :(. Not sure there's much that can
be done about that though...
0006-0007: Two short converted tests that run much faster using this
framework
0008: Shows off the background_sql functionality, which is much easier
to use and more robust than Perl its background_psql
0009: Conversion of the 001_aio test, which is the most unreadable Perl
test I could find (sorry Andres). It becomes, less buggy, more
readable (imo), shorter and faster when written in Python.
No reason to be sorry, I hated writing it.
0010-0012: Some more tests that are faster and/or easier to read+write
in Python.
What's your plan for validating that these do not actually reduce test
coverage substantially?
I think this needs some systematic work. E.g. build with coverage, clear all
coverage, running the perl test, collect coverage, clear coverage, run the
python test, compare the two coverages, including branch coverage.
Greetings,
Andres Freund
On Wed, 19 Aug 2026 at 15:16, Andres Freund <andres@anarazel.de> wrote:
-1. I don't see why we would want to do that. Printing 10k lines onto the fake
terminal in CI makes no sense whatsoever. If the pytest errors are unreadably
verbose, we need to fix that, not display more unusable output.
I agree that 10k is way too much, and also not necessary. Some of the
failed output has been very verbose though, and I have some ideas on
how to fix that (silence some commands, limit size of logs shown in
the output, etc). I'll work on that for the next version. However, the
Meson default of 100 lines is very small. I think we should bump that
to a few hundred e.g. 300-500 lines. There's quite a bit of *useful*
output that we can show that still easily overflows a 100 line limit:
1. Pytest shows the stack trace of an error
2. Values of local variables
3. Multiline diff view of the diff between two lists if an asserted
comparison on them fails
4. stderr, stdout and python logging
5. PostgreSQL logs of the servers
0003: Installing 32-bit python, can be removed after pg-vm-images PR is
merged-1. As commented on that PR just now, this is a maintenance burden going
forward that likely Bilal or I will have to pay.
I responded in that PR, let's discuss there further. I'm fine using
another approach, but we need a solution for 32bit.
0004: Basic pytest support for the build system and CI.
0005: Introduction of the actual framework that can manage Postgres
servers and send queriesPhew, that's quite a bit of code to review :(. Not sure there's much that can
be done about that though...
Yeah, I'm also not sure how to make it much better. I guess some
methods on the PostgresServer could be left out of the initial commit
and added in a follow-up one. But I don't think that would improve the
situation much in the end.
No reason to be sorry, I hated writing it.
I can only imagine...
0010-0012: Some more tests that are faster and/or easier to read+write
in Python.What's your plan for validating that these do not actually reduce test
coverage substantially?I think this needs some systematic work. E.g. build with coverage, clear all
coverage, running the perl test, collect coverage, clear coverage, run the
python test, compare the two coverages, including branch coverage.
To be clear on my intent with this patchset: I DO NOT want to make
converting all/most/some perl tests a goal of this patchset. The ones
converted here are purely intended as a review-aid to show the pytest
framework in action. i.e. so reviewers can see what a test would look
like, as well as serving as a proof that it supports the functionality
we need to write a wide range of tests. Merging support for writing
Pytest tests would be a huge win on its own, even without converting
any tests. Tests for new features could then be written using it. And
more practically speaking: I don't want to spend my energy discussing
the details and difficulties of converting tests until we have a
pytest framework in main.
But responding to your thoughts on it anyway: I agree that if/when we
start converting Perl tests to Python, we should have do coverage
measurements