RFC: adding pytest as a supported test framework
Hi all,
For the v18 cycle, I would like to try to get pytest [1]https://docs.pytest.org/ in as a
supported test driver, in addition to the current offerings.
(I'm tempted to end the email there.)
We had an unconference session at PGConf.dev [2]https://wiki.postgresql.org/wiki/PGConf.dev_2024_Developer_Unconference#New_testing_frameworks around this topic.
There seemed to be a number of nodding heads and some growing
momentum. So here's a thread to try to build wider consensus. If you
have a competing or complementary test proposal in mind, heads up!
== Problem Statement(s) ==
1. We'd like to rerun a failing test by itself.
2. It'd be helpful to see _what_ failed without poring over logs.
These two got the most nodding heads of the points I presented. (#1
received tongue-in-cheek applause.) I think most modern test
frameworks are going to give you these things, but unfortunately we
don't have them.
Additionally,
3. Many would like to use modern developer tooling during testing
(language servers! autocomplete! debuggers! type checking!) and we
can't right now.
4. It'd be great to split apart client-side tests from server-side
tests. Driving Postgres via psql all the time is fine for acceptance
testing, but it becomes a big problem when you need to test how
clients talk to servers with incompatible feature sets, or how a peer
behaves when talking to something buggy.
5. Personally, I want to implement security features test-first (for
high code coverage and other benefits), and our Perl-based tests are
usually too cumbersome for that.
== Why pytest? ==
From the small and biased sample at the unconference session, it looks
like a number of people have independently settled on pytest in their
own projects. In my opinion, pytest occupies a nice space where it
solves some of the above problems for us, and it gives us plenty of
tools to solve the other problems without too much pain.
Problem 1 (rerun failing tests): One architectural roadblock to this
in our Test::More suite is that tests depend on setup that's done by
previous tests. pytest allows you to declare each test's setup
requirements via pytest fixtures, letting the test runner build up the
world exactly as it needs to be for a single isolated test. These
fixtures may be given a "scope" so that multiple tests may share the
same setup for performance or other reasons.
Problem 2 (seeing what failed): pytest does this via assertion
introspection and very detailed failure reporting. If you haven't seen
this before, take a look at the pytest homepage [1]https://docs.pytest.org/; there's an
example of a full log.
Problem 3 (modern tooling): We get this from Python's very active
developer base.
Problems 4 (splitting client and server tests) and 5 (making it easier
to write tests first) aren't really Python- or pytest-specific, but I
have done both quite happily in my OAuth work [3]https://github.com/jchampio/pg-pytest-suite, and I've since
adapted that suite multiple times to develop and test other proposals
on this list, like LDAP/SCRAM, client encryption, direct SSL, and
compression.
Python's standard library has lots of power by itself, with very good
documentation. And virtualenvs and better package tooling have made it
much easier, IMO, to avoid the XKCD dependency tangle [4]https://xkcd.com/1987/ of the
2010s. When it comes to third-party packages, which I think we're
probably going to want in moderation, we would still need to discuss
supply chain safety. Python is not as mature here as, say, Go.
== A Plan ==
Even if everyone were on board immediately, there's a lot of work to
do. I'd like to add pytest in a more probationary status, so we can
iron out the inevitable wrinkles. My proposal would be:
1. Commit bare-bones support in our Meson setup for running pytest, so
everyone can kick the tires independently.
2. Add a test for something that we can't currently exercise.
3. Port a test from a place where the maintenance is terrible, to see
if we can improve it.
If we hate it by that point, no harm done; tear it back out. Otherwise
we keep rolling forward.
Thoughts? Suggestions?
Thanks,
--Jacob
[1]: https://docs.pytest.org/
[2]: https://wiki.postgresql.org/wiki/PGConf.dev_2024_Developer_Unconference#New_testing_frameworks
[3]: https://github.com/jchampio/pg-pytest-suite
[4]: https://xkcd.com/1987/
Hi!
On Mon, Jun 10, 2024 at 9:46 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
Thoughts? Suggestions?
Thank you for working on this.
Do you think you could re-use something from testgres[1] package?
Links.
1. https://github.com/postgrespro/testgres
------
Regards,
Alexander Korotkov
Supabase
Hi,
Just for context for the rest the email: I think we desperately need to move
off perl for tests. The infrastructure around our testing is basically
unmaintained and just about nobody that started doing dev stuff in the last 10
years learned perl.
On 2024-06-10 11:46:00 -0700, Jacob Champion wrote:
4. It'd be great to split apart client-side tests from server-side
tests. Driving Postgres via psql all the time is fine for acceptance
testing, but it becomes a big problem when you need to test how
clients talk to servers with incompatible feature sets, or how a peer
behaves when talking to something buggy.
That seems orthogonal to using pytest vs something else?
== Why pytest? ==
From the small and biased sample at the unconference session, it looks
like a number of people have independently settled on pytest in their
own projects. In my opinion, pytest occupies a nice space where it
solves some of the above problems for us, and it gives us plenty of
tools to solve the other problems without too much pain.
We might be able to alleviate that by simply abstracting it away, but I found
pytest's testrunner pretty painful. Oodles of options that are not very well
documented and that often don't work because they are very specific to some
situations, without that being explained.
Problem 1 (rerun failing tests): One architectural roadblock to this
in our Test::More suite is that tests depend on setup that's done by
previous tests. pytest allows you to declare each test's setup
requirements via pytest fixtures, letting the test runner build up the
world exactly as it needs to be for a single isolated test. These
fixtures may be given a "scope" so that multiple tests may share the
same setup for performance or other reasons.
OTOH, that's quite likely to increase overall test times very
significantly. Yes, sometimes that can be avoided with careful use of various
features, but often that's hard, and IME is rarely done rigiorously.
Problem 2 (seeing what failed): pytest does this via assertion
introspection and very detailed failure reporting. If you haven't seen
this before, take a look at the pytest homepage [1]; there's an
example of a full log.
That's not really different than what the perl tap test stuff allows. We
indeed are bad at utilizing it, but I'm not sure that switching languages will
change that.
I think part of the problem is that the information about what precisely
failed is often much harder to collect when testing multiple servers
interacting than when doing localized unit tests.
I think we ought to invest a bunch in improving that, I'd hope that a lot of
that work would be largely independent of the language the tests are written
in.
Python's standard library has lots of power by itself, with very good
documentation. And virtualenvs and better package tooling have made it
much easier, IMO, to avoid the XKCD dependency tangle [4] of the
2010s.
Ugh, I think this is actually python's weakest area. There's about a dozen
package managers and "python distributions", that are at best half compatible,
and the documentation situation around this is *awful*.
When it comes to third-party packages, which I think we're
probably going to want in moderation, we would still need to discuss
supply chain safety. Python is not as mature here as, say, Go.
What external dependencies are you imagining?
== A Plan ==
Even if everyone were on board immediately, there's a lot of work to
do. I'd like to add pytest in a more probationary status, so we can
iron out the inevitable wrinkles. My proposal would be:1. Commit bare-bones support in our Meson setup for running pytest, so
everyone can kick the tires independently.
2. Add a test for something that we can't currently exercise.
3. Port a test from a place where the maintenance is terrible, to see
if we can improve it.If we hate it by that point, no harm done; tear it back out. Otherwise
we keep rolling forward.
I think somewhere between 1 and 4 a *substantial* amount of work would be
required to provide a bunch of the infrastructure that Cluster.pm etc
provide. Otherwise we'll end up with a lot of copy pasted code between tests.
Greetings,
Andres Freund
On 2024-06-10 Mo 16:04, Andres Freund wrote:
Hi,
Just for context for the rest the email: I think we desperately need to move
off perl for tests. The infrastructure around our testing is basically
unmaintained and just about nobody that started doing dev stuff in the last 10
years learned perl.
Andres,
I get that you don't like perl. But it's hard for me to take this
terribly seriously. "desperately" seems like massive overstatement at
best. As for what up and coming developers learn, they mostly don't
learn C either, and that's far more critical to what we do.
I'm not sure what part of the testing infrastructure you think is
unmaintained. For example, the last release of Test::Simple was all the
way back on April 25.
Maybe there are some technical superiorities about what Jacob is
proposing, enough for us to add it to our armory. I'll keep an open mind
on that.
But let's not throw the baby out with the bathwater. Quite apart from
anything else, a wholesale rework of the test infrastructure would make
backpatching more painful.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
On Mon, 10 Jun 2024 at 20:46, Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
For the v18 cycle, I would like to try to get pytest [1] in as a
supported test driver, in addition to the current offerings.
Huge +1 from me (but I'm definitely biased here)
Thoughts? Suggestions?
I think the most important thing is that we make it easy for people to
use this thing, and use it in a "correct" way. I have met very few
people that actually like writing tests, so I think it's very
important to make the barrier to do so as low as possible.
For the PgBouncer repo I created my own pytest based test suite more
~1.5 years ago now. I tried to make it as easy as possible to write
tests there, and it has worked out quite well imho. I don't think it
makes sense to copy all things I did there verbatim, because some of
it is quite specific to testing PgBouncer. But I do think there's
quite a few things that could probably be copied (or at least inspire
what you do). Some examples:
1. helpers to easily run shell commands, most importantly setting
check=True by default[1]https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L83-L131
2. helper to get a free tcp port[2]https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L210-L233
3. helper to check if the log contains a specific string[3]https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L1125-L1143
4. automatically show PG logs on test failure[4]https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L1075-L1103
5. helpers to easily run sql commands (psycopg interface isn't very
user friendly imho for the common case)[5]https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L326-L338
6. startup/teardown cleanup logic[6]https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L546-L642
[1]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L83-L131
[2]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L210-L233
[3]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L1125-L1143
[4]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L1075-L1103
[5]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L326-L338
[6]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L546-L642
On Mon, 10 Jun 2024 at 22:04, Andres Freund <andres@anarazel.de> wrote:
Problem 1 (rerun failing tests): One architectural roadblock to this
in our Test::More suite is that tests depend on setup that's done by
previous tests. pytest allows you to declare each test's setup
requirements via pytest fixtures, letting the test runner build up the
world exactly as it needs to be for a single isolated test. These
fixtures may be given a "scope" so that multiple tests may share the
same setup for performance or other reasons.OTOH, that's quite likely to increase overall test times very
significantly. Yes, sometimes that can be avoided with careful use of various
features, but often that's hard, and IME is rarely done rigiorously.
You definitely want to cache things like initdb and "pg_ctl start".
But that's fairly easy to do with some startup/teardown logic. For
PgBouncer I create a dedicated schema for each test that needs to
create objects and automatically drop that schema at the end of the
test[6]https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L546-L642 (including any temporary objects outside of schemas like
users/replication slots). You can even choose not to clean up certain
large schemas if they are shared across multiple tests.
Problem 2 (seeing what failed): pytest does this via assertion
introspection and very detailed failure reporting. If you haven't seen
this before, take a look at the pytest homepage [1]; there's an
example of a full log.That's not really different than what the perl tap test stuff allows. We
indeed are bad at utilizing it, but I'm not sure that switching languages will
change that.
It's not about allowing, it's about doing the thing that you want by
default. The following code
assert a == b
will show you the actual values of both a and b when the test fails,
instead of saying something like "false is not true". Ofcourse you can
provide a message here too, like with perl its ok function, but even
when you don't the output is helpful.
I think part of the problem is that the information about what precisely
failed is often much harder to collect when testing multiple servers
interacting than when doing localized unit tests.I think we ought to invest a bunch in improving that, I'd hope that a lot of
that work would be largely independent of the language the tests are written
in.
Well, as you already noted no-one that started doing dev stuff in the
last 10 years knows Perl nor wants to learn it. So a large part of the
community tries to touch the current perl test suite as little as
possible. I personally haven't tried to improve anything about our
perl testing framework, even though I'm normally very much into
improving developer tooling.
Python's standard library has lots of power by itself, with very good
documentation. And virtualenvs and better package tooling have made it
much easier, IMO, to avoid the XKCD dependency tangle [4] of the
2010s.Ugh, I think this is actually python's weakest area. There's about a dozen
package managers and "python distributions", that are at best half compatible,
and the documentation situation around this is *awful*.
I definitely agree this is Python its weakest area. But since venv is
part of the python standard library it's much better. I have the
following short blurb in PgBouncer its test README[7]https://github.com/pgbouncer/pgbouncer/blob/master/test/README.md and it has
worked for all contributors so far:
# create a virtual environment (only needed once)
python3 -m venv env
# activate the environment. You will need to activate this environment in
# your shell every time you want to run the tests. (so it's needed once per
# shell).
source env/bin/activate
# Install the dependencies (only needed once, or whenever extra dependencies
# get added to requirements.txt)
pip install -r requirements.txt
[7]: https://github.com/pgbouncer/pgbouncer/blob/master/test/README.md
I think somewhere between 1 and 4 a *substantial* amount of work would be
required to provide a bunch of the infrastructure that Cluster.pm etc
provide. Otherwise we'll end up with a lot of copy pasted code between tests.
Totally agreed, that we should have a fairly decent base to work on
top of. I think we should at least port a few tests to show that the
base has at least the most basic functionality.
On Mon, 10 Jun 2024 at 22:47, Andrew Dunstan <andrew@dunslane.net> wrote:
As for what up and coming developers learn, they mostly don't learn C either, and that's far more critical to what we do.
I think many up and coming devs have at least touched C somewhere
(e.g. in university). And because it's more critical to the project
and also to many other low level projects, they don't mind learning it
so much if they don't know it yet. But I, for example, try to write as
few Perl tests as possible, because getting good at Perl has pretty
much no use to me outside of writing tests for postgres.
(I do personally think that official Rust support in Postgres would
probably be a good thing, but that is a whole other discussion that
I'd like to save for some other day)
But let's not throw the baby out with the bathwater. Quite apart from anything else, a wholesale rework of the test infrastructure would make backpatching more painful.
Backporting test improvements to decrease backporting pain is
something we don't look badly upon afaict (Citus its test suite breaks
semi-regularly on minor PG version updates due to some slight output
changes introduced by e.g. an updated version of the isolationtester).
Hi,
On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote:
On 2024-06-10 Mo 16:04, Andres Freund wrote:
Hi,
Just for context for the rest the email: I think we desperately need to move
off perl for tests. The infrastructure around our testing is basically
unmaintained and just about nobody that started doing dev stuff in the last 10
years learned perl.
Andres,
I get that you don't like perl.
I indeed don't particularly like perl - but that's really not the main
issue. I've already learned [some of] it. What is the main issue is that I've
also watched several newer folks try to write tests in it, and it was not
pretty.
But it's hard for me to take this terribly seriously. "desperately" seems
like massive overstatement at best.
Shrug.
As for what up and coming developers learn, they mostly don't learn C
either, and that's far more critical to what we do.
C is a a lot more useful to to them than perl. And it's actually far more
widely known these days than perl. C does teach you some reasonably
low-level-ish understanding of hardware. There are gazillions of programs
written in C that we'll have to maintain for decades. I don't think that's
comparably true for perl.
I'm not sure what part of the testing infrastructure you think is
unmaintained. For example, the last release of Test::Simple was all the way
back on April 25.
IPC::Run is quite buggy and basically just maintained by Noah these days.
Greetings,
Andres Freund
On 2024-06-10 Mo 21:49, Andres Freund wrote:
Hi,
On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote:
On 2024-06-10 Mo 16:04, Andres Freund wrote:
Hi,
Just for context for the rest the email: I think we desperately need to move
off perl for tests. The infrastructure around our testing is basically
unmaintained and just about nobody that started doing dev stuff in the last 10
years learned perl.Andres,
I get that you don't like perl.
I indeed don't particularly like perl - but that's really not the main
issue. I've already learned [some of] it. What is the main issue is that I've
also watched several newer folks try to write tests in it, and it was not
pretty.
Hmm. I've done webinars in the past about how to write TAP tests for
PostgreSQL, maybe I need to beef that up some.
I'm not sure what part of the testing infrastructure you think is
unmaintained. For example, the last release of Test::Simple was all the way
back on April 25.IPC::Run is quite buggy and basically just maintained by Noah these days.
Yes, that's true. I think the biggest pain point is possibly the
recovery tests.
Some time ago I did some work on wrapping libpq using the perl FFI
module. It worked pretty well, and would mean we could probably avoid
many uses of IPC::Run, and would probably be substantially more
efficient (no fork required). It wouldn't avoid all uses of IPC::Run,
though.
But my point was mainly that while a new framework might have value, I
don't think we need to run out and immediately rewrite several hundred
TAP tests. Let's pick the major pain points and address those.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
On Mon, Jun 10, 2024 at 1:04 PM Andres Freund <andres@anarazel.de> wrote:
Just for context for the rest the email: I think we desperately need to move
off perl for tests. The infrastructure around our testing is basically
unmaintained and just about nobody that started doing dev stuff in the last 10
years learned perl.
Okay. Personally, I'm going to try to stay out of discussions around
subtracting Perl and focus on adding Python, for a bunch of different
reasons:
- Tests aren't cheap, but in my experience, the maintenance-cost math
for tests is a lot different than the math for implementations.
- I don't personally care for Perl, but having tests in any form is
usually better than not having them.
- Trying to convince people to get rid of X while adding Y is a good
way to make sure Y never happens.
On 2024-06-10 11:46:00 -0700, Jacob Champion wrote:
4. It'd be great to split apart client-side tests from server-side
tests. Driving Postgres via psql all the time is fine for acceptance
testing, but it becomes a big problem when you need to test how
clients talk to servers with incompatible feature sets, or how a peer
behaves when talking to something buggy.That seems orthogonal to using pytest vs something else?
Yes, I think that's fair. It's going to be hard not to talk about
"things that pytest+Python don't give us directly but are much easier
to build" in all of this (and I tried to call that out in the next
section, maybe belatedly). I think I'm going to have to convince both
a group of people who want to ask "why pytest in particular?" and a
group of people who ask "why isn't what we have good enough?"
== Why pytest? ==
From the small and biased sample at the unconference session, it looks
like a number of people have independently settled on pytest in their
own projects. In my opinion, pytest occupies a nice space where it
solves some of the above problems for us, and it gives us plenty of
tools to solve the other problems without too much pain.We might be able to alleviate that by simply abstracting it away, but I found
pytest's testrunner pretty painful. Oodles of options that are not very well
documented and that often don't work because they are very specific to some
situations, without that being explained.
Hm. There are a bunch of them, but I've never needed to go through the
oodles of options. Anything in particular that caused problems?
Problem 1 (rerun failing tests): One architectural roadblock to this
in our Test::More suite is that tests depend on setup that's done by
previous tests. pytest allows you to declare each test's setup
requirements via pytest fixtures, letting the test runner build up the
world exactly as it needs to be for a single isolated test. These
fixtures may be given a "scope" so that multiple tests may share the
same setup for performance or other reasons.OTOH, that's quite likely to increase overall test times very
significantly. Yes, sometimes that can be avoided with careful use of various
features, but often that's hard, and IME is rarely done rigiorously.
Well, scopes are pretty front and center when you start building
pytest fixtures, and the complicated longer setups will hopefully
converge correctly early on and be reused everywhere else. I imagine
no one wants to build cluster setup from scratch.
On a slight tangent, is this not a problem today? I mean... part of my
personal long-term goal is in increasing test hygiene, which is going
to take some shifts in practice. As long as review keeps the quality
of the tests fairly high, I see the inevitable "our tests take too
long" problem as a good one. That's true no matter what framework we
use, unless the framework is so bad that no one uses it and the
runtime is trivial. If we're worried that people will immediately
start exploding the runtime and no one will notice during review,
maybe we can have some infrastructure flag how much a patch increased
it?
Problem 2 (seeing what failed): pytest does this via assertion
introspection and very detailed failure reporting. If you haven't seen
this before, take a look at the pytest homepage [1]; there's an
example of a full log.That's not really different than what the perl tap test stuff allows. We
indeed are bad at utilizing it, but I'm not sure that switching languages will
change that.
Jelte already touched on this, but I wanted to hammer on the point: If
no one, not even the developers who chose and like Perl, is using
Test::More in a way that's maintainable, I would prefer to use a
framework that does maintainable things by default so that you have to
try really hard to screw it up. It is possible to screw up `assert
actual == expected`, but it takes more work than doing it the right
way.
I think part of the problem is that the information about what precisely
failed is often much harder to collect when testing multiple servers
interacting than when doing localized unit tests.I think we ought to invest a bunch in improving that, I'd hope that a lot of
that work would be largely independent of the language the tests are written
in.
We do a lot more acceptance testing than internal testing, which came
up as a major complaint from me and others during the unconference.
One of the reasons people avoid writing internal tests in Perl is
because it's very painful to find a rhythm with Test::More. From
experience test-driving the OAuth work, I'm *very* happy with the
development cycle that pytest gave me.
Other languages _could_ do that, sure. It's a simple matter of programming...
Ugh, I think this is actually python's weakest area. There's about a dozen
package managers and "python distributions", that are at best half compatible,
and the documentation situation around this is *awful*.
So... don't support the half-compatible stuff? I thought this
conversation was still going on with Windows Perl (ActiveState?
Strawberry?) but everyone just seems to pick what works for them and
move on to better things to do.
Modern CPython includes pip and venv. Done. If someone comes to us
with some horrible Anaconda setup wanting to know why their duct tape
doesn't work, can't we just tell them no?
When it comes to third-party packages, which I think we're
probably going to want in moderation, we would still need to discuss
supply chain safety. Python is not as mature here as, say, Go.What external dependencies are you imagining?
The OAuth pytest suite makes extensive use of
- psycopg, to easily drive libpq;
- construct, for on-the-wire packet representations and manipulation; and
- pyca/cryptography, for easy generation of certificates and manual
crypto testing.
I'd imagine each would need considerable discussion, if there is
interest in doing the same things that I do with them.
I think somewhere between 1 and 4 a *substantial* amount of work would be
required to provide a bunch of the infrastructure that Cluster.pm etc
provide. Otherwise we'll end up with a lot of copy pasted code between tests.
Possibly, yes. I think it depends on what you want to test first, and
there's a green-field aspect of hope/anxiety/ennui, too. Are you
trying to port the acceptance-test framework that we already have, or
are you trying to build a framework that can handle the things we
can't currently test? Will it be easier to refactor duplication into
shared fixtures when the language doesn't encourage an infinite number
of ways to do things? Or will we have to keep on top of it to avoid
pain?
--Jacob
On Mon, Jun 10, 2024 at 12:26 PM Alexander Korotkov
<aekorotkov@gmail.com> wrote:
Thank you for working on this.
Do you think you could re-use something from testgres[1] package?
Possibly? I think we're all coming at this with our own bags of tricks
and will need to carve off pieces to port, contribute, or reimplement.
Does testgres have something in particular you'd like to see the
Postgres tests support?
Thanks,
--Jacob
On Mon, Jun 10, 2024 at 06:49:11PM -0700, Andres Freund wrote:
On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote:
On 2024-06-10 Mo 16:04, Andres Freund wrote:
Just for context for the rest the email: I think we desperately need to move
off perl for tests. The infrastructure around our testing is basically
unmaintained and just about nobody that started doing dev stuff in the last 10
years learned perl.
As for what up and coming developers learn, they mostly don't learn C
either, and that's far more critical to what we do.C is a a lot more useful to to them than perl. And it's actually far more
widely known these days than perl.
If we're going to test in a non-Perl language, I'd pick C over Python. There
would be several other unlikely-community-choice languages I'd pick over
Python (C#, Java, C++). We'd need a library like today's Perl
PostgreSQL::Test to make C-language tests nice, but the same would apply to
any new language.
I also want the initial scope to be the new language coexisting with the
existing Perl tests. If a bulk translation ever happens, it should happen
long after the debut of the new framework. That said, I don't much trust a
human-written bulk language translation to go through without some tests
accidentally ceasing to test what they test in Perl today.
On 2024-06-11 Tu 19:48, Noah Misch wrote:
On Mon, Jun 10, 2024 at 06:49:11PM -0700, Andres Freund wrote:
On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote:
On 2024-06-10 Mo 16:04, Andres Freund wrote:
Just for context for the rest the email: I think we desperately need to move
off perl for tests. The infrastructure around our testing is basically
unmaintained and just about nobody that started doing dev stuff in the last 10
years learned perl.As for what up and coming developers learn, they mostly don't learn C
either, and that's far more critical to what we do.C is a a lot more useful to to them than perl. And it's actually far more
widely known these days than perl.If we're going to test in a non-Perl language, I'd pick C over Python. There
would be several other unlikely-community-choice languages I'd pick over
Python (C#, Java, C++). We'd need a library like today's Perl
PostgreSQL::Test to make C-language tests nice, but the same would apply to
any new language.
Indeed. We've invested quite a lot of effort on that infrastructure. I
guess people can learn from what we've done so a second language might
be easier to support.
(Java would be my pick from your unlikely set, but I can see the
attraction of Python.)
I also want the initial scope to be the new language coexisting with the
existing Perl tests. If a bulk translation ever happens, it should happen
long after the debut of the new framework. That said, I don't much trust a
human-written bulk language translation to go through without some tests
accidentally ceasing to test what they test in Perl today.
+1
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
On Wed, 12 Jun 2024 at 01:48, Noah Misch <noah@leadboat.com> wrote:
If we're going to test in a non-Perl language, I'd pick C over Python. There
would be several other unlikely-community-choice languages I'd pick over
Python (C#, Java, C++).
My main goals of this thread are:
1. Allowing people to quickly write tests
2. Have those tests do what the writer intended them to do
3. Have good error reporting by default
Those goals indeed don't necesitate python.
But I think those are really hard to achieve with any C based
framework, and probably with C++ too. Also manual memory management in
tests seems to add tons of complexity for basically no benefit.
I think C#, Java, Go, Rust, Kotlin, and Swift would be acceptable
choices for me (and possibly some more). They allow some type of
introspection, they have a garbage collector, and their general
tooling is quite good.
But I think a dynamically typed scripting language is much more
fitting for writing tests like this. I love static typing for
production code, but imho it really doesn't have much benefit for
tests.
As scripting languages go, the ones that are still fairly heavily in
use are Javascript, Python, Ruby, and PHP. I think all of those could
probably work, but my personal order of preference would be Python,
Ruby, Javascript, PHP.
Finally, I'm definitely biased towards using Python myself. But I
think there's good reasons for that:
1. In the data space, that Postgres in, Python is very heavily used for analysis
2. Everyone coming out of university these days knows it to some extent
3. Multiple people in the community have been writing Postgres related
tests in python and have enjoyed doing so (me[1]https://github.com/pgbouncer/pgbouncer/tree/master/test, Jacob[2]https://github.com/jchampio/pg-pytest-suite,
Alexander[3]https://github.com/postgrespro/testgres)
What language people want to write tests in is obviously very
subjective. And obviously we cannot allow every language for writing
tests. But I think if ~25% of the community prefers to write their
tests in Python. Then that should be enough of a reason to allow them
to do so.
TO CLARIFY: This thread is not a proposal to replace Perl with Python.
It's a proposal to allow people to also write tests in Python.
I also want the initial scope to be the new language coexisting with the
existing Perl tests. If a bulk translation ever happens, it should happen
long after the debut of the new framework. That said, I don't much trust a
human-written bulk language translation to go through without some tests
accidentally ceasing to test what they test in Perl today.
I definitely don't think we should rewrite all the tests that we have
in Perl today into some other language. But I do think that whatever
language we choose, that language should make it as least as easy to
write tests, as easy to read them and as easy to see that they are
testing the intended thing, as is currently the case for Perl.
Rewriting a few Perl tests into the new language, even if not merging
the rewrite, is a good way of validating that imho.
PS. For PgBouncer I actually hand-rewrote all the tests that we had in
bash (which is the worst testing language ever) in Python and doing so
actually found more bugs in PgBouncer code that our bash tests
wouldn't catch. So it's not necessarily the case that you lose
coverage by rewriting tests.
[1]: https://github.com/pgbouncer/pgbouncer/tree/master/test
[2]: https://github.com/jchampio/pg-pytest-suite
[3]: https://github.com/postgrespro/testgres
On Tue, Jun 11, 2024 at 5:31 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
On Mon, Jun 10, 2024 at 12:26 PM Alexander Korotkov
<aekorotkov@gmail.com> wrote:Thank you for working on this.
Do you think you could re-use something from testgres[1] package?Possibly? I think we're all coming at this with our own bags of tricks
and will need to carve off pieces to port, contribute, or reimplement.
Does testgres have something in particular you'd like to see the
Postgres tests support?
Generally, testgres was initially designed as Python analogue of what
we have in src/test/perl/PostgreSQL/Test. In particular its
testgres.PostgresNode is analogue of PostgreSQL::Test::Cluster. It
comes under PostgreSQL License. So, I wonder if we could revise it
and fetch most part of it into our source tree.
------
Regards,
Alexander Korotkov
Supabase
On Wed, Jun 12, 2024 at 2:48 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
On Tue, Jun 11, 2024 at 5:31 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:On Mon, Jun 10, 2024 at 12:26 PM Alexander Korotkov
<aekorotkov@gmail.com> wrote:Thank you for working on this.
Do you think you could re-use something from testgres[1] package?Possibly? I think we're all coming at this with our own bags of tricks
and will need to carve off pieces to port, contribute, or reimplement.
Does testgres have something in particular you'd like to see the
Postgres tests support?Generally, testgres was initially designed as Python analogue of what
we have in src/test/perl/PostgreSQL/Test. In particular its
testgres.PostgresNode is analogue of PostgreSQL::Test::Cluster. It
comes under PostgreSQL License. So, I wonder if we could revise it
and fetch most part of it into our source tree.
Plus testgres exists from 2016 and already have quite amount of use
cases. This is what I quickly found on github.
https://github.com/adjust/pg_querylog
https://github.com/postgrespro/pg_pathman
https://github.com/lanterndata/lantern
https://github.com/orioledb/orioledb
https://github.com/cbandy/pgtwixt
https://github.com/OpenNTI/nti.testing
https://github.com/postgrespro/pg_probackup
https://github.com/postgrespro/rum
------
Regards,
Alexander Korotkov
Supabase
On Wed, 12 Jun 2024 at 01:48, Noah Misch <noah@leadboat.com> wrote:
If we're going to test in a non-Perl language, I'd pick C over Python.
<snip>
We'd need a library like today's Perl
PostgreSQL::Test to make C-language tests nice, but the same would apply to
any new language.
P.P.S. We already write tests in C, we use it for testing libpq[1]https://github.com/postgres/postgres/blob/master/src/test/modules/libpq_pipeline/libpq_pipeline.c.
I'd personally definitely welcome a C library to make those tests
nicer to write, because I've written a fair bit of those in the past
and currently it's not very fun to do.
[1]: https://github.com/postgres/postgres/blob/master/src/test/modules/libpq_pipeline/libpq_pipeline.c
On Jun 12, 2024, at 6:40 AM, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
I think C#, Java, Go, Rust, Kotlin, and Swift would be acceptable
choices for me (and possibly some more). They allow some type of
introspection, they have a garbage collector, and their general
tooling is quite good.
Having used Python for 15+ years and then abandoned it for all projects I would
say the single most important points for a long term project like Postgres is,
not necessarily in order, package stability, package depth, semantic versioning,
available resources, and multiprocessor support.
The reason I abandoned Python was for the constant API breaks in packages. Yes,
python is a great language to teach in school for a one-time class project, but
that is not Postgres’s use-case. Remember that Fortran and Pascal were the
darlings for teaching in school prior to Python and no-one uses them any more.
Yes Python innovates fast and is fashionable. But again, not Postgres’s use-case.
I believe that anyone coming out of school these days would have a relatively
easy transition to any of Go, Rust, Kotlin, Swift, etc. In other words, any of
the modern languages. In addition, the language should scale well to
multiprocessors, because parallel testing is becoming more important every day.
If the Postgres project is going to pick a new language for testing, it should
pick a language for the next 50 years based on the projects needs.
Python is good for package depth and resource availability, but fails IMO in the
other categories. My experience with python where the program flow can change
because of non-visible characters is a terrible way to write robust long term
maintainable code. Because of this most of the modern languages are going to be
closer in style to Postgres’s C code base than Python.
Jelte Fennema-Nio:
As scripting languages go, the ones that are still fairly heavily in
use are Javascript, Python, Ruby, and PHP. I think all of those could
probably work, but my personal order of preference would be Python,
Ruby, Javascript, PHP.Finally, I'm definitely biased towards using Python myself. But I
think there's good reasons for that:
1. In the data space, that Postgres in, Python is very heavily used for analysis
2. Everyone coming out of university these days knows it to some extent
3. Multiple people in the community have been writing Postgres related
tests in python and have enjoyed doing so (me[1], Jacob[2],
Alexander[3])
PostgREST also uses pytest for integration tests - and that was a very
good decision compared to the bash based tests we had before.
One more argument for Python compared to the other mentioned scripting
languages: Python is already a development dependency via meson. None of
the other 3 are. In a future where meson will be the only build system,
we will have python "for free" already.
Best,
Wolfgang
Hi,
On 2024-06-11 08:04:57 -0400, Andrew Dunstan wrote:
Some time ago I did some work on wrapping libpq using the perl FFI module.
It worked pretty well, and would mean we could probably avoid many uses of
IPC::Run, and would probably be substantially more efficient (no fork
required). It wouldn't avoid all uses of IPC::Run, though.
FWIW, I'd *love* to see work on this continue. The reduction in test runtime
on windows is substantial and would shorten the hack->CI->fail->hack loop a
good bit shorter. And save money.
But my point was mainly that while a new framework might have value, I don't
think we need to run out and immediately rewrite several hundred TAP tests.
Oh, yea. That's not at all feasible to just do in one go.
Greetings,
Andres Freund
On Wed, 12 Jun 2024 at 15:49, FWS Neil <neil@fairwindsoft.com> wrote:
I believe that anyone coming out of school these days would have a relatively
easy transition to any of Go, Rust, Kotlin, Swift, etc. In other words, any of
the modern languages.
Agreed, which is why I said they'd be acceptable to me. But I think
one important advantage of Python is that it's clear that many people
in the community are willing to write tests in it. At PGConf.dev there
were a lot of people in the unconference session about this. Also many
people already wrote a Postgres testing framework for python, and are
using it (see list of projects that Alexander shared). I haven't seen
that level of willingness to write tests for any of those other
languages (yet).
In addition, the language should scale well to
multiprocessors, because parallel testing is becoming more important every day.
<snip>
Python is good for package depth and resource availability, but fails IMO in the
other categories.
You can easily pin packages or call a different function based on the
version of the package, so I'm not sure what the problem is with
package stability. Also chances are we'll pull in very little external
packages and rely mostly on the stdlib (which is quite stable).
Regarding parallelised running of tests, I agree that's very
important. And indeed normally parallelism in python can be a pain
(although async/await makes I/O parallelism a lot better at least).
But running pytest tests in parallel is extremely easy by using
pytest-xdist[1]https://pypi.org/project/pytest-xdist/, so I don't really think there's an issue for this
specific Python usecase.
My experience with python where the program flow can change
because of non-visible characters is a terrible way to write robust long term
maintainable code. Because of this most of the modern languages are going to be
closer in style to Postgres’s C code base than Python.
I'm assuming this is about spaces vs curly braces for blocks? Now that
we have auto formatters for every modern programming language I indeed
prefer curly braces myself too. But honestly that's pretty much a tabs
vs spaces discussion.
[1]: https://pypi.org/project/pytest-xdist/
Show quoted text
On Wed, 12 Jun 2024 at 15:49, FWS Neil <neil@fairwindsoft.com> wrote:
On Jun 12, 2024, at 6:40 AM, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
I think C#, Java, Go, Rust, Kotlin, and Swift would be acceptable
choices for me (and possibly some more). They allow some type of
introspection, they have a garbage collector, and their general
tooling is quite good.Having used Python for 15+ years and then abandoned it for all projects I would
say the single most important points for a long term project like Postgres is,
not necessarily in order, package stability, package depth, semantic versioning,
available resources, and multiprocessor support.The reason I abandoned Python was for the constant API breaks in packages. Yes,
python is a great language to teach in school for a one-time class project, but
that is not Postgres’s use-case. Remember that Fortran and Pascal were the
darlings for teaching in school prior to Python and no-one uses them any more.Yes Python innovates fast and is fashionable. But again, not Postgres’s use-case.
I believe that anyone coming out of school these days would have a relatively
easy transition to any of Go, Rust, Kotlin, Swift, etc. In other words, any of
the modern languages. In addition, the language should scale well to
multiprocessors, because parallel testing is becoming more important every day.If the Postgres project is going to pick a new language for testing, it should
pick a language for the next 50 years based on the projects needs.Python is good for package depth and resource availability, but fails IMO in the
other categories. My experience with python where the program flow can change
because of non-visible characters is a terrible way to write robust long term
maintainable code. Because of this most of the modern languages are going to be
closer in style to Postgres’s C code base than Python.
