List TAP test files in makefiles

I agree that this difference between meson and autotools builds is kind of problematic.

On 23 Aug 2025, at 11:09, Peter Eisentraut <peter@eisentraut.org> wrote:

The obvious solution is to also require the makefiles to list TAP files explicitly, which is what I'm proposing here.

What is the downside of the approach where meson uses t/*.pl wildcard?

Best regards, Andrey Borodin.

Hi,

On 2025-08-23 11:57:37 +0500, Andrey Borodin wrote:

I agree that this difference between meson and autotools builds is kind of problematic.

On 23 Aug 2025, at 11:09, Peter Eisentraut <peter@eisentraut.org> wrote:

The obvious solution is to also require the makefiles to list TAP files explicitly, which is what I'm proposing here.

What is the downside of the approach where meson uses t/*.pl wildcard?

In meson you can't do wildcards at "configure" time, since wildcards do not
allow reliable detection of when re-configure is needed. You could do it by
running prove to run the tap tests, like make does, but that would
considerably slow down the tests, as prove has either no parallelism or
independent parallelism from the make's (or ninjas).

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:

On 2025-08-23 11:57:37 +0500, Andrey Borodin wrote:

What is the downside of the approach where meson uses t/*.pl wildcard?

In meson you can't do wildcards at "configure" time, since wildcards do not
allow reliable detection of when re-configure is needed.

But ... what "re-configuration" is needed when adding a new test file?

This proposed fix adds more manual maintenance effort, and I really
doubt that it will do much to solve the ostensible problem that
people forget to update the meson.build files. We should be striving
to make the meson system as easy to use as autoconf/make, not trying
to attain parity by making the latter experience worse.

regards, tom lane

Hi,

On 2025-08-23 10:17:59 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

On 2025-08-23 11:57:37 +0500, Andrey Borodin wrote:

What is the downside of the approach where meson uses t/*.pl wildcard?

In meson you can't do wildcards at "configure" time, since wildcards do not
allow reliable detection of when re-configure is needed.

But ... what "re-configuration" is needed when adding a new test file?

In meson, not autoconf/make. The test runner doesn't parse the meson.build
files, that's done once when running configure. After that there's a table of
tests that need to be run that gets used when you type ninja/meson test.

We should be striving to make the meson system as easy to use as
autoconf/make, not trying to attain parity by making the latter experience
worse.

FWIW, I find the autoconf/make test run experience completely unusable. It
literally is made me embark on getting away from it. I don't understand how
people stand it.

Greetings,

Andres Freund

On 2025-08-23 Sa 10:38 AM, Andres Freund wrote:

Hi,

On 2025-08-23 10:17:59 -0400, Tom Lane wrote:

We should be striving to make the meson system as easy to use as
autoconf/make, not trying to attain parity by making the latter experience
worse.

FWIW, I find the autoconf/make test run experience completely unusable. It
literally is made me embark on getting away from it. I don't understand how
people stand it.

Honestly, they both have annoyances and points where it's easy to get
into the weeds. There are definitely cases where meson has left me
scratching my head.

But I think this is really analogous to adding a new object file. That
needs to be added to the Makefile as well as the meson.build file, and I
don't see why tests shouldn't work the same. That might be slightly
annoying, but it would be way down on my list.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Andres Freund <andres@anarazel.de> writes:

FWIW, I find the autoconf/make test run experience completely unusable. It
literally is made me embark on getting away from it. I don't understand how
people stand it.

Interesting perspective, because from where I sit the testing aspect
is the weakest part of our meson setup: it seems to be very much
all-or-nothing, whereas the makefiles allow you to run any particular
test suite you want. Maybe this is just a lack of documentation?
There is still not a word in

https://www.postgresql.org/docs/devel/regress-run.html

about how to accomplish any of those effects under meson.

I find the lack of any equivalent to installcheck to be
particularly distressing, since it means you can't repeat
a test without paying for "make temp-install" every time.
Admittedly that doesn't work for a test that needs particular
non-default installation settings, but that's a minority
of our tests.

regards, tom lane

Hi,

On 2025-08-23 12:11:51 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

FWIW, I find the autoconf/make test run experience completely unusable. It
literally is made me embark on getting away from it. I don't understand how
people stand it.

Interesting perspective, because from where I sit the testing aspect
is the weakest part of our meson setup: it seems to be very much
all-or-nothing, whereas the makefiles allow you to run any particular
test suite you want.

Meson definitely allows to run just some of the test. You can list all tests
with
meson test --list

And you can run testsuites (like all the recovery tests) with
meson test --suite recovery
or individual tests with
meson test test_misc/003_check_guc

Maybe this is just a lack of documentation?

Yea. Partially because that's generic information about meson (i.e. not
postgres specific stuff) and partially because there were some doc patches
that we ended up not finding agreement upon approach-wise.

I find the lack of any equivalent to installcheck to be
particularly distressing, since it means you can't repeat
a test without paying for "make temp-install" every time.

If you just want to skip the install (reusing the prior install), you can do
that with
meson test --no-suite setup

That's basically the equivalent of NO_TEMP_INSTALL (which isn't documented
either :( )

Admittedly that doesn't work for a test that needs particular
non-default installation settings, but that's a minority
of our tests.

You can run all tests against a running postgres by doing
meson test --setup running

Unless somebody else volunteers (please!), I guess I should write up a
patch...

Greetings,

Andres Freund

On 2025-Aug-25, Andres Freund wrote:

Unless somebody else volunteers (please!), I guess I should write up a
patch...

Maybe it doesn't have to be a patch -- we have some info on command
lines to use for testing at
https://wiki.postgresql.org/wiki/Meson#Test_related_commands
which can perhaps be updated and filled in with additional content.

There may be other outdated things in that page, which perhaps should be
updated. For instance, I would rip out the entire "Why and What"
section, which feels no longer relevant.

I'm a bit short of time at present, but in a few weeks I can give this a
look. I'm happy if somebody else does it though.

--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
Si no sabes adonde vas, es muy probable que acabes en otra parte.

On 23.08.25 08:09, Peter Eisentraut wrote:

meson.build files have to list TAP test files explicitly.  Makefiles
have been relying on a t/*.pl wildcard.  As a consequence, it is
traditional now that many developers who are primarily using make forget
to add any new TAP test files to the respective meson.build file.

The obvious solution is to also require the makefiles to list TAP files
explicitly, which is what I'm proposing here.

To list the test files, I'm re-using the make variable TAP_TESTS that is
already in use by extensions.  Extensions up to now would just write
something like

    TAP_TESTS = 1

Starting with this patch, they would have to write

    TAP_TESTS = t/001_foo.pl t/002_bar.pl

This makes it somewhat backward compatible.  (For example, with the
latter formulation, PG<=18 would still read it as enabling TAP tests,
but it would ignore the actual file list.)

(I have considered other variants: We already have the make variable
PROVE_TESTS that you can use to override the list of tests from the make
command line.  But if you use that one to list the tests in an extension
makefile, then that extension would have to use both PROVE_TESTS and
TAP_TESTS (to enable TAP tests overall), which seems pretty confusing. I
think the solution I ended up with is compact and intuitive.)

In the attached patch, I have arranged it so that the interesting
changes are in the first three files, the rest is just mechanical.

In the meantime, here is a new version to adapt to the addition of TAP
tests for libpq-oauth.

On Sat, Aug 23, 2025 at 10:17:59AM -0400, Tom Lane wrote:

This proposed fix adds more manual maintenance effort, and I really
doubt that it will do much to solve the ostensible problem that
people forget to update the meson.build files.

This was my reaction, too. I suspect folks will update either Makefile or
meson.build based on what they use and then still forget to update the
other one.

--
nathan

On Fri, Oct 24, 2025 at 10:49:53AM -0500, Nathan Bossart wrote:

On Sat, Aug 23, 2025 at 10:17:59AM -0400, Tom Lane wrote:

This proposed fix adds more manual maintenance effort, and I really
doubt that it will do much to solve the ostensible problem that
people forget to update the meson.build files.

This was my reaction, too. I suspect folks will update either Makefile or
meson.build based on what they use and then still forget to update the
other one.

This is not a problem limited to committers: it is a worse an issue
for patch authors and reviewers. I've seen it again literally three
days ago, where a test was missing from meson, and the CI reported
green without testing the patch in most environments where we use only
meson, aka Windows.

IMO, we should tackle this problem the other way around: what would it
take to have meson build a list of the TAP files in t/ based on an
automated rule when tests.tap is defined?

Another less invasive solution for the code tree would be to have
meson cross-check that the list of files it defines matches with
what's on disk. With the drop of Makefiles and ./configure at some
point in mind, I am not convinced that the t/*.pl rule needs to
change, but meson could be made more reliable by making sure that
what's listed matches with what we have in the source tree. That
would prevent a lot of mistakes.
--
Michael

On 2025-Oct-28, Michael Paquier wrote:

IMO, we should tackle this problem the other way around: what would it
take to have meson build a list of the TAP files in t/ based on an
automated rule when tests.tap is defined?

Maybe what we need is a single source of truth that is used both by
Makefiles and meson: say, a file named "testlist" which contains lines
like

regress = foo bar baz
tap = t/001_foo.pl t/002_bar.pl

(possibly with backslash-escaped newlines) and so on, which is processed
by each build system in the way that it needs. This way, whichever
system is used by a developer, either the test will run because they
changed this one file, or it won't.

I don't know how much scaffolding is needed to make this work, but I
don't think it would be too bad.

IIRC Andres has complained about the Makefiles using t/*.pl to determine
the tests to run. This would also solve that problem.

--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"I suspect most samba developers are already technically insane...
Of course, since many of them are Australians, you can't tell." (L. Torvalds)

Hi,

On Tue, 28 Oct 2025 at 07:19, Michael Paquier <michael@paquier.xyz> wrote:

Another less invasive solution for the code tree would be to have
meson cross-check that the list of files it defines matches with
what's on disk. With the drop of Makefiles and ./configure at some
point in mind, I am not convinced that the t/*.pl rule needs to
change, but meson could be made more reliable by making sure that
what's listed matches with what we have in the source tree. That
would prevent a lot of mistakes.

I tried to implement this but it didn't quite work, please see details below.

I introduced a new test, meson_tap_test_check, in the meson build
system. This test runs first in the test suite and compares two file
sets:

1- Files on disk matching the t/*.pl wildcard.
2- The output of "meson introspect ${pg_root_dir} --tests" filtered
with the same t/*.pl wildcard.

The test subtracts the second list from the first and fails if there
is any difference.

However, this approach didn’t work because some tests are not
registered if their dependencies (or such) aren’t found. For example,
'001_sepgsql.pl' test isn't registered if the 'selinux' dependency is
missing and that causes a difference between #1 and #2. Given that, I
think it would be better if we register all tests and skip those whose
dependencies aren’t found, rather than not registering them when the
dependency isn’t found.

The attached patch shows what I implemented.

--
Regards,
Nazir Bilal Yavuz
Microsoft

On 01.09.25 07:09, Peter Eisentraut wrote:

On 23.08.25 08:09, Peter Eisentraut wrote:

meson.build files have to list TAP test files explicitly.  Makefiles
have been relying on a t/*.pl wildcard.  As a consequence, it is
traditional now that many developers who are primarily using make
forget to add any new TAP test files to the respective meson.build file.

The obvious solution is to also require the makefiles to list TAP
files explicitly, which is what I'm proposing here.

To list the test files, I'm re-using the make variable TAP_TESTS that
is already in use by extensions.  Extensions up to now would just
write something like

     TAP_TESTS = 1

Starting with this patch, they would have to write

     TAP_TESTS = t/001_foo.pl t/002_bar.pl

This makes it somewhat backward compatible.  (For example, with the
latter formulation, PG<=18 would still read it as enabling TAP tests,
but it would ignore the actual file list.)

(I have considered other variants: We already have the make variable
PROVE_TESTS that you can use to override the list of tests from the
make command line.  But if you use that one to list the tests in an
extension makefile, then that extension would have to use both
PROVE_TESTS and TAP_TESTS (to enable TAP tests overall), which seems
pretty confusing. I think the solution I ended up with is compact and
intuitive.)

In the attached patch, I have arranged it so that the interesting
changes are in the first three files, the rest is just mechanical.

In the meantime, here is a new version to adapt to the addition of TAP
tests for libpq-oauth.

It looks like there is no consensus to go into this direction, and none
of the discussed alternatives are taking on a concrete shape, so I'm
withdrawing this patch.