pg_regress: schedule multi-line test groups
Hi Hackers,
This was motivated by two rebases on the SLOPE patch [1] this week due to
changes in parallel_schedule. Also I find much easier to parse the list
using
this syntax, if we stick with one test per line we get the benefit of
counting
tests by looking at the line numbers.
I used a `goto` to keep the number of touched lines small, and facilitate
your review.
Attachments:
v1-0001-pg_regress-multi-line-test-groups.patchapplication/octet-stream; name=v1-0001-pg_regress-multi-line-test-groups.patchDownload+278-26
On Thu, Sep 17, 2026 at 08:57:51AM +0100, Alexandre Felipe wrote:
This was motivated by two rebases on the SLOPE patch [1] this week due to
changes in parallel_schedule. Also I find much easier to parse the list
using this syntax, if we stick with one test per line we get the benefit of
counting tests by looking at the line numbers.I used a `goto` to keep the number of touched lines small, and facilitate
your review.
I'd +1 something like that in the schedule files, even if we only
limit the number of tests per line to be up to 20.
+test:
+ boolean
+ char
+ name
At least, let's make the whitespace vs tab policy consistent. ;)
--
Michael
On Thu, Sep 17, 2026 at 9:03 AM Michael Paquier <michael@paquier.xyz> wrote:
On Thu, Sep 17, 2026 at 08:57:51AM +0100, Alexandre Felipe wrote:
This was motivated by two rebases on the SLOPE patch [1] this week due to
changes in parallel_schedule. Also I find much easier to parse the list
using this syntax, if we stick with one test per line we get the benefitof
counting tests by looking at the line numbers.
I used a `goto` to keep the number of touched lines small, and facilitate
your review.
I'd +1 something like that in the schedule files, even if we only
limit the number of tests per line to be up to 20.
Actually, I think we should support arbitrary size test groups and run them
on a pool
from a queue that would possibly make the regression faster as today we
always
wait for the slowest test in each group.
+test: + boolean + char + nameAt least, let's make the whitespace vs tab policy consistent. ;)
I will blame the editor, I hit tab and it adds spaces :) I started manually
but then I saw that
what I was doing was dumb and error prone, and did the rest with find and
replace (with \t)
now all using spaces.
Regards,
Alexandre
On 17 Sep 2026, at 10:43, Alexandre Felipe <o.alexandre.felipe@gmail.com> wrote:
Actually, I think we should support arbitrary size test groups and run them on a pool
from a queue that would possibly make the regression faster as today we always
wait for the slowest test in each group.
I know of ongoing, as of yet unpublished, work in this area to improve this
scheduling which will also change the syntax. There is a bit complexity to it
than just picking from a pool since there are inter-suite dependencies.
+ /*
+ * Found `test: # no tests` treat it as a multiline test group
+ */
if (num_tests == 0)
{
- bail("syntax error in schedule file \"%s\" line %d: %s",
- schedule, line_num, scbuf);
+ multiline_test = true;
+ }
This will allow empty test groups which we currently treat as an error. This
schedule:
test: test_setup\n
test:\n
\n
test:\n
Parses and runs with all tests succeeded:
# initializing database system by copying initdb template
# using temp instance on port 40058 with PID 71845
ok 1 - test_setup 448 ms
# parallel group (0 tests):
# parallel group (0 tests):
1..1
# All 1 tests passed.
# test succeeded
Without the patch an empty group will throw an error, and I think it makes
sense to preserve that behaviour.
Also, if we are to change the accepted syntax, why limit to single line or
multiline, why not accept any whitespace separated name between ^test: and
(^test|EOF)? Something like the below:
test: boolean
char
name
--
Daniel Gustafsson
Thank you for your looking into it Daniel
On Fri, Sep 18, 2026 at 10:31 AM Daniel Gustafsson <daniel@yesql.se> wrote:
On 17 Sep 2026, at 10:43, Alexandre Felipe <o.alexandre.felipe@gmail.com>
wrote:
Actually, I think we should support arbitrary size test groups and run
them on a pool
from a queue that would possibly make the regression faster as today we
always
wait for the slowest test in each group.
I know of ongoing, as of yet unpublished, work in this area to improve this
scheduling which will also change the syntax. There is a bit complexity
to it
than just picking from a pool since there are inter-suite dependencies.
An easy gain would be to somehow transpose execution, currently we put
together
things to indicate that they are independent, it would be more readable if
we used
the lines to indicate dependency.
The right way to handle dependency is by writing a dependency tree and doing
a topological sort, like makefiles, but to keep the syntax backward
compatible
we could do something like this
# tests that have to run in isolation
test: setup_test
test: sanity_check
# tests that have to respect certain order
test sequentially: create_am psql
# where a test should wait for multiple tests that can run in parallel
test sequentially: (geometry create_index_spgist hash_index brin) amutils
# tests that can mix with other tests
test concurrently: compression compression_lz4 compression_pglz cluster
But please, let's first get the multiline syntax over the fence.
v2 is logging the effective concurrency of the groups during the execution
(we can't optimise what we don't measure).
e.g:
# effective concurrency 5.80 / 18
# effective concurrency 3.83 / 15
+ /*
+ * Found `test: # no tests` treat it as a multiline test group + */ if (num_tests == 0) { - bail("syntax error in schedule file \"%s\" line %d: %s", - schedule, line_num, scbuf); + multiline_test = true; + }This will allow empty test groups which we currently treat as an error.
This
schedule:
Fixed, also improved the error messages
Also, if we are to change the accepted syntax, why limit to single line or
multiline, why not accept any whitespace separated name between ^test: and
(^test|EOF)? Something like the below:test: boolean
char
name
Minimalism. in v2 I am trying to do something more like what you described
The schedule syntax in v2 should be
* Schedule = (Blank | Comment | Group)*
* Group = "test:" (Line | Comment) (Indent Line)*
* Line = token (Space+ token)* Comment?
* Blank = '\n'
* Comment = '#' [^\n]* \n
* Indent = [\t ]+
But I won't be surprised if you find inconsistencies between that and what
was actually implemented at this stage.
Regards,
v2 failed on CFBot with error: 'INFINITY' undeclared.
v2.1 does the same as v2 but without using INFINITY.
On Sun, Sep 20, 2026 at 8:50 AM Alexandre Felipe
<o.alexandre.felipe@gmail.com> wrote:
Show quoted text
Thank you for your looking into it Daniel
On Fri, Sep 18, 2026 at 10:31 AM Daniel Gustafsson <daniel@yesql.se> wrote:
On 17 Sep 2026, at 10:43, Alexandre Felipe <o.alexandre.felipe@gmail.com> wrote:
Actually, I think we should support arbitrary size test groups and run them on a pool
from a queue that would possibly make the regression faster as today we always
wait for the slowest test in each group.I know of ongoing, as of yet unpublished, work in this area to improve this
scheduling which will also change the syntax. There is a bit complexity to it
than just picking from a pool since there are inter-suite dependencies.An easy gain would be to somehow transpose execution, currently we put together
things to indicate that they are independent, it would be more readable if we used
the lines to indicate dependency.The right way to handle dependency is by writing a dependency tree and doing
a topological sort, like makefiles, but to keep the syntax backward compatible
we could do something like this# tests that have to run in isolation
test: setup_test
test: sanity_check
# tests that have to respect certain order
test sequentially: create_am psql
# where a test should wait for multiple tests that can run in parallel
test sequentially: (geometry create_index_spgist hash_index brin) amutils
# tests that can mix with other tests
test concurrently: compression compression_lz4 compression_pglz clusterBut please, let's first get the multiline syntax over the fence.
v2 is logging the effective concurrency of the groups during the execution
(we can't optimise what we don't measure).
e.g:
# effective concurrency 5.80 / 18
# effective concurrency 3.83 / 15+ /*
+ * Found `test: # no tests` treat it as a multiline test group + */ if (num_tests == 0) { - bail("syntax error in schedule file \"%s\" line %d: %s", - schedule, line_num, scbuf); + multiline_test = true; + }This will allow empty test groups which we currently treat as an error. This
schedule:Fixed, also improved the error messages
Also, if we are to change the accepted syntax, why limit to single line or
multiline, why not accept any whitespace separated name between ^test: and
(^test|EOF)? Something like the below:test: boolean
char
nameMinimalism. in v2 I am trying to do something more like what you described
The schedule syntax in v2 should be
* Schedule = (Blank | Comment | Group)*
* Group = "test:" (Line | Comment) (Indent Line)*
* Line = token (Space+ token)* Comment?
* Blank = '\n'
* Comment = '#' [^\n]* \n
* Indent = [\t ]+But I won't be surprised if you find inconsistencies between that and what
was actually implemented at this stage.Regards,
Hi All,
I just received an email about this requiring a rebase.
Is there any chance this will be committed, or should I abandon it?
Regards,
Alexandre
On Sun, Sep 20, 2026 at 6:56 PM Alexandre Felipe <
o.alexandre.felipe@gmail.com> wrote:
Show quoted text
v2 failed on CFBot with error: 'INFINITY' undeclared.
v2.1 does the same as v2 but without using INFINITY.On Sun, Sep 20, 2026 at 8:50 AM Alexandre Felipe
<o.alexandre.felipe@gmail.com> wrote:Thank you for your looking into it Daniel
On Fri, Sep 18, 2026 at 10:31 AM Daniel Gustafsson <daniel@yesql.se>
wrote:
On 17 Sep 2026, at 10:43, Alexandre Felipe <
o.alexandre.felipe@gmail.com> wrote:
Actually, I think we should support arbitrary size test groups and
run them on a pool
from a queue that would possibly make the regression faster as today
we always
wait for the slowest test in each group.
I know of ongoing, as of yet unpublished, work in this area to improve
this
scheduling which will also change the syntax. There is a bit
complexity to it
than just picking from a pool since there are inter-suite dependencies.
An easy gain would be to somehow transpose execution, currently we put
together
things to indicate that they are independent, it would be more readable
if we used
the lines to indicate dependency.
The right way to handle dependency is by writing a dependency tree and
doing
a topological sort, like makefiles, but to keep the syntax backward
compatible
we could do something like this
# tests that have to run in isolation
test: setup_test
test: sanity_check
# tests that have to respect certain order
test sequentially: create_am psql
# where a test should wait for multiple tests that can run in parallel
test sequentially: (geometry create_index_spgist hash_index brin) amutils
# tests that can mix with other tests
test concurrently: compression compression_lz4 compression_pglz clusterBut please, let's first get the multiline syntax over the fence.
v2 is logging the effective concurrency of the groups during the
execution
(we can't optimise what we don't measure).
e.g:
# effective concurrency 5.80 / 18
# effective concurrency 3.83 / 15+ /*
+ * Found `test: # no tests` treat it as a
multiline test group
+ */ if (num_tests == 0) { - bail("syntax error in schedule file\"%s\" line %d: %s",
- schedule, line_num, scbuf); + multiline_test = true; + }This will allow empty test groups which we currently treat as an
error. This
schedule:
Fixed, also improved the error messages
Also, if we are to change the accepted syntax, why limit to single line
or
multiline, why not accept any whitespace separated name between ^test:
and
(^test|EOF)? Something like the below:
test: boolean
char
nameMinimalism. in v2 I am trying to do something more like what you
described
The schedule syntax in v2 should be
* Schedule = (Blank | Comment | Group)*
* Group = "test:" (Line | Comment) (Indent Line)*
* Line = token (Space+ token)* Comment?
* Blank = '\n'
* Comment = '#' [^\n]* \n
* Indent = [\t ]+But I won't be surprised if you find inconsistencies between that and
what
was actually implemented at this stage.
Regards,
On 13 Nov 2026, at 10:13, Alexandre Felipe <o.alexandre.felipe@gmail.com> wrote:
Hi All,
I just received an email about this requiring a rebase.
Is there any chance this will be committed, or should I abandon it?
There is definitely a chance, I have it on my radar for when the 19 release is
out, so it would be great if it was rebased.
--
Daniel Gustafsson
Thank you Daniel,
v2.2 - rebased: Conflicts caused by tests added to parallel_scheule.
Regards,
Alexandre
Hello
This looks like a good feature, but I think the parser needs some improvements:
1. A test name directly followed by # is silently dropped
test: a b#note
or
test:
a
b# depends on a
Will only run 'a'
2. An empty or comment-only schedule file now fails
3. The --max-concurrent-tests error reports the wrong line
test:
a
b
c
d
test: a
With --max-concurrent-tests=3 this reports "line 6: test: a". With a trailing comment after the group it reports the comment line instead. Maybe it should point to the start of the test?
4. (2) and (3) both come from the goto structure. Wouldn't a run_test_group() helper be simpler?
5.
EFFECTIVE CONCURRENCY
=====================
This also add for each group a note about the average execution
concurrency achieved during the test.
# parallel group (2 tests): brin_bloom brin_multi
# effective concurrency 1.46 / 2
Wouldn't it be better to separate this part into a separate patch file?
On Fri, Nov 13, 2026 at 4:10 PM Zsolt Parragi <zsolt.parragi@percona.com>
wrote:
Hello
Hello
Thank you for joining.
This looks like a good feature, but I think the parser needs some
improvements:1. A test name directly followed by # is silently dropped
test: a b#note
or
test:
a
b# depends on aWill only run 'a'
Fixed, updated syntax at the function doc comment.
2. An empty or comment-only schedule file now fails
Yes, intentionally, the patch preamble mentions that.
3. The --max-concurrent-tests error reports the wrong line
test:
a
b
c
d
test: aWith --max-concurrent-tests=3 this reports "line 6: test: a". With a
trailing comment after the group it reports the comment line instead.
Maybe it should point to the start of the test?
upstream has two conditions with the same error message, and I failed
to distinguish them.
Now there is only one message, will show
in schedule file "..." line 5, test d: max concurrency (3) exceeded
I think this is better than the pointing to the start of the test, but
if that is still preferred it can easily be changed.
4. (2) and (3) both come from the goto structure. Wouldn't a
run_test_group() helper be simpler?
It would, I was trying to keep the *patch* as small as possible
I suggested some functional changes in this thread on 2026-09-20 [1]/messages/by-id/CAE8JnxPycpY5mk7C-rNa4jeMQQXG1H5mobB10Oa_CaQs1nBvUg@mail.gmail.com
I recorded a video presenting a mockup of of that in [2]https://youtu.be/FHjQEuDoxFw
5.
EFFECTIVE CONCURRENCY
=====================
This also add for each group a note about the average execution
concurrency achieved during the test.# parallel group (2 tests): brin_bloom brin_multi
# effective concurrency 1.46 / 2Wouldn't it be better to separate this part into a separate patch file?
v2.3 puts effective concurrency logging in a separate patch.
[1]: /messages/by-id/CAE8JnxPycpY5mk7C-rNa4jeMQQXG1H5mobB10Oa_CaQs1nBvUg@mail.gmail.com
/messages/by-id/CAE8JnxPycpY5mk7C-rNa4jeMQQXG1H5mobB10Oa_CaQs1nBvUg@mail.gmail.com
[2]: https://youtu.be/FHjQEuDoxFw
Attachments:
t253819_11index.htmltext/html; charset=US-ASCII; name=index.htmlDownload
v2.3-0001-pg_regress-multi-line-test-groups.patchapplication/octet-stream; name=v2.3-0001-pg_regress-multi-line-test-groups.patchDownload+339-50
v2.3-0002-pg_regress-report-group-effective-concurrency.patchapplication/octet-stream; name=v2.3-0002-pg_regress-report-group-effective-concurrency.patchDownload+27-1
On Sun, 15 Nov 2026, Alexandre Felipe <o.alexandre.felipe@gmail.com> wrote:
Yes, intentionally, the patch preamble mentions that.
ops, I missed that, sorry.
1. Now with 2.3:
$ cat s
test: a
test: b
$ pg_regress ... --schedule=s --max-concurrent-tests=0
# in schedule file "s" line 1, test a: max concurrency (0) exceeded
Bail out!
While --help still says that 0 means unlimited. (and --max-concurrent-tests=abc is also 0, but that's a different issue)
2. long commented out lines can run tests
test:
a
# xxxx...(1019 x)...xxxx b
"All 2 tests passed"
+/*
+ * Maximum supported number of concurrent tests
+ */
+#define MAX_PARALLEL_TESTS 100
This is now defined twice.
On Mon, Nov 16, 2026 at 7:46 PM Zsolt Parragi <zsolt.parragi@percona.com>
wrote:
1. Now with 2.3:
$ pg_regress ... --schedule=s --max-concurrent-tests=0
# in schedule file "s" line 1, test a: max concurrency (0) exceeded
Bail out!While --help still says that 0 means unlimited. (and
--max-concurrent-tests=abc is also 0, but that's a different issue)
Now,
regress$ ./pg_regress --schedule=parallel_schedule --max-concurrent-tests=0
# --max-concurrent-tests=0 expected to be integer in range 1-100
Bail out!%
regress$ ./pg_regress --schedule=parallel_schedule --max-concurrent-tests=x
# --max-concurrent-tests=x expected to be integer in range 1-100
Bail out!%
regress$ ./pg_regress --schedule=parallel_schedule
--max-concurrent-tests=101
# --max-concurrent-tests=101 expected to be integer in range 1-100
Bail out!%
regress$ ./pg_regress --schedule=parallel_schedule
--max-concurrent-tests=100
# using postmaster on Unix socket, default port
regress$ ./pg_regress --help
...
--max-concurrent-tests=N maximum number of concurrent tests in
schedule
default: 100
2. long commented out lines can run tests
test:
a
# xxxx...(1019 x)...xxxx b"All 2 tests passed"
Fixing (this was an old bug).
sizeof(scbuf) = 1024, so maximum line length is 1022.
this has to handle two cases: (i) in the middle of the file
the string can be 1023 long including a final line separator
(ii) the last line doesn't have a \n separator, and can't be
1023 long. Since fgets stops when the buffer it will read at
most 1023 bytes, 1 byte is reserved for the null terminator
+/*
+ * Maximum supported number of concurrent tests + */ +#define MAX_PARALLEL_TESTS 100This is now defined twice.
Ups, I think I removed now...
---
Alexandre Felipe
Attachments:
t253819_13v2.4-0001-pg_regress-multi-line-test-groups.patchapplication/octet-stream; name=v2.4-0001-pg_regress-multi-line-test-groups.patchDownload+351-54
v2.4-0002-pg_regress-report-group-effective-concurrency.patchapplication/octet-stream; name=v2.4-0002-pg_regress-report-group-effective-concurrency.patchDownload+27-1
On Wed, Nov 18, 2026 at 08:25:32AM +0000, Alexandre Felipe wrote:
+ * Maximum supported number of concurrent tests + */ +#define MAX_PARALLEL_TESTS 100This is now defined twice.
Ups, I think I removed now...
+test:
+ boolean
+ char
+ name
While reading through this patch, I am wondering if it would not be
better to support backslashes instead of an arbitrary amount of spaces
a-la-yaml for each group, like:
+test: \
+ boolean \
+ char \
+ name \
And that may actually make the patch simpler overall with less
hardcoded rules arount "test:". I can see that 0001 adds two more of
them, reducing the code readability. That would also work as a
backward-compatible grammar. Perhaps I'm too pedantic, so feel free
to overwrite me, of course.
0002 provides a cute metric, but I doubt that it is strictly
necessary. Performance matters for grouping, but that's not the only
reason why we have groups: we want some organization as well.
--
Michael