pg_regress: schedule multi-line test groups

Started by Alexandre Felipe2 months ago14 messageshackers
Jump to latest
#1Alexandre Felipe
o.alexandre.felipe@gmail.com

Hi Hackers,

This was motivated by two rebases on the SLOPE patch [1] this week due to
changes in parallel_schedule. Also I find much easier to parse the list
using
this syntax, if we stick with one test per line we get the benefit of
counting
tests by looking at the line numbers.

I used a `goto` to keep the number of touched lines small, and facilitate
your review.

1. https://commitfest.postgresql.org/patch/6479/

Attachments:

v1-0001-pg_regress-multi-line-test-groups.patchapplication/octet-stream; name=v1-0001-pg_regress-multi-line-test-groups.patchDownload+278-26
#2Michael Paquier
michael@paquier.xyz
In reply to: Alexandre Felipe (#1)
Re: pg_regress: schedule multi-line test groups

On Thu, Sep 17, 2026 at 08:57:51AM +0100, Alexandre Felipe wrote:

This was motivated by two rebases on the SLOPE patch [1] this week due to
changes in parallel_schedule. Also I find much easier to parse the list
using this syntax, if we stick with one test per line we get the benefit of
counting tests by looking at the line numbers.

I used a `goto` to keep the number of touched lines small, and facilitate
your review.

I'd +1 something like that in the schedule files, even if we only
limit the number of tests per line to be up to 20.

+test:
+    boolean
+	char
+	name

At least, let's make the whitespace vs tab policy consistent. ;)
--
Michael

#3Alexandre Felipe
o.alexandre.felipe@gmail.com
In reply to: Michael Paquier (#2)
Re: pg_regress: schedule multi-line test groups

On Thu, Sep 17, 2026 at 9:03 AM Michael Paquier <michael@paquier.xyz> wrote:

On Thu, Sep 17, 2026 at 08:57:51AM +0100, Alexandre Felipe wrote:

This was motivated by two rebases on the SLOPE patch [1] this week due to
changes in parallel_schedule. Also I find much easier to parse the list
using this syntax, if we stick with one test per line we get the benefit

of

counting tests by looking at the line numbers.

I used a `goto` to keep the number of touched lines small, and facilitate
your review.

I'd +1 something like that in the schedule files, even if we only

limit the number of tests per line to be up to 20.

Actually, I think we should support arbitrary size test groups and run them
on a pool
from a queue that would possibly make the regression faster as today we
always
wait for the slowest test in each group.

+test:
+    boolean
+       char
+       name

At least, let's make the whitespace vs tab policy consistent. ;)

I will blame the editor, I hit tab and it adds spaces :) I started manually
but then I saw that
what I was doing was dumb and error prone, and did the rest with find and
replace (with \t)
now all using spaces.

Regards,
Alexandre

Attachments:

t253819_3
v1.1-0001-pg_regress-multi-line-test-groups.patchapplication/octet-stream; name=v1.1-0001-pg_regress-multi-line-test-groups.patchDownload+276-26
#4Daniel Gustafsson
daniel@yesql.se
In reply to: Alexandre Felipe (#3)
Re: pg_regress: schedule multi-line test groups

On 17 Sep 2026, at 10:43, Alexandre Felipe <o.alexandre.felipe@gmail.com> wrote:

Actually, I think we should support arbitrary size test groups and run them on a pool
from a queue that would possibly make the regression faster as today we always
wait for the slowest test in each group.

I know of ongoing, as of yet unpublished, work in this area to improve this
scheduling which will also change the syntax. There is a bit complexity to it
than just picking from a pool since there are inter-suite dependencies.

	+		/*
	+		 * Found `test: # no tests` treat it as a multiline test group
	+		 */
	 		if (num_tests == 0)
	 		{
	-			bail("syntax error in schedule file \"%s\" line %d: %s",
	-				 schedule, line_num, scbuf);
	+			multiline_test = true;
	+		}

This will allow empty test groups which we currently treat as an error. This
schedule:

test: test_setup\n
test:\n
\n
test:\n

Parses and runs with all tests succeeded:

# initializing database system by copying initdb template
# using temp instance on port 40058 with PID 71845
ok 1 - test_setup 448 ms
# parallel group (0 tests):
# parallel group (0 tests):
1..1
# All 1 tests passed.
# test succeeded

Without the patch an empty group will throw an error, and I think it makes
sense to preserve that behaviour.

Also, if we are to change the accepted syntax, why limit to single line or
multiline, why not accept any whitespace separated name between ^test: and
(^test|EOF)? Something like the below:

test: boolean
char
name

--
Daniel Gustafsson

#5Alexandre Felipe
o.alexandre.felipe@gmail.com
In reply to: Daniel Gustafsson (#4)
Re: pg_regress: schedule multi-line test groups

Thank you for your looking into it Daniel

On Fri, Sep 18, 2026 at 10:31 AM Daniel Gustafsson <daniel@yesql.se> wrote:

On 17 Sep 2026, at 10:43, Alexandre Felipe <o.alexandre.felipe@gmail.com>

wrote:

Actually, I think we should support arbitrary size test groups and run

them on a pool

from a queue that would possibly make the regression faster as today we

always

wait for the slowest test in each group.

I know of ongoing, as of yet unpublished, work in this area to improve this
scheduling which will also change the syntax. There is a bit complexity
to it
than just picking from a pool since there are inter-suite dependencies.

An easy gain would be to somehow transpose execution, currently we put
together
things to indicate that they are independent, it would be more readable if
we used
the lines to indicate dependency.

The right way to handle dependency is by writing a dependency tree and doing
a topological sort, like makefiles, but to keep the syntax backward
compatible
we could do something like this

# tests that have to run in isolation
test: setup_test
test: sanity_check
# tests that have to respect certain order
test sequentially: create_am psql
# where a test should wait for multiple tests that can run in parallel
test sequentially: (geometry create_index_spgist hash_index brin) amutils
# tests that can mix with other tests
test concurrently: compression compression_lz4 compression_pglz cluster

But please, let's first get the multiline syntax over the fence.

v2 is logging the effective concurrency of the groups during the execution
(we can't optimise what we don't measure).
e.g:
# effective concurrency 5.80 / 18
# effective concurrency 3.83 / 15

+ /*

+                * Found `test: # no tests` treat it as a
multiline test group
+                */
if (num_tests == 0)
{
-                       bail("syntax error in schedule file \"%s\"
line %d: %s",
-                                schedule, line_num, scbuf);
+                       multiline_test = true;
+               }

This will allow empty test groups which we currently treat as an error.
This
schedule:

Fixed, also improved the error messages

Also, if we are to change the accepted syntax, why limit to single line or

multiline, why not accept any whitespace separated name between ^test: and
(^test|EOF)? Something like the below:

test: boolean
char
name

Minimalism. in v2 I am trying to do something more like what you described

The schedule syntax in v2 should be

* Schedule = (Blank | Comment | Group)*
* Group = "test:" (Line | Comment) (Indent Line)*
* Line = token (Space+ token)* Comment?
* Blank = '\n'
* Comment = '#' [^\n]* \n
* Indent = [\t ]+

But I won't be surprised if you find inconsistencies between that and what
was actually implemented at this stage.

Regards,

Attachments:

t253819_5
v2-0001-pg_regress-multi-line-test-groups.patchapplication/octet-stream; name=v2-0001-pg_regress-multi-line-test-groups.patchDownload+345-37
#6Alexandre Felipe
o.alexandre.felipe@gmail.com
In reply to: Alexandre Felipe (#5)
Re: pg_regress: schedule multi-line test groups

v2 failed on CFBot with error: 'INFINITY' undeclared.
v2.1 does the same as v2 but without using INFINITY.

On Sun, Sep 20, 2026 at 8:50 AM Alexandre Felipe
<o.alexandre.felipe@gmail.com> wrote:

Show quoted text

Thank you for your looking into it Daniel

On Fri, Sep 18, 2026 at 10:31 AM Daniel Gustafsson <daniel@yesql.se> wrote:

On 17 Sep 2026, at 10:43, Alexandre Felipe <o.alexandre.felipe@gmail.com> wrote:

Actually, I think we should support arbitrary size test groups and run them on a pool
from a queue that would possibly make the regression faster as today we always
wait for the slowest test in each group.

I know of ongoing, as of yet unpublished, work in this area to improve this
scheduling which will also change the syntax. There is a bit complexity to it
than just picking from a pool since there are inter-suite dependencies.

An easy gain would be to somehow transpose execution, currently we put together
things to indicate that they are independent, it would be more readable if we used
the lines to indicate dependency.

The right way to handle dependency is by writing a dependency tree and doing
a topological sort, like makefiles, but to keep the syntax backward compatible
we could do something like this

# tests that have to run in isolation
test: setup_test
test: sanity_check
# tests that have to respect certain order
test sequentially: create_am psql
# where a test should wait for multiple tests that can run in parallel
test sequentially: (geometry create_index_spgist hash_index brin) amutils
# tests that can mix with other tests
test concurrently: compression compression_lz4 compression_pglz cluster

But please, let's first get the multiline syntax over the fence.

v2 is logging the effective concurrency of the groups during the execution
(we can't optimise what we don't measure).
e.g:
# effective concurrency 5.80 / 18
# effective concurrency 3.83 / 15

+ /*

+                * Found `test: # no tests` treat it as a multiline test group
+                */
if (num_tests == 0)
{
-                       bail("syntax error in schedule file \"%s\" line %d: %s",
-                                schedule, line_num, scbuf);
+                       multiline_test = true;
+               }

This will allow empty test groups which we currently treat as an error. This
schedule:

Fixed, also improved the error messages

Also, if we are to change the accepted syntax, why limit to single line or

multiline, why not accept any whitespace separated name between ^test: and
(^test|EOF)? Something like the below:

test: boolean
char
name

Minimalism. in v2 I am trying to do something more like what you described

The schedule syntax in v2 should be

* Schedule = (Blank | Comment | Group)*
* Group = "test:" (Line | Comment) (Indent Line)*
* Line = token (Space+ token)* Comment?
* Blank = '\n'
* Comment = '#' [^\n]* \n
* Indent = [\t ]+

But I won't be surprised if you find inconsistencies between that and what
was actually implemented at this stage.

Regards,

Attachments:

t253819_6
v2.1-0001-pg_regress-multi-line-test-groups.patchapplication/octet-stream; name=v2.1-0001-pg_regress-multi-line-test-groups.patchDownload+346-37
#7Alexandre Felipe
o.alexandre.felipe@gmail.com
In reply to: Alexandre Felipe (#6)
Re: pg_regress: schedule multi-line test groups

Hi All,

I just received an email about this requiring a rebase.

Is there any chance this will be committed, or should I abandon it?

Regards,
Alexandre

On Sun, Sep 20, 2026 at 6:56 PM Alexandre Felipe <
o.alexandre.felipe@gmail.com> wrote:

Show quoted text

v2 failed on CFBot with error: 'INFINITY' undeclared.
v2.1 does the same as v2 but without using INFINITY.

On Sun, Sep 20, 2026 at 8:50 AM Alexandre Felipe
<o.alexandre.felipe@gmail.com> wrote:

Thank you for your looking into it Daniel

On Fri, Sep 18, 2026 at 10:31 AM Daniel Gustafsson <daniel@yesql.se>

wrote:

On 17 Sep 2026, at 10:43, Alexandre Felipe <

o.alexandre.felipe@gmail.com> wrote:

Actually, I think we should support arbitrary size test groups and

run them on a pool

from a queue that would possibly make the regression faster as today

we always

wait for the slowest test in each group.

I know of ongoing, as of yet unpublished, work in this area to improve

this

scheduling which will also change the syntax. There is a bit

complexity to it

than just picking from a pool since there are inter-suite dependencies.

An easy gain would be to somehow transpose execution, currently we put

together

things to indicate that they are independent, it would be more readable

if we used

the lines to indicate dependency.

The right way to handle dependency is by writing a dependency tree and

doing

a topological sort, like makefiles, but to keep the syntax backward

compatible

we could do something like this

# tests that have to run in isolation
test: setup_test
test: sanity_check
# tests that have to respect certain order
test sequentially: create_am psql
# where a test should wait for multiple tests that can run in parallel
test sequentially: (geometry create_index_spgist hash_index brin) amutils
# tests that can mix with other tests
test concurrently: compression compression_lz4 compression_pglz cluster

But please, let's first get the multiline syntax over the fence.

v2 is logging the effective concurrency of the groups during the

execution

(we can't optimise what we don't measure).
e.g:
# effective concurrency 5.80 / 18
# effective concurrency 3.83 / 15

+ /*

+ * Found `test: # no tests` treat it as a

multiline test group

+                */
if (num_tests == 0)
{
-                       bail("syntax error in schedule file

\"%s\" line %d: %s",

-                                schedule, line_num, scbuf);
+                       multiline_test = true;
+               }

This will allow empty test groups which we currently treat as an

error. This

schedule:

Fixed, also improved the error messages

Also, if we are to change the accepted syntax, why limit to single line

or

multiline, why not accept any whitespace separated name between ^test:

and

(^test|EOF)? Something like the below:

test: boolean
char
name

Minimalism. in v2 I am trying to do something more like what you

described

The schedule syntax in v2 should be

* Schedule = (Blank | Comment | Group)*
* Group = "test:" (Line | Comment) (Indent Line)*
* Line = token (Space+ token)* Comment?
* Blank = '\n'
* Comment = '#' [^\n]* \n
* Indent = [\t ]+

But I won't be surprised if you find inconsistencies between that and

what

was actually implemented at this stage.

Regards,

#8Daniel Gustafsson
daniel@yesql.se
In reply to: Alexandre Felipe (#7)
Re: pg_regress: schedule multi-line test groups

On 13 Nov 2026, at 10:13, Alexandre Felipe <o.alexandre.felipe@gmail.com> wrote:

Hi All,

I just received an email about this requiring a rebase.

Is there any chance this will be committed, or should I abandon it?

There is definitely a chance, I have it on my radar for when the 19 release is
out, so it would be great if it was rebased.

--
Daniel Gustafsson

#9Alexandre Felipe
o.alexandre.felipe@gmail.com
In reply to: Daniel Gustafsson (#8)
Re: pg_regress: schedule multi-line test groups

Thank you Daniel,

v2.2 - rebased: Conflicts caused by tests added to parallel_scheule.

Regards,
Alexandre

Attachments:

t253819_9
v2.2-0001-pg_regress-multi-line-test-groups.patchapplication/octet-stream; name=v2.2-0001-pg_regress-multi-line-test-groups.patchDownload+349-37
#10Zsolt Parragi
zsolt.parragi@percona.com
In reply to: Alexandre Felipe (#9)
Re: pg_regress: schedule multi-line test groups

Hello

This looks like a good feature, but I think the parser needs some improvements:

1. A test name directly followed by # is silently dropped

test: a b#note

or

test:
a
b# depends on a

Will only run 'a'

2. An empty or comment-only schedule file now fails

3. The --max-concurrent-tests error reports the wrong line

test:
a
b
c
d
test: a

With --max-concurrent-tests=3 this reports "line 6: test: a". With a trailing comment after the group it reports the comment line instead. Maybe it should point to the start of the test?

4. (2) and (3) both come from the goto structure. Wouldn't a run_test_group() helper be simpler?

5.

EFFECTIVE CONCURRENCY
=====================
This also add for each group a note about the average execution
concurrency achieved during the test.

# parallel group (2 tests): brin_bloom brin_multi
# effective concurrency 1.46 / 2

Wouldn't it be better to separate this part into a separate patch file?

#11Alexandre Felipe
o.alexandre.felipe@gmail.com
In reply to: Zsolt Parragi (#10)
Re: pg_regress: schedule multi-line test groups

On Fri, Nov 13, 2026 at 4:10 PM Zsolt Parragi <zsolt.parragi@percona.com>
wrote:

Hello

Hello
Thank you for joining.

This looks like a good feature, but I think the parser needs some
improvements:

1. A test name directly followed by # is silently dropped

test: a b#note

or

test:
a
b# depends on a

Will only run 'a'

Fixed, updated syntax at the function doc comment.

2. An empty or comment-only schedule file now fails

Yes, intentionally, the patch preamble mentions that.

3. The --max-concurrent-tests error reports the wrong line

test:
a
b
c
d
test: a

With --max-concurrent-tests=3 this reports "line 6: test: a". With a
trailing comment after the group it reports the comment line instead.
Maybe it should point to the start of the test?

upstream has two conditions with the same error message, and I failed
to distinguish them.

Now there is only one message, will show

in schedule file "..." line 5, test d: max concurrency (3) exceeded

I think this is better than the pointing to the start of the test, but
if that is still preferred it can easily be changed.

4. (2) and (3) both come from the goto structure. Wouldn't a

run_test_group() helper be simpler?

It would, I was trying to keep the *patch* as small as possible
I suggested some functional changes in this thread on 2026-09-20 [1]/messages/by-id/CAE8JnxPycpY5mk7C-rNa4jeMQQXG1H5mobB10Oa_CaQs1nBvUg@mail.gmail.com
I recorded a video presenting a mockup of of that in [2]https://youtu.be/FHjQEuDoxFw

5.

EFFECTIVE CONCURRENCY
=====================
This also add for each group a note about the average execution
concurrency achieved during the test.

# parallel group (2 tests): brin_bloom brin_multi
# effective concurrency 1.46 / 2

Wouldn't it be better to separate this part into a separate patch file?

v2.3 puts effective concurrency logging in a separate patch.

[1]: /messages/by-id/CAE8JnxPycpY5mk7C-rNa4jeMQQXG1H5mobB10Oa_CaQs1nBvUg@mail.gmail.com
/messages/by-id/CAE8JnxPycpY5mk7C-rNa4jeMQQXG1H5mobB10Oa_CaQs1nBvUg@mail.gmail.com
[2]: https://youtu.be/FHjQEuDoxFw

Attachments:

t253819_11
index.htmltext/html; charset=US-ASCII; name=index.htmlDownload
v2.3-0001-pg_regress-multi-line-test-groups.patchapplication/octet-stream; name=v2.3-0001-pg_regress-multi-line-test-groups.patchDownload+339-50
v2.3-0002-pg_regress-report-group-effective-concurrency.patchapplication/octet-stream; name=v2.3-0002-pg_regress-report-group-effective-concurrency.patchDownload+27-1
#12Zsolt Parragi
zsolt.parragi@percona.com
In reply to: Alexandre Felipe (#11)
Re: pg_regress: schedule multi-line test groups

On Sun, 15 Nov 2026, Alexandre Felipe <o.alexandre.felipe@gmail.com> wrote:

Yes, intentionally, the patch preamble mentions that.

ops, I missed that, sorry.

1. Now with 2.3:

$ cat s
test: a
test: b
$ pg_regress ... --schedule=s --max-concurrent-tests=0
# in schedule file "s" line 1, test a: max concurrency (0) exceeded
Bail out!

While --help still says that 0 means unlimited. (and --max-concurrent-tests=abc is also 0, but that's a different issue)

2. long commented out lines can run tests

test:
a
# xxxx...(1019 x)...xxxx b

"All 2 tests passed"

+/*
+ * Maximum supported number of concurrent tests
+ */
+#define MAX_PARALLEL_TESTS 100

This is now defined twice.

#13Alexandre Felipe
o.alexandre.felipe@gmail.com
In reply to: Zsolt Parragi (#12)
Re: pg_regress: schedule multi-line test groups

On Mon, Nov 16, 2026 at 7:46 PM Zsolt Parragi <zsolt.parragi@percona.com>
wrote:

1. Now with 2.3:

$ pg_regress ... --schedule=s --max-concurrent-tests=0
# in schedule file "s" line 1, test a: max concurrency (0) exceeded
Bail out!

While --help still says that 0 means unlimited. (and
--max-concurrent-tests=abc is also 0, but that's a different issue)

Now,
regress$ ./pg_regress --schedule=parallel_schedule --max-concurrent-tests=0
# --max-concurrent-tests=0 expected to be integer in range 1-100
Bail out!%

regress$ ./pg_regress --schedule=parallel_schedule --max-concurrent-tests=x
# --max-concurrent-tests=x expected to be integer in range 1-100
Bail out!%

regress$ ./pg_regress --schedule=parallel_schedule
--max-concurrent-tests=101
# --max-concurrent-tests=101 expected to be integer in range 1-100
Bail out!%

regress$ ./pg_regress --schedule=parallel_schedule
--max-concurrent-tests=100
# using postmaster on Unix socket, default port

regress$ ./pg_regress --help
...
--max-concurrent-tests=N maximum number of concurrent tests in
schedule
default: 100

2. long commented out lines can run tests

test:
a
# xxxx...(1019 x)...xxxx b

"All 2 tests passed"

Fixing (this was an old bug).

sizeof(scbuf) = 1024, so maximum line length is 1022.
this has to handle two cases: (i) in the middle of the file
the string can be 1023 long including a final line separator
(ii) the last line doesn't have a \n separator, and can't be
1023 long. Since fgets stops when the buffer it will read at
most 1023 bytes, 1 byte is reserved for the null terminator

+/*

+ * Maximum supported number of concurrent tests
+ */
+#define MAX_PARALLEL_TESTS 100

This is now defined twice.

Ups, I think I removed now...

---
Alexandre Felipe

Attachments:

t253819_13
v2.4-0001-pg_regress-multi-line-test-groups.patchapplication/octet-stream; name=v2.4-0001-pg_regress-multi-line-test-groups.patchDownload+351-54
v2.4-0002-pg_regress-report-group-effective-concurrency.patchapplication/octet-stream; name=v2.4-0002-pg_regress-report-group-effective-concurrency.patchDownload+27-1
#14Michael Paquier
michael@paquier.xyz
In reply to: Alexandre Felipe (#13)
Re: pg_regress: schedule multi-line test groups

On Wed, Nov 18, 2026 at 08:25:32AM +0000, Alexandre Felipe wrote:

+ * Maximum supported number of concurrent tests
+ */
+#define MAX_PARALLEL_TESTS 100

This is now defined twice.

Ups, I think I removed now...

+test:
+    boolean
+    char
+    name
While reading through this patch, I am wondering if it would not be
better to support backslashes instead of an arbitrary amount of spaces
a-la-yaml for each group, like:
+test: \
+    boolean \
+    char \
+    name \

And that may actually make the patch simpler overall with less
hardcoded rules arount "test:". I can see that 0001 adds two more of
them, reducing the code readability. That would also work as a
backward-compatible grammar. Perhaps I'm too pedantic, so feel free
to overwrite me, of course.

0002 provides a cute metric, but I doubt that it is strictly
necessary. Performance matters for grouping, but that's not the only
reason why we have groups: we want some organization as well.
--
Michael