Environment variable to disable diffs file output

On Mon, 6 Apr 2026 at 09:56, Илья Чердаков <i.cherdakov.pg@gmail.com> wrote:

I'd like to add a environment variable to the patch to
disable diff output. Sometimes you have to be creative
with exploratory testing, modifying test files, etc.,
and multiple diff outputs can become a nuisance. This
doesn't change the default behavior, but it does give
you control over diff output in tests for specific
scenarios where a large number of failures are expected
(mass runs or exploratory testing with changing tests).

I'm fine with adding the abilitity to configure what pg_regress should
print, but it's unclear to me what's special about the diff output
compared to all the other output? i.e. what is the output you would
actually like to see for your use case? Piping everything to /dev/null
would silence all output except for the exit code, but I guess you
want some output. i.e. what do you do with the output that you get? Do
you only want to know which tests failed? If so, do you care about
which tests are passing?

On 6 Apr 2026, at 09:55, Илья Чердаков <i.cherdakov.pg@gmail.com> wrote:

I'd like to add a environment variable to the patch to
disable diff output. Sometimes you have to be creative
with exploratory testing, modifying test files, etc.,
and multiple diff outputs can become a nuisance. This
doesn't change the default behavior, but it does give
you control over diff output in tests for specific
scenarios where a large number of failures are expected
(mass runs or exploratory testing with changing tests).

The env usage and implementation are somewhat similar
to PG_REGRESS_DIFF_OPTS
PG_REGRESS_DISABLE_DIFFS_OUTPUT=1 make check -s.

For now, I suggest to simply disabling it, but I'm also
considering passing the number of lines to output via env
or via flag pg_regress instead of completely disabling it.
Something like
PG_REGRESS_DIFFS_NUMBER_LINES_OUTPUT=20 make check
and then when an error occurs, 20 lines will be output
instead of the default 80. Setting it to 0 disables
diffs output.

I'm not sure I entirely understand the problem. If you expect lots of
failures, but also don't want to see the test failures, what is the use of
running the tests? Why not run the subset you actually care about and expand
that set as testing fixes bugs/issues?

--
Daniel Gustafsson

06.04.2026 16:47, Jelte Fennema-Nio wrote:

I'm fine with adding the abilitity to configure what pg_regress should
print, but it's unclear to me what's special about the diff output
compared to all the other output? i.e. what is the output you would
actually like to see for your use case? Piping everything to /dev/null
would silence all output except for the exit code, but I guess you
want some output. i.e. what do you do with the output that you get? Do
you only want to know which tests failed? If so, do you care about >

which tests are passing?

Yes, redirecting all output to /dev/null isn't exactly what I need.

Take a specific case, for example:
When making numerous changes to tests as part of extensive stability
testing (specifically, investigating crashes), the majority of test
files may be modified. This results in diffs output for practically
every single test, which turns into an unreadable jumble during the
test run. In this case, I am not interested in all the diffs
the ones associated with a crash. I can inspect those specific cases
manually later, without needing the diffs output. Therefore, a flag
like `PG_REGRESS_DIFF_OPTS='q'` does not suit my needs. And to find
out which tests caused the crash (exit code 1 or 2) I need the
entire output.

06.04.2026 21:11, Daniel Gustafsson wrote:

I'm not sure I entirely understand the problem. If you expect lots of
failures, but also don't want to see the test failures, what is the

use of

running the tests? Why not run the subset you actually care about and

expand

that set as testing fixes bugs/issues?

As part of exploratory testing:
1. Output mismatches are considered normal behavior.
2. Exploratory testing almost always affects all tests. Cases where
it is necessary to run a small subset of tests are rare.

Another case where multiple tests may fail is testing a patch that
changes cluster parameters.For example, a patch for the scheduler
has been released, and I need to ensure that it doesn't cause crashes
when changing the 'random_page_cost' parameter. This parameter
significantly changes the plan output, leading to multiple crashes and,
consequently, the diffs output. While I'm not interested in the diffs
themselves, I do want to know that there haven't been any crashes.
In this case, we need to run not just a small group of tests,
but everything we have.

These are just a couple of examples where disabling diffs output
would be useful.

---

Also, if the diffs within a single directory run are very large,
the output starts to become cluttered with lines like:

    #(diff output truncated and silencing output for
                            further failing tests...)

Which also creates confusion and interferes with output analysis.
Example

#<long diffs output>
not ok 1     - test_setup                                133 ms
# parallel group (20 tests, in groups of 1):  boolean char name ...
# (diff output truncated and silencing output for further failing tests...)
not ok 2     + boolean                                    14 ms
# (diff output truncated and silencing output for further failing tests...)
not ok 3     + char                                        8 ms
# (diff output truncated and silencing output for further failing tests...)
not ok 4     + name                                        8 ms
# (diff output truncated and silencing output for further failing tests...)
<etc.>

---
Best regards,
Ilya Cherdakov
Postgres Professional: https://postgrespro.com