Contributing test cases to improve coverage
Hello All,
I am working on a project that aims to produce test cases that
improve mutation coverage of a dbms's test suite.
The rough workflow of the project goes as follows:
(a) apply mutation at a souce code level
(b) compile and check if the mutated installation passed existing testsuite
(c) If not, fuzz the mutated installation with SQL fuzzer
(d) if a fuzzor successfully produce a test case that crash or trigger bugs
in the mutated installation, use a reduction tools to reduce the test case
(e) add the reduced test case to existing test suite
For postgres, I am looking at adding test cases to test suite in
test/src/regress/. I have gone through (a)-(e), and managed to produced
some test cases. As an example, I claim the test case
```
CREATE RECURSIVE VIEW a(b) AS SELECT'' ;
SELECT FROM a WHERE NULL;
```
could kill the following mutation at optimizer/plan/setrefs.c, 502:5--502:33
Original binary operator expression:
```
rte->rtekind == RTE_SUBQUERY
````
Replacement expression:
```
(rte->rtekind) >= RTE_SUBQUERY
```
I have a few questions about adding these test cases:
(a) The regression test suite is run by a parallel scheduler, with some
test cases dependent on previous test cases. If I just add my test case as
part of the parallel scheduler’s tests, it might not work, since previous
test cases in the scheduler might already create the same table, for
instance.
(b) How do I get my test cases reviewed and ultimately included in a future
release of PostgreSQL?
Thank you for your time.
Regards,
Jon
J F <jonathanfoo0523@gmail.com> writes:
For postgres, I am looking at adding test cases to test suite in
test/src/regress/. I have gone through (a)-(e), and managed to produced
some test cases. As an example, I claim the test case
```
CREATE RECURSIVE VIEW a(b) AS SELECT'' ;
SELECT FROM a WHERE NULL;
```
could kill the following mutation at optimizer/plan/setrefs.c, 502:5--502:33
Original binary operator expression:
```
rte->rtekind == RTE_SUBQUERY
````
Replacement expression:
```
(rte->rtekind) >= RTE_SUBQUERY
```
I am quite confused about what is the point of this. You have not
found any actual bug, nor have you demonstrated that this test case
could discover a likely future bug that wouldn't be detected another
way. Moreover, it seems like the process would lead to some very
large number of equally marginal test cases. We aren't likely to
accept such a patch, because we are concerned about keeping down the
runtime of the test suite.
regards, tom lane
I am quite confused about what is the point of this. You have not
found any actual bug, nor have you demonstrated that this test case
could discover a likely future bug that wouldn't be detected another
way. Moreover, it seems like the process would lead to some very
large number of equally marginal test cases. We aren't likely to
accept such a patch, because we are concerned about keeping down the
runtime of the test suite.regards, tom lane
The point of this project is to improve the coverage of PostgreSQL’s
preexisting test suite. Writing a test suite to achieve close to 100%
coverage is challenging, but I have proposed a workflow to automate this
process.
I assert that no test case in the regression test suite currently covers
the comparator in the expression rte->rtekind == RTE_SUBQUERY. I propose
adding a new test case that addresses exactly this. In the future, if
someone accidentally modifies the operator to become >=, it will trigger
incorrect behavior when certain queries are executed. This test case will
catch that issue.
I get that the test cases in /regress are likely reserved for actual bugs
found and are designed to run quickly. Would it be a good idea to have a
separate, more rigorous test suite that runs longer but provides better
code coverage?
Regards,
Jon
On Thu, Jun 13, 2024 at 1:22 AM J F <jonathanfoo0523@gmail.com> wrote:
I am quite confused about what is the point of this. You have not
found any actual bug, nor have you demonstrated that this test case
could discover a likely future bug that wouldn't be detected another
way. Moreover, it seems like the process would lead to some very
large number of equally marginal test cases. We aren't likely to
accept such a patch, because we are concerned about keeping down the
runtime of the test suite.regards, tom lane
The point of this project is to improve the coverage of PostgreSQL’s
preexisting test suite. Writing a test suite to achieve close to 100%
coverage is challenging, but I have proposed a workflow to automate this
process.
We monitor code coverage. But this is doing more than that. It will find
out the places in code, which if changed, will cause bugs. That seems
useful to avoid refactoring mistakes, esp. when people rely on regression
tests to tell whether their code changes are sane. But in PostgreSQL, we
rely heavily on reviewers and committers to do that instead of tests.
Still, the tests produced by this tool will help catch bugs that human eyes
can not. As Tom said, managing that battery of tests may not be worth it.
Basically, I am flip-flopping on the usefulness of this effort.
I assert that no test case in the regression test suite currently covers
the comparator in the expression rte->rtekind == RTE_SUBQUERY. I propose
adding a new test case that addresses exactly this. In the future, if
someone accidentally modifies the operator to become >=, it will trigger
incorrect behavior when certain queries are executed. This test case will
catch that issue.
Usually PostgreSQL developers know that rtekind is an enum so they are very
very unlikely to use anything other than == and !=. Such a change will be
caught by a reviewer. I think many of the tests that this tool will produce
will fall in this category.
I get that the test cases in /regress are likely reserved for actual bugs
found and are designed to run quickly. Would it be a good idea to have a
separate, more rigorous test suite that runs longer but provides better
code coverage?There are practical difficulties like maintaining the expected outputs for
such a large battery of tests. But maybe some external project could.
BTW, have you considered perl tests, isolation tests etc. Tests in regress/
do not cover many subsystems e.g. replication.
--
Best Wishes,
Ashutosh Bapat
On 12.06.24 18:44, J F wrote:
(a) The regression test suite is run by a parallel scheduler, with some
test cases dependent on previous test cases. If I just add my test case
as part of the parallel scheduler’s tests, it might not work, since
previous test cases in the scheduler might already create the same
table, for instance.
Yes, you need to take care of that somehow. Some test files put all
their test objects in a schema. Others are careful to drop all test
objects at the end. Or you just have to pick non-conflicting names.
(b) How do I get my test cases reviewed and ultimately included in a
future release of PostgreSQL?
Perhaps start with
https://wiki.postgresql.org/wiki/Development_information
and in particular