On Judging the Value of Tests
Dear PG Experts,
I am entirely very new to PG development.
Currently, I’m studying the test suites, and I feel the more experienced PG devs here could provide some insights on testing because I could not find concrete answers in the Postgres documentations.
1. How do you judge when a test suite is acceptable to be added to Postgres OSS source code? (How do you judge the value of a test suite?)
2. Besides code coverage test, are there other ways/techniques/tools to measure how much value a test would have?
This is coming from a perspective that in the current code coverage (https://coverage.postgresql.org/), some functions were covered because the function was called/run in some tests, while the function/lines may also not necessarily be thoroughly/directly tested so the coverage skips some lines of code inside a block.
3. Is there a standard way of writing tests on the source code that we should follow, like when should test be written in TAP/SQL/C formats and how long should it be?
� I know that TAP test is for client program tests, SQL for regression tests with sql queries, but there are instances also where tests are also written in C like isolation tests, etc. How do we best judge which language is preferred to use when writing tests for Postgres components? How long should a test be when proposing them to hackers page?
4. In the src/test/examples directory (which are all libpq tests), why is the “examples” directory not included when building postgres? (Why weren't these libpq tests added to src/interface/libpq/test or in regression test suite instead?) In short, how to know where (in which file/directory in source code) to put a test?
I’d like to apologize for the series of questions, but I’d really like to get insights from the experienced devs.
It's alright if you don't have answers to all questions as this might take much of your time.
Thank you very much.
Regards,
Kirk Jamison
On 22 November 2017 at 08:43, Jankirk.Vincent., Jamison <
k.jamison@jp.fujitsu.com> wrote:
Dear PG Experts,
I am entirely very new to PG development.
Currently, I’m studying the test suites, and I feel the more experienced
PG devs here could provide some insights on testing because I could not
find concrete answers in the Postgres documentations.1. How do you judge when a test suite is acceptable to be added to
Postgres OSS source code? (How do you judge the value of a test suite?)
Make your argument for it, and see if others agree. There's no formal
process.
Inputs into the decision making process include:
* How much coverage of previously untested functionality it adds
* How much code coverage it adds
* How long the test takes to run, especially considering the slow buildfarm
boxes and development turnaround time
* Whether the test fits into one of the existing suites we run routinely,
or requires separate steps
* How much work will be required to maintain the test
3. Is there a standard way of writing tests on the source code that
we should follow, like when should test be written in TAP/SQL/C formats and
how long should it be?① I know that TAP test is for client program tests, SQL for
regression tests with sql queries, but there are instances also where tests
are also written in C like isolation tests, etc. How do we best judge which
language is preferred to use when writing tests for Postgres components?
How long should a test be when proposing them to hackers page?
In general, prefer pg_regress if you can use it. Isolation tests for
concurrency issues. TAP tests if you can't write it with pg_regress or
isolation tester. Test modules only if you really must.
4. In the src/test/examples directory (which are all libpq tests),
why is the “examples” directory not included when building postgres? (Why
weren't these libpq tests added to src/interface/libpq/test or in
regression test suite instead?) In short, how to know where (in which
file/directory in source code) to put a test?
Dunno, sorry.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
On Wed, Nov 22, 2017 at 10:31 AM, Craig Ringer <craig@2ndquadrant.com> wrote:
On 22 November 2017 at 08:43, Jankirk.Vincent., Jamison
<k.jamison@jp.fujitsu.com> wrote:I am entirely very new to PG development.
Currently, I’m studying the test suites, and I feel the more experienced
PG devs here could provide some insights on testing because I could not find
concrete answers in the Postgres documentations.1. How do you judge when a test suite is acceptable to be added to
Postgres OSS source code? (How do you judge the value of a test suite?)Make your argument for it, and see if others agree. There's no formal
process.Inputs into the decision making process include:
* How much coverage of previously untested functionality it adds
* How much code coverage it adds
* How long the test takes to run, especially considering the slow buildfarm
boxes and development turnaround time
* Whether the test fits into one of the existing suites we run routinely, or
requires separate steps
* How much work will be required to maintain the test
Even with those factors in mind there is a case-by-case variability.
Sometimes it is not worth initializing a new instance with initdb for
a small test. Parallelization of tests help on large machines by on
buildfarm members the size of a RPI things get time-consuming.
Priority is usually given into things that do not consume much time,
so as any developer can run the full set of tests quickly on a laptop.
I have for example a small alias to run the full set of basic tests
(make check-world -j 4 PROVE_FLAGS="-j 4") and this finishes within
two minutes if I recall correctly, TAP tests included. So runtime is
really an important matter.
3. Is there a standard way of writing tests on the source code that
we should follow, like when should test be written in TAP/SQL/C formats and
how long should it be?① I know that TAP test is for client program tests, SQL for
regression tests with sql queries, but there are instances also where tests
are also written in C like isolation tests, etc. How do we best judge which
language is preferred to use when writing tests for Postgres components? How
long should a test be when proposing them to hackers page?In general, prefer pg_regress if you can use it. Isolation tests for
concurrency issues. TAP tests if you can't write it with pg_regress or
isolation tester. Test modules only if you really must.
This really depends on what you want to achieve, so this comes back to
a case-by-case. Regression tests are fine for basic tests, where one
session is enough to prove your feature. Isolation tests are helpful
when testing concurrent behaviors across multple sessions. TAP tests
are useful when more than one instance is needed or when you need to
test cluster-wide configurations or client binaries. Having tests in
patches proposed definitely helps people in understanding more how
something works. Sometimes things get discarded but they can really
help in making a point. Lately, while working on channel binding
support for SCRAM, TAP tests have for example proved to help:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=9288d62bb4b6f302bf13bb2fed3783b61385f315#patch13.
This needs a SSL setup with a specific configuration for pg_hba.conf,
and proved with just a lookup at the tests in the patch that the
feature proposed was able to achieve what it aimed at. So it proved a
point.
4. In the src/test/examples directory (which are all libpq tests),
why is the “examples” directory not included when building postgres? (Why
weren't these libpq tests added to src/interface/libpq/test or in regression
test suite instead?) In short, how to know where (in which file/directory in
source code) to put a test?Dunno, sorry.
Those are mainly present as example programs.
--
Michael
Craig Ringer <craig@2ndquadrant.com> writes:
On 22 November 2017 at 08:43, Jankirk.Vincent., Jamison <
k.jamison@jp.fujitsu.com> wrote:1. How do you judge when a test suite is acceptable to be added to
Postgres OSS source code? (How do you judge the value of a test suite?)
Make your argument for it, and see if others agree. There's no formal
process.
Yeah. Also realize that we've been accreting test cases for ~20 years,
and there's never been any master plan about what to test. So there's a
lot of individual test cases of varying quality/value, and you shouldn't
necessarily take any one existing test as gospel. I don't hold that out
as a great example of software engineering, but it's the truth.
Inputs into the decision making process include:
* How much coverage of previously untested functionality it adds
* How much code coverage it adds
* How long the test takes to run, especially considering the slow buildfarm
boxes and development turnaround time
* Whether the test fits into one of the existing suites we run routinely,
or requires separate steps
* How much work will be required to maintain the test
Also, portability/repeatability. If it doesn't produce the same answers,
with very high probability, across all the platforms we support, it isn't
going to last long.
3. Is there a standard way of writing tests on the source code that
we should follow, like when should test be written in TAP/SQL/C formats and
how long should it be?
In general, prefer pg_regress if you can use it. Isolation tests for
concurrency issues. TAP tests if you can't write it with pg_regress or
isolation tester. Test modules only if you really must.
Right. This ties in closely to the incremental runtime needed for one
additional test case. A new query or two in an existing SQL regression
script adds little overhead. A new TAP test adds quite a bit.
4. In the src/test/examples directory (which are all libpq tests),
why is the “examples” directory not included when building postgres?
I've always taken those to be more documentation, or sample code, than
test cases. I'm not sure why they're under src/test/ at all.
regards, tom lane
1. How do you judge when a test suite is acceptable to be added to
Postgres OSS source code? (How do you judge the value of a test
suite?)Make your argument for it, and see if others agree. There's no formal
process.Yeah. Also realize that we've been accreting test cases for ~20 years,
and there's never been any master plan about what to test. So there's a
lot of individual test cases of varying quality/value, and you shouldn't
necessarily take any one existing test as gospel. I don't hold that out
as a great example of software engineering, but it's the truth.Inputs into the decision making process include:
* How much coverage of previously untested functionality it adds
* How much code coverage it adds
* How long the test takes to run, especially considering the slow
buildfarm boxes and development turnaround time
* Whether the test fits into one of the existing suites we run
routinely, or requires separate steps
* How much work will be required to maintain the testEven with those factors in mind there is a case-by-case variability.
Sometimes it is not worth initializing a new instance with initdb for
a small test. Parallelization of tests help on large machines by on
buildfarm members the size of a RPI things get time-consuming.
Priority is usually given into things that do not consume much time,
so as any developer can run the full set of tests quickly on a laptop.
I have for example a small alias to run the full set of basic tests
(make check-world -j 4 PROVE_FLAGS="-j 4") and this finishes within
two minutes if I recall correctly, TAP tests included. So runtime is
really an important matter.Also, portability/repeatability. If it doesn't produce the same answers,
with very high probability, across all the platforms we support, it isn't
going to last long.
Thank you everyone for your valuable input.
I will take note of all your advice. I think it's because I've noticed that a lot of functions and lines are still not covered by tests (https://coverage.postgresql.org/), so I feel like I want to work on tests first while I'm still learning the PG internals and hacking.
3. Is there a standard way of writing tests on the source code that
we should follow, like when should test be written in TAP/SQL/C
formats and how long should it be?In general, prefer pg_regress if you can use it. Isolation tests for
concurrency issues. TAP tests if you can't write it with pg_regress
or isolation tester. Test modules only if you really must.Right. This ties in closely to the incremental runtime needed for one
additional test case. A new query or two in an existing SQL regression
script adds little overhead. A new TAP test adds quite a bit.
I understand that tests should not take that long when running.
For example, how about when proposing tests that just test
parameter settings of a function?
Let's say, for example, a libpq main function where in different test cases
the params are null, empty, invalid, or valid.
Are these test types common or recommended?
4. In the src/test/examples directory (which are all libpq tests),
why is the “examples” directory not included when building postgres?Dunno, sorry.
Those are mainly present as example programs.
I've always taken those to be more documentation, or sample code, than
test cases. I'm not sure why they're under src/test/ at all.
Since these example codes are only documentations, so should these be removed in the source code instead? Or should these examples code be revised and be moved to a test directory in libpq?
Thanks again!
Regards,
Kirk Jamison
On Fri, Nov 24, 2017 at 10:27 AM, Jankirk.Vincent., Jamison
<k.jamison@jp.fujitsu.com> wrote:
I will take note of all your advice. I think it's because I've noticed that a lot of functions and lines are still not covered by tests (https://coverage.postgresql.org/), so I feel like I want to work on tests first while I'm still learning the PG internals and hacking.
That's a good idea to get more familiar with the internals of
Postgres. Don't forget to submit any patch you would like to get
integrated into -hackers and to get it registered in the commit fest
application. Please see here for more details:
https://wiki.postgresql.org/wiki/Submitting_a_Patch
--
Michael