Grouping isolationtester tests in the schedule
The list of tests in src/test/isolation/isolation_schedule has grown
over the years. Originally, they were all related to Serializable
Snapshot Isolation, but there are different kinds of concurrency tests
there now. More tests is good, but the schedule file has grown into a
big inscrutable list with zero comments.
I propose to categorize the tests and add some divider comments to the
file, see attached.
- Heikki
Attachments:
On Wed, Aug 7, 2019 at 11:28 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
The list of tests in src/test/isolation/isolation_schedule has grown
over the years. Originally, they were all related to Serializable
Snapshot Isolation, but there are different kinds of concurrency tests
there now. More tests is good, but the schedule file has grown into a
big inscrutable list with zero comments.
+1
I propose to categorize the tests and add some divider comments to the
file, see attached.
I think I'd put nowait and skip locked under a separate category "FOR
UPDATE" or "row locking" or something, but maybe that's just me... can
you call that stuff DML?
--
Thomas Munro
https://enterprisedb.com
On 07/08/2019 14:42, Thomas Munro wrote:
On Wed, Aug 7, 2019 at 11:28 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
The list of tests in src/test/isolation/isolation_schedule has grown
over the years. Originally, they were all related to Serializable
Snapshot Isolation, but there are different kinds of concurrency tests
there now. More tests is good, but the schedule file has grown into a
big inscrutable list with zero comments.+1
I propose to categorize the tests and add some divider comments to the
file, see attached.I think I'd put nowait and skip locked under a separate category "FOR
UPDATE" or "row locking" or something, but maybe that's just me... can
you call that stuff DML?
Yeah, I guess SELECT FOR UPDATE isn't really DML. Separate "Row locking"
category works for me. Or maybe "Concurrent DML and row locking". There
is also DML in some of those tests.
- Heikki
Heikki Linnakangas <hlinnaka@iki.fi> writes:
The list of tests in src/test/isolation/isolation_schedule has grown
over the years. Originally, they were all related to Serializable
Snapshot Isolation, but there are different kinds of concurrency tests
there now. More tests is good, but the schedule file has grown into a
big inscrutable list with zero comments.
I propose to categorize the tests and add some divider comments to the
file, see attached.
+1 for concept, didn't review your divisions.
Something related I've been wondering about is whether we could
parallelize the isolation tests. A difficulty here is that the
slowest ones tend to also be timing-sensitive, such that running
them in parallel would increase the risk of failure. But we
could likely get at least some improvement.
regards, tom lane
On 2019-Aug-07, Tom Lane wrote:
Something related I've been wondering about is whether we could
parallelize the isolation tests. A difficulty here is that the
slowest ones tend to also be timing-sensitive, such that running
them in parallel would increase the risk of failure. But we
could likely get at least some improvement.
Yeah, there's some improvement to be had there. We've discussed it
previously:
/messages/by-id/20180124231006.z7spaz5gkzbdvob5@alvherre.pgsql
I'm not really happy about this grouping if we mean we're restricted in
how we can make tests run in parallel.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 07/08/2019 18:52, Alvaro Herrera wrote:
On 2019-Aug-07, Tom Lane wrote:
Something related I've been wondering about is whether we could
parallelize the isolation tests. A difficulty here is that the
slowest ones tend to also be timing-sensitive, such that running
them in parallel would increase the risk of failure. But we
could likely get at least some improvement.Yeah, there's some improvement to be had there. We've discussed it
previously:
/messages/by-id/20180124231006.z7spaz5gkzbdvob5@alvherre.pgsqlI'm not really happy about this grouping if we mean we're restricted in
how we can make tests run in parallel.
The elephant in the room is the 'timeouts' test, which takes about 40
seconds, out of a total runtime of 90 seconds. So we'd really want to
run that in parallel with everything else. Or split 'timeouts' into
multiple tests that could run in parallel. I don't think grouping the
rest of the tests differently will make much difference to how easy or
hard that is.
In any case, we can scramble the list again later, if that's needed for
running the tests in parallel, and we think it's worth it. Until then, a
more logical grouping and some comments would be nice.
- Heikki
On 2019-Aug-07, Heikki Linnakangas wrote:
The elephant in the room is the 'timeouts' test, which takes about 40
seconds, out of a total runtime of 90 seconds. So we'd really want to run
that in parallel with everything else. Or split 'timeouts' into multiple
tests that could run in parallel.
Hmm, that test has 8 permutations, five second each ... if we split it
in 3 and run those in parallel, we reduce the total isolation runtime
by 25 seconds even if we do *nothing else at all*. If we tweak the
other things, I think we could make the whole set run in about 30
seconds total in a normal machine.
Splitting the test never crossed my mind.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Heikki Linnakangas <hlinnaka@iki.fi> writes:
The elephant in the room is the 'timeouts' test, which takes about 40
seconds, out of a total runtime of 90 seconds. So we'd really want to
run that in parallel with everything else. Or split 'timeouts' into
multiple tests that could run in parallel. I don't think grouping the
rest of the tests differently will make much difference to how easy or
hard that is.
The problem in "timeouts" is that it has to use drearily long timeouts
to be sure that the behavior will be stable even on really slow machines
(think CLOBBER_CACHE_ALWAYS or valgrind --- it can take seconds for them
to reach a waiting state that other machines reach quickly). If we run
such tests in parallel with anything else, that risks re-introducing the
instability. I'm not very sure what we can do about that. But you might
be right that unless we can solve that, there's not going to be much to be
gained from parallelizing the rest.
I wonder if there's some way to scale the timeout values based on
machine speed? But how would the test get that info?
regards, tom lane
On 2019-Aug-07, Tom Lane wrote:
The problem in "timeouts" is that it has to use drearily long timeouts
to be sure that the behavior will be stable even on really slow machines
(think CLOBBER_CACHE_ALWAYS or valgrind --- it can take seconds for them
to reach a waiting state that other machines reach quickly). If we run
such tests in parallel with anything else, that risks re-introducing the
instability. I'm not very sure what we can do about that. But you might
be right that unless we can solve that, there's not going to be much to be
gained from parallelizing the rest.
It runs 8 different permutations serially. If we run the same
permutations in parallel, it would finish much quicker, and we wouldn't
run it in parallel with anything that would take up CPU time, since
they're all just sleeping.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
On 2019-Aug-07, Tom Lane wrote:
The problem in "timeouts" is that it has to use drearily long timeouts
to be sure that the behavior will be stable even on really slow machines
(think CLOBBER_CACHE_ALWAYS or valgrind --- it can take seconds for them
to reach a waiting state that other machines reach quickly). If we run
such tests in parallel with anything else, that risks re-introducing the
instability. I'm not very sure what we can do about that. But you might
be right that unless we can solve that, there's not going to be much to be
gained from parallelizing the rest.
It runs 8 different permutations serially. If we run the same
permutations in parallel, it would finish much quicker, and we wouldn't
run it in parallel with anything that would take up CPU time, since
they're all just sleeping.
Wrong ... they're *not* just sleeping, in the problem cases. They're
eating cycles due to CLOBBER_CACHE_ALWAYS or valgrind. They're on their
way to sleeping; but they have to get there before the timeout elapses,
or the test shows unexpected results.
Admittedly, as long as you've got more CPUs than tests, it should still
be OK. But if you don't, boom.
regards, tom lane
On Wed, Aug 07, 2019 at 03:17:02PM +0300, Heikki Linnakangas wrote:
On 07/08/2019 14:42, Thomas Munro wrote:
I think I'd put nowait and skip locked under a separate category "FOR
UPDATE" or "row locking" or something, but maybe that's just me... can
you call that stuff DML?Yeah, I guess SELECT FOR UPDATE isn't really DML. Separate "Row locking"
category works for me. Or maybe "Concurrent DML and row locking". There is
also DML in some of those tests.
Or would it make sense to group the nowait and skip-locked portion
with the multixact group, then keep the DML-specific stuff together?
There is a test called update-locked-tuple which could enter into the
"row locking" group, and the skip-locked tests have references to
multixact locks. So I think that I would group all that into a single
group: "multixact and row locking".
--
Michael