Need a builtin way to run all tests faster manner
Hi,
Right now check-world, a sensible thing to run before commits that
aren't very narrow, takes a long time. To the point it seems to impact
development velocity enough that it's more sensible to just skip it, and
rely on the buildfarm.
That's not a great solution.
A large amount of the time is actually spent doing completely redundant
initdb, cluster start/stop work, and running tests serially that could
be run in parallel.
A single check-world on my machine takes over 20min. That's just not
realistic to run without hurting development pace.
We can avoid a lot of redundant work (skip redundant initdb & cluster
start/stop), and we can quite easily parallelize others.
The problem is that doing so isn't something entirely trivially
scriptable, e.g. make installcheck-world doesn't run all tests, and it
doesn't do so in parallel. Scripting it locally also has the issue that
it's very easy to not notice new stuff being added by others.
As an example of the speedups, here's the comparison for contrib:
make -C contrib: 2m21.056s
make -C contrib installcheck 0m30.672s
make -C contrib -j16 -s -Otarget installcheck USE_MODULE_DB=1 0m10.418s
that's not an entirely fair comparison however, because test_decoding
doesn't to installcheck, but the general principle holds.
This is obviously a large difference.
A lot of the slow tap tests could be run serially, recoverying a lot
more time.
There's also some tests simply taking way too long, e.g. the pg_dump
tests just do largely redundant tests for 30s.
I'm not quite sure what the best way to attack this is, but I think we
need to do something.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres,
* Andres Freund (andres@anarazel.de) wrote:
There's also some tests simply taking way too long, e.g. the pg_dump
tests just do largely redundant tests for 30s.
About half of the time in the pg_dump case, at least, appears to be in
010_dump_connstr.pl. I've not looked into it, but based on the test
names it does look like some of those tests might be redundant to what
is already being covered in 002_pg_dump.pl. Of course, the way the TAP
tests run, they require an initdb and startup of the postmaster, and
that's a somewhat fixed amount of time that any TAP test is going to
take.
I'm all for improving things certainly, though, well, while the pg_dump
tests do a lot, they also try to cover a good bit of the code in
pg_dump.c which is quite large. I wouldn't want to reduce our code
coverage just to make the regression tests go faster. The changes to
the pg_dump tests that I'm hoping to push soon will at least reduce the
output some, if not the length of time taken, while providing more code
coverage.
All that said, 30s out of 20m (which seems to more-or-less match what I
get locally too) makes me really wonder if that's the place that we need
to focus. Is it really the longest running test we have and it's just
that we have tons of them that are doing initdb over and over again?
I'm not quite sure what the best way to attack this is, but I think we
need to do something.
I tend to agree with this, though I haven't got any great answers,
unfortunately.
Thanks!
Stephen
Stephen Frost <sfrost@snowman.net> writes:
* Andres Freund (andres@anarazel.de) wrote:
I'm not quite sure what the best way to attack this is, but I think we
need to do something.
I tend to agree with this, though I haven't got any great answers,
unfortunately.
I don't want to reduce test coverage either. I think the most painless
way to improve matters would just be to work harder on running tests in
parallel. I think most devs these days do most of their work on 4- or
8-core machines, yet almost everything except the core regression tests
is depressingly serial. I think we could likely get a 2x or better
reduction in total runtime without too much work just by attacking that.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2017-03-06 19:45:27 -0500, Tom Lane wrote:
Stephen Frost <sfrost@snowman.net> writes:
* Andres Freund (andres@anarazel.de) wrote:
I'm not quite sure what the best way to attack this is, but I think we
need to do something.I tend to agree with this, though I haven't got any great answers,
unfortunately.I don't want to reduce test coverage either. I think the most painless
way to improve matters would just be to work harder on running tests in
parallel. I think most devs these days do most of their work on 4- or
8-core machines, yet almost everything except the core regression tests
is depressingly serial. I think we could likely get a 2x or better
reduction in total runtime without too much work just by attacking that.
A lot more probably, based on my preliminary tests / my local test
script.
I'm just not quite sure what the best way is to make it easier to run
tests in parallel within the tree.
The best I can come up so far is a toplevel target that creates the temp
install, starts a cluster and then runs the 'installcheck-or-check'
target on all the subdirectories via recursion. Individual makefiles can
either use the pre-existing cluster (most of of contrib for example), or
use the temporary install and run their pre-existing check target using
that (the tap tests, test_decoding, ...).
Requires editing a bunch of Makefiles to take advantage. But I don't
really see anything that doesn't require that.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
FWIW, +1 on improving matters here.
Andres Freund wrote:
The best I can come up so far is a toplevel target that creates the temp
install, starts a cluster and then runs the 'installcheck-or-check'
target on all the subdirectories via recursion. Individual makefiles can
either use the pre-existing cluster (most of of contrib for example), or
use the temporary install and run their pre-existing check target using
that (the tap tests, test_decoding, ...).
I think a toplevel installcheck-or-check target is a good first step
(though definitely lets find a better name). Just being able to run all
tests without the need for 95% of pointless initdb's would be helpful
enough.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
FWIW, +1 on improving matters here.
+1 also.
I don't see what's wrong with relying on buildfarm though; testing is
exactly what its there for.
If we had a two-stage process, where committers can issue "trial
commits" as a way of seeing if the build farm likes things. If they
do, we can push to the main repo.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Simon Riggs <simon@2ndquadrant.com> writes:
On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
FWIW, +1 on improving matters here.
+1 also.
I don't see what's wrong with relying on buildfarm though; testing is
exactly what its there for.If we had a two-stage process, where committers can issue "trial
commits" as a way of seeing if the build farm likes things. If they
do, we can push to the main repo.
In perl we do this by having the smoke testers (build farm) pick up
branches with a specific prefix (smoke-me/ in our case, typically
smoke-me/<username>/<topic>), in addition to the blead (master) and
maint-x.y (release) branches.
--
"I use RMS as a guide in the same way that a boat captain would use
a lighthouse. It's good to know where it is, but you generally
don't want to find yourself in the same spot." - Tollef Fog Heen
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 03/07/2017 07:58 AM, Simon Riggs wrote:
On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
FWIW, +1 on improving matters here.
+1 also.
I don't see what's wrong with relying on buildfarm though; testing is
exactly what its there for.If we had a two-stage process, where committers can issue "trial
commits" as a way of seeing if the build farm likes things. If they
do, we can push to the main repo.
I'm happy to work on this. Not quite sure how it would work, but I'm
open to any suggestions.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Tue, Mar 7, 2017 at 7:24 AM, Andrew Dunstan <
andrew.dunstan@2ndquadrant.com> wrote:
On 03/07/2017 07:58 AM, Simon Riggs wrote:
On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:
FWIW, +1 on improving matters here.
+1 also.
I don't see what's wrong with relying on buildfarm though; testing is
exactly what its there for.If we had a two-stage process, where committers can issue "trial
commits" as a way of seeing if the build farm likes things. If they
do, we can push to the main repo.I'm happy to work on this. Not quite sure how it would work, but I'm
open to any suggestions.
Assuming the intention is a service for *committers only*, I suggest
setting up a separate (closed) git repository that committers can push to
and a separate set of BF animals could work from. We could just have it do
all branches in it, no need to filter that way as long as we keep it a
separate repo.
There have also on and off been discussions about building arbitrary
patches as they are sent to the mailinglists. Doing that without any
committer (or other trusted party) as a filter is a completely different
challenge of course, given that it basically amounts to downloading and
running random code off the internet.
But doing just the first would make it a lot easier, and probably still be
of good value.
An in-between could be to hook something off the CF app, but one important
question is how important covering many platforms is. Since we already have
good functionality for doing that in the buildfarm, it makes sense to
utilize that if we can.
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
On 2017-03-07 20:58:35 +0800, Simon Riggs wrote:
On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
FWIW, +1 on improving matters here.
+1 also.
I don't see what's wrong with relying on buildfarm though; testing is
exactly what its there for.If we had a two-stage process, where committers can issue "trial
commits" as a way of seeing if the build farm likes things. If they
do, we can push to the main repo.
Personally that's not addressing my main concern, which is that the
latency of getting done with some patch/topic takes a long while. If I
have to wait for the buildfarm to check some preliminary patch, I still
have to afterwards work on pushing it to master. And very likely my
local check would finish a lot faster than a bunch of buildfarm animals
- I have after all a plenty powerfull machine, lots of cores, fast ssd,
lots of memory, ...
So I really want faster end-to-end test, not less cpu time spent on my
own machine.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Tue, Mar 7, 2017 at 11:23 PM, Andres Freund <andres@anarazel.de> wrote:
Personally that's not addressing my main concern, which is that the
latency of getting done with some patch/topic takes a long while. If I
have to wait for the buildfarm to check some preliminary patch, I still
have to afterwards work on pushing it to master. And very likely my
local check would finish a lot faster than a bunch of buildfarm animals
- I have after all a plenty powerfull machine, lots of cores, fast ssd,
lots of memory, ...So I really want faster end-to-end test, not less cpu time spent on my
own machine.
Yeah. I think the buildfarm-for-test-commits or maybe
buildfarm-for-approved-branches-belonging-to-people-we-basically-trust
idea isn't a bad one, but it's not a substitute for $SUBJECT.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 3/6/17 19:53, Andres Freund wrote:
I'm just not quite sure what the best way is to make it easier to run
tests in parallel within the tree.
make check-world -j2 seems to run fine for me.
With higher -j I appear to be running out of memory or disks space, so I
haven't checked that any further, but it seems possible.
You can also run prove with a -j option.
And we could parallelize some of the contrib/pl tests, e.g., plpython.
(The problem is that parallel make and parallel tests together might
explode a bit, so we might want some way to control which aspect we
parallelize.)
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2017-03-08 16:32:49 -0500, Peter Eisentraut wrote:
On 3/6/17 19:53, Andres Freund wrote:
I'm just not quite sure what the best way is to make it easier to run
tests in parallel within the tree.make check-world -j2 seems to run fine for me.
Hm, I at least used to get a lot of spurious failures with this. I
e.g. don't think the free port selection is race free. Also that
doesn't solve the issue of the time spent in repetitive initdbs - but
that's mainly in contrib, and we probably solve that there by having a
special target in contrib/ building a cluster once.
You can also run prove with a -j option.
Ah, interesting.
(The problem is that parallel make and parallel tests together might
explode a bit, so we might want some way to control which aspect we
parallelize.)
Yea, I indeed see that. Probably could do a top-level prerequisite for
check on a check-world subset that includes docs?
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, Mar 9, 2017 at 6:49 AM, Andres Freund <andres@anarazel.de> wrote:
On 2017-03-08 16:32:49 -0500, Peter Eisentraut wrote:
On 3/6/17 19:53, Andres Freund wrote:
I'm just not quite sure what the best way is to make it easier to run
tests in parallel within the tree.make check-world -j2 seems to run fine for me.
Hm, I at least used to get a lot of spurious failures with this. I
e.g. don't think the free port selection is race free.
pg_regress is running each Postgres instance in a separate Unix socket
directory, so the port selection is not problem. PostgresNode.pm is
also careful about that.
Also that
doesn't solve the issue of the time spent in repetitive initdbs - but
that's mainly in contrib, and we probably solve that there by having a
special target in contrib/ building a cluster once.
Yeah, you can count the growing src/test/modules in that as well.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 3/8/17 16:49, Andres Freund wrote:
make check-world -j2 seems to run fine for me.
Hm, I at least used to get a lot of spurious failures with this. I
e.g. don't think the free port selection is race free.
I was also not sure about that, but as Michael has pointed out, that
doesn't matter anymore, because it now uses a private socket directory.
I have been pounding it a bit, and every so often the test_decoding
tests fail in mysterious ways, but otherwise it seems to work fine. I'm
curious what you are seeing.
Combining make -j10 and prove -j4, I get the run time down to 2 minutes
and a bit, from 20+ minutes. Using the -O option if you have GNU make
=4 is also useful to get some more sane output.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 3/7/17 9:52 PM, Magnus Hagander wrote:
There have also on and off been discussions about building arbitrary
patches as they are sent to the mailinglists. Doing that without any
committer (or other trusted party) as a filter is a completely different
challenge of course, given that it basically amounts to downloading and
running random code off the internet.
Perhaps https://travis-ci.org/ or something similar could be used for
this. That avoids any issues about random code.
--
Jim Nasby, Chief Data Architect, OpenSCG
http://OpenSCG.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Mar 10, 2017 at 3:27 AM, Jim Nasby <jim.nasby@openscg.com> wrote:
On 3/7/17 9:52 PM, Magnus Hagander wrote:
There have also on and off been discussions about building arbitrary
patches as they are sent to the mailinglists. Doing that without any
committer (or other trusted party) as a filter is a completely different
challenge of course, given that it basically amounts to downloading and
running random code off the internet.Perhaps https://travis-ci.org/ or something similar could be used for
this. That avoids any issues about random code.
AFAIK travis-ci would require us to use github as our hoster for all those
things, and embrace that workflow, they don't support anything else.
There might be others that do, just not travis.
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
Magnus Hagander <magnus@hagander.net> writes:
AFAIK travis-ci would require us to use github as our hoster for all those
things, and embrace that workflow, they don't support anything else.There might be others that do, just not travis.
It merely requires the repository to exist on GitHub, and postgresql.git
is already mirrored to https://github.com/postgres/postgres. If there
was a .travis.yml in the repo, people who fork it could easily enable
Travis-CI for their fork, even the official repo isn't hooked up.
- ilmari
--
- Twitter seems more influential [than blogs] in the 'gets reported in
the mainstream press' sense at least. - Matt McLeod
- That'd be because the content of a tweet is easier to condense down
to a mainstream media article. - Calle Dybedahl
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Mar 10, 2017 at 11:29 AM, Dagfinn Ilmari Mannsåker <
ilmari@ilmari.org> wrote:
Magnus Hagander <magnus@hagander.net> writes:
AFAIK travis-ci would require us to use github as our hoster for all
those
things, and embrace that workflow, they don't support anything else.
There might be others that do, just not travis.
It merely requires the repository to exist on GitHub, and postgresql.git
is already mirrored to https://github.com/postgres/postgres. If there
was a .travis.yml in the repo, people who fork it could easily enable
Travis-CI for their fork, even the official repo isn't hooked up.
That's true. It would require a bunch of additional branches to make it
useful in core though, but it śeems like it could be a useful thing for
people to enable in their own personal forks/branches if they use github
for their largest feature development. Could be a good idea to for example
ave an example yml file somewhere on the wiki that people could put into
their branches.
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
On 3/10/17 13:02, Magnus Hagander wrote:
for their largest feature development. Could be a good idea to for
example ave an example yml file somewhere on the wiki that people could
put into their branches.
https://github.com/petere/postgresql/blob/travis/.travis.yml
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers