WIP patch for avoiding duplicate initdb runs during "make check"
Yesterday I spent a bit of time on an idea that we've talked about
before, which is to not run initdb over and over again in contexts like
"make check-world", or even just during "make check" in contrib or in the
recovery or subscription tests. The idea would be to do it once and
then copy the created data directory tree into place for each test
run. Thus we might trade this:
$ time initdb -D $PGDATA -N
...
real 0m1.235s
user 0m0.896s
sys 0m0.341s
for this:
$ time cp -a $PGDATA junk
real 0m0.146s
user 0m0.008s
sys 0m0.136s
The attached patch is far from being committable, but it's complete enough
to get an idea of what sort of overall performance gain we might expect.
On my workstation, I've lately been doing check-world like this:
time make -s check-world -j4 PROVE_FLAGS='-j4'
and what I see is that HEAD takes about this long:
real 2m1.514s
user 2m8.113s
sys 1m15.993s
and with this patch I get
real 1m42.549s
user 0m42.888s
sys 0m46.982s
(These are median-of-3-runs; the real time bounces around a fair bit,
the CPU times not very much.)
On my laptop, a simple non-parallelized "time make check-world" takes
real 9m12.813s
user 2m15.113s
sys 1m25.631s
and with patch
real 8m3.785s
user 1m19.024s
sys 1m13.707s
So those are respectable gains, but not earth-shattering. I'm not
sure if it's worth putting more work into it right now. The main
thing that's needed to make it committable is to get rid of the
system("cp -a ...") call in favor of something more portable.
I looked longingly at our existing copydir() function but it seems
pretty tied to the backend environment. It wouldn't be hard to
make a version for frontend, but I'm not going to expend that work
right now unless people are excited enough to want to commit this
now rather than later (or not at all).
(Other unfinished work: teaching the MSVC scripts to use this,
and teaching pg_upgrade's test script to use it.)
regards, tom lane
Attachments:
avoid-duplicate-initdb-calls-0.1.patchtext/x-diff; charset=us-ascii; name=avoid-duplicate-initdb-calls-0.1.patchDownload+73-70
On 2 July 2017 at 18:33, Tom Lane <tgl@sss.pgh.pa.us> wrote:
system("cp -a ...") call in favor of something more portable.
If we're ok with using Perl there's File::Copy::Recursive::dircopy()
which does exactly that.
--
greg
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Jul 3, 2017 at 9:25 PM, Greg Stark <stark@mit.edu> wrote:
On 2 July 2017 at 18:33, Tom Lane <tgl@sss.pgh.pa.us> wrote:
system("cp -a ...") call in favor of something more portable.
If we're ok with using Perl there's File::Copy::Recursive::dircopy()
which does exactly that.
This stuff needs to support perl down to 5.8.0, and that's a reason
behind having src/test/perl/RecursiveCopy.pm. So I would suggest just
to use that. cp is not portable on Windows as well, that's a recipe
for non-portable code there.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Tom Lane wrote:
(Other unfinished work: teaching the MSVC scripts to use this,
and teaching pg_upgrade's test script to use it.)
Maybe it'd be simpler to rewrite pg_upgrade tests using PostgresNode
instead?
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Jul 3, 2017 at 10:02 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
Tom Lane wrote:
(Other unfinished work: teaching the MSVC scripts to use this,
and teaching pg_upgrade's test script to use it.)Maybe it'd be simpler to rewrite pg_upgrade tests using PostgresNode
instead?
You are looking for this patch:
https://commitfest.postgresql.org/14/1101/
And for this thread:
/messages/by-id/CAB7nPqRdaN1A1YNjxNL9T1jUEWct8ttqq29dNv8W_o37+e8wfA@mail.gmail.com
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Michael Paquier <michael.paquier@gmail.com> writes:
On Mon, Jul 3, 2017 at 9:25 PM, Greg Stark <stark@mit.edu> wrote:
On 2 July 2017 at 18:33, Tom Lane <tgl@sss.pgh.pa.us> wrote:
system("cp -a ...") call in favor of something more portable.
If we're ok with using Perl there's File::Copy::Recursive::dircopy()
which does exactly that.
This stuff needs to support perl down to 5.8.0, and that's a reason
behind having src/test/perl/RecursiveCopy.pm. So I would suggest just
to use that. cp is not portable on Windows as well, that's a recipe
for non-portable code there.
I can't see going this path in pg_regress, because then you would have
exactly zero test functionality in a non-Perl build. What I had in
mind was a frontend-friendly version of backend/storage/file/copydir.c,
either just dropped into pg_regress.c or put in src/common/.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Jul 3, 2017 at 11:00 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Michael Paquier <michael.paquier@gmail.com> writes:
On Mon, Jul 3, 2017 at 9:25 PM, Greg Stark <stark@mit.edu> wrote:
On 2 July 2017 at 18:33, Tom Lane <tgl@sss.pgh.pa.us> wrote:
system("cp -a ...") call in favor of something more portable.
If we're ok with using Perl there's File::Copy::Recursive::dircopy()
which does exactly that.This stuff needs to support perl down to 5.8.0, and that's a reason
behind having src/test/perl/RecursiveCopy.pm. So I would suggest just
to use that. cp is not portable on Windows as well, that's a recipe
for non-portable code there.
This was under the assumption of "if we use perl" :)
I can't see going this path in pg_regress, because then you would have
exactly zero test functionality in a non-Perl build.
Indeed, release tarballs don't need perl to work. So that's a no-go.
What I had in
mind was a frontend-friendly version of backend/storage/file/copydir.c,
either just dropped into pg_regress.c or put in src/common/.
+1 for src/common/.
--
Michael--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers