make install getting slower

On 12/4/18 12:31 PM, Andres Freund wrote:

Hi,

best of three of
time make -s install INSTALL=/bin/true

9.3: 1.53s
9.4: 1.55s
9.5: 1.90s
9.6: 2.10s
10: 2.45s
11: 2.81s
master: 2.99s

Obviously a part of that can be explained by the tree getting bigger,
but that can't be all of it.

It also gets a bit worse if LLVM is enabled (due to the bitcode files
getting installed), but that's "just" 0.3s for me. We should optimize
that for fewer install invocations / using cp, but that seems seperate.

I think this also has gotten more pronounced with newer OS versions, as
the overhead of the spectre/meltdown mitigations have increased the cost
of syscalls and forks.

Obviously this doesn't terribly matter for actual installations, but I
do find it fairly annoying when running regression tests...

Wouldn't using -j option with make help this?

JD

Greetings,

Andres Freund

--
Command Prompt, Inc. || http://the.postgres.company/ || @cmdpromptinc
*** A fault and talent of mine is to tell it exactly how it is. ***
PostgreSQL centered full stack support, consulting and development.
Advocate: @amplifypostgres || Learn: https://postgresconf.org
***** Unless otherwise stated, opinions are my own. *****

Hi,

On 2018-12-04 12:31:48 -0800, Andres Freund wrote:

best of three of
time make -s install INSTALL=/bin/true

9.3: 1.53s
9.4: 1.55s
9.5: 1.90s
9.6: 2.10s
10: 2.45s
11: 2.81s
master: 2.99s

Obviously a part of that can be explained by the tree getting bigger,
but that can't be all of it.

It also gets a bit worse if LLVM is enabled (due to the bitcode files
getting installed), but that's "just" 0.3s for me. We should optimize
that for fewer install invocations / using cp, but that seems seperate.

I think this also has gotten more pronounced with newer OS versions, as
the overhead of the spectre/meltdown mitigations have increased the cost
of syscalls and forks.

Obviously this doesn't terribly matter for actual installations, but I
do find it fairly annoying when running regression tests...

As a comparison, I'd a tree that had the cmake patchset applied around
(~1.5 yo tree). Using the ninja generator gets a clean build to
0m0.073s, a first install to 0m0.201s and a repeat install to 0m0.170s.

Greetings,

Andres Freund

On 04/12/2018 21:31, Andres Freund wrote:

Obviously this doesn't terribly matter for actual installations, but I
do find it fairly annoying when running regression tests...

One possible workaround:
/messages/by-id/5d14d20c-b1f6-69c8-7465-df8adb424aea@2ndquadrant.com

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 04/12/2018 21:56, Andres Freund wrote:

As a comparison, I'd a tree that had the cmake patchset applied around
(~1.5 yo tree). Using the ninja generator gets a clean build to
0m0.073s, a first install to 0m0.201s and a repeat install to 0m0.170s.

Yeah, I've been playing with meson+ninja, and those run times are amazing.

I'm not confident whether a high-level build system generator like cmake
or meson would ever give us the flexibility we need. I'm just thinking
of the most recent episode where we had to override the timestamp
handling of ranlib of macOS. There are many others like it.

But I think that we realistically could generate ninja files using some
Perl code not unlike the one that does the catalog .dat to .bki
transformation. Just a thought at this point.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On December 7, 2018 8:04:10 AM PST, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote:

On 04/12/2018 21:56, Andres Freund wrote:

As a comparison, I'd a tree that had the cmake patchset applied

around

(~1.5 yo tree). Using the ninja generator gets a clean build to
0m0.073s, a first install to 0m0.201s and a repeat install to

0m0.170s.

Yeah, I've been playing with meson+ninja, and those run times are
amazing.

It's not just the the faster times due to better tools, a non recursive implementation also allows for much cleaner dependencies then we have right now. Which also increases speed...

I'm not confident whether a high-level build system generator like
cmake
or meson would ever give us the flexibility we need. I'm just thinking
of the most recent episode where we had to override the timestamp
handling of ranlib of macOS. There are many others like it.

I don't know much about meson, but I don't see why cmake can't handle that kind of thing.

But I think that we realistically could generate ninja files using some
Perl code not unlike the one that does the catalog .dat to .bki
transformation. Just a thought at this point.

That seems not great, because we would lose the ability to generate mscv files from the same source. Or whatever the eventual replacement of ninja will be.

Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

On 07/12/2018 18:34, Andres Freund wrote:

But I think that we realistically could generate ninja files using some
Perl code not unlike the one that does the catalog .dat to .bki
transformation. Just a thought at this point.

That seems not great, because we would lose the ability to generate mscv files from the same source. Or whatever the eventual replacement of ninja will be.

We already generate msvc files using custom Perl code. Right now we
generated them by parsing makefiles in a hackish way. If we were to
change that to parsing a well-structured ".dat" file, that seems
entirely doable and wouldn't be worse than what we do now, I think.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services