windows build slow due to windows.h includes
Hi,
For the AIO stuff I needed to build postgres for windows. And I was a bit
horrified by the long compile times. At first I was ready to blame the MS
compiler for being slow, until I noticed that using mingw gcc from linux to
cross compile to windows is also a *lot* slower than building for linux.
I found some blog-post-documented-only compiler flags [1]https://aras-p.info/blog/2019/01/21/Another-cool-MSVC-flag-d1reportTime/, most importantly
/d1reportTime. Which shows that the include processing of postgres.h takes
0.6s [2]postgres.c Include Headers: Count: 483 c:\Users\anfreund\src\postgres\src\include\postgres.h: 0.561795s c:\Users\anfreund\src\postgres\src\include\c.h: 0.556991s c:\Users\anfreund\src\postgres\src\include\postgres_ext.h: 0.000488s c:\Users\anfreund\src\postgres\src\include\pg_config_ext.h: 0.000151s c:\Users\anfreund\src\postgres\src\include\pg_config.h: 0.000551s c:\Users\anfreund\src\postgres\src\include\pg_config_manual.h: 0.000286s c:\Users\anfreund\src\postgres\src\include\pg_config_os.h: 0.014283s C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\crtdefs.h: 0.009727s ... c:\Users\anfreund\src\postgres\src\include\port\win32_port.h: 0.487469s C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\um\winsock2.h: 0.449373s ... C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\um\windows.h: 0.439666s
Basically all the time in a debug windows build is spent parsing windows.h and
related headers. Argh.
The amount of stuff we include in win32_port.h and declare is pretty absurd
imo. There's really no need to expose the whole backend to all of it. Most of
it should just be needed in a few port/ files and a few select users.
But that's too much work for my taste. As it turns out there's a partial
solution to windows.h being just so damn big, the delightfully named
WIN32_LEAN_AND_MEAN.
This reduces the non-incremental buildtime in my 8 core windows VM from 187s to
140s. Cross compiling from linux it's
master:
real 0m53.807s
user 22m16.930s
sys 2m50.264s
WIN32_LEAN_AND_MEAN
real 0m32.956s
user 12m17.773s
sys 1m52.313s
Still far from !windows compile times, but still not a bad improvement.
Most of the compile time after this is still spent doing parsing /
preprocessing. I sidetracked myself into looking at precompiled headers, but
it's not trivial to do that right unfortunately.
I think it'd be good if win32_port.h were slimmed down, and more of its
contents were moved into fake "port/win32/$name-of-unix-header" style headers
or such.
Greetings,
Andres Freund
[1]: https://aras-p.info/blog/2019/01/21/Another-cool-MSVC-flag-d1reportTime/
[2]: postgres.c Include Headers: Count: 483 c:\Users\anfreund\src\postgres\src\include\postgres.h: 0.561795s c:\Users\anfreund\src\postgres\src\include\c.h: 0.556991s c:\Users\anfreund\src\postgres\src\include\postgres_ext.h: 0.000488s c:\Users\anfreund\src\postgres\src\include\pg_config_ext.h: 0.000151s c:\Users\anfreund\src\postgres\src\include\pg_config.h: 0.000551s c:\Users\anfreund\src\postgres\src\include\pg_config_manual.h: 0.000286s c:\Users\anfreund\src\postgres\src\include\pg_config_os.h: 0.014283s C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\crtdefs.h: 0.009727s ... c:\Users\anfreund\src\postgres\src\include\port\win32_port.h: 0.487469s C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\um\winsock2.h: 0.449373s ... C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\um\windows.h: 0.439666s
postgres.c
Include Headers:
Count: 483
c:\Users\anfreund\src\postgres\src\include\postgres.h: 0.561795s
c:\Users\anfreund\src\postgres\src\include\c.h: 0.556991s
c:\Users\anfreund\src\postgres\src\include\postgres_ext.h: 0.000488s
c:\Users\anfreund\src\postgres\src\include\pg_config_ext.h: 0.000151s
c:\Users\anfreund\src\postgres\src\include\pg_config.h: 0.000551s
c:\Users\anfreund\src\postgres\src\include\pg_config_manual.h: 0.000286s
c:\Users\anfreund\src\postgres\src\include\pg_config_os.h: 0.014283s
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\crtdefs.h: 0.009727s
...
c:\Users\anfreund\src\postgres\src\include\port\win32_port.h: 0.487469s
C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\um\winsock2.h: 0.449373s
...
C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\um\windows.h: 0.439666s
Attachments:
win32_lean_and_mean.difftext/x-diff; charset=us-asciiDownload
diff --git i/src/include/port/win32_port.h w/src/include/port/win32_port.h
index 05c5a534420..b5a44519d98 100644
--- i/src/include/port/win32_port.h
+++ w/src/include/port/win32_port.h
@@ -43,6 +43,8 @@
#define _WINSOCKAPI_
#endif
+#define WIN32_LEAN_AND_MEAN
+
#include <winsock2.h>
#include <ws2tcpip.h>
#include <windows.h>
On 9/21/21 3:30 PM, Andres Freund wrote:
Hi,
For the AIO stuff I needed to build postgres for windows. And I was a bit
horrified by the long compile times. At first I was ready to blame the MS
compiler for being slow, until I noticed that using mingw gcc from linux to
cross compile to windows is also a *lot* slower than building for linux.I found some blog-post-documented-only compiler flags [1], most importantly
/d1reportTime. Which shows that the include processing of postgres.h takes
0.6s [2]Basically all the time in a debug windows build is spent parsing windows.h and
related headers. Argh.The amount of stuff we include in win32_port.h and declare is pretty absurd
imo. There's really no need to expose the whole backend to all of it. Most of
it should just be needed in a few port/ files and a few select users.But that's too much work for my taste. As it turns out there's a partial
solution to windows.h being just so damn big, the delightfully named
WIN32_LEAN_AND_MEAN.This reduces the non-incremental buildtime in my 8 core windows VM from 187s to
140s. Cross compiling from linux it's
master:
real 0m53.807s
user 22m16.930s
sys 2m50.264s
WIN32_LEAN_AND_MEAN
real 0m32.956s
user 12m17.773s
sys 1m52.313s
Nice!
I also see references to VC_EXTRALEAN which defines this and some other
stuff that might make things even faster.
Worth investigating.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
Hi,
On 2021-09-21 16:13:55 -0400, Andrew Dunstan wrote:
I also see references to VC_EXTRALEAN which defines this and some other
stuff that might make things even faster.
I don't think that's relevant to "us", just mfc apps (which we gladly
aren't). From what I can see we'd have to actually clean up our includes to
not have windows.h everywhere or use precompiled headers to benefit further.
Greetings,
Andres Freund
Em ter., 21 de set. de 2021 às 16:30, Andres Freund <andres@anarazel.de>
escreveu:
Hi,
For the AIO stuff I needed to build postgres for windows. And I was a bit
horrified by the long compile times. At first I was ready to blame the MS
compiler for being slow, until I noticed that using mingw gcc from linux to
cross compile to windows is also a *lot* slower than building for linux.I found some blog-post-documented-only compiler flags [1], most importantly
/d1reportTime. Which shows that the include processing of postgres.h takes
0.6s [2]Basically all the time in a debug windows build is spent parsing windows.h
and
related headers. Argh.The amount of stuff we include in win32_port.h and declare is pretty absurd
imo. There's really no need to expose the whole backend to all of it. Most
of
it should just be needed in a few port/ files and a few select users.But that's too much work for my taste. As it turns out there's a partial
solution to windows.h being just so damn big, the delightfully named
WIN32_LEAN_AND_MEAN.
+1
But I did a quick dirty test here, and removed windows.h in win32_port.h,
and compiled normally with msvc 2019 (64 bit), would it work with mingw
cross compile?
regards,
Ranier Vilela
Hi,
On 2021-09-21 20:26:36 -0300, Ranier Vilela wrote:
Em ter., 21 de set. de 2021 �s 16:30, Andres Freund <andres@anarazel.de>
escreveu:But that's too much work for my taste. As it turns out there's a partial
solution to windows.h being just so damn big, the delightfully named
WIN32_LEAN_AND_MEAN.+1
But I did a quick dirty test here, and removed windows.h in win32_port.h,
and compiled normally with msvc 2019 (64 bit), would it work with mingw
cross compile?
That's likely only because winsock indirectly includes windows.h - because of
that it won't actually reduce compile time. And you can't remove the other
headers that indirectly include windows.h without causing compilation errors.
Greetings,
Andres Freund
On Tue, Sep 21, 2021 at 12:30:35PM -0700, Andres Freund wrote:
solution to windows.h being just so damn big, the delightfully named
WIN32_LEAN_AND_MEAN.This reduces the non-incremental buildtime in my 8 core windows VM from 187s to
140s. Cross compiling from linux it's
master:
real 0m53.807s
user 22m16.930s
sys 2m50.264s
WIN32_LEAN_AND_MEAN
real 0m32.956s
user 12m17.773s
sys 1m52.313s
+1, great win for a one-liner.
On Wed, Sep 22, 2021 at 1:56 AM Andres Freund <andres@anarazel.de> wrote:
On 2021-09-21 20:26:36 -0300, Ranier Vilela wrote:
Em ter., 21 de set. de 2021 às 16:30, Andres Freund <andres@anarazel.de>
escreveu:But that's too much work for my taste. As it turns out there's a
partial
solution to windows.h being just so damn big, the delightfully named
WIN32_LEAN_AND_MEAN.+1
But I did a quick dirty test here, and removed windows.h in win32_port.h,
and compiled normally with msvc 2019 (64 bit), would it work with mingw
cross compile?That's likely only because winsock indirectly includes windows.h - because
of
that it won't actually reduce compile time. And you can't remove the other
headers that indirectly include windows.h without causing compilation
errors.You are right about winsock2.h including some parts of windows.h, please
see note in [1]https://docs.microsoft.com/en-us/windows/win32/winsock/creating-a-basic-winsock-application. You could move the windows.h inclusion for clarity:
+ #ifndef WIN32_LEAN_AND_MEAN
+ #define WIN32_LEAN_AND_MEAN
+ #endif
+
+ #include <windows.h>
#include <winsock2.h>
#include <ws2tcpip.h>
- #include <windows.h>[1]: https://docs.microsoft.com/en-us/windows/win32/winsock/creating-a-basic-winsock-application
https://docs.microsoft.com/en-us/windows/win32/winsock/creating-a-basic-winsock-application
Regards,
Juan José Santamaría Flecha
On Wed, Sep 22, 2021 at 11:14 AM Noah Misch <noah@leadboat.com> wrote:
On Tue, Sep 21, 2021 at 12:30:35PM -0700, Andres Freund wrote:
solution to windows.h being just so damn big, the delightfully named
WIN32_LEAN_AND_MEAN.This reduces the non-incremental buildtime in my 8 core windows VM from 187s to
140s. Cross compiling from linux it's
master:
real 0m53.807s
user 22m16.930s
sys 2m50.264s
WIN32_LEAN_AND_MEAN
real 0m32.956s
user 12m17.773s
sys 1m52.313s+1, great win for a one-liner.
+1. It reduced the build time of Postgres from "Time Elapsed
00:01:57.60" to "Time Elapsed 00:01:38.11" in my Windows env. (Win 10,
MSVC 2019).
--
With Regards,
Amit Kapila.
On Wed, Sep 22, 2021 at 02:14:59PM +0530, Amit Kapila wrote:
On Wed, Sep 22, 2021 at 11:14 AM Noah Misch <noah@leadboat.com> wrote:
+1, great win for a one-liner.
+1. It reduced the build time of Postgres from "Time Elapsed
00:01:57.60" to "Time Elapsed 00:01:38.11" in my Windows env. (Win 10,
MSVC 2019).
That's nice. Great find!
--
Michael