Speed up build on Windows by generating symbol definition in batch

Started by Peifeng Qiualmost 7 years ago5 messages

pqiu@pivotal.io

almost 7 years ago

1 attachment(s)

Hi, hackers.

Build process on Windows includes compiling all source into object files,
linking them to binaries, and generating export symbol definitions, etc.
When I watched the whole build process with a task manager, I discovered
that a lot of time was spent on generating export symbol definitions,
without consuming much CPU or IO.
The script that doing this is src/tools/msvc/gendef.pl, it enumerates the
whole directory for ".obj" files and call dumpbin utility to generate
".sym" files one by one like this:

dumpbin /symbols /out:a.sym a.obj >NUL

Actually the dumpbin utility accepts a wildcard file name, so we can
generate the export symbols of all ".obj" files in batch.

dumpbin /symbols /out:all.sym *.obj >NUL

This will avoid wasting time by creating and destroying dumpbin process
repeatedly and can speed up the build process considerably.
I've tested on my 4-core 8-thread Intel i7 CPU. I've set MSBFLAGS=/m to
ensure it can utilize all CPU cores.
Building without this patch takes about 370 seconds. Building with this
patch takes about 200 seconds. That's almost 2x speed up.

Best regards,
Peifeng Qiu

Attachments:

generate-symbols-in-batch-on-windows-v1.patchapplication/octet-stream; name=generate-symbols-in-batch-on-windows-v1.patchDownload

diff --git a/src/tools/msvc/gendef.pl b/src/tools/msvc/gendef.pl
index 77c3a775b0..3212627270 100644
--- a/src/tools/msvc/gendef.pl
+++ b/src/tools/msvc/gendef.pl
@@ -12,17 +12,6 @@ my @def;
 # src/tools/msvc/gendef.pl
 #
 
-sub dumpsyms
-{
-	my ($objfile, $symfile) = @_;
-	my ($symvol, $symdirs, $symbase) = splitpath($symfile);
-	my $tmpfile = catpath($symvol, $symdirs, "symbols.out");
-	system("dumpbin /symbols /out:$tmpfile $_ >NUL")
-	  && die "Could not call dumpbin";
-	rename($tmpfile, $symfile);
-	return;
-}
-
 # Given a symbol file path, loops over its contents
 # and returns a list of symbols of interest as a dictionary
 # of 'symbolname' -> symtype, where symtype is:
@@ -176,16 +165,10 @@ if (-f $deffile
 print "Generating $defname.DEF from directory $ARGV[0], platform $platform\n";
 
 my %def = ();
+my $symfile = "$ARGV[0]/all.sym";
+system("dumpbin /symbols /out:$symfile $ARGV[0]/*obj >NUL");
+extract_syms($symfile, \%def);
 
-while (<$ARGV[0]/*.obj>)    ## no critic (RequireGlobFunction);
-{
-	my $objfile = $_;
-	my $symfile = $objfile;
-	$symfile =~ s/\.obj$/.sym/i;
-	dumpsyms($objfile, $symfile);
-	print ".";
-	extract_syms($symfile, \%def);
-}
 print "\n";
 
 writedef($deffile, $platform, \%def);

Noah Misch

noah@leadboat.com

almost 7 years ago

In reply to: Peifeng Qiu (#1)

Re: Speed up build on Windows by generating symbol definition in batch

On Sat, Mar 30, 2019 at 03:42:39PM +0900, Peifeng Qiu wrote:

When I watched the whole build process with a task manager, I discovered
that a lot of time was spent on generating export symbol definitions,
without consuming much CPU or IO.
The script that doing this is src/tools/msvc/gendef.pl, it enumerates the
whole directory for ".obj" files and call dumpbin utility to generate
".sym" files one by one like this:

dumpbin /symbols /out:a.sym a.obj >NUL

Actually the dumpbin utility accepts a wildcard file name, so we can
generate the export symbols of all ".obj" files in batch.

dumpbin /symbols /out:all.sym *.obj >NUL

This will avoid wasting time by creating and destroying dumpbin process
repeatedly and can speed up the build process considerably.
I've tested on my 4-core 8-thread Intel i7 CPU. I've set MSBFLAGS=/m to
ensure it can utilize all CPU cores.
Building without this patch takes about 370 seconds. Building with this
patch takes about 200 seconds. That's almost 2x speed up.

I, too, get a strong improvement, from 201s to 149s. I can confirm it yields
identical *.def files. Thanks for identifying this improvement.

- my ($objfile, $symfile) = @_;
- my ($symvol, $symdirs, $symbase) = splitpath($symfile);
- my $tmpfile = catpath($symvol, $symdirs, "symbols.out");

You removed the last use of File::Spec::Functions, so remove its "use"
statement.

- system("dumpbin /symbols /out:$tmpfile $_ >NUL")
- && die "Could not call dumpbin";

This error handling was crude, but don't replace it with zero error handling.

- rename($tmpfile, $symfile);

Keep the use of a temporary file, too.

+system("dumpbin /symbols /out:$symfile $ARGV[0]/*obj >NUL");

That should be *.obj, not *obj.

Peifeng Qiu

pqiu@pivotal.io

almost 7 years ago

In reply to: Noah Misch (#2)

1 attachment(s)

Re: Speed up build on Windows by generating symbol definition in batch

Thanks for reviewing!
I've updated the patch according to your comments.

Best regards,
Peifeng Qiu

On Sun, Apr 7, 2019 at 2:31 PM Noah Misch <noah@leadboat.com> wrote:

Show quoted text

On Sat, Mar 30, 2019 at 03:42:39PM +0900, Peifeng Qiu wrote:

When I watched the whole build process with a task manager, I discovered
that a lot of time was spent on generating export symbol definitions,
without consuming much CPU or IO.
The script that doing this is src/tools/msvc/gendef.pl, it enumerates

the

whole directory for ".obj" files and call dumpbin utility to generate
".sym" files one by one like this:

dumpbin /symbols /out:a.sym a.obj >NUL

Actually the dumpbin utility accepts a wildcard file name, so we can
generate the export symbols of all ".obj" files in batch.

dumpbin /symbols /out:all.sym *.obj >NUL

This will avoid wasting time by creating and destroying dumpbin process
repeatedly and can speed up the build process considerably.
I've tested on my 4-core 8-thread Intel i7 CPU. I've set MSBFLAGS=/m to
ensure it can utilize all CPU cores.
Building without this patch takes about 370 seconds. Building with this
patch takes about 200 seconds. That's almost 2x speed up.

I, too, get a strong improvement, from 201s to 149s. I can confirm it
yields
identical *.def files. Thanks for identifying this improvement.

- my ($objfile, $symfile) = @_;
- my ($symvol, $symdirs, $symbase) = splitpath($symfile);
- my $tmpfile = catpath($symvol, $symdirs, "symbols.out");

You removed the last use of File::Spec::Functions, so remove its "use"
statement.

- system("dumpbin /symbols /out:$tmpfile $_ >NUL")
- && die "Could not call dumpbin";

This error handling was crude, but don't replace it with zero error
handling.

- rename($tmpfile, $symfile);

Keep the use of a temporary file, too.

+system("dumpbin /symbols /out:$symfile $ARGV[0]/*obj >NUL");

That should be *.obj, not *obj.

Attachments:

generate-symbols-in-batch-on-windows-v2.patchapplication/octet-stream; name=generate-symbols-in-batch-on-windows-v2.patchDownload

diff --git a/src/tools/msvc/gendef.pl b/src/tools/msvc/gendef.pl
index 77c3a775b0..7ca78f3b38 100644
--- a/src/tools/msvc/gendef.pl
+++ b/src/tools/msvc/gendef.pl
@@ -1,7 +1,6 @@
 use strict;
 use warnings;
 use 5.8.0;
-use File::Spec::Functions qw(splitpath catpath);
 use List::Util qw(max);
 
 my @def;
@@ -12,17 +11,6 @@ my @def;
 # src/tools/msvc/gendef.pl
 #
 
-sub dumpsyms
-{
-	my ($objfile, $symfile) = @_;
-	my ($symvol, $symdirs, $symbase) = splitpath($symfile);
-	my $tmpfile = catpath($symvol, $symdirs, "symbols.out");
-	system("dumpbin /symbols /out:$tmpfile $_ >NUL")
-	  && die "Could not call dumpbin";
-	rename($tmpfile, $symfile);
-	return;
-}
-
 # Given a symbol file path, loops over its contents
 # and returns a list of symbols of interest as a dictionary
 # of 'symbolname' -> symtype, where symtype is:
@@ -176,16 +164,13 @@ if (-f $deffile
 print "Generating $defname.DEF from directory $ARGV[0], platform $platform\n";
 
 my %def = ();
+my $symfile = "$ARGV[0]/all.sym";
+my $tmpfile = "$ARGV[0]/tmp.sym";
+system("dumpbin /symbols /out:$tmpfile $ARGV[0]/*.obj >NUL")
+	&& die "Could not call dumpbin";
+rename($tmpfile, $symfile);
+extract_syms($symfile, \%def);
 
-while (<$ARGV[0]/*.obj>)    ## no critic (RequireGlobFunction);
-{
-	my $objfile = $_;
-	my $symfile = $objfile;
-	$symfile =~ s/\.obj$/.sym/i;
-	dumpsyms($objfile, $symfile);
-	print ".";
-	extract_syms($symfile, \%def);
-}
 print "\n";
 
 writedef($deffile, $platform, \%def);

Noah Misch

noah@leadboat.com

over 6 years ago

In reply to: Peifeng Qiu (#3)

Re: Speed up build on Windows by generating symbol definition in batch

On Wed, Apr 10, 2019 at 02:27:26PM +0800, Peifeng Qiu wrote:

I've updated the patch according to your comments.

Looks good. Thanks. I plan to push this on Saturday.

David Rowley

david.rowley@2ndquadrant.com

over 6 years ago

In reply to: Noah Misch (#4)

Re: Speed up build on Windows by generating symbol definition in batch

On Mon, 29 Apr 2019 at 14:50, Noah Misch <noah@leadboat.com> wrote:

On Wed, Apr 10, 2019 at 02:27:26PM +0800, Peifeng Qiu wrote:

I've updated the patch according to your comments.

Looks good. Thanks. I plan to push this on Saturday.

I didn't really look at the patch in detail, but on testing it on a
windows machine with vs2017, it took built time from 5:00 to 4:10.

--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services