Building PostgreSQL extensions on Windows

Started by David Rowleyover 5 years ago3 messages
#1David Rowley
dgrowleyml@gmail.com
1 attachment(s)

Hi,

I've heard from a few people that building PostgreSQL extensions on
Windows is a bit of a pain. I've heard from these people that their
solution was to temporarily add their extension as a contrib module
and have the extension building code take care of creating and
building the Visual Studio project file.

I also have to say, I do often use Visual Studio myself for PostgreSQL
development, but when it comes to testing something with an extension,
I've always avoided the problem and moved over to Linux.

I thought about how we might improve this situation. The easiest way
I could think to do this was to just reuse the code that builds the
Visual Studio project files for contrib modules and write a Perl
script which calls those functions. Now, these functions, for those
who have never looked there before, they do use the PGXS compatible
Makefile as a source of truth and build the VS project file from that.
I've attached a very rough PoC patch which attempts to do this.

The script just takes the directory of the Makefile as the first
argument, and optionally the path to pg_config.exe as the 2nd
argument. If that happens to be in the PATH environment variable then
that can be left out.

You end up with:

X:\pg_src\src\tools\msvc>perl extbuild.pl
X:\pg_src\contrib\auto_explain X:\pg\bin
Makefile dir = X:\pg_src\contrib\auto_explain
Postgres include dir = X:\pg\include
Building = Release
Detected hardware platform: x64

...

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:01.13

For now, I've only tested this on a few contrib modules. It does need
more work to properly build ones with a list of "MODULES" in the
Makefile. It seems to work ok on the MODULE_big ones that I tested. It
needs a bit more work to get the resource file paths working properly
for PROGRAM.

Before I go and invest more time in this, I'd like to get community
feedback about the idea. Is this something that we'd want? Does it
seem maintainable enough to have in core? Is there a better way to do
it?

David

Attachments:

building_extensions_on_windows_poc.patchapplication/octet-stream; name=building_extensions_on_windows_poc.patchDownload
diff --git a/src/tools/msvc/MSBuildProject.pm b/src/tools/msvc/MSBuildProject.pm
index ebb169e201..5bbab6ac29 100644
--- a/src/tools/msvc/MSBuildProject.pm
+++ b/src/tools/msvc/MSBuildProject.pm
@@ -373,7 +373,11 @@ EOF
 	print $f <<EOF;
     </Link>
     <ResourceCompile>
-      <AdditionalIncludeDirectories>src\\include;\%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
+      <AdditionalIncludeDirectories>
+EOF
+	print $f "$self->{includes}";
+	print $f <<EOF;
+src\\include;\%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
     </ResourceCompile>
 EOF
 	if ($self->{builddef})
@@ -381,7 +385,7 @@ EOF
 		print $f <<EOF;
     <PreLinkEvent>
       <Message>Generate DEF file</Message>
-      <Command>perl src\\tools\\msvc\\gendef.pl $cfgname\\$self->{name} $self->{platform}</Command>
+      <Command>perl $self->{abspath}src\\tools\\msvc\\gendef.pl $cfgname\\$self->{name} $self->{platform}</Command>
     </PreLinkEvent>
 EOF
 	}
diff --git a/src/tools/msvc/Project.pm b/src/tools/msvc/Project.pm
index 20f79b382b..6f69807532 100644
--- a/src/tools/msvc/Project.pm
+++ b/src/tools/msvc/Project.pm
@@ -34,6 +34,7 @@ sub _new
 		disablewarnings       => '4018;4244;4273;4102;4090;4267',
 		disablelinkerwarnings => '',
 		platform              => $solution->{platform},
+		abspath               => '',
 	};
 
 	bless($self, $classname);
@@ -317,19 +318,18 @@ sub AddDirResourceFile
 sub AddResourceFile
 {
 	my ($self, $dir, $desc, $ico) = @_;
-
 	my ($sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst) =
 	  localtime(time);
 	my $d = sprintf("%02d%03d", ($year - 100), $yday);
 
-	if (Solution::IsNewer("$dir/win32ver.rc", 'src/port/win32ver.rc'))
+	if (Solution::IsNewer("$dir/win32ver.rc", "$self->{abspath}src/port/win32ver.rc"))
 	{
 		print "Generating win32ver.rc for $dir\n";
-		open(my $i, '<', 'src/port/win32ver.rc')
+		open(my $i, '<', "$self->{abspath}src/port/win32ver.rc")
 		  || confess "Could not open win32ver.rc";
 		open(my $o, '>', "$dir/win32ver.rc")
 		  || confess "Could not write win32ver.rc";
-		my $icostr = $ico ? "IDI_ICON ICON \"src/port/$ico.ico\"" : "";
+		my $icostr = $ico ? "IDI_ICON ICON \"$self->{abspath}src/port/$ico.ico\"" : "";
 		while (<$i>)
 		{
 			s/FILEDESC/"$desc"/gm;
diff --git a/src/tools/msvc/extbuild.pl b/src/tools/msvc/extbuild.pl
new file mode 100644
index 0000000000..baf2d1de10
--- /dev/null
+++ b/src/tools/msvc/extbuild.pl
@@ -0,0 +1,161 @@
+# src/tools/msvc/extbuild.pl
+
+package Mkvcbuild;
+
+use strict;
+use warnings;
+use Cwd;
+use File::Basename;
+
+use Mkvcbuild;
+
+my $solution;
+my $bconf     = $ENV{CONFIG}   || "Release";
+my $msbflags  = $ENV{MSBFLAGS} || "";
+my $makefiledir = $ARGV[0]       || "";
+my $pgconfigdir = $ARGV[1]      || "";
+my $pgincludedir = "";	# filled by output from pg_config.exe
+my $pglibdir = "";	# filled by output from pg_config.exe
+
+sub ProcessPgConfigOutput
+{
+	my $pgconfigdir = shift;
+	my $pgconfig;
+	my $output;
+
+	if ($pgconfigdir ne '')
+	{
+		$pgconfig = $pgconfigdir . "/pg_config.exe";
+	}
+	else
+	{
+		$pgconfig = "pg_config.exe";
+	}
+
+	if (-e $pgconfig)
+	{
+		# Run the pg_config command
+		$output = `$pgconfig`;
+
+		# parse the stdout of the pg_config command and extract some details.
+		if ($output =~ /^INCLUDEDIR\s*=\s*(.*)$/mg)
+		{
+			$pgincludedir = $1;
+		}
+
+		if ($output =~ /^LIBDIR\s*=\s*(.*)$/mg)
+		{
+			$pglibdir = $1;
+		}
+	}
+	else
+	{
+		die("Cannot find $pgconfig\n");
+	}
+}
+
+
+# Create the vcxproj file for the extension
+sub BuildExtensionProjectFile
+{
+	my $subdir = shift;
+	my $scriptdir = shift;
+	my $mf     = Project::read_file("$subdir/Makefile");
+	my $name;
+	my $proj;
+	my $visualStudioVersion = VSObjectFactory::DetermineVisualStudioVersion();
+	my $solution = CreateSolution($visualStudioVersion);
+	if ($mf =~ /^MODULE_big\s*=\s*(.*)$/mg)
+	{
+		$name = $1;
+		$proj = VSObjectFactory::CreateProject($visualStudioVersion, $name, 'dll', $solution);
+	}
+	elsif ($mf =~ /^MODULES\s*=\s*(.*)$/mg)
+	{
+		foreach my $mod (split /\s+/, $1)
+		{
+			$proj = VSObjectFactory::CreateProject($visualStudioVersion, $mod, 'dll', $solution);
+			my $filename = $mod . '.c';
+			$proj->AddFile("$subdir/$filename");
+		}
+	}
+	elsif ($mf =~ /^PROGRAM\s*=\s*(.*)$/mg)
+	{
+		$name = $1;
+		$proj = VSObjectFactory::CreateProject($visualStudioVersion, $1, 'exe', $solution);
+	}
+	else
+	{
+		die("Could not determine extension type for $subdir\n");
+	}
+	$proj->{abspath} = $scriptdir . "\\..\\..\\..\\";
+	$proj->AddDirResourceFile($subdir);
+	$proj->AddDir($subdir);
+	$proj->AddIncludeDir($pgincludedir);
+	$proj->AddIncludeDir("$pgincludedir/server");
+	$proj->AddIncludeDir("$pgincludedir/server/port/win32");
+	$proj->AddIncludeDir("$pgincludedir/server/port/win32_msvc");
+	$proj->AddLibrary($pglibdir . '/postgres.lib');
+	$proj->AddLibrary($pglibdir . '/libpq.lib');
+
+	#$proj->AddReference("$pglibdir/postgres");
+	# Are there any output data files to build?
+	GenerateContribSqlFiles($subdir, $mf);
+	$proj->Save();
+	return $name;
+}
+
+
+my $num_args = $#ARGV + 1;
+
+if ($num_args < 1)
+{
+	print("Usage: extbuild.pl <path_to_make_file> [path_to_pg_config] [DEBUG]");
+	exit;
+}
+
+# buildenv.pl is for specifying the build environment settings
+# it should contain lines like:
+# $ENV{PATH} = "c:/path/to/bison/bin;$ENV{PATH}";
+
+if (-e "src/tools/msvc/buildenv.pl")
+{
+	do "./src/tools/msvc/buildenv.pl";
+}
+elsif (-e "./buildenv.pl")
+{
+	do "./buildenv.pl";
+}
+
+if ($num_args >= 3)
+{
+	if (uc($ARGV[2]) eq 'DEBUG')
+	{
+		$bconf = "Debug";
+	}
+	elsif (uc($ARGV[2]) eq 'RELEASE')
+	{
+		$bconf = "Release";
+	}
+}
+
+my $name;
+
+my $scriptdir = dirname(Cwd::abs_path(__FILE__));
+chdir $makefiledir;
+
+ProcessPgConfigOutput($pgconfigdir);
+
+print("Makefile dir         = $makefiledir\n");
+print("Postgres include dir = $pgincludedir\n");
+print("Building             = $bconf\n");
+
+$name = BuildExtensionProjectFile($makefiledir, $scriptdir);
+
+system(
+	"msbuild $makefiledir/$name.vcxproj /verbosity:normal $msbflags /p:Configuration=$bconf"
+);
+
+
+
+
#2Joe Conway
mail@joeconway.com
In reply to: David Rowley (#1)
Re: Building PostgreSQL extensions on Windows

On 6/11/20 6:42 PM, David Rowley wrote:

I've heard from a few people that building PostgreSQL extensions on
Windows is a bit of a pain. I've heard from these people that their
solution was to temporarily add their extension as a contrib module
and have the extension building code take care of creating and
building the Visual Studio project file.

Yep -- that is exactly how I have been building PL/R on Windows for many years,
and it is painful.

I thought about how we might improve this situation. The easiest way
I could think to do this was to just reuse the code that builds the
Visual Studio project files for contrib modules and write a Perl
script which calls those functions. Now, these functions, for those
who have never looked there before, they do use the PGXS compatible
Makefile as a source of truth and build the VS project file from that.
I've attached a very rough PoC patch which attempts to do this.

The script just takes the directory of the Makefile as the first
argument, and optionally the path to pg_config.exe as the 2nd
argument. If that happens to be in the PATH environment variable then
that can be left out.

You end up with:

X:\pg_src\src\tools\msvc>perl extbuild.pl
X:\pg_src\contrib\auto_explain X:\pg\bin
Makefile dir = X:\pg_src\contrib\auto_explain
Postgres include dir = X:\pg\include
Building = Release
Detected hardware platform: x64

...

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:01.13

For now, I've only tested this on a few contrib modules. It does need
more work to properly build ones with a list of "MODULES" in the
Makefile. It seems to work ok on the MODULE_big ones that I tested. It
needs a bit more work to get the resource file paths working properly
for PROGRAM.

Before I go and invest more time in this, I'd like to get community
feedback about the idea. Is this something that we'd want? Does it
seem maintainable enough to have in core? Is there a better way to do
it?

Sounds very useful to me -- I'll give it a try with PL/R this weekend.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

#3Dmitry Igrishin
dmitigr@gmail.com
In reply to: David Rowley (#1)
Re: Building PostgreSQL extensions on Windows

Hi David,

No pain with CMake. It's pretty easy to use it in Windows for PostgreSQL
extensions. Example, https://github.com/dmitigr/pgnso

On Fri, 12 Jun 2020, 01:43 David Rowley, <dgrowleyml@gmail.com> wrote:

Show quoted text

Hi,

I've heard from a few people that building PostgreSQL extensions on
Windows is a bit of a pain. I've heard from these people that their
solution was to temporarily add their extension as a contrib module
and have the extension building code take care of creating and
building the Visual Studio project file.

I also have to say, I do often use Visual Studio myself for PostgreSQL
development, but when it comes to testing something with an extension,
I've always avoided the problem and moved over to Linux.

I thought about how we might improve this situation. The easiest way
I could think to do this was to just reuse the code that builds the
Visual Studio project files for contrib modules and write a Perl
script which calls those functions. Now, these functions, for those
who have never looked there before, they do use the PGXS compatible
Makefile as a source of truth and build the VS project file from that.
I've attached a very rough PoC patch which attempts to do this.

The script just takes the directory of the Makefile as the first
argument, and optionally the path to pg_config.exe as the 2nd
argument. If that happens to be in the PATH environment variable then
that can be left out.

You end up with:

X:\pg_src\src\tools\msvc>perl extbuild.pl
X:\pg_src\contrib\auto_explain X:\pg\bin
Makefile dir = X:\pg_src\contrib\auto_explain
Postgres include dir = X:\pg\include
Building = Release
Detected hardware platform: x64

...

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:01.13

For now, I've only tested this on a few contrib modules. It does need
more work to properly build ones with a list of "MODULES" in the
Makefile. It seems to work ok on the MODULE_big ones that I tested. It
needs a bit more work to get the resource file paths working properly
for PROGRAM.

Before I go and invest more time in this, I'd like to get community
feedback about the idea. Is this something that we'd want? Does it
seem maintainable enough to have in core? Is there a better way to do
it?

David