Supporting src/test/modules in MSVC builds
Hi all,
As mentioned previously
(CAB7nPqSCphafxS2RzeB-FBccjqiKnQXJhLoZtkggiM1MF5Xv4A@mail.gmail.com),
attached are patches to add support for src/test/modules in MSVC
builds, modules whose tests are not supported since they have been
moved from contrib/:
- 0001 adds support for the build portion.
- 0002 adds support for the installation. I arrived at the conclusion
that those modules should be installed by default, because
contribcheck relies on the fact that tests should be run on a server
already running, and modulescheck should do the same IMO.
- 0003 adds a new target modulescheck in vcregress.
Patches 0001 and 0003 are based on some previous work from Alvaro,
that I modified slightly after testing them. Note that this split is
done to test each step on the build farm for safety.
Regards,
--
Michael
Attachments:
0001-Include-modules-of-src-test-modules-in-build.patchtext/x-diff; charset=US-ASCII; name=0001-Include-modules-of-src-test-modules-in-build.patchDownload
From 1b58eba30d0347a7718a0db18ac6ae2c02888aaf Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Wed, 15 Apr 2015 22:14:33 -0700
Subject: [PATCH 1/3] Include modules of src/test/modules in build
commit_ts, being only a module used for test purposes, is simply ignored
in the process.
---
src/tools/msvc/Mkvcbuild.pm | 33 +++++++++++++++++++--------------
1 file changed, 19 insertions(+), 14 deletions(-)
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index e4dbebf..986f3b3 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -28,7 +28,7 @@ my $libpgcommon;
my $postgres;
my $libpq;
-# Set of variables for contrib modules
+# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
my @contrib_uselibpq =
('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
@@ -50,7 +50,7 @@ my $contrib_extraincludes =
my $contrib_extrasource = {
'cube' => [ 'contrib\cube\cubescan.l', 'contrib\cube\cubeparse.y' ],
'seg' => [ 'contrib\seg\segscan.l', 'contrib\seg\segparse.y' ], };
-my @contrib_excludes = ('pgcrypto', 'intagg', 'sepgsql');
+my @contrib_excludes = ('pgcrypto', 'commit_ts', 'intagg', 'sepgsql');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
@@ -564,15 +564,18 @@ sub mkvcbuild
my $mf = Project::read_file('contrib/pgcrypto/Makefile');
GenerateContribSqlFiles('pgcrypto', $mf);
- opendir($D, 'contrib') || croak "Could not opendir on contrib!\n";
- while (my $d = readdir($D))
+ foreach my $subdir ('contrib', 'src/test/modules')
{
- next if ($d =~ /^\./);
- next unless (-f "contrib/$d/Makefile");
- next if (grep { /^$d$/ } @contrib_excludes);
- AddContrib($d);
+ opendir($D, $subdir) || croak "Could not opendir on $subdir!\n";
+ while (my $d = readdir($D))
+ {
+ next if ($d =~ /^\./);
+ next unless (-f "$subdir/$d/Makefile");
+ next if (grep { /^$d$/ } @contrib_excludes);
+ AddContrib($subdir, $d);
+ }
+ closedir($D);
}
- closedir($D);
$mf =
Project::read_file('src\backend\utils\mb\conversion_procs\Makefile');
@@ -689,14 +692,15 @@ sub AddSimpleFrontend
# Add a simple contrib project
sub AddContrib
{
+ my $subdir = shift;
my $n = shift;
- my $mf = Project::read_file('contrib\\' . $n . '\Makefile');
+ my $mf = Project::read_file("$subdir/$n/Makefile");
if ($mf =~ /^MODULE_big\s*=\s*(.*)$/mg)
{
my $dn = $1;
my $proj =
- $solution->AddProject($dn, 'dll', 'contrib', 'contrib\\' . $n);
+ $solution->AddProject($dn, 'dll', 'contrib', "$subdir/$n");
$proj->AddReference($postgres);
AdjustContribProj($proj);
}
@@ -705,8 +709,9 @@ sub AddContrib
foreach my $mod (split /\s+/, $1)
{
my $proj =
- $solution->AddProject($mod, 'dll', 'contrib', 'contrib\\' . $n);
- $proj->AddFile('contrib\\' . $n . '\\' . $mod . '.c');
+ $solution->AddProject($mod, 'dll', 'contrib', "$subdir/$n");
+ my $filename = $mod . '.c';
+ $proj->AddFile($subdir . '\\' . $n . '\\' . $mod . '.c');
$proj->AddReference($postgres);
AdjustContribProj($proj);
}
@@ -714,7 +719,7 @@ sub AddContrib
elsif ($mf =~ /^PROGRAM\s*=\s*(.*)$/mg)
{
my $proj =
- $solution->AddProject($1, 'exe', 'contrib', 'contrib\\' . $n);
+ $solution->AddProject($1, 'exe', 'contrib', "$subdir/$n");
AdjustContribProj($proj);
}
else
--
2.3.5
0002-Support-installation-of-test-modules-in-MSVC.patchtext/x-diff; charset=US-ASCII; name=0002-Support-installation-of-test-modules-in-MSVC.patchDownload
From e901543c2fa728646dca13a66979a6f0619dd87f Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Wed, 15 Apr 2015 23:13:14 -0700
Subject: [PATCH 2/3] Support installation of test modules in MSVC
Note that those modules are needed in the installation because like
contribcheck, modulescheck does not create a cluster from scratch
on where to run the tests. It seems also good to include them to be
able to perform those tests using a central installation path.
---
src/tools/msvc/Install.pm | 151 +++++++++++++++++++++++++---------------------
1 file changed, 81 insertions(+), 70 deletions(-)
diff --git a/src/tools/msvc/Install.pm b/src/tools/msvc/Install.pm
index 93a6724..c82743d 100644
--- a/src/tools/msvc/Install.pm
+++ b/src/tools/msvc/Install.pm
@@ -433,97 +433,108 @@ sub CopyContribFiles
print "Copying contrib data files...";
my $D;
- opendir($D, 'contrib') || croak "Could not opendir on contrib!\n";
- while (my $d = readdir($D))
+ foreach my $subdir ('contrib', 'src/test/modules')
{
- next if ($d =~ /^\./);
- next unless (-f "contrib/$d/Makefile");
- next
- if ($insttype eq "client" && !grep { $_ eq $d } @client_contribs);
+ opendir($D, $subdir) || croak "Could not opendir on $subdir!\n";
+ while (my $d = readdir($D))
+ {
+ # These configuration-based exclusions must match vcregress.pl
+ next if ($d eq "uuid-ossp" && !defined($config->{uuid}));
+ next if ($d eq "sslinfo" && !defined($config->{openssl}));
+ next if ($d eq "xml2" && !defined($config->{xml}));
+ next if ($d eq "sepgsql");
+ CopySubdirFiles($subdir, $d, $config, $target);
+ }
+ }
+ print "\n";
+}
- # these configuration-based exclusions must match vcregress.pl
- next if ($d eq "uuid-ossp" && !defined($config->{uuid}));
- next if ($d eq "sslinfo" && !defined($config->{openssl}));
- next if ($d eq "xml2" && !defined($config->{xml}));
- next if ($d eq "sepgsql");
+sub CopySubdirFiles
+{
+ my $subdir = shift;
+ my $module = shift;
+ my $config = shift;
+ my $target = shift;
- my $mf = read_file("contrib/$d/Makefile");
- $mf =~ s{\\\r?\n}{}g;
+ return if ($module =~ /^\./);
+ return unless (-f "$subdir/$module/Makefile");
+ return
+ if ($insttype eq "client" && !grep { $_ eq $module } @client_contribs);
- # Note: we currently don't support setting MODULEDIR in the makefile
- my $moduledir = 'contrib';
+ my $mf = read_file("$subdir/$module/Makefile");
+ $mf =~ s{\\\r?\n}{}g;
- my $flist = '';
- if ($mf =~ /^EXTENSION\s*=\s*(.*)$/m) { $flist .= $1 }
- if ($flist ne '')
- {
- $moduledir = 'extension';
- $flist = ParseAndCleanRule($flist, $mf);
+ # Note: we currently don't support setting MODULEDIR in the makefile
+ my $moduledir = 'contrib';
- foreach my $f (split /\s+/, $flist)
- {
- lcopy(
- 'contrib/' . $d . '/' . $f . '.control',
- $target . '/share/extension/' . $f . '.control'
- ) || croak("Could not copy file $f.control in contrib $d");
+ my $flist = '';
+ if ($mf =~ /^EXTENSION\s*=\s*(.*)$/m) { $flist .= $1 }
+ if ($flist ne '')
+ {
+ $moduledir = 'extension';
+ $flist = ParseAndCleanRule($flist, $mf);
+
+ foreach my $f (split /\s+/, $flist)
+ {
+ lcopy(
+ "$subdir/$module/$f.control",
+ "$target/share/extension/$f.control"
+ ) || croak("Could not copy file $f.control in contrib $module");
print '.';
- }
}
+ }
- $flist = '';
- if ($mf =~ /^DATA_built\s*=\s*(.*)$/m) { $flist .= $1 }
- if ($mf =~ /^DATA\s*=\s*(.*)$/m) { $flist .= " $1" }
- $flist =~ s/^\s*//; # Remove leading spaces if we had only DATA_built
+ $flist = '';
+ if ($mf =~ /^DATA_built\s*=\s*(.*)$/m) { $flist .= $1 }
+ if ($mf =~ /^DATA\s*=\s*(.*)$/m) { $flist .= " $1" }
+ $flist =~ s/^\s*//; # Remove leading spaces if we had only DATA_built
- if ($flist ne '')
- {
- $flist = ParseAndCleanRule($flist, $mf);
+ if ($flist ne '')
+ {
+ $flist = ParseAndCleanRule($flist, $mf);
- foreach my $f (split /\s+/, $flist)
- {
- lcopy('contrib/' . $d . '/' . $f,
- $target . '/share/' . $moduledir . '/' . basename($f))
- || croak("Could not copy file $f in contrib $d");
+ foreach my $f (split /\s+/, $flist)
+ {
+ lcopy("$subdir/$module/$f",
+ "$target/share/$moduledir/" . basename($f))
+ || croak("Could not copy file $f in contrib $module");
print '.';
- }
}
+ }
- $flist = '';
- if ($mf =~ /^DATA_TSEARCH\s*=\s*(.*)$/m) { $flist .= $1 }
- if ($flist ne '')
- {
- $flist = ParseAndCleanRule($flist, $mf);
+ $flist = '';
+ if ($mf =~ /^DATA_TSEARCH\s*=\s*(.*)$/m) { $flist .= $1 }
+ if ($flist ne '')
+ {
+ $flist = ParseAndCleanRule($flist, $mf);
- foreach my $f (split /\s+/, $flist)
- {
- lcopy('contrib/' . $d . '/' . $f,
- $target . '/share/tsearch_data/' . basename($f))
- || croak("Could not copy file $f in contrib $d");
- print '.';
- }
+ foreach my $f (split /\s+/, $flist)
+ {
+ lcopy("$subdir/$module/$f",
+ "$target/share/tsearch_data/" . basename($f))
+ || croak("Could not copy file $f in $subdir $module");
+ print '.';
}
+ }
- $flist = '';
- if ($mf =~ /^DOCS\s*=\s*(.*)$/mg) { $flist .= $1 }
- if ($flist ne '')
- {
- $flist = ParseAndCleanRule($flist, $mf);
+ $flist = '';
+ if ($mf =~ /^DOCS\s*=\s*(.*)$/mg) { $flist .= $1 }
+ if ($flist ne '')
+ {
+ $flist = ParseAndCleanRule($flist, $mf);
- # Special case for contrib/spi
- $flist =
+ # Special case for contrib/spi
+ $flist =
"autoinc.example insert_username.example moddatetime.example refint.example timetravel.example"
- if ($d eq 'spi');
- foreach my $f (split /\s+/, $flist)
- {
- lcopy('contrib/' . $d . '/' . $f,
- $target . '/doc/' . $moduledir . '/' . $f)
- || croak("Could not copy file $f in contrib $d");
- print '.';
- }
+ if ($module eq 'spi');
+ foreach my $f (split /\s+/, $flist)
+ {
+ lcopy("$subdir/$module/$f",
+ "$target/doc/$moduledir/$f")
+ || croak("Could not copy file $f in contrib $module");
+ print '.';
}
}
- closedir($D);
- print "\n";
}
sub ParseAndCleanRule
--
2.3.5
0003-Add-new-target-modulescheck-for-vcregress.patchtext/x-diff; charset=US-ASCII; name=0003-Add-new-target-modulescheck-for-vcregress.patchDownload
From 7f9d8612293947fca0ae1aebf0f75655dece8bf6 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Wed, 15 Apr 2015 22:45:53 -0700
Subject: [PATCH 3/3] Add new target modulescheck for vcregress
This allows an MSVC build to run regression tests related to modules
in src/test/modules.
---
doc/src/sgml/install-windows.sgml | 1 +
src/tools/msvc/vcregress.pl | 68 ++++++++++++++++++++++++++-------------
2 files changed, 46 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/install-windows.sgml b/doc/src/sgml/install-windows.sgml
index 9b77648..d154b44 100644
--- a/doc/src/sgml/install-windows.sgml
+++ b/doc/src/sgml/install-windows.sgml
@@ -436,6 +436,7 @@ $ENV{CONFIG}="Debug";
<userinput>vcregress installcheck</userinput>
<userinput>vcregress plcheck</userinput>
<userinput>vcregress contribcheck</userinput>
+<userinput>vcregress modulescheck</userinput>
<userinput>vcregress ecpgcheck</userinput>
<userinput>vcregress isolationcheck</userinput>
<userinput>vcregress upgradecheck</userinput>
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 4812a03..c3143ac 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -31,7 +31,7 @@ if (-e "src/tools/msvc/buildenv.pl")
my $what = shift || "";
if ($what =~
-/^(check|installcheck|plcheck|contribcheck|ecpgcheck|isolationcheck|upgradecheck)$/i
+/^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck)$/i
)
{
$what = uc $what;
@@ -49,7 +49,7 @@ copy("$Config/autoinc/autoinc.dll", "src/test/regress");
copy("$Config/regress/regress.dll", "src/test/regress");
copy("$Config/dummy_seclabel/dummy_seclabel.dll", "src/test/regress");
-$ENV{PATH} = "../../../$Config/libpq;../../$Config/libpq;$ENV{PATH}";
+$ENV{PATH} = "$topdir/$Config/libpq;$topdir/$Config/libpq;$ENV{PATH}";
my $schedule = shift;
unless ($schedule)
@@ -76,6 +76,7 @@ my %command = (
INSTALLCHECK => \&installcheck,
ECPGCHECK => \&ecpgcheck,
CONTRIBCHECK => \&contribcheck,
+ MODULESCHECK => \&modulescheck,
ISOLATIONCHECK => \&isolationcheck,
UPGRADECHECK => \&upgradecheck,);
@@ -213,10 +214,39 @@ sub plcheck
chdir "../../..";
}
-sub contribcheck
+sub subdircheck
{
- chdir "../../../contrib";
+ my $subdir = shift;
+ my $module = shift;
my $mstat = 0;
+
+ if ( ! -d "$module/sql" ||
+ ! -d "$module/expected" ||
+ ( ! -f "$module/GNUmakefile" && ! -f "$module/Makefile"))
+ {
+ return;
+ }
+ chdir $module;
+ print
+ "============================================================\n";
+ print "Checking $module\n";
+ my @tests = fetchTests();
+ my @opts = fetchRegressOpts();
+ my @args = (
+ "$topdir/$Config/pg_regress/pg_regress",
+ "--psqldir=$topdir/$Config/psql",
+ "--dbname=contrib_regression", @opts, @tests);
+ system(@args);
+ my $status = $? >> 8;
+ $mstat ||= $status;
+ chdir "..";
+
+ exit $mstat if $mstat;
+}
+
+sub contribcheck
+{
+ chdir "$topdir/contrib";
foreach my $module (glob("*"))
{
# these configuration-based exclusions must match Install.pm
@@ -225,28 +255,20 @@ sub contribcheck
next if ($module eq "xml2" && !defined($config->{xml}));
next if ($module eq "sepgsql");
- next
- unless -d "$module/sql"
- && -d "$module/expected"
- && (-f "$module/GNUmakefile" || -f "$module/Makefile");
- chdir $module;
- print
- "============================================================\n";
- print "Checking $module\n";
- my @tests = fetchTests();
- my @opts = fetchRegressOpts();
- my @args = (
- "../../$Config/pg_regress/pg_regress",
- "--psqldir=../../$Config/psql",
- "--dbname=contrib_regression", @opts, @tests);
- system(@args);
- my $status = $? >> 8;
- $mstat ||= $status;
- chdir "..";
+ subdircheck("$topdir/contrib", $module);
}
- exit $mstat if $mstat;
}
+sub modulescheck
+{
+ chdir "$topdir/src/test/modules";
+ foreach my $module (glob("*"))
+ {
+ subdircheck("$topdir/src/test/modules", $module);
+ }
+}
+
+
# Run "initdb", then reconfigure authentication.
sub standard_initdb
{
--
2.3.5
On 04/16/2015 02:46 AM, Michael Paquier wrote:
Hi all,
As mentioned previously
(CAB7nPqSCphafxS2RzeB-FBccjqiKnQXJhLoZtkggiM1MF5Xv4A@mail.gmail.com),
attached are patches to add support for src/test/modules in MSVC
builds, modules whose tests are not supported since they have been
moved from contrib/:
- 0001 adds support for the build portion.
- 0002 adds support for the installation. I arrived at the conclusion
that those modules should be installed by default, because
contribcheck relies on the fact that tests should be run on a server
already running, and modulescheck should do the same IMO.
- 0003 adds a new target modulescheck in vcregress.Patches 0001 and 0003 are based on some previous work from Alvaro,
that I modified slightly after testing them. Note that this split is
done to test each step on the build farm for safety.
Regards,
Thanks for doing this. It looks good, and if you've tested it I'm
satisfied. I suggest that we apply patches 1 and 2 immediately. AUIU
they don't require any changes to the buildfarm, as the MSVC build
process will automatically build and install it with these changes. Then
if all goes well we can apply the third patch and I'll fix the buildfarm
client for the forthcoming release to run the tests on MSVC builds.
Nothing will break in the meantime - the tests just won't get run until
the new client version is deployed.
cheers
andrew
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andrew Dunstan wrote:
Thanks for doing this.
Yes, much appreciated.
It looks good, and if you've tested it I'm satisfied. I suggest that
we apply patches 1 and 2 immediately. AUIU they don't require any
changes to the buildfarm, as the MSVC build process will automatically
build and install it with these changes.
Okay, I just pushed patches 1 and 2.
Then if all goes well we can apply the third patch and I'll fix the
buildfarm client for the forthcoming release to run the tests on MSVC
builds. Nothing will break in the meantime - the tests just won't get
run until the new client version is deployed.
Will wait for a day or so to see what the Win buildfarm members say.
--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Apr 17, 2015 at 4:47 AM, Alvaro Herrera wrote:
Andrew Dunstan wrote:
It looks good, and if you've tested it I'm satisfied. I suggest that
we apply patches 1 and 2 immediately. AUIU they don't require any
changes to the buildfarm, as the MSVC build process will automatically
build and install it with these changes.Okay, I just pushed patches 1 and 2.
Cool, thanks.
Then if all goes well we can apply the third patch and I'll fix the
buildfarm client for the forthcoming release to run the tests on MSVC
builds. Nothing will break in the meantime - the tests just won't get
run until the new client version is deployed.Will wait for a day or so to see what the Win buildfarm members say.
currawong and mastodon seem satisfied.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 04/16/2015 07:42 PM, Michael Paquier wrote:
Then if all goes well we can apply the third patch and I'll fix the
buildfarm client for the forthcoming release to run the tests on MSVC
builds. Nothing will break in the meantime - the tests just won't get
run until the new client version is deployed.Will wait for a day or so to see what the Win buildfarm members say.
currawong and mastodon seem satisfied.
Yeah. I think we can push the third one - I am just about ready to
release the new buildfarm client.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andrew Dunstan wrote:
On 04/16/2015 07:42 PM, Michael Paquier wrote:
Then if all goes well we can apply the third patch and I'll fix the
buildfarm client for the forthcoming release to run the tests on MSVC
builds. Nothing will break in the meantime - the tests just won't get
run until the new client version is deployed.Will wait for a day or so to see what the Win buildfarm members say.
currawong and mastodon seem satisfied.
Yeah. I think we can push the third one - I am just about ready to release
the new buildfarm client.
Pushed.
--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Apr 17, 2015 at 11:41 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
Andrew Dunstan wrote:
On 04/16/2015 07:42 PM, Michael Paquier wrote:
Then if all goes well we can apply the third patch and I'll fix the
buildfarm client for the forthcoming release to run the tests on MSVC
builds. Nothing will break in the meantime - the tests just won't get
run until the new client version is deployed.Will wait for a day or so to see what the Win buildfarm members say.
currawong and mastodon seem satisfied.
Yeah. I think we can push the third one - I am just about ready to release
the new buildfarm client.Pushed.
And currawong is satisfied with this patch and the new buildfarm code,
test modules being run as testmodules-install-check-C:
http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=currawong&dt=2015-04-18%2014%3A51%3A14
So this closes the loop for this thread.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Michael Paquier wrote:
And currawong is satisfied with this patch and the new buildfarm code,
test modules being run as testmodules-install-check-C:
http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=currawong&dt=2015-04-18%2014%3A51%3A14
So this closes the loop for this thread.
Yay! Thanks, Michael and Andrew.
--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers