improved parallel make support
I have worked on some improvements on how we handle recursive make in
our makefiles. Most places uses for loops, which has some
disadvantages: parallel make doesn't work across directories, make -k
doesn't work, and make -q doesn't work. Instead, I went with the
approach that we already use in the src/backend directory, where we call
the subordinate makes as target prerequisites.
Note that because with this, parallel make really works, the rule
dependencies must be correct. This has always been the case, but now it
really shows up. A frequent issue is that this sort of thing no longer
works:
all: submake-libpgport zic
zic: $(ZICOBJS)
because this relies on the "all" target to execute its prerequisites in
order. Instead, you need to write it like this:
all: zic
zic: $(ZICOBJS) | submake-libpgport
(The bar is necessary so that zic isn't considered constantly out of
date because it depends on a phony target.)
This patch requires GNU make 3.80, because of the above "|" feature and
the $(eval) function. Version 3.80 is dated October 2002, so it should
be no problem, but I do occasionally read of make 3.79 around here;
maybe it's time to get rid of that. I did put in a check that makes the
build fail right away if a wrong version of make is used.
Attachments:
parallel-make.patchtext/x-patch; charset=UTF-8; name=parallel-make.patchDownload+154-169
Peter Eisentraut <peter_e@gmx.net> writes:
This patch requires GNU make 3.80, because of the above "|" feature and
the $(eval) function. Version 3.80 is dated October 2002, so it should
be no problem, but I do occasionally read of make 3.79 around here;
maybe it's time to get rid of that. I did put in a check that makes the
build fail right away if a wrong version of make is used.
Do we have a handle on how many buildfarm members this will break?
(fwiw, my hpux box is running 3.79.1)
regards, tom lane
On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
Peter Eisentraut <peter_e@gmx.net> writes:
This patch requires GNU make 3.80, because of the above "|" feature and
the $(eval) function. Version 3.80 is dated October 2002, so it should
be no problem, but I do occasionally read of make 3.79 around here;
maybe it's time to get rid of that. I did put in a check that makes the
build fail right away if a wrong version of make is used.Do we have a handle on how many buildfarm members this will break?
I suppose we don't. One way to find out would be to commit just this
bit
+# We need the $(eval) function, which is available in GNU make 3.80.
+# That also happens to be the version where the .VARIABLES variable
+# was introduced, so this is a simple check.
+ifndef .VARIABLES
+$(error GNU make 3.80 or newer is required)
+endifwith a $(warning) instead, and let it run for a bit.
Peter Eisentraut <peter_e@gmx.net> writes:
On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
Do we have a handle on how many buildfarm members this will break?
I suppose we don't. One way to find out would be to commit just this
bit
+# We need the $(eval) function, which is available in GNU make 3.80. +# That also happens to be the version where the .VARIABLES variable +# was introduced, so this is a simple check. +ifndef .VARIABLES +$(error GNU make 3.80 or newer is required) +endif
with a $(warning) instead, and let it run for a bit.
+1
regards, tom lane
On ons, 2010-11-03 at 16:34 +0200, Peter Eisentraut wrote:
On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
Do we have a handle on how many buildfarm members this will break?
I suppose we don't. One way to find out would be to commit just this
bit+# We need the $(eval) function, which is available in GNU make 3.80. +# That also happens to be the version where the .VARIABLES variable +# was introduced, so this is a simple check. +ifndef .VARIABLES +$(error GNU make 3.80 or newer is required) +endifwith a $(warning) instead, and let it run for a bit.
So far, two machines have reported an older make version:
dawn_bat
narwhal
both of the mingw type. Andrew, Dave, could you see about upgrading the
GNU make installation there?
There are a few machines that haven't build in five days or more, but
based on their operating system version, it is fairly safe to assume
that they have an up-to-date version.
On 11/06/2010 07:35 AM, Peter Eisentraut wrote:
So far, two machines have reported an older make version:
dawn_bat
narwhalboth of the mingw type. Andrew, Dave, could you see about upgrading the
GNU make installation there?
dawn_bat is done.
cheers
andrew
On Sat, Nov 6, 2010 at 4:35 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
On ons, 2010-11-03 at 16:34 +0200, Peter Eisentraut wrote:
On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
Do we have a handle on how many buildfarm members this will break?
I suppose we don't. One way to find out would be to commit just this
bit+# We need the $(eval) function, which is available in GNU make 3.80. +# That also happens to be the version where the .VARIABLES variable +# was introduced, so this is a simple check. +ifndef .VARIABLES +$(error GNU make 3.80 or newer is required) +endifwith a $(warning) instead, and let it run for a bit.
So far, two machines have reported an older make version:
dawn_bat
narwhalboth of the mingw type. Andrew, Dave, could you see about upgrading the
GNU make installation there?
Narwhal should be OK now.
/D
On tis, 2010-11-09 at 03:54 -0800, Dave Page wrote:
Narwhal should be OK now.
The build has issues now, possibly related to the make upgrade.
On 11/10/2010 10:32 AM, Peter Eisentraut wrote:
On tis, 2010-11-09 at 03:54 -0800, Dave Page wrote:
Narwhal should be OK now.
The build has issues now, possibly related to the make upgrade.
Yeah, it's complaining about not finding bison, but configure managed to
find bison just fine. Are you sure the right make was installed? It
looks suspicious because it's not talking about msys virtual maths like
the old make did. It needs to be make-3.81-3-msys-1.0.13
<http://sourceforge.net/projects/mingw/files/MSYS/make/make-3.81-3/make-3.81-3-msys-1.0.13-bin.tar.lzma/download>
You'll need another couple of libraries as well (libiconv and libintl)
if they are not already installed. Making this change took me a while to
get right on dawn_bat.
cheers
andrew
On Wed, Nov 10, 2010 at 6:13 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
Yeah, it's complaining about not finding bison, but configure managed to
find bison just fine. Are you sure the right make was installed? It looks
suspicious because it's not talking about msys virtual maths like the old
make did. It needs to be make-3.81-3-msys-1.0.13
<http://sourceforge.net/projects/mingw/files/MSYS/make/make-3.81-3/make-3.81-3-msys-1.0.13-bin.tar.lzma/download>
You'll need another couple of libraries as well (libiconv and libintl) if
they are not already installed. Making this change took me a while to get
right on dawn_bat.
I installed the latest make from gnu.org (which I've now uninstalled).
The Msys installation on this box is old, and doesn't support the lzma
packages used by the latest releases - and from what I can tell, it
would take a major upgrade of the installation to get that support.
I'm not sure thats a path I want to go down, as I have no idea how
much will break if I do that, and I don't exactly have much in the way
of spare time to fix it if that happens.
I'm currently leaning towards removing the 9.1 build from the machine;
on a purely selfish note, I have no interest in mingw/msys builds
anymore anyway. However, I'm open to suggestions if anyone knows a
relatively safe way to resolve this.
/D
On 11/11/2010 06:58 AM, Dave Page wrote:
On Wed, Nov 10, 2010 at 6:13 PM, Andrew Dunstan<andrew@dunslane.net> wrote:
Yeah, it's complaining about not finding bison, but configure managed to
find bison just fine. Are you sure the right make was installed? It looks
suspicious because it's not talking about msys virtual maths like the old
make did. It needs to be make-3.81-3-msys-1.0.13
<http://sourceforge.net/projects/mingw/files/MSYS/make/make-3.81-3/make-3.81-3-msys-1.0.13-bin.tar.lzma/download>
You'll need another couple of libraries as well (libiconv and libintl) if
they are not already installed. Making this change took me a while to get
right on dawn_bat.I installed the latest make from gnu.org (which I've now uninstalled).
The Msys installation on this box is old, and doesn't support the lzma
packages used by the latest releases - and from what I can tell, it
would take a major upgrade of the installation to get that support.
I'm not sure thats a path I want to go down, as I have no idea how
much will break if I do that, and I don't exactly have much in the way
of spare time to fix it if that happens.I'm currently leaning towards removing the 9.1 build from the machine;
on a purely selfish note, I have no interest in mingw/msys builds
anymore anyway. However, I'm open to suggestions if anyone knows a
relatively safe way to resolve this.
No, all you need to unpack those is the basic-bsdtar package. But to
save you the pain of all this, I have copied the three objects I
installed to get this working on my likewise pretty old Msys to where
you can get them. Just grab
<http://developer.postgresql.org/~adunstan/msys-make.tgz>
As a matter of policy, I do not want to drop support for a FOSS build
tool chain on Windows if at all avoidable.
cheers
andrew
On Thu, Nov 11, 2010 at 1:04 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
No, all you need to unpack those is the basic-bsdtar package.
Ahh, OK. That seems to be in the MinGW (compiler) section of the
downloads for some reason.
But to save
you the pain of all this, I have copied the three objects I installed to get
this working on my likewise pretty old Msys to where you can get them. Just
grab
<http://developer.postgresql.org/~adunstan/msys-make.tgz>
Thanks - installed.
As a matter of policy, I do not want to drop support for a FOSS build tool
chain on Windows if at all avoidable.
Nor I, however I only have limited time to dedicate to that goal.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Dave Page wrote:
Thanks - installed.
As a matter of policy, I do not want to drop support for a FOSS build tool
chain on Windows if at all avoidable.Nor I, however I only have limited time to dedicate to that goal.
One thing to think about is that since PostGIS uses MingW/PGXS on
Windows, we use MingW builds in order to generate the Makefiles we need
(there is no native MSVC build for Windows). Not being able to do this
would cause us great inconvenience :(
ATB,
Mark.
--
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063
Sirius Labs: http://www.siriusit.co.uk/labs
On 11/11/2010 11:43 AM, Mark Cave-Ayland wrote:
Dave Page wrote:
Thanks - installed.
As a matter of policy, I do not want to drop support for a FOSS
build tool
chain on Windows if at all avoidable.Nor I, however I only have limited time to dedicate to that goal.
One thing to think about is that since PostGIS uses MingW/PGXS on
Windows, we use MingW builds in order to generate the Makefiles we
need (there is no native MSVC build for Windows). Not being able to do
this would cause us great inconvenience :(
Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and
isn't its Windows version build with MSVC?
cheers
andrew
On Thu, Nov 11, 2010 at 4:51 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and isn't
its Windows version build with MSVC?
Yes - it's a PITA as we have to have a dummy build of the server in
mingw/msys to compile PostGIS and Slony. We're probably going to be
looking at that in the not-to-distant future as we want 64bit builds
of both and will be using VC++.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Dave Page wrote:
On Thu, Nov 11, 2010 at 4:51 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and isn't
its Windows version build with MSVC?Yes - it's a PITA as we have to have a dummy build of the server in
mingw/msys to compile PostGIS and Slony. We're probably going to be
looking at that in the not-to-distant future as we want 64bit builds
of both and will be using VC++.
Just for the record, a lot of work was done in the 1.4 release series to
make MSVC builds possible, and indeed several people have reported success:
http://postgis.refractions.net/pipermail/postgis-devel/2009-March/005102.html
http://postgis.refractions.net/pipermail/postgis-devel/2010-September/010299.html
The two main outstanding issues as I see it are:
1) The GTK-based GUI for shp2pgsql (although if someone wanted to
sponsor work to convert to wxWidgets to bring us in line with pgAdmin,
that would be strongly considered).
2) Maintenance of the MSVC build system. So far we have had some
complaints about not using MSVC, but then no-one has stepped up to
maintain the build system for it. Forcing all existing developers to
suddenly start maintaining the Windows build is a total non-starter.
My hope is that one day CMake will enable us to come up with a universal
solution, but we're some way from that yet.
ATB,
Mark.
--
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063
Sirius Labs: http://www.siriusit.co.uk/labs
On Thu, Nov 11, 2010 at 5:19 PM, Mark Cave-Ayland
<mark.cave-ayland@siriusit.co.uk> wrote:
Dave Page wrote:
On Thu, Nov 11, 2010 at 4:51 PM, Andrew Dunstan <andrew@dunslane.net>
wrote:Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and
isn't
its Windows version build with MSVC?Yes - it's a PITA as we have to have a dummy build of the server in
mingw/msys to compile PostGIS and Slony. We're probably going to be
looking at that in the not-to-distant future as we want 64bit builds
of both and will be using VC++.Just for the record, a lot of work was done in the 1.4 release series to
make MSVC builds possible, and indeed several people have reported success:http://postgis.refractions.net/pipermail/postgis-devel/2009-March/005102.html
http://postgis.refractions.net/pipermail/postgis-devel/2010-September/010299.html
Cool - that will help.
The two main outstanding issues as I see it are:
1) The GTK-based GUI for shp2pgsql (although if someone wanted to sponsor
work to convert to wxWidgets to bring us in line with pgAdmin, that would be
strongly considered).
:-)
2) Maintenance of the MSVC build system. So far we have had some complaints
about not using MSVC, but then no-one has stepped up to maintain the build
system for it. Forcing all existing developers to suddenly start maintaining
the Windows build is a total non-starter.
Unless you're making major architectural changes, it shouldn't take
any real effort to add/remove the occasional source file. I'm sure
there are folks that could be persuaded to do that occasionally.
My hope is that one day CMake will enable us to come up with a universal
solution, but we're some way from that yet.
We used CMake for a couple of projects, but ended up abandoning it for
new stuff. It just didn't work as nicely as we wanted.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 11/11/2010 03:19 PM, Dave Page wrote:
My hope is that one day CMake will enable us to come up with a universal
solution, but we're some way from that yet.We used CMake for a couple of projects, but ended up abandoning it for
new stuff. It just didn't work as nicely as we wanted.
Yes, it's been discussed before here too and didn't really go anywhere :-(
cheers
andrew