HEAD doesn't cope with libraries in non-default locations

Tom Lane wrote:

CVS tip fails with
./configure --with-openssl \
--with-includes=/usr/local/ssl/include --with-libs=/usr/local/ssl/lib

...
make[3]: Entering directory `/home/postgres/pgsql/src/interfaces/libpq'
...
/usr/ccs/bin/ld +h libpq.sl.4 -b +b /home/postgres/testversion/lib fe-auth.o fe-connect.o fe-exec.o fe-misc.o fe-print.o fe-lobj.o fe-protocol2.o fe-protocol3.o pqexpbuffer.o pqsignal.o fe-secure.o md5.o ip.o wchar.o encnames.o noblock.o pgstrcasecmp.o thread.o getaddrinfo.o -lssl -lcrypto `gcc -L../../../src/port -L/usr/local/ssl/lib -Wl,-z -Wl,+b -Wl,/home/postgres/testversion/lib -print-libgcc-file-name` -L../../../src/port -L/usr/local/ssl/lib -o libpq.sl.4
/usr/ccs/bin/ld: Can't find library for -lssl
make[3]: *** [libpq.sl.4] Error 1

It appears that somebody has changed things so that the -L switches
appear after the -l switches (ie, too late). I'm too tired to
investigate now, but my money is on Autoconf 2.59 being the problem ...

I wonder if it was this commit. I am attaching the patch so you can
test to see if it fixes it. If it does, please let us know.

---------------------------------------------------------------------------

revision 1.91
date: 2005/07/02 23:28:22; author: momjian; state: Exp; lines: +2 -2

A quick look shows that when you use --with-libraries=/foo/bar the
generated link line for libraries says

-L/foo/bar -lpq

and it should probably be the other way around (as it is for the
executables).

So I suspect we need some makefile tuning.

You were correct. This patch fixes it.

Jim C. Nasby

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Tom Lane wrote:

It appears that somebody has changed things so that the -L switches
appear after the -l switches (ie, too late). I'm too tired to
investigate now, but my money is on Autoconf 2.59 being the problem ...

I wonder if it was this commit. I am attaching the patch so you can
test to see if it fixes it. If it does, please let us know.

! SHLIB_LINK := $(filter -L%, $(LDFLAGS)) $(SHLIB_LINK)
...
! SHLIB_LINK := $(SHLIB_LINK) $(filter -L%, $(LDFLAGS))

Urgh. This was considered a good idea why exactly?

regards, tom lane

Tom Lane wrote:

It appears that somebody has changed things so that the -L switches
appear after the -l switches (ie, too late). I'm too tired to
investigate now, but my money is on Autoconf 2.59 being the problem ...

Perhaps this:
http://archives.postgresql.org/pgsql-hackers/2005-07/msg00085.php

-O

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Tom Lane wrote:

It appears that somebody has changed things so that the -L switches
appear after the -l switches (ie, too late). I'm too tired to
investigate now, but my money is on Autoconf 2.59 being the problem ...

I wonder if it was this commit. I am attaching the patch so you can
test to see if it fixes it. If it does, please let us know.

! SHLIB_LINK := $(filter -L%, $(LDFLAGS)) $(SHLIB_LINK)
...
! SHLIB_LINK := $(SHLIB_LINK) $(filter -L%, $(LDFLAGS))

Urgh. This was considered a good idea why exactly?

I didn't like it myself but it fixed a problem with a build farm
machine, and no one objected, so I applied it, but it seemed pretty
strange to be reversing those. I will reverse the patch.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian said:

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Tom Lane wrote:

It appears that somebody has changed things so that the -L switches
appear after the -l switches (ie, too late). I'm too tired to
investigate now, but my money is on Autoconf 2.59 being the problem
...

I wonder if it was this commit. I am attaching the patch so you can
test to see if it fixes it. If it does, please let us know.

! SHLIB_LINK := $(filter -L%, $(LDFLAGS)) $(SHLIB_LINK)
...
! SHLIB_LINK := $(SHLIB_LINK) $(filter -L%, $(LDFLAGS))

Urgh. This was considered a good idea why exactly?

I didn't like it myself but it fixed a problem with a build farm
machine, and no one objected, so I applied it, but it seemed pretty
strange to be reversing those. I will reverse the patch.

I was also slightly dubious about it. However, we do still need to solve the
problem that the patch addressed. Buildfarm members platypus and cuckoo are
currently failing because dblink is picking up the wrong libpq (and it
appears that incorrect libraries are also being picked up in the ecpg
libraries, although this isn't causing a buildfarm failure.)

Alternatively, if we can't say --with-libraries=/foo/bar when /foo/bar
contains possibly conflicting libraries, that should be tested for at
configure time.

cheers

andrew

Andrew Dunstan wrote:

I was also slightly dubious about it. However, we do still need to
solve the problem that the patch addressed. Buildfarm members
platypus and cuckoo are currently failing because dblink is picking
up the wrong libpq (and it appears that incorrect libraries are also
being picked up in the ecpg libraries, although this isn't causing a
buildfarm failure.)

We have four pieces of information when linking a shared library:

B: in-tree libraries that we might need (in case of ecpglib: libpq)
A: path to those in-tree libraries
D: external libraries that we might need (in case of ecpglib in my case:
-lcrypt -lm)
C: path to those external libraries (e.g., /usr/local/lib)

On the linker command line, we need this information in one of the
following two orders:

A B C D
A C B D

The Makefile.shlib receives from the respective main makefile "A B D" in
SHLIB_LINK and would have to insert "C" in the middle somewhere.
Currently, the actual behavior is "C A B D" and the failed patch wanted
to do "A B D C", both of which are wrong.

So either we code up some intelligence to put the "C" in the right
position or we have to pass down "A B" and "D" separately from the main
makefile.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

On Mon, Jul 04, 2005 at 04:14:51PM +1200, Oliver Jowett wrote:

Tom Lane wrote:

It appears that somebody has changed things so that the -L switches
appear after the -l switches (ie, too late). I'm too tired to
investigate now, but my money is on Autoconf 2.59 being the problem ...

Perhaps this:
http://archives.postgresql.org/pgsql-hackers/2005-07/msg00085.php

Would be my guess as well...

So should includes/linking for PostgreSQL-built libs be handled
seperately? The original problem was that if --with-libraries was
specified then libpq etc would be linked from the system locations
instead of from within the build.
--
Jim C. Nasby, Database Consultant decibel@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"

Jim C. Nasby wrote:

On Mon, Jul 04, 2005 at 04:14:51PM +1200, Oliver Jowett wrote:

Tom Lane wrote:

It appears that somebody has changed things so that the -L switches
appear after the -l switches (ie, too late). I'm too tired to
investigate now, but my money is on Autoconf 2.59 being the problem ...

Perhaps this:
http://archives.postgresql.org/pgsql-hackers/2005-07/msg00085.php

Would be my guess as well...

So should includes/linking for PostgreSQL-built libs be handled
seperately? The original problem was that if --with-libraries was
specified then libpq etc would be linked from the system locations
instead of from within the build.

Yes, I think so. Peter Eisentraut's analysis strikes me as being correct.

cheers

andrew

On Mon, Jul 04, 2005 at 05:58:27PM +0200, Peter Eisentraut wrote:

Andrew Dunstan wrote:

I was also slightly dubious about it. However, we do still need to
solve the problem that the patch addressed. Buildfarm members
platypus and cuckoo are currently failing because dblink is picking
up the wrong libpq (and it appears that incorrect libraries are also
being picked up in the ecpg libraries, although this isn't causing a
buildfarm failure.)

We have four pieces of information when linking a shared library:

B: in-tree libraries that we might need (in case of ecpglib: libpq)
A: path to those in-tree libraries

Is A even represented in the build at all right now? ISTM it's not, so
simply adding it in front of C might suffice. What would be a reasonable
way to add that to the makefiles?

D: external libraries that we might need (in case of ecpglib in my case:
-lcrypt -lm)
C: path to those external libraries (e.g., /usr/local/lib)

On the linker command line, we need this information in one of the
following two orders:

A B C D
A C B D

The Makefile.shlib receives from the respective main makefile "A B D" in
SHLIB_LINK and would have to insert "C" in the middle somewhere.
Currently, the actual behavior is "C A B D" and the failed patch wanted
to do "A B D C", both of which are wrong.

So either we code up some intelligence to put the "C" in the right
position or we have to pass down "A B" and "D" separately from the main
makefile.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

--
Jim C. Nasby, Database Consultant decibel@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"

Jim C. Nasby wrote:

B: in-tree libraries that we might need (in case of ecpglib: libpq)
A: path to those in-tree libraries

Is A even represented in the build at all right now? ISTM it's not,

Sure it is. How else would anything like psql and pg_dump find libpq?

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

I wrote:

So either we code up some intelligence to put the "C" in the right
position or we have to pass down "A B" and "D" separately from the
main makefile.

The following patch might just do the former. Please try it out.

diff -ur ../cvs-pgsql/src/Makefile.shlib ./src/Makefile.shlib
--- ../cvs-pgsql/src/Makefile.shlib     2005-07-04 16:32:57.000000000 +0200
+++ ./src/Makefile.shlib        2005-07-05 22:02:10.556162778 +0200
@@ -240,7 +240,7 @@
   SHLIB_LINK           += -ltermcap -lstdc++.r4 -lbind -lsocket -L/boot/develop/lib/x86
 endif

-SHLIB_LINK := $(filter -L%, $(LDFLAGS)) $(SHLIB_LINK)
+SHLIB_LINK := $(filter -L%, $(SHLIB_LINK)) $(filter -L%, $(LDFLAGS)) $(filter-out -L%, $(SHLIB_LINK))
 ifeq ($(enable_rpath), yes)
 SHLIB_LINK += $(rpath)
 endif

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

On Tue, Jul 05, 2005 at 10:09:19PM +0200, Peter Eisentraut wrote:

I wrote:

So either we code up some intelligence to put the "C" in the right
position or we have to pass down "A B" and "D" separately from the
main makefile.

The following patch might just do the former. Please try it out.

diff -ur ../cvs-pgsql/src/Makefile.shlib ./src/Makefile.shlib
--- ../cvs-pgsql/src/Makefile.shlib     2005-07-04 16:32:57.000000000 +0200
+++ ./src/Makefile.shlib        2005-07-05 22:02:10.556162778 +0200
@@ -240,7 +240,7 @@
SHLIB_LINK           += -ltermcap -lstdc++.r4 -lbind -lsocket -L/boot/develop/lib/x86
endif

-SHLIB_LINK := $(filter -L%, $(LDFLAGS)) $(SHLIB_LINK)
+SHLIB_LINK := $(filter -L%, $(SHLIB_LINK)) $(filter -L%, $(LDFLAGS)) $(filter-out -L%, $(SHLIB_LINK))
ifeq ($(enable_rpath), yes)
SHLIB_LINK += $(rpath)
endif

Worked on platypus:
http://pgbuildfarm.org/cgi-bin/show_log.pl?nm=platypus&amp;dt=2005-07-05%2022:03:35
--
Jim C. Nasby, Database Consultant decibel@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"

Could we please get this patch applied? It seems like the right thing to do.

cheers

andrew

Jim C. Nasby wrote:

Patch applied. Thanks.

---------------------------------------------------------------------------

Peter Eisentraut wrote:

I wrote:

So either we code up some intelligence to put the "C" in the right
position or we have to pass down "A B" and "D" separately from the
main makefile.

The following patch might just do the former. Please try it out.

diff -ur ../cvs-pgsql/src/Makefile.shlib ./src/Makefile.shlib
--- ../cvs-pgsql/src/Makefile.shlib     2005-07-04 16:32:57.000000000 +0200
+++ ./src/Makefile.shlib        2005-07-05 22:02:10.556162778 +0200
@@ -240,7 +240,7 @@
SHLIB_LINK           += -ltermcap -lstdc++.r4 -lbind -lsocket -L/boot/develop/lib/x86
endif

-SHLIB_LINK := $(filter -L%, $(LDFLAGS)) $(SHLIB_LINK)
+SHLIB_LINK := $(filter -L%, $(SHLIB_LINK)) $(filter -L%, $(LDFLAGS)) $(filter-out -L%, $(SHLIB_LINK))
ifeq ($(enable_rpath), yes)
SHLIB_LINK += $(rpath)
endif

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

This patch seems to have broken builds on Windows and other boxes (e.g.
buildfarm's octopus, a FreeBSD box). Maybe this should be reverted until
we find a more robust solution :-(

cheers

andrew

Bruce Momjian wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

This patch seems to have broken builds on Windows and other boxes (e.g.
buildfarm's octopus, a FreeBSD box). Maybe this should be reverted until
we find a more robust solution :-(

The only thing I see any evidence for is a broken version of gmake on
octopus.

gmake[3]: Entering directory `/raid0/buildfarm/buildfarm/HEAD/pgsql.54583/src/backend/utils/mb/conversion_procs/ascii_and_mic'
../../../../../../src/Makefile.shlib:250: *** missing separator. Stop.
gmake[3]: Leaving directory `/raid0/buildfarm/buildfarm/HEAD/pgsql.54583/src/backend/utils/mb/conversion_procs/ascii_and_mic'
gmake[2]: *** [all] Error 2

If there were a genuine syntax error in that command, we'd all be seeing
this.

What gmake version is octopus using, anyway?

regards, tom lane

Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

This patch seems to have broken builds on Windows and other boxes (e.g.
buildfarm's octopus, a FreeBSD box). Maybe this should be reverted until
we find a more robust solution :-(

The only thing I see any evidence for is a broken version of gmake on
octopus.

gmake[3]: Entering directory `/raid0/buildfarm/buildfarm/HEAD/pgsql.54583/src/backend/utils/mb/conversion_procs/ascii_and_mic'
../../../../../../src/Makefile.shlib:250: *** missing separator. Stop.
gmake[3]: Leaving directory `/raid0/buildfarm/buildfarm/HEAD/pgsql.54583/src/backend/utils/mb/conversion_procs/ascii_and_mic'
gmake[2]: *** [all] Error 2

If there were a genuine syntax error in that command, we'd all be seeing
this.

What gmake version is octopus using, anyway?

I wondered about that. Certainly the compiler is very old indeed.

Jim?

Meanwhile, we are now choking on building plperl for Windows, at least
with the ActiveState perl port, where we were not before.
Makefile.global gets these settings:

PERL = "/c/perl/bin//perl"
perl_archlibexp = C:\Perl\lib
perl_privlibexp = C:\Perl\lib
perl_useshrplib = yes
perl_embed_ldflags = -nologo -nodefaultlib -debug -opt:ref,icf
-libpath:"C:\Perl\lib\CORE" -machine:x86 C:\Perl\lib\CORE\perl58.lib

and we see this error:

dllwrap -o libplperl.dll --dllname libplperl.dll --def plperl.def plperl.o spi_internal.o SPI.o -L -L../../../src/backend -L../../../src/port -L/c/tcl/lib C:/Perl/lib/CORE -lperl58 -lpostgres
c:\mingw\bin\..\lib\gcc-lib\mingw32\3.2.3\..\..\..\..\mingw32\bin\ld.exe: cannot open C:/Perl/lib/CORE: Permission denied
c:\mingw\bin\dllwrap.exe: c:\mingw\bin\gcc exited with status 1

which looks very odd indeed, especially:

-L -L../../../src/backend -L../../../src/port -L/c/tcl/lib C:/Perl/lib/CORE -lperl58

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

which looks very odd indeed, especially:

-L -L../../../src/backend -L../../../src/port -L/c/tcl/lib C:/Perl/lib/CORE -lperl58

Ah, I see the problem:

ifeq ($(PORTNAME), win32)
perl_archlibexp := $(subst \,/,$(perl_archlibexp))
perl_privlibexp := $(subst \,/,$(perl_privlibexp))
perl_embed_ldflags := -L $(perl_archlibexp)/CORE -lperl58
^^^^^^^^^^^^^^^^^^^^^^^^^^
override CPPFLAGS += -DPLPERL_HAVE_UID_GID
endif

The filter hack depends on -L and the following argument to not be
space-separated. We made the no-space assumption before for -L in
LDFLAGS, but not for -L in SHLIB_LINK.

I've removed the space in CVS tip, we'll see where that takes us.

regards, tom lane

Jim C. Nasby wrote:

Turns out there was a cvs conflict. Doh!

Ouch. I have repeatedly warned buildfarm owners not to make any changes
or run builds in buildfarm's local CVS repo. Use a copy if necessary.

Hmm... would probably be a good idea to have the script check for
conflicts and throw a CVS error if it finds any. Here's what CVS does on
a conflict:

RCS file: /projects/cvsroot/pgsql/src/Makefile.shlib,v
retrieving revision 1.93
retrieving revision 1.94
Merging differences between 1.93 and 1.94 into Makefile.shlib
rcsmerge: warning: conflicts during merge
cvs update: conflicts found in Makefile.shlib
C Makefile.shlib

It also creates a .# file. I'm not really sure what the best way to test
for this would be.

I can look for it. It's a pity that cvs doesn't exit with a non-zero
status when it does this. I am assuming we don't ever have such files in
CVS.

BTW, it would probably be handy to have logs available for all steps of
the build, even if they succeed.

They are available locally in the directory lastrun-logs. I made a
deliberate decision not to clog up the server with them. At roughly 1Mb
per build it seemed quite unnecessary - I wanted to keep traffic down
and server storage requirements modest.

In any case, I've cleared the conflict and I'm running a build right
now.

OK, cool.

cheers

andrew