Fix linking of OpenLDAP libraries

"Albe Laurenz" <all@adv.magwien.gv.at> writes:

# The backend doesn't need everything that's in LIBS, however
! LIBS := $(filter-out -lz -lreadline -ledit -ltermcap -lncurses -lcurses -lldap_r $(PTHREAD_LIBS), $(LIBS))

This seems pretty risky. What if PTHREAD_LIBS contains -L switches?
They'd get removed even if needed for other libraries.

It would probably be safer not to put LDAP into LIBS at all, but invent
two new macros for configure to set, say LDAP_LIBS and LDAP_LIBS_R,
and add these to the link lines in the backend and libpq respectively.

regards, tom lane

Tom Lane wrote on September 04, 2006:

"Albe Laurenz" <all@adv.magwien.gv.at> writes:

# The backend doesn't need everything that's in LIBS, however
! LIBS := $(filter-out -lz -lreadline -ledit -ltermcap -lncurses

-lcurses -lldap_r $(PTHREAD_LIBS), $(LIBS))

This seems pretty risky. What if PTHREAD_LIBS contains -L switches?
They'd get removed even if needed for other libraries.

It would probably be safer not to put LDAP into LIBS at all, but

invent

two new macros for configure to set, say LDAP_LIBS and LDAP_LIBS_R,
and add these to the link lines in the backend and libpq respectively.

That seems like a good idea.
I'll try to work it out and resubmit the fix.

Yours,
Laurenz Albe

Tom Lane wrote:

I have realized that my modifications in configure.in and
src/interfaces/libpq/Makefile to link libpq against
OpenLDAP are buggy.

Here is a proposed patch to fix it.

[...]

# The backend doesn't need everything that's in LIBS, however
! LIBS := $(filter-out -lz -lreadline -ledit -ltermcap
-lncurses -lcurses -lldap_r $(PTHREAD_LIBS), $(LIBS))

This seems pretty risky. What if PTHREAD_LIBS contains -L switches?
They'd get removed even if needed for other libraries.

It would probably be safer not to put LDAP into LIBS at all,
but invent two new macros for configure to set, say LDAP_LIBS
and LDAP_LIBS_R, and add these to the link lines in the backend
and libpq respectively.

Here is a new patch that replaces the previous one; it adds two
macros LDAP_LIBS_FE and LDAP_LIBS_BE for frontend and backend,
respectively.

I did not only add them to the Makefile for interfaces/libpq,
but also everywhere something is linked against libpq in case
somebody links static.
I don't know if that's necessary, or if static builds are
supported - if not, the changes to those Makefiles should
perhaps not be applied.

Tested on Linux, AIX and Windows.

Yours,
Laurenz Albe

"Albe Laurenz" <all@adv.magwien.gv.at> writes:

I did not only add them to the Makefile for interfaces/libpq,
but also everywhere something is linked against libpq in case
somebody links static.

I intensely dislike that part of the patch, but will work on applying
the rest.

If we do need to start mentioning all of libpq's dependencies everywhere
it's linked, I think it's time for a generic solution to that, instead
of hacking each such place over again every time a new dependency pops up.
But at the moment I'm unconvinced that we need to do it.

regards, tom lane

On Fri, Sep 08, 2006 at 03:20:00PM -0400, Tom Lane wrote:

"Albe Laurenz" <all@adv.magwien.gv.at> writes:

I did not only add them to the Makefile for interfaces/libpq,
but also everywhere something is linked against libpq in case
somebody links static.

If we do need to start mentioning all of libpq's dependencies everywhere
it's linked, I think it's time for a generic solution to that, instead
of hacking each such place over again every time a new dependency pops up.

The business of having to include every single dependancy when linking
static is quite irritating. It has almost reached the point where
people are just giving up static linking because it's too much of a
pain.

However, if we do want to support it, the way you do it is by extending
pg_config to do something like:

pg_config --dynamic-lick => returns -lpq
pg_config --static-link => returns -lpq <plus other libs>

That way only people who actually want static linking need be bothered.

Have a ncie day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

"Albe Laurenz" <all@adv.magwien.gv.at> writes:

Here is a new patch that replaces the previous one; it adds two
macros LDAP_LIBS_FE and LDAP_LIBS_BE for frontend and backend,
respectively.

I did not only add them to the Makefile for interfaces/libpq,
but also everywhere something is linked against libpq in case
somebody links static.

Applied, but without that last part. It builds OK for me on Darwin,
which is moderately picky about that sort of thing, but someone should
try AIX.

regards, tom lane

tgl@sss.pgh.pa.us (Tom Lane) wrote:

"Albe Laurenz" <all@adv.magwien.gv.at> writes:

Here is a new patch that replaces the previous one; it adds two
macros LDAP_LIBS_FE and LDAP_LIBS_BE for frontend and backend,
respectively.

I did not only add them to the Makefile for interfaces/libpq,
but also everywhere something is linked against libpq in case
somebody links static.

Applied, but without that last part. It builds OK for me on Darwin,
which is moderately picky about that sort of thing, but someone should
try AIX.

I'd like to, but that has to take second place to Slony-I activity
next week; I'll see about putting that on my ToDo list. (After SSL,
readline...)
--
output = ("cbbrowne" "@" "gmail.com")
http://linuxdatabases.info/info/emacs.html
Send messages calling for fonts not available to the recipient(s).
This can (in the case of Zmail) totally disable the user's machine and
mail system for up to a whole day in some circumstances.
-- from the Symbolics Guidelines for Sending Mail

Here is a new patch that replaces the previous one; it adds two
macros LDAP_LIBS_FE and LDAP_LIBS_BE for frontend and backend,
respectively.

I did not only add them to the Makefile for interfaces/libpq,
but also everywhere something is linked against libpq in case
somebody links static.

Applied, but without that last part. It builds OK for me on Darwin,
which is moderately picky about that sort of thing, but someone should
try AIX.

I'll do that today.

Laurenz Albe

Tom Lane wrote:

Here is a new patch that replaces the previous one; it adds two
macros LDAP_LIBS_FE and LDAP_LIBS_BE for frontend and backend,
respectively.

I did not only add them to the Makefile for interfaces/libpq,
but also everywhere something is linked against libpq in case
somebody links static.

Applied, but without that last part. It builds OK for me on Darwin,
which is moderately picky about that sort of thing, but someone should
try AIX.

It builds fine on AIX 5.3 as long as you tell it to link with
libpq.so. Static builds against libpq.a will fail.

Should -lldap or -lldap_r be added to pg_config --libs?
If yes, which of them?

Yours,
Laurenz Albe

On Mon, Sep 11, 2006 at 12:13:29PM +0200, Albe Laurenz wrote:

Applied, but without that last part. It builds OK for me on Darwin,
which is moderately picky about that sort of thing, but someone should
try AIX.

It builds fine on AIX 5.3 as long as you tell it to link with
libpq.so. Static builds against libpq.a will fail.

Should -lldap or -lldap_r be added to pg_config --libs?
If yes, which of them?

Static links are going to require it on every platform, not just AIX.
The question do we want to ask is how easy do we want to make static
linking, because the same treatment will have to apply to -lssl,
-lcrypto, -lkrb5, -lk5crypto and quite possibly others. Do we really
want to go there?

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

"Albe Laurenz" <all@adv.magwien.gv.at> writes:

Tom Lane wrote:

Applied, but without that last part. It builds OK for me on Darwin,
which is moderately picky about that sort of thing, but someone should
try AIX.

It builds fine on AIX 5.3 as long as you tell it to link with
libpq.so. Static builds against libpq.a will fail.

Hm. We have been assuming that AIX's problem is that dynamic libraries
don't remember their dependencies properly, but maybe the real issue is
that it prefers static over dynamic libraries? If so, what we ought to
be doing is adding the prefer-dynamic-libraries switch to the default
LDFLAGS on that platform.

Should -lldap or -lldap_r be added to pg_config --libs?

You have a mistaken idea of the purpose of pg_config --libs. It exists
to record what LIBS was at build time, not more, not less. It is
certainly not intended as a guide to how to link libpq.

regards, tom lane

Martijn van Oosterhout <kleptog@svana.org> writes:

Static links are going to require it on every platform, not just AIX.
The question do we want to ask is how easy do we want to make static
linking, because the same treatment will have to apply to -lssl,
-lcrypto, -lkrb5, -lk5crypto and quite possibly others. Do we really
want to go there?

Well, we already have a solution for static linking within the PG build
environment, the question is do we wish to export it. Given the lack of
complaints to date, I think not. Do we really want to encourage anyone
to statically link libraries that don't belong to their project? It's
not only the build-time dependency hell, it's the difficulty of
installing library updates. There's a reason why dynamic link is the
default on all modern platforms.

regards, tom lane

Tom Lane wrote:

It builds fine on AIX 5.3 as long as you tell it to link with
libpq.so. Static builds against libpq.a will fail.

Hm. We have been assuming that AIX's problem is that dynamic

libraries

don't remember their dependencies properly, but maybe the real issue

is

that it prefers static over dynamic libraries? If so, what we ought

to

be doing is adding the prefer-dynamic-libraries switch to the default
LDFLAGS on that platform.

AIX shared libraries know their dependencies well, and you don't
have to specify them again when linking against the library.

Let me expand a little on some of the peculiarities of
shared libraries on AIX:

- A normal AIX shared library is called libXX.a
It is an 'ar' archive that contains the shared object(s).
- In static linking mode, the shared object will be
included in the executable. In dynamic linking mode
(the default) only a reference to the shared object is
included.
- AIX can also link against shared objects called
libXX.so (like other UNIXen).
- When it resolves a -lXX flag, the linker searches
the -L list for libXX.a and libXX.so.
It prefers libXX.a over libXX.so unless invoked
with the flag -brtl.

So the linker does not prefer static over dynamic libraries,
it just perfers libXX.a over libXX.so.

In our case, we have libpq.a and libpq.so in the same directory,
so unless you link with -brtl you will get a static link
(because libpq.a is a static library).

To illustrate, let me include the output of 'ldd psql' for
a dynamically linked PostgreSQL:

8.2/bin/psql needs:
/postgres/8.2/lib/libpq.so
/usr/lib/libpthread.a(shr_xpg5_64.o)
/usr/lib/libc.a(shr_64.o)
/usr/local/lib/libldap_r.a(libldap_r-2.3.so.0)
/unix
/usr/lib/libcrypt.a(shr_64.o)
/usr/lib/libs.a(shr_64.o)

See
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/c
om.ibm.aix.cmds/doc/aixcmds3/ld.htm

Should -brtl be added to src/template/aix?

Should -lldap or -lldap_r be added to pg_config --libs?

You have a mistaken idea of the purpose of pg_config --libs.
It exists to record what LIBS was at build time, not more,
not less. It is certainly not intended as a guide to how
to link libpq.

*nods* Thanks for the clarification.

Yours,
Laurenz Albe

"Albe Laurenz" <all@adv.magwien.gv.at> writes:

Let me expand a little on some of the peculiarities of
shared libraries on AIX:

- A normal AIX shared library is called libXX.a
It is an 'ar' archive that contains the shared object(s).

Ah, so the problem really boils down to funny naming conventions.
If they use ".a" for both shared and static libraries, how does anyone
tell the difference?

So the linker does not prefer static over dynamic libraries,
it just perfers libXX.a over libXX.so.

In our case, we have libpq.a and libpq.so in the same directory,
so unless you link with -brtl you will get a static link
(because libpq.a is a static library).

I wonder whether we ought to suppress building (or at least installing)
our .a libraries at all on AIX. Adding -btrl to LDFLAGS would help
within the context of our own build, but external clients that link
to libpq without saying that are going to get undesirable results.

I think there's a reasonable argument that by installing a .a file that
isn't a shared library, we are violating the platform's conventions.

Should -brtl be added to src/template/aix?

Sounds that way, but that'll only help for psql and other stuff built
within our build. Could you try this against CVS tip:

* add -brtl to LDFLAGS in the template
* Remove the AIX-specific hack on $(libpq) at lines 349-354 of
src/Makefile.global.in
* see if it configures and builds

regards, tom lane

Tom Lane wrote:

"Albe Laurenz" <all@adv.magwien.gv.at> writes:

Let me expand a little on some of the peculiarities of
shared libraries on AIX:

- A normal AIX shared library is called libXX.a
It is an 'ar' archive that contains the shared object(s).

Ah, so the problem really boils down to funny naming conventions.
If they use ".a" for both shared and static libraries, how does anyone
tell the difference?

It sounds to me like there is no difference. Notice how his example
ldd output shows dependencies on specific .o entries within the
various .a files that reside on the system, rather than on the .a
files as a whole. If those entries had been statically linked then
they wouldn't have shown up in the ldd output at all.

So the difference is no longer between static libraries and shared
libraries -- they're now just libraries. The only difference is how
you link to them.

What IBM has done here is very sensible, and is really what the other
Unixes should have done from the beginning: a library is just a
library, and what differs is how it's used.

--
Kevin Brown kevin@sysexperts.com

Kevin Brown wrote:

Let me expand a little on some of the peculiarities of
shared libraries on AIX:

- A normal AIX shared library is called libXX.a
It is an 'ar' archive that contains the shared object(s).

Ah, so the problem really boils down to funny naming conventions.
If they use ".a" for both shared and static libraries, how does

anyone

tell the difference?

It sounds to me like there is no difference. Notice how his example
ldd output shows dependencies on specific .o entries within the
various .a files that reside on the system, rather than on the .a
files as a whole. If those entries had been statically linked then
they wouldn't have shown up in the ldd output at all.

That is not entirely true.
The difference between a static and a shared library on AIX
is that the *.o files in a dynamic library are dynamic objects,
produced by the linker (what is called *.so in Linux), and the
*.o files in a static library are the output of the compiler
(what is called *.o in Linux).

What IS true is that you can do a static build against a dynamic
library. Against a static library you can only do static builds.

Yours,
Laurenz Albe

Tom Lane wrote:

In our case, we have libpq.a and libpq.so in the same directory,
so unless you link with -brtl you will get a static link
(because libpq.a is a static library).

I wonder whether we ought to suppress building (or at least

installing)

our .a libraries at all on AIX. Adding -btrl to LDFLAGS would help
within the context of our own build, but external clients that link
to libpq without saying that are going to get undesirable results.

I think there's a reasonable argument that by installing a .a file

that

isn't a shared library, we are violating the platform's conventions.

The natural way in AIX would be:
- Create libpq.so
- Create libpq.a by 'rm -f libpq.a; ar -rc libpq.a libpq.so'
- Install only libpq.a

For a static build on AIX, you have to specify all the libraries and
give the linker -bstatic and -bI:/lib/syscalls.exp

Should -brtl be added to src/template/aix?

Sounds that way, but that'll only help for psql and other stuff built
within our build. Could you try this against CVS tip:

* add -brtl to LDFLAGS in the template
* Remove the AIX-specific hack on $(libpq) at lines 349-354 of
src/Makefile.global.in
* see if it configures and builds

I have done that (see the attached patch) and it works fine.
I don't have the native AIX C compiler, so I could only test
it with gcc.

I have taken the liberty to modify the static link line
in Makefile.global.in to contain the LDAP libraries, I hope
that's appropriate.

Yours,
Laurenz Albe

"Albe Laurenz" <all@adv.magwien.gv.at> writes:

Tom Lane wrote:

I think there's a reasonable argument that by installing a .a file that
isn't a shared library, we are violating the platform's conventions.

The natural way in AIX would be:
- Create libpq.so
- Create libpq.a by 'rm -f libpq.a; ar -rc libpq.a libpq.so'
- Install only libpq.a

Hm. This seems possible with some moderate hacking on Makefile.shlib
(certainly it'd be no more invasive than the existing Windows-specific
platform variants). However, looking at what's already in
Makefile.shlib for AIX makes me doubt the above claim a bit, because
AFAICS libpq.so is produced from libpq.a on that platform. Is it
possible that the rules have changed across AIX versions, and that the
code in there now is needful for older versions?

Another issue with installing only .a is that there's no provision
for versioning in .a library names ... what happens to someone who
needs two generations of libpq on his machine?

regards, tom lane

Tom Lane wrote:

The natural way in AIX would be:
- Create libpq.so
- Create libpq.a by 'rm -f libpq.a; ar -rc libpq.a libpq.so'
- Install only libpq.a

Hm. This seems possible with some moderate hacking on Makefile.shlib
(certainly it'd be no more invasive than the existing Windows-specific
platform variants). However, looking at what's already in
Makefile.shlib for AIX makes me doubt the above claim a bit, because
AFAICS libpq.so is produced from libpq.a on that platform. Is it
possible that the rules have changed across AIX versions, and that the
code in there now is needful for older versions?

I don't think that this behaviour has changed. I remember it from
AIX 4.3.2.

Of course libpq.so is created from (the static) libpq.a.
But once you have the dynamic library, you can link statically
against it.

Another issue with installing only .a is that there's no provision
for versioning in .a library names ... what happens to someone who
needs two generations of libpq on his machine?

Use different directories and set LIBPATH?
I don't know if there is a canonical way to do that. I'll investigate.

Yours,
Laurenz Albe