static libpq (and other libraries) overwritten on aix

On Wed, Aug 17, 2022 at 3:02 PM Andres Freund <andres@anarazel.de> wrote:

2) Do we care about static libraries not suriving on AIX? There could also be
a race in the buildrules leading to sometimes static libs sometimes shared
libs winning, I think.

Instead of overwriting the same file, can we not use different
filenames for different things?

--
Robert Haas
EDB: http://www.enterprisedb.com

Hi,

On 2022-08-17 15:28:18 -0400, Robert Haas wrote:

On Wed, Aug 17, 2022 at 3:02 PM Andres Freund <andres@anarazel.de> wrote:

2) Do we care about static libraries not suriving on AIX? There could also be
a race in the buildrules leading to sometimes static libs sometimes shared
libs winning, I think.

Instead of overwriting the same file, can we not use different
filenames for different things?

Not easily, as far as I understand. The way one customarily links to shared
libraries on aix is to have an .a archive containing the shared library. That
way the -lpq picks up libpq.a, which then triggers the shared library to be
referenced.

E.g.
andres@gcc119:[/home/andres/src/postgres/build-ac]$ LIBPATH=$(pwd)/src/interfaces/libpq ldd src/bin/scripts/clusterdb
src/bin/scripts/clusterdb needs:
/usr/lib/libc.a(shr_64.o)
/usr/lib/libpthread.a(shr_xpg5_64.o)
/usr/lib/libreadline.a(libreadline.so.6)
/home/andres/src/postgres/build-ac/src/interfaces/libpq/libpq.a(libpq.so.5)
/unix
/usr/lib/libcrypt.a(shr_64.o)
/usr/lib/libcurses.a(shr42_64.o)
/usr/lib/libpthreads.a(shr_xpg5_64.o)

Note the .a(libpq.so.5) bit.

Unfortunately that's exactly how one links to a static library as well.

So we'd have to change the name used as -l$this between linking to a shared
libpq and a static libpq.

Greetings,

Andres Freund

The AIX sections of Makefile.shlib misuse the terms "static" and "shared".
Imagine s/static library/library ending in .a/ and s/shared library/library
ending in .so/. That yields an accurate description of the makefile rules.

On Wed, Aug 17, 2022 at 12:01:54PM -0700, Andres Freund wrote:

Two questions:
1) Do we continue building static libraries for libpq etc?

Essentially, we don't build static libpq today, and we should continue not
building it. (The first-built libpq.a is static, but that file is an
implementation detail of the makefile rules. The surviving libpq.a is a
normal AIX shared library.)

2) Do we care about static libraries not suriving on AIX?

No.

There could also be
a race in the buildrules leading to sometimes static libs sometimes shared
libs winning, I think.

Not since commit e8564ef, to my knowledge.

Along the lines of Robert's comment, it could be a nice code beautification to
use a different suffix for the short-lived .a file. Perhaps _so_inputs.a.

I found this useful years ago:
https://web.archive.org/web/20151003130212/http://seriousbirder.com/blogs/aix-shared-and-static-libraries-explained/

On Thu, Aug 18, 2022 at 12:59 AM Noah Misch <noah@leadboat.com> wrote:

Along the lines of Robert's comment, it could be a nice code beautification to
use a different suffix for the short-lived .a file. Perhaps _so_inputs.a.

Yeah, this is the kind of thing I was thinking about.

--
Robert Haas
EDB: http://www.enterprisedb.com

On 2022-08-18 Th 10:10, Robert Haas wrote:

On Thu, Aug 18, 2022 at 12:59 AM Noah Misch <noah@leadboat.com> wrote:

Along the lines of Robert's comment, it could be a nice code beautification to
use a different suffix for the short-lived .a file. Perhaps _so_inputs.a.

Yeah, this is the kind of thing I was thinking about.

+1 for that and clarifying Makefile.shlib.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 2022-08-17 21:59:29 -0700, Noah Misch wrote:

The AIX sections of Makefile.shlib misuse the terms "static" and "shared".

Imagine s/static library/library ending in .a/ and s/shared library/library
ending in .so/. That yields an accurate description of the makefile rules.

I realize that aspect.

My point is that we currently, across most of the other platforms, support
building a "proper" static library, and install it too. But on AIX (and I
think mingw), we don't, but without an explicit comment about not doing so. In
fact, the all-static-lib target on those platforms will build a non-static
library, which seems not great.

There could also be
a race in the buildrules leading to sometimes static libs sometimes shared
libs winning, I think.

Not since commit e8564ef, to my knowledge.

I'd missed that the $(stlib): ... bit is not defined due to haslibarule being
defined...

Along the lines of Robert's comment, it could be a nice code beautification to
use a different suffix for the short-lived .a file. Perhaps _so_inputs.a.

Agreed, it'd be an improvement.

Afaict we could just stop building the intermediary static lib. Afaict the
MKLDEXPORT path isn't needed for libraries without an exports.txt because the
linker defaults to exporting "most" symbols, and for symbols with an
exports.txt we don't need it either.

The only path that really needs MKLDEXPORT is postgres. Not really for the
export side either, but for the import side.

Greetings,

Andres Freund

On Thu, Aug 18, 2022 at 09:03:57AM -0700, Andres Freund wrote:

My point is that we currently, across most of the other platforms, support
building a "proper" static library, and install it too. But on AIX (and I
think mingw), we don't, but without an explicit comment about not doing so. In
fact, the all-static-lib target on those platforms will build a non-static
library, which seems not great.

Yep. If someone had just pushed a correct patch to make AIX match our
GNU/Linux static linking assistance, I wouldn't be arguing to revert that
patch. At the same time, if someone asks me to choose high-value projects for
20 people, doing more for static linking on AIX won't be on the list.

On 2022-08-17 21:59:29 -0700, Noah Misch wrote:

Along the lines of Robert's comment, it could be a nice code beautification to
use a different suffix for the short-lived .a file. Perhaps _so_inputs.a.

Agreed, it'd be an improvement.

Afaict we could just stop building the intermediary static lib. Afaict the
MKLDEXPORT path isn't needed for libraries without an exports.txt because the
linker defaults to exporting "most" symbols

If that works, great.

Hi,

On 2022-08-18 22:56:43 -0700, Noah Misch wrote:

On 2022-08-17 21:59:29 -0700, Noah Misch wrote:

Along the lines of Robert's comment, it could be a nice code beautification to
use a different suffix for the short-lived .a file. Perhaps _so_inputs.a.

Agreed, it'd be an improvement.

Afaict we could just stop building the intermediary static lib. Afaict the
MKLDEXPORT path isn't needed for libraries without an exports.txt because the
linker defaults to exporting "most" symbols

If that works, great.

I looked at that. It's not too hard to make it work. But while doing so I
encountered some funny bits.

As far as I can tell the way we build shared libraries on aix with gcc isn't
correct:

Without -shared gcc won't know that it's building a shared library, which
afaict will prevent gcc from generating correct unwind info and we end up with
a statically linked copy of libgcc each time.

The naive thing of just adding -shared fails, but that's our fault:

ldd pgoutput.so
pgoutput.so needs:
Cannot find libgcc_s.a(shr.o)
/usr/lib/libc.a(shr_64.o)
/unix
/usr/lib/libcrypt.a(shr_64.o)

Makefile.aix has:
# -blibpath must contain ALL directories where we should look for libraries
libpath := $(shell echo $(subst -L,:,$(filter -L/%,$(LDFLAGS))) | sed -e's/ //g'):/usr/lib:/lib

but that's insufficient for gcc, because it won't find gcc's runtime lib. We
could force a build of the statically linked libgcc, but once it knows it's
generating with a shared library, a static libgcc unfortunately blows up the
size of the output considerably.

So I think we need something like

ifeq ($(GCC), yes)
libpath := $(libpath):$(dir $(shell gcc -print-libgcc-file-name))
endif

although deferring the computation of that would be nicer, but would require
some cleanup before.

With that libraries do shrink a bit. E.g. cube.so goes from 140k to 96k.

Afaict there's no reason to generate lib<name>.a for extension .so's, right?

We have plenty of detritus that's vaguely AIX related. The common.mk rule to
generate SUBSYS.o isn't used (mea culpa), and backend/Makefile's postgres.o
rule hasn't been used for well over 20 years.

I'll send in a patch series tomorrow, too tired for today.

Greetings,

Andres Freund

On Sat, Aug 20, 2022 at 01:35:22AM -0700, Andres Freund wrote:

Afaict there's no reason to generate lib<name>.a for extension .so's, right?

Right. We install cube.so, not any *cube.a file.

Hi,

On 2022-08-20 01:35:22 -0700, Andres Freund wrote:

I'll send in a patch series tomorrow, too tired for today.

Here it goes.

0001 aix: Fix SHLIB_EXPORTS reference in VPATH builds

That's mostly so I could even build. It's not quite right in the sense that
we don't depend on the file, but that's a preexisting issue. Could be folded
in with 0005, which fixes that aspect. Or it could be backpatched as the
minimal fix.

0002 Remove SUBSYS.o rule in common.mk, hasn't been used in a long time
0003 Remove rule to generate postgres.o, not needed for 20+ years

Both obvious, I think.

0004 aix: when building with gcc, tell gcc we're building a shared library

That's the gcc -shared issue I explained in the email I'm replying to.

We should probably consider building executables with -shared-libgcc too,
that shrinks them a decent amount (e.g. 1371684 -> 1126765 for psql). But
I've not done that here.

0005 aix: No need to use mkldexport when we want to export all symbols

This makes the building of shared libraries a lot more similar to other
platforms. Export files are only used when an exports.txt is present and
there's no more intermediary static libraries.

0006 configure: Expand -fvisibility checks to more compilers, add -qvisibility

This isn't strictly speaking part of the same "thread" of work, but I don't
want to touch aix more often than I have too... I'll post it in the other
thread too.

I did just test that this passes at least some tests on aix with xlc and
solaris with sunpro.

Greetings,

Andres

Hi,

On 2022-08-20 10:42:13 -0700, Andres Freund wrote:

On 2022-08-20 01:35:22 -0700, Andres Freund wrote:

I'll send in a patch series tomorrow, too tired for today.

Here it goes.

0001 aix: Fix SHLIB_EXPORTS reference in VPATH builds

That's mostly so I could even build. It's not quite right in the sense that
we don't depend on the file, but that's a preexisting issue. Could be folded
in with 0005, which fixes that aspect. Or it could be backpatched as the
minimal fix.

0002 Remove SUBSYS.o rule in common.mk, hasn't been used in a long time
0003 Remove rule to generate postgres.o, not needed for 20+ years

Both obvious, I think.

Pushed these, given that they're all pretty trivial.

0004 aix: when building with gcc, tell gcc we're building a shared library

That's the gcc -shared issue I explained in the email I'm replying to.

We should probably consider building executables with -shared-libgcc too,
that shrinks them a decent amount (e.g. 1371684 -> 1126765 for psql). But
I've not done that here.

0005 aix: No need to use mkldexport when we want to export all symbols

This makes the building of shared libraries a lot more similar to other
platforms. Export files are only used when an exports.txt is present and
there's no more intermediary static libraries.

0006 configure: Expand -fvisibility checks to more compilers, add -qvisibility

This isn't strictly speaking part of the same "thread" of work, but I don't
want to touch aix more often than I have too... I'll post it in the other
thread too.

I did just test that this passes at least some tests on aix with xlc and
solaris with sunpro.

Any comments here?

Greetings,

Andres Freund

On Wed, Aug 24, 2022 at 08:43:04PM -0700, Andres Freund wrote:

On 2022-08-20 10:42:13 -0700, Andres Freund wrote:

0004 aix: when building with gcc, tell gcc we're building a shared library

That's the gcc -shared issue I explained in the email I'm replying to.

We should probably consider building executables with -shared-libgcc too,
that shrinks them a decent amount (e.g. 1371684 -> 1126765 for psql). But
I've not done that here.

0005 aix: No need to use mkldexport when we want to export all symbols

This makes the building of shared libraries a lot more similar to other
platforms. Export files are only used when an exports.txt is present and
there's no more intermediary static libraries.

0006 configure: Expand -fvisibility checks to more compilers, add -qvisibility

This isn't strictly speaking part of the same "thread" of work, but I don't
want to touch aix more often than I have too... I'll post it in the other
thread too.

I did just test that this passes at least some tests on aix with xlc and
solaris with sunpro.

Any comments here?

I don't know much about them, but they sound like the sort of thing that can't
cause subtle bugs. If they build and test the first time, they're probably
valid. You may as well push them.