pgsql: aix: No need to use mkldexport when we want to export all symbol

Andres Freund <andres@anarazel.de> writes:

aix: No need to use mkldexport when we want to export all symbols

hmm, early returns not good:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoverfly&amp;dt=2022-09-10%2003%3A31%3A29

The backend seems to build OK, but our first try to make a .so
falls over:

gmake[3]: Entering directory '/scratch/nm/farm/xlc64v16/HEAD/pgsql.build/src/backend/utils/mb/conversion_procs/cyrillic_and_mic'
/opt/IBM/xlc/16.1.0/bin/xlc_r -D_LARGE_FILES=1 -DRANDOMIZE_ALLOCATED_MEMORY -qnoansialias -g -O2 -qmaxmem=33554432 -qsuppress=1500-010:1506-995 -qsuppress=1506-010:1506-416:1506-450:1506-480:1506-481:1506-492:1506-944:1506-1264 -qinfo=all:nocnd:noeff:noext:nogot:noini:noord:nopar:noppc:norea:nouni:nouse -qinfo=nounset -I../../../../../../src/include -I/home/nm/sw/nopath/icu58.3-64/include -I/home/nm/sw/nopath/libxml2-64/include/libxml2 -I/home/nm/sw/nopath/uuid-64/include -I/home/nm/sw/nopath/openldap-64/include -I/home/nm/sw/nopath/icu58.3-64/include -I/home/nm/sw/nopath/libxml2-64/include -c -o cyrillic_and_mic.o cyrillic_and_mic.c
/opt/IBM/xlc/16.1.0/bin/xlc_r -D_LARGE_FILES=1 -DRANDOMIZE_ALLOCATED_MEMORY -qnoansialias -g -O2 -qmaxmem=33554432 -qsuppress=1500-010:1506-995 -qsuppress=1506-010:1506-416:1506-450:1506-480:1506-481:1506-492:1506-944:1506-1264 -qinfo=all:nocnd:noeff:noext:nogot:noini:noord:nopar:noppc:norea:nouni:nouse -qinfo=nounset -o cyrillic_and_mic.so cyrillic_and_mic.o -L../../../../../../src/port -L../../../../../../src/common -L/home/nm/sw/nopath/libxml2-64/lib -L/home/nm/sw/nopath/icu58.3-64/lib -L/home/nm/sw/nopath/uuid-64/lib -L/home/nm/sw/nopath/openldap-64/lib -L/home/nm/sw/nopath/icu58.3-64/lib -L/home/nm/sw/nopath/libxml2-64/lib -Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -Wl,-bI:../../../../../../src/backend/postgres.imp
ld: 0711-244 ERROR: No csects or exported symbols have been saved.
../../../../../../src/Makefile.shlib:279: recipe for target 'cyrillic_and_mic.so' failed
gmake[3]: *** [cyrillic_and_mic.so] Error 8

regards, tom lane

Hi,

On September 9, 2022 9:14:26 PM PDT, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andres Freund <andres@anarazel.de> writes:

aix: No need to use mkldexport when we want to export all symbols

hmm, early returns not good:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoverfly&amp;dt=2022-09-10%2003%3A31%3A29

The backend seems to build OK, but our first try to make a .so
falls over:

ld: 0711-244 ERROR: No csects or exported symbols have been saved.
../../../../../../src/Makefile.shlib:279: recipe for target 'cyrillic_and_mic.so' failed
gmake[3]: *** [cyrillic_and_mic.so] Error 8

I'd tested that maybe two weeks ago, so I'm a bit surprised. I'll try to reproduce.

Andres

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Hi,

On 2022-09-09 21:16:38 -0700, Andres Freund wrote:

On September 9, 2022 9:14:26 PM PDT, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andres Freund <andres@anarazel.de> writes:

aix: No need to use mkldexport when we want to export all symbols

hmm, early returns not good:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoverfly&amp;dt=2022-09-10%2003%3A31%3A29

The backend seems to build OK, but our first try to make a .so
falls over:

ld: 0711-244 ERROR: No csects or exported symbols have been saved.
../../../../../../src/Makefile.shlib:279: recipe for target 'cyrillic_and_mic.so' failed
gmake[3]: *** [cyrillic_and_mic.so] Error 8

I'd tested that maybe two weeks ago, so I'm a bit surprised. I'll try to reproduce.

A basic build with xlc 16.1 still succeeds building world-bin. Including
obviously cyrillic_and_mic.so

../configure PERL=perl64 --cache ../config-ac-xlc.cache --with-perl CC=/opt/IBM/xlc/16.1.0/bin/xlc_r CFLAGS="-qmaxmem=33554432 -qsuppress=1500-010:1506-995 -qsuppress=1506-010:1506-416:1506-450:1506-480:1506-481:1506-492:1506-944:1506-1264 -qinfo=all:nocnd:noeff:noext:nogot:noini:noord:nopar:noppc:norea:nouni:nouse -qinfo=nounset"

with OBJECT_MODE=64 previously set.

I'll try a build closer to Noah's configure options. Most likely candidate
seems --enable-debug.

Greetings,

Andres Freund

Hi,

On 2022-09-09 21:37:49 -0700, Andres Freund wrote:

On 2022-09-09 21:16:38 -0700, Andres Freund wrote:

On September 9, 2022 9:14:26 PM PDT, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andres Freund <andres@anarazel.de> writes:

aix: No need to use mkldexport when we want to export all symbols

hmm, early returns not good:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoverfly&amp;dt=2022-09-10%2003%3A31%3A29

The backend seems to build OK, but our first try to make a .so
falls over:

ld: 0711-244 ERROR: No csects or exported symbols have been saved.
../../../../../../src/Makefile.shlib:279: recipe for target 'cyrillic_and_mic.so' failed
gmake[3]: *** [cyrillic_and_mic.so] Error 8

I'd tested that maybe two weeks ago, so I'm a bit surprised. I'll try to reproduce.

A basic build with xlc 16.1 still succeeds building world-bin. Including
obviously cyrillic_and_mic.so

../configure PERL=perl64 --cache ../config-ac-xlc.cache --with-perl CC=/opt/IBM/xlc/16.1.0/bin/xlc_r CFLAGS="-qmaxmem=33554432 -qsuppress=1500-010:1506-995 -qsuppress=1506-010:1506-416:1506-450:1506-480:1506-481:1506-492:1506-944:1506-1264 -qinfo=all:nocnd:noeff:noext:nogot:noini:noord:nopar:noppc:norea:nouni:nouse -qinfo=nounset"

with OBJECT_MODE=64 previously set.

I'll try a build closer to Noah's configure options. Most likely candidate
seems --enable-debug.

Hm. In a new tree it doesn't work with HEAD. But it did previously survive a
gmake clean && gmake world-bin. I don't see why the configure cache or
existing buildtree should play a role here, given a make clean?

Either way, the symbol visibility patch from
/messages/by-id/20220901211935.k6v7mt3mjhnfbhnr@awork3.anarazel.de
repairs the build, even with a new build tree.

Looking at that patch I don't like this bit overly much:
+if test -n "${CFLAGS_SL_MODULE}"; then
+  AC_DEFINE([HAVE_VISIBILITY_ATTRIBUTE], 1,
+            [Define to 1 if your compiler knows the visibility("hidden") attribute.])
+fi

Seems too fragile, somebody could test for another attribute to add to
CFLAGS_SL_MODULE and it'd break. I'm inclined to just set a
pgac_have_visibility_attr in the two branches and set it from
pgac_cv_prog_CC_cflags__fvisibility_hidden
pgac_cv_prog_CC_cflags__qvisibility_hidden
?

It seem worth applying the -qvisibility patch and seeing whether that fixes
the buildfarm? I guess alternatively we could revert the mkldexport patch,
apply the -qvisibility patch and then apply the mkldexport thing again
afterwards?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:

It seem worth applying the -qvisibility patch and seeing whether that fixes
the buildfarm?

Worth a try.

I guess alternatively we could revert the mkldexport patch,
apply the -qvisibility patch and then apply the mkldexport thing again
afterwards?

Seems pretty academic, unless -qvisibility doesn't fix it.

regards, tom lane

Hi,

On 2022-09-10 01:32:52 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

It seem worth applying the -qvisibility patch and seeing whether that fixes
the buildfarm?

Worth a try.

Done now.

Looks like we also could get away with -Wl,-bexpall or such, but of course
selective exporting is preferrable. Just noting that in case an older xlc
version makes trouble.

Greetings,

Andres Freund

Hi,

On 2022-09-09 22:57:36 -0700, Andres Freund wrote:

On 2022-09-10 01:32:52 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

It seem worth applying the -qvisibility patch and seeing whether that fixes
the buildfarm?

Worth a try.

Done now.

Seems to have done the trick for at least hoverfly. And wrasse (sunpro) is
still happy. Still need to wait for the older xlc's (mandril, hornet)
though...

Greetings,

Andres Freund

Hi,

On 2022-09-10 01:19:44 -0700, Andres Freund wrote:

On 2022-09-09 22:57:36 -0700, Andres Freund wrote:

On 2022-09-10 01:32:52 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

It seem worth applying the -qvisibility patch and seeing whether that fixes
the buildfarm?

Worth a try.

Done now.

Seems to have done the trick for at least hoverfly. And wrasse (sunpro) is
still happy. Still need to wait for the older xlc's (mandril, hornet)
though...

At least hornet isn't happy:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hornet&amp;dt=2022-09-10%2021%3A44%3A00

Looks like the support for -qvisibility came with xlc 13.1, released April 28, 2014:
https://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&amp;infotype=an&amp;appname=iSource&amp;supplier=897&amp;letternum=ENUS214-162

-qvisibility option specifies visibility attributes for entities. Entity
visibility attributes describe whether and how entities defined in one
module can be referenced or used in other modules. Visibility attributes
affect entities with external linkage only, and cannot increase the
visibility of other entities.

Since xlc 13.1 supports all still supported AIX versions I'm inclined to think
we should increase our requirement to 13.1 rather than revert back to the
higher complexity way of building modules. We could also try to fall back to
using -Wl,b,expall for extension modules.

Greetings,

Andres Freund

On Sun, Sep 11, 2022 at 11:19:25AM -0700, Andres Freund wrote:

On 2022-09-10 01:19:44 -0700, Andres Freund wrote:

On 2022-09-09 22:57:36 -0700, Andres Freund wrote:

On 2022-09-10 01:32:52 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

It seem worth applying the -qvisibility patch and seeing whether that fixes
the buildfarm?

Worth a try.

Done now.

Seems to have done the trick for at least hoverfly. And wrasse (sunpro) is
still happy. Still need to wait for the older xlc's (mandril, hornet)
though...

At least hornet isn't happy:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hornet&amp;dt=2022-09-10%2021%3A44%3A00

Looks like the support for -qvisibility came with xlc 13.1, released April 28, 2014:
https://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&amp;infotype=an&amp;appname=iSource&amp;supplier=897&amp;letternum=ENUS214-162

Last two sungazer (gcc64) runs failed with a suspicious "missing magic block":
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&amp;dt=2022-09-12%2006%3A24%3A53
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&amp;dt=2022-09-10%2018%3A09%3A11

Both runs had the latest commits discussed above. tern (gcc32) passed, but it
doesn't use --with-perl.

-qvisibility option specifies visibility attributes for entities. Entity
visibility attributes describe whether and how entities defined in one
module can be referenced or used in other modules. Visibility attributes
affect entities with external linkage only, and cannot increase the
visibility of other entities.

Since xlc 13.1 supports all still supported AIX versions I'm inclined to think
we should increase our requirement to 13.1 rather than revert back to the
higher complexity way of building modules.

Perhaps. That would demolish two buildfarm members. What does PostgreSQL
lose if you revert fe6a64a?

We could also try to fall back to
using -Wl,b,expall for extension modules.

I have no knowledge of that option.

Hi,

On 2022-09-12 20:38:45 -0700, Noah Misch wrote:

On Sun, Sep 11, 2022 at 11:19:25AM -0700, Andres Freund wrote:

On 2022-09-10 01:19:44 -0700, Andres Freund wrote:

On 2022-09-09 22:57:36 -0700, Andres Freund wrote:

On 2022-09-10 01:32:52 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

It seem worth applying the -qvisibility patch and seeing whether that fixes
the buildfarm?

Worth a try.

Done now.

Seems to have done the trick for at least hoverfly. And wrasse (sunpro) is
still happy. Still need to wait for the older xlc's (mandril, hornet)
though...

At least hornet isn't happy:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hornet&amp;dt=2022-09-10%2021%3A44%3A00

Looks like the support for -qvisibility came with xlc 13.1, released April 28, 2014:
https://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&amp;infotype=an&amp;appname=iSource&amp;supplier=897&amp;letternum=ENUS214-162

Last two sungazer (gcc64) runs failed with a suspicious "missing magic block":
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&amp;dt=2022-09-12%2006%3A24%3A53
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&amp;dt=2022-09-10%2018%3A09%3A11

Both runs had the latest commits discussed above. tern (gcc32) passed, but it
doesn't use --with-perl.

I'll check it out tomorrow. I configured perl on the other AIX gcc compile
farm machine building 64bit with both gcc and xlc, IIRC. I can't check rn,
they seem to be down?

What is "perl64-for-gcc"?

-qvisibility option specifies visibility attributes for entities. Entity
visibility attributes describe whether and how entities defined in one
module can be referenced or used in other modules. Visibility attributes
affect entities with external linkage only, and cannot increase the
visibility of other entities.

Since xlc 13.1 supports all still supported AIX versions I'm inclined to think
we should increase our requirement to 13.1 rather than revert back to the
higher complexity way of building modules.

Perhaps. That would demolish two buildfarm members.

Or you could update them to a slightly newer xlc? IIRC the machine has 13.x
available?

What does PostgreSQL lose if you revert fe6a64a?

A somewhat readable Makefile.shlib. The prior state is complicated, specific
to AIX. I've several times spent quite a bit of time understanding it - not
helped by the complete lack of comments. It's not frequent enough to stay in
my brain unfortunately, so I'll have to relearn it again next time. And we'll
need to continue supporting Makefile.shlib for quite a while, so ...

I do not understand what value it has to run an ancient compiler on a very
niche, quite different, OS. I can see some value to supporting a larger number
of operating systems and compiles - but I can't see any sort of benefit in
also supporting old versions of those compilers.

Greetings,

Andres Freund

On Mon, Sep 12, 2022 at 11:11:23PM -0700, Andres Freund wrote:

On 2022-09-12 20:38:45 -0700, Noah Misch wrote:

On Sun, Sep 11, 2022 at 11:19:25AM -0700, Andres Freund wrote:

On 2022-09-10 01:19:44 -0700, Andres Freund wrote:

On 2022-09-09 22:57:36 -0700, Andres Freund wrote:

On 2022-09-10 01:32:52 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

It seem worth applying the -qvisibility patch and seeing whether that fixes
the buildfarm?

Worth a try.

Done now.

Seems to have done the trick for at least hoverfly. And wrasse (sunpro) is
still happy. Still need to wait for the older xlc's (mandril, hornet)
though...

At least hornet isn't happy:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hornet&amp;dt=2022-09-10%2021%3A44%3A00

Looks like the support for -qvisibility came with xlc 13.1, released April 28, 2014:
https://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&amp;infotype=an&amp;appname=iSource&amp;supplier=897&amp;letternum=ENUS214-162

Last two sungazer (gcc64) runs failed with a suspicious "missing magic block":
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&amp;dt=2022-09-12%2006%3A24%3A53
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&amp;dt=2022-09-10%2018%3A09%3A11

Both runs had the latest commits discussed above. tern (gcc32) passed, but it
doesn't use --with-perl.

I'll check it out tomorrow. I configured perl on the other AIX gcc compile
farm machine building 64bit with both gcc and xlc, IIRC. I can't check rn,
they seem to be down?

They're up.

What is "perl64-for-gcc"?

===
#! /bin/sh

real_perl=perl64

case $* in
'-MExtUtils::Embed -e ldopts' | \
'-MConfig -e print $Config{ccdlflags}')
$real_perl "$@" | sed 's/-b/-Wl,-b/g' ;;
*)
exec $real_perl "$@" ;;
esac
===

-qvisibility option specifies visibility attributes for entities. Entity
visibility attributes describe whether and how entities defined in one
module can be referenced or used in other modules. Visibility attributes
affect entities with external linkage only, and cannot increase the
visibility of other entities.

Since xlc 13.1 supports all still supported AIX versions I'm inclined to think
we should increase our requirement to 13.1 rather than revert back to the
higher complexity way of building modules.

Perhaps. That would demolish two buildfarm members.

Or you could update them to a slightly newer xlc? IIRC the machine has 13.x
available?

I don't see 13.x, but I might not know where to look.

What does PostgreSQL lose if you revert fe6a64a?

A somewhat readable Makefile.shlib. The prior state is complicated, specific
to AIX. I've several times spent quite a bit of time understanding it - not
helped by the complete lack of comments. It's not frequent enough to stay in
my brain unfortunately, so I'll have to relearn it again next time. And we'll
need to continue supporting Makefile.shlib for quite a while, so ...

I'm voting for a revert, then. Code beautification is nice when it just
works, but desupporting this compiler to land this particular code
beautification makes the net value go negative for me. I can understand your
feeling differently based on your recent experiences.

Hi,

On 2022-09-12 23:11:23 -0700, Andres Freund wrote:

I'll check it out tomorrow. I configured perl on the other AIX gcc compile
farm machine building 64bit with both gcc and xlc, IIRC. I can't check rn,
they seem to be down?

Seems to just have been a temporary connectivity issue.

After battling a bit with bison flex (system m4 seems incompatible with system
flex?), solved by using your binaries, I suceeded building and testing plperl
on 346990ae2e

~/src/postgres/configure FLEX=/home/nm/bin/flex BISON=/home/nm/sw/bison/bin/bison CXX='g++ -maix64' CC='gcc -maix64' PERL=perl64 --cache ../config-ac-gcc.cache --enable-cassert --enable-debug --with-perl

gmake -j8 -s world-bin && gmake -j8 -s checkprep && gmake -j8 -s temp-install
cd src/pl/plperl/
gmake check NO_TEMP_INSTALL=1
...
======================
All 13 tests passed.
======================

"my" linking of plperl.so:

gcc -maix64 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2 -fvisibility=hidden -o plperl.so plperl.o SPI.o Util.o -L../../../src/port -L../../../src/common -Wl,-blibpath:'/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE:/usr/lib:/lib:/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/8.3.0/' -Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -shared -fvisibility=hidden -L/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE -lperl -lpthread -lbind -lnsl -ldl -lld -lm -lcrypt -lpthreads -lc -Wl,-bI:../../../src/backend/postgres.imp

sungazer:
wrap-gcc-8.3 -D_THREAD_SAFE=1 -D_LARGE_FILES=1 -maix64 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2 -fvisibility=hidden -o plperl.so plperl.o SPI.o Util.o -L../../../src/port -L../../../src/common -L/home/nm/sw/nopath/libxml2-64/lib -L/home/nm/sw/nopath/icu58.2-64/lib -L/home/nm/sw/nopath/uuid-64/lib -L/home/nm/sw/nopath/openldap-64/lib -L/home/nm/sw/nopath/icu58.2-64/lib -L/home/nm/sw/nopath/libxml2-64/lib -Wl,-blibpath:'/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE:/home/nm/sw/nopath/libxml2-64/lib:/home/nm/sw/nopath/icu58.2-64/lib:/home/nm/sw/nopath/uuid-64/lib:/home/nm/sw/nopath/openldap-64/lib:/home/nm/sw/nopath/icu58.2-64/lib:/home/nm/sw/nopath/libxml2-64/lib:/usr/lib:/lib:/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/8.3.0/' -Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -shared -fvisibility=hidden -Wl,-bE:/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE/perl.exp -Wl,-bE:/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE/perl.exp -Wl,-brtl -Wl,-bdynamic -Wl,-b64 -L/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE -lperl -lpthread -lbind -lnsl -ldl -lld -lm -lcrypt -lpthreads -lc -Wl,-bI:../../../src/backend/postgres.imp

I suspect the problem is the -Wl,-bE:/.../perl.exp - that'll obviously
restrict the exports of plperl.so to the ones of perl, which obviously is
wrong. I would have assumed that Peter's recent plperl changes should have
gotten rid of the -Wl,-bE:?

Greetings,

Andres Freund

Hi,

On 2022-09-12 23:39:04 -0700, Noah Misch wrote:

On Mon, Sep 12, 2022 at 11:11:23PM -0700, Andres Freund wrote:

On 2022-09-12 20:38:45 -0700, Noah Misch wrote:

On Sun, Sep 11, 2022 at 11:19:25AM -0700, Andres Freund wrote:
Both runs had the latest commits discussed above. tern (gcc32) passed, but it
doesn't use --with-perl.

I'll check it out tomorrow. I configured perl on the other AIX gcc compile
farm machine building 64bit with both gcc and xlc, IIRC. I can't check rn,
they seem to be down?

They're up.

Yep, seems to have been temporary.

What is "perl64-for-gcc"?

===
#! /bin/sh

real_perl=perl64

case $* in
'-MExtUtils::Embed -e ldopts' | \
'-MConfig -e print $Config{ccdlflags}')
$real_perl "$@" | sed 's/-b/-Wl,-b/g' ;;
*)
exec $real_perl "$@" ;;
esac
===

I guess that defeats our plperl flags logic - the wrapper shouldn't be needed
anymore today, see my parallel message about the problem likely being caused
by -Wl,-bE:.

-qvisibility option specifies visibility attributes for entities. Entity
visibility attributes describe whether and how entities defined in one
module can be referenced or used in other modules. Visibility attributes
affect entities with external linkage only, and cannot increase the
visibility of other entities.

Since xlc 13.1 supports all still supported AIX versions I'm inclined to think
we should increase our requirement to 13.1 rather than revert back to the
higher complexity way of building modules.

Perhaps. That would demolish two buildfarm members.

Or you could update them to a slightly newer xlc? IIRC the machine has 13.x
available?

I don't see 13.x, but I might not know where to look.

Looks like it's not available on gcc111, just on gcc119, there it's at
/opt/IBM/xlc/13.1.3/bin/xlc

What does PostgreSQL lose if you revert fe6a64a?

A somewhat readable Makefile.shlib. The prior state is complicated, specific
to AIX. I've several times spent quite a bit of time understanding it - not
helped by the complete lack of comments. It's not frequent enough to stay in
my brain unfortunately, so I'll have to relearn it again next time. And we'll
need to continue supporting Makefile.shlib for quite a while, so ...

I'm voting for a revert, then. Code beautification is nice when it just
works, but desupporting this compiler to land this particular code
beautification makes the net value go negative for me. I can understand your
feeling differently based on your recent experiences.

IMO this more than just a beautification - the difference is functional in
nature. I'd agree if it just were a reformating or such, but this isn't that.

I just tested, and with the mlkdexport.sh approach we end up with all symbols
exported (-fvisibility=hidden) in extension .so's - AIX will be the only
platform unable to restrict symbol exports. That's bound to lead to problems
we'll only have on AIX.

The more special-case-y some platform is, the higher the cost of supporting it
is. I'm somewhat ok with supporting niche stuff, but then the people desiring
that support also need to do the work of making that bearable for everyone
else.

Greetings,

Andres Freund

Hi,

On 2022-09-13 00:02:53 -0700, Andres Freund wrote:

On 2022-09-12 23:11:23 -0700, Andres Freund wrote:

I'll check it out tomorrow. I configured perl on the other AIX gcc compile
farm machine building 64bit with both gcc and xlc, IIRC. I can't check rn,
they seem to be down?

Seems to just have been a temporary connectivity issue.

After battling a bit with bison flex (system m4 seems incompatible with system
flex?), solved by using your binaries, I suceeded building and testing plperl
on 346990ae2e

~/src/postgres/configure FLEX=/home/nm/bin/flex BISON=/home/nm/sw/bison/bin/bison CXX='g++ -maix64' CC='gcc -maix64' PERL=perl64 --cache ../config-ac-gcc.cache --enable-cassert --enable-debug --with-perl

gmake -j8 -s world-bin && gmake -j8 -s checkprep && gmake -j8 -s temp-install
cd src/pl/plperl/
gmake check NO_TEMP_INSTALL=1
...
======================
All 13 tests passed.
======================

"my" linking of plperl.so:

gcc -maix64 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2 -fvisibility=hidden -o plperl.so plperl.o SPI.o Util.o -L../../../src/port -L../../../src/common -Wl,-blibpath:'/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE:/usr/lib:/lib:/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/8.3.0/' -Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -shared -fvisibility=hidden -L/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE -lperl -lpthread -lbind -lnsl -ldl -lld -lm -lcrypt -lpthreads -lc -Wl,-bI:../../../src/backend/postgres.imp

sungazer:
wrap-gcc-8.3 -D_THREAD_SAFE=1 -D_LARGE_FILES=1 -maix64 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2 -fvisibility=hidden -o plperl.so plperl.o SPI.o Util.o -L../../../src/port -L../../../src/common -L/home/nm/sw/nopath/libxml2-64/lib -L/home/nm/sw/nopath/icu58.2-64/lib -L/home/nm/sw/nopath/uuid-64/lib -L/home/nm/sw/nopath/openldap-64/lib -L/home/nm/sw/nopath/icu58.2-64/lib -L/home/nm/sw/nopath/libxml2-64/lib -Wl,-blibpath:'/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE:/home/nm/sw/nopath/libxml2-64/lib:/home/nm/sw/nopath/icu58.2-64/lib:/home/nm/sw/nopath/uuid-64/lib:/home/nm/sw/nopath/openldap-64/lib:/home/nm/sw/nopath/icu58.2-64/lib:/home/nm/sw/nopath/libxml2-64/lib:/usr/lib:/lib:/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/8.3.0/' -Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -shared -fvisibility=hidden -Wl,-bE:/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE/perl.exp -Wl,-bE:/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE/perl.exp -Wl,-brtl -Wl,-bdynamic -Wl,-b64 -L/usr/opt/perl5/lib64/5.28.1/aix-thread-multi-64all/CORE -lperl -lpthread -lbind -lnsl -ldl -lld -lm -lcrypt -lpthreads -lc -Wl,-bI:../../../src/backend/postgres.imp

I suspect the problem is the -Wl,-bE:/.../perl.exp - that'll obviously
restrict the exports of plperl.so to the ones of perl, which obviously is
wrong. I would have assumed that Peter's recent plperl changes should have
gotten rid of the -Wl,-bE:?

The spurious -Wl,-bE's are the cause of all these warnings btw:

ld: 0711-415 WARNING: Symbol ASCII_TO_NEED is already exported.
ld: 0711-415 WARNING: Symbol boot_DynaLoader is already exported.
ld: 0711-415 WARNING: Symbol NATIVE_TO_NEED is already exported.
ld: 0711-415 WARNING: Symbol Perl__add_range_to_invlist is already exported.
ld: 0711-415 WARNING: Symbol Perl__core_swash_init is already exported.
[~1150 more]
ld: 0711-415 WARNING: Symbol PL_Yes is already exported.
ld: 0711-415 WARNING: Symbol PL_Zero is already exported.

Greetings,

Andres Freund

On 2022-09-13 00:15:55 -0700, Andres Freund wrote:

On 2022-09-12 23:39:04 -0700, Noah Misch wrote:

On Mon, Sep 12, 2022 at 11:11:23PM -0700, Andres Freund wrote:

-qvisibility option specifies visibility attributes for entities. Entity
visibility attributes describe whether and how entities defined in one
module can be referenced or used in other modules. Visibility attributes
affect entities with external linkage only, and cannot increase the
visibility of other entities.

Since xlc 13.1 supports all still supported AIX versions I'm inclined to think
we should increase our requirement to 13.1 rather than revert back to the
higher complexity way of building modules.

Perhaps. That would demolish two buildfarm members.

Or you could update them to a slightly newer xlc? IIRC the machine has 13.x
available?

I don't see 13.x, but I might not know where to look.

Looks like it's not available on gcc111, just on gcc119, there it's at
/opt/IBM/xlc/13.1.3/bin/xlc

What does PostgreSQL lose if you revert fe6a64a?

A somewhat readable Makefile.shlib. The prior state is complicated, specific
to AIX. I've several times spent quite a bit of time understanding it - not
helped by the complete lack of comments. It's not frequent enough to stay in
my brain unfortunately, so I'll have to relearn it again next time. And we'll
need to continue supporting Makefile.shlib for quite a while, so ...

I'm voting for a revert, then. Code beautification is nice when it just
works, but desupporting this compiler to land this particular code
beautification makes the net value go negative for me. I can understand your
feeling differently based on your recent experiences.

IMO this more than just a beautification - the difference is functional in
nature. I'd agree if it just were a reformating or such, but this isn't that.

I just tested, and with the mlkdexport.sh approach we end up with all symbols
exported (-fvisibility=hidden) in extension .so's - AIX will be the only
platform unable to restrict symbol exports. That's bound to lead to problems
we'll only have on AIX.

The more special-case-y some platform is, the higher the cost of supporting it
is. I'm somewhat ok with supporting niche stuff, but then the people desiring
that support also need to do the work of making that bearable for everyone
else.

Worth noting that xlc 12.1 appears to be EOL:

https://www.ibm.com/support/pages/unsupported-c-and-c-compilers-and-debuggers-links-latest-updates

"This page contains links to the final updates (Fix Packs, PTFs, etc) for the
non-supported IBM C and C++ compiler and debugger products. This document is
provided "AS IS" since those compilers are no longer supported.
...
XL C for AIX Fix Pack 23 (May 2021 PTF) for 12.1".

The successor, 13.1.0 is from 06 June 2014, which in turn has been superceeded
by 16.1 (16.1.0 14 December 2018) and 17.1 (17.1.0 17 September 2021).

Greetings,

Andres Freund

Hi,

On 2022-09-11 11:19:25 -0700, Andres Freund wrote:

We could also try to fall back to using -Wl,b,expall for extension modules.

expall doesn't work, because of our use of identifiers in reserved namespaces,
e.g. _PG_init:

expall
Exports all global symbols, except imported symbols, unreferenced symbols defined in archive members, and symbols beginning with an underscore
(_). You can export additional symbols by listing them in an export file or using the expfull option. This option does not affect symbols
exported by the autoexp option.

However, there also is 'expfull' - IIRC that causes problems when used for
postgres, but appears to be fine for .so's.

So we could work around the xlc 12.1 issue with something like the attached
patch. It passes at some of the tests, with both 32 and 64bit xlc 12.1, will
have to wait a while to see more

I think it'd be considerably better to just not support xlc < 13.1 though.

This won't fix sungazer, that just needs the perl wrapper removed...

Greetings,

Andres Freund

Hi,

On 2022-09-13 11:25:30 -0700, Andres Freund wrote:

On 2022-09-11 11:19:25 -0700, Andres Freund wrote:

We could also try to fall back to using -Wl,b,expall for extension modules.

expall doesn't work, because of our use of identifiers in reserved namespaces,
e.g. _PG_init:

expall
Exports all global symbols, except imported symbols, unreferenced symbols defined in archive members, and symbols beginning with an underscore
(_). You can export additional symbols by listing them in an export file or using the expfull option. This option does not affect symbols
exported by the autoexp option.

However, there also is 'expfull' - IIRC that causes problems when used for
postgres, but appears to be fine for .so's.

So we could work around the xlc 12.1 issue with something like the attached
patch. It passes at some of the tests, with both 32 and 64bit xlc 12.1, will
have to wait a while to see more

Both passed check-world.

- Andres

Andres Freund <andres@anarazel.de> writes:

So we could work around the xlc 12.1 issue with something like the attached
patch. It passes at some of the tests, with both 32 and 64bit xlc 12.1, will
have to wait a while to see more

Shouldn't that be more like

+  if test "$pgac_cv_prog_CC_cflags__qvisibility_hidden" != "yes"; then
+    CFLAGS_SL_MODULE='$CFLAGS_SL_MODULE -Wl,-b,expfull'
+  fi

to avoid losing whatever we found out before that?

I think it'd be considerably better to just not support xlc < 13.1 though.

A three-line patch doesn't seem like an unreasonable thing to carry,
at least till these systems go out of support. We've jumped through
much higher hoops in the past to support niche platforms.

regards, tom lane

Hi,

On 2022-09-13 22:45:46 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

So we could work around the xlc 12.1 issue with something like the attached
patch. It passes at some of the tests, with both 32 and 64bit xlc 12.1, will
have to wait a while to see more

Shouldn't that be more like

+  if test "$pgac_cv_prog_CC_cflags__qvisibility_hidden" != "yes"; then
+    CFLAGS_SL_MODULE='$CFLAGS_SL_MODULE -Wl,-b,expfull'
+  fi

to avoid losing whatever we found out before that?

Yes, you're right. There's nothing today, but we shouldn't assume that.

I think it'd be considerably better to just not support xlc < 13.1 though.

A three-line patch doesn't seem like an unreasonable thing to carry,
at least till these systems go out of support. We've jumped through
much higher hoops in the past to support niche platforms.

xlc 12.1 *is* out of support and there's a newer compiler installed.

The amount of lines itself doesn't bother me. What does is that it makes an
already complicated platform more complicated: All symbols from extension
.so's are exported, but only for old and unsupported versions of the
proprietary compiler.

Greetings,

Andres Freund