Lower priority of the configure option --with-libraries causes wrong lib resolution
Hello,
I am having issues building Postgres due to the `./configure` script. I
found a workaround but I would appreciate a more reliable fix.
I use Arch Linux 64bit and want to build Postgres with the latest libicu
version which is version 77.1. Unfortunately the Arch package is still
at version 76 currently [0]https://archlinux.org/packages/core/x86_64/icu/. To solve this, I built libicu 77.1 locally.
I also built the latest libxml2 with the latest libicu to ensure compat.
This means that I have a local directory with shared objects and include
files for libicu 77. Let's say that these dependencies are in
`/postgres-deps/lib` and `/postgres-deps/include`. I also still have the
system-level ICU version 76 present in `/usr/lib`.
My goal is to build Postgres with the libicu version from `libicuuc.so`
(77) instead of the one from `/usr/lib` (76). Both directories contain a
file named `libicuuc.so`.
To achieve it, I called the `configure` script with the arguments
`--with-libraries` and `--with-includes`, as such: `./configure
--with-libraries=/postgres-deps/lib
--with-includes=/postgres-deps/include ...` (+ some extra `--with` flags
to enable features, irrelevant to this issue I think).
I then initiate the build with `LD_LIBRARY_PATH="/postgres-deps/lib "
make all`. The C compilation into object code succeeds using the libicu
77 includes, but then it fails to link with many errors such as:
```
/usr/bin/ld: commands/collationcmds.o: in function `get_icu_locale_comment':
collationcmds.c:(.text+0x1a20): undefined reference to
`uloc_getDisplayName_77'
```
The failing command is fairly long so I shortened it to focus on the
main part:
```
gcc [...COMPILER_FLAGS] [...OBJECT_FILES] -L../../src/port
-L../../src/common -L/usr/lib -L/postgres-deps/lib -Wl,--as-needed
-Wl,--export-dynamic -lzstd -llz4 -lxslt -lxml2 -lpam -lssl -lcrypto
-lgssapi_krb5 -lz -lm -lldap -licui18n -licuuc -lsystemd -o postgres
```
In particular, notice that the lib locations give priority to the system
directory instead of the `--with-libraries` directory that I passed to
`./configure`: `-L/usr/lib -L/postgres-deps/lib`. Since the command
requests `-licuuc`, the file `/usr/lib/libicuuc.so` (version 76) is
matched first and the file `/usr/lib/libicuuc.so` (77) is ignored.
Swapping the two flags so the order is `-L/postgres-deps/lib -L/usr/lib`
fixes the build. I consider it very surprising that that libraries
passed with `--with-libraries` have lower priority, I would expect
explicitly requested libraries to have the highest priority.
I've searched a bit to find where both locations are inserted.
First, `/usr/lib` is added in the clang configuration [1]https://github.com/postgres/postgres/blob/6d6480066c1a96c7130b97b1139fdada9d484f80/configure#L5197. It is
retrieved from `/usr/bin/llvm-config --ldflags`.
Second, the `--with-libraries` directories are collected into
`$LIBDIRS`, with a check to verify that the dirs exists, this adds
`/postgres-deps/lib` [2]https://github.com/postgres/postgres/blob/6d6480066c1a96c7130b97b1139fdada9d484f80/configure#L8104.
Finally, $LIBDIRS is _appended at the end of LDFLAGS_.using
`LDFLAGS="$LDFLAGS $LIBDIRS"` [3]https://github.com/postgres/postgres/blob/6d6480066c1a96c7130b97b1139fdada9d484f80/configure#L9824.
My workaround is to update the configure script to instead prepend
`LIBDIRS` at the start of `LDFLAGS` using `LDFLAGS="$LIBDIRS $LDFLAGS"`.
With this change, the lib dir order is `-L/postgres-deps/lib -L/usr/lib`
and the version 77 of libicuuc.so is picked by the linker. The build
completes and I get a fully functional Postgres (at least it passes the
test suite of my application depending on it).
My expectation that moving the lib directories passed using
`--with-libraries` before any other linker flags makes sense, however
I'm not an expert in this area and I guess that there may be situations
where having the CLI libs first may cause issues. I can send a patch
swapping the application order or `LIBDIRS` as described in my
workaround if it makes sense. If there is a better solution to give
higher priority to the libs passed through the configure CLI, I would
gladly use it.
[0]: https://archlinux.org/packages/core/x86_64/icu/
[1]: https://github.com/postgres/postgres/blob/6d6480066c1a96c7130b97b1139fdada9d484f80/configure#L5197
https://github.com/postgres/postgres/blob/6d6480066c1a96c7130b97b1139fdada9d484f80/configure#L5197
[2]: https://github.com/postgres/postgres/blob/6d6480066c1a96c7130b97b1139fdada9d484f80/configure#L8104
https://github.com/postgres/postgres/blob/6d6480066c1a96c7130b97b1139fdada9d484f80/configure#L8104
[3]: https://github.com/postgres/postgres/blob/6d6480066c1a96c7130b97b1139fdada9d484f80/configure#L9824
https://github.com/postgres/postgres/blob/6d6480066c1a96c7130b97b1139fdada9d484f80/configure#L9824
Thank you in advance,
Charles "demurgos" Samborski
Charles Samborski <demurgos@demurgos.net> writes:
Swapping the two flags so the order is `-L/postgres-deps/lib -L/usr/lib`
fixes the build. I consider it very surprising that that libraries
passed with `--with-libraries` have lower priority, I would expect
explicitly requested libraries to have the highest priority.
Ugh.
First, `/usr/lib` is added in the clang configuration [1]. It is
retrieved from `/usr/bin/llvm-config --ldflags`.
I would pin the blame here. This code should not be messing with
the global LDFLAGS. -L switches from llvm-config should probably
go into LLVM_LIBS instead, so that they're only applied while
linking llvmjit.so. (Compare the handling of -L switches from
python or perl: those go into python_libspec or perl_embed_ldflags,
they're not applied globally.)
Can you check whether a change along that line fixes the problem
in your environment?
My workaround is to update the configure script to instead prepend
`LIBDIRS` at the start of `LDFLAGS` using `LDFLAGS="$LIBDIRS $LDFLAGS"`.
I don't think we'd accept that: it seems about as likely to break
builds as fix them. Notably, users might've themselves modified
LDFLAGS to contain -L switches. While that's not good practice
IMO, we shouldn't make changes that are more sweeping than
necessary.
regards, tom lane
I wrote:
I would pin the blame here. This code should not be messing with
the global LDFLAGS.
Looking closer, we've made the same mistake elsewhere.
I think we need something like the attached to ensure
that -L switches coming from libraries' configure helpers
don't override user-specified directories.
regards, tom lane
Attachments:
v1-avoid-overriding-libdirs.patchtext/x-diff; charset=us-ascii; name=v1-avoid-overriding-libdirs.patchDownload+9-9
I wrote:
I think we need something like the attached to ensure
that -L switches coming from libraries' configure helpers
don't override user-specified directories.
I looked this over again, and was about ready to commit it when
I realized that we have essentially the same problem for -I
switches. If there's a --with-includes switch, then the -I
switches from that should override anything supplied by external
configuration helpers, but we didn't reliably get that right.
(Some of those bugs are new in v18, but others are older.)
However, -I switches pointing to in-tree directories should
override all of those. We fixed one instance of that problem
a few months ago in cb36f8ec2, but I was dismayed to find that
there are more. This could result in build failures thanks to
pulling in the wrong version of some Postgres header.
Hence, 0001 attached is the same as before (but now with
a commit message), and then 0002 tackles the problems with -I
switches.
regards, tom lane