BUG #19727: pg-combinebackup fails to link

PG Bug reporting form <noreply@postgresql.org> writes:

When I build pg-combineback from postgress 18.6 (in Yocto, cross-compiled on
aarch64 to aarch64), it fails to link:

aarch64-poky-linux-ld: src/common/libpgcommon.a(jsonapi.o): in function
`json_lex_number':
src/common/jsonapi.c:2379:(.text+0x168): undefined reference to
`appendBinaryPQExpBuffer'

You seem to be using a broken linker :-(

pg_combinebackup doesn't use any of the functions in
src/common/jsonapi.c. Therefore, the jsonapi.o member of
libpgcommon.a shouldn't get pulled into the link, and its
unsatisfied dependency on libpq.so shouldn't matter.
Apparently, whatever build toolchain you're using has
forgotten this decades-old principle of .a library usage.

regards, tom lane

That's a good point. This is GCC 16.2 and binutils 2.47...

Ross Burton <ross@burtonini.com> writes:

That's a good point. This is GCC 16.2 and binutils 2.47...

Hm. The newest Linux installation I have at hand is Fedora 44
with gcc 16.2 + binutils 2.46, and it doesn't show this behavior.

Can you check on whether gcc is invoking /usr/bin/ld? If it's
calling some other linker maybe that's the cause.

I'm suspicious in any case that there's something odd about either
your installation or your environment. I don't recall ever hearing
a similar complaint, so this linking behavior can't be common.

regards, tom lane

On Tue, 29 Sept 2026 at 16:30, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Hm. The newest Linux installation I have at hand is Fedora 44
with gcc 16.2 + binutils 2.46, and it doesn't show this behavior.

Can you check on whether gcc is invoking /usr/bin/ld? If it's
calling some other linker maybe that's the cause.

I'm suspicious in any case that there's something odd about either
your installation or your environment. I don't recall ever hearing
a similar complaint, so this linking behavior can't be common.

It's not calling the host ld. Interestingly, this doesn't happen
reliably, I needed to do a loop of builds with make --shuffle to make
it fail. So that suggests there's a missing Makefile dependency
somewhere that normally doesn't cause problems.

I'll dig a little more later, but now I need to cook dinner. :)
Ross

I now have a build tree that works and a build tree that breaks, from
the same machine and environment, simply by re-running the build with
make --shuffle.

The working jsonapi.o has functions like appendStringInfoChar,
resetStringInfo, etc. The one that fails to link has
appendPQExpBufferChar, resetPQExpBuffer, etc.

This is a makefile dependency bug exposed in parallel builds, and can
be reproduced on demand with a minimal reproducer. From a clean but
configured tree, in src/common:

$ make jsonapi.o
$ readelf -sW jsonapi.o | grep reset
64: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND resetStringInfo

This builds a jsonapi.o using the direct target, and
JSONAPI_USE_PQEXPBUFFER is not defined.

$ make clean
$ make jsonapi_shlib.o
$ readelf -sW jsonapi.o | grep reset
65: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND resetPQExpBuffer

This builds jsonapi_shlib.o, which adds JSONAPI_USE_PQEXPBUFFER to
CPPFLAGS. However jsonapi.o hasn't been built yet so that is also
built, but with the PQEXPBUFFER define set this time.

My builds are with -j32 but also --shuffle, which is very good at
shaking out Makefiles which have dependencies that work via luck: if a
target has dependencies A B C, then normal make would be A then B then
C, but --shuffle might build C then B then A.

Ross

Ross Burton <ross@burtonini.com> writes:

I now have a build tree that works and a build tree that breaks, from
the same machine and environment, simply by re-running the build with
make --shuffle.

The working jsonapi.o has functions like appendStringInfoChar,
resetStringInfo, etc. The one that fails to link has
appendPQExpBufferChar, resetPQExpBuffer, etc.

This is a makefile dependency bug exposed in parallel builds, and can
be reproduced on demand with a minimal reproducer. From a clean but
configured tree, in src/common:

$ make jsonapi.o
$ readelf -sW jsonapi.o | grep reset
64: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND resetStringInfo

This builds a jsonapi.o using the direct target, and
JSONAPI_USE_PQEXPBUFFER is not defined.

$ make clean
$ make jsonapi_shlib.o
$ readelf -sW jsonapi.o | grep reset
65: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND resetPQExpBuffer

This builds jsonapi_shlib.o, which adds JSONAPI_USE_PQEXPBUFFER to
CPPFLAGS. However jsonapi.o hasn't been built yet so that is also
built, but with the PQEXPBUFFER define set this time.

Very interesting! So the problem is that the scope of the effects of
this:

jsonapi_shlib.o: override CPPFLAGS += -DJSONAPI_USE_PQEXPBUFFER

isn't terribly well-defined. Looking at the rest of that Makefile,
I'm also suspicious about

$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS

Can you break that part? If not, maybe the problem is the
gratuitous-looking "override"?

regards, tom lane

I wrote:

Ross Burton <ross@burtonini.com> writes:

This is a makefile dependency bug exposed in parallel builds, and can
be reproduced on demand with a minimal reproducer.

Very interesting! So the problem is that the scope of the effects of
this:
jsonapi_shlib.o: override CPPFLAGS += -DJSONAPI_USE_PQEXPBUFFER
isn't terribly well-defined.

I dug in the gmake manual and verified that they document this
behavior and point out that it can produce inconsistent results.
So we need to fix it. The attached seems to do the trick, but
now I wonder where else there might be similar issues. I did
a few rounds of "make -s -j32 --shuffle" and didn't see any
other compilation failures, but that isn't a very thorough test.

regards, tom lane