make -C libpq check fails obscurely if tap tests are disabled

On 2022-07-20 We 13:23, Justin Pryzby wrote:

make -C ./src/interfaces/libpq check
PATH=... && @echo "TAP tests not enabled. Try configuring with --enable-tap-tests"
/bin/sh: 1: @echo: not found

make is telling the shell to run "@echo" , rather than running "echo" silently.

Since:

commit 6b04abdfc5e0653542ac5d586e639185a8c61a39
Author: Andres Freund <andres@anarazel.de>
Date: Sat Feb 26 16:51:47 2022 -0800

Run tap tests in src/interfaces/libpq.

Yeah. It's a bit ugly but I think the attached would fix it.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On 2022-Jul-20, Andrew Dunstan wrote:

On 2022-07-20 We 13:23, Justin Pryzby wrote:

PATH=... && @echo "TAP tests not enabled. Try configuring with --enable-tap-tests"
/bin/sh: 1: @echo: not found

make is telling the shell to run "@echo" , rather than running "echo" silently.

Yeah. It's a bit ugly but I think the attached would fix it.

Here's a different take. Just assign the variable separately.

--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"No renuncies a nada. No te aferres a nada."

On 2022-07-21 Th 04:53, Alvaro Herrera wrote:

On 2022-Jul-20, Andrew Dunstan wrote:

On 2022-07-20 We 13:23, Justin Pryzby wrote:

PATH=... && @echo "TAP tests not enabled. Try configuring with --enable-tap-tests"
/bin/sh: 1: @echo: not found

make is telling the shell to run "@echo" , rather than running "echo" silently.

Yeah. It's a bit ugly but I think the attached would fix it.

Here's a different take. Just assign the variable separately.

Nice, I didn't know you could do that.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On 2022-Jul-21, Andrew Dunstan wrote:

On 2022-07-21 Th 04:53, Alvaro Herrera wrote:

Here's a different take. Just assign the variable separately.

Nice, I didn't know you could do that.

It's not very common -- we do have some target-specific variable
assignments, but none of them use 'export'. I saw somewhere that this
works from Make 3.77 onwards, and we require 3.80, so it should be okay.
The buildfarm will tell us ...

--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"Industry suffers from the managerial dogma that for the sake of stability
and continuity, the company should be independent of the competence of
individual employees." (E. Dijkstra)

On 2022-Jul-22, Alvaro Herrera wrote:

It's not very common -- we do have some target-specific variable
assignments, but none of them use 'export'. I saw somewhere that this
works from Make 3.77 onwards, and we require 3.80, so it should be okay.
The buildfarm will tell us ...

Hm, so prairiedog didn't like this:

make -C libpq all
Makefile:146: *** multiple target patterns. Stop.

but I don't understand which part it is upset about. The rules are:

check installcheck: export PATH := $(CURDIR)/test:$(PATH)

check: test-build all
$(prove_check)

installcheck: test-build all
$(prove_installcheck)

I think "multiple target patterns" means it doesn't like the fact that
there are two colons in the first line. But if I use a recursive
assignment (PATH = ...), that of course doesn't work because PATH appears on
both sides of the assignment:

Makefile:146: *** Recursive variable 'PATH' references itself (eventually). Stop.

Now, maybe that colon is not the issue and perhaps the problem can be
solved by splitting the rule:

check: export PATH := $(CURDIR)/test:$(PATH)
installcheck: export PATH := $(CURDIR)/test:$(PATH)

According to 32568.1536241083@sss.pgh.pa.us, prairiedog is on Make 3.80.
Hmmm.

--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
Voy a acabar con todos los humanos / con los humanos yo acabaré
voy a acabar con todos (bis) / con todos los humanos acabaré ¡acabaré! (Bender)

Alvaro Herrera <alvherre@alvh.no-ip.org> writes:

On 2022-Jul-22, Alvaro Herrera wrote:

It's not very common -- we do have some target-specific variable
assignments, but none of them use 'export'. I saw somewhere that this
works from Make 3.77 onwards, and we require 3.80, so it should be okay.
The buildfarm will tell us ...

Hm, so prairiedog didn't like this:
According to 32568.1536241083@sss.pgh.pa.us, prairiedog is on Make 3.80.

Yeah, it is. I looked at the gmake manual on that machine, and its
description of "export" seems about the same as what I see in a
modern version. So it should work ... but we've found bugs in 3.80
before.

Let me poke at it and see if there's a variant that works.
The wording of the message suggests that maybe breaking it into
two separate rules would help.

regards, tom lane

I wrote:

Yeah, it is. I looked at the gmake manual on that machine, and its
description of "export" seems about the same as what I see in a
modern version.

Um ... I was not looking in the right place. The description of
"target-specific variables" does not say you can use "export",
whereas the modern manual mentions that specifically. I found
a relevant entry in their changelog:

2002-10-13 Paul D. Smith <psmith@gnu.org>
...
* read.c (eval): Fix Bug #1391: allow "export" keyword in
target-specific variable definitions. Check for it and set an
"exported" flag.
* doc/make.texi (Target-specific): Document the ability to use
"export".

So it'll work in 3.81 (released 2006) and later, but not 3.80.

TBH my inclination here is to move our goalposts to say "we support
gmake 3.81 and later". It's possible that prairiedog's copy of 3.80 is
the last one left in the wild, and nearly certain that it's the last
one left that anyone would try to build PG with. (I see gmake 3.81 in
the next macOS version, 10.5.) I doubt it'd take long to install a newer
version on prairiedog.

Alternatively, we could use Andrew's hacky solution from upthread.

regards, tom lane

I wrote:

So it'll work in 3.81 (released 2006) and later, but not 3.80.

Confirmed that things are fine with 3.81.

TBH my inclination here is to move our goalposts to say "we support
gmake 3.81 and later".

Barring objections, I'll push the attached patch. I suppose we
could undo whatever dumbing-down was done in _create_recursive_target,
but is it worth troubling with?

regards, tom lane

On 2022-Jul-22, Tom Lane wrote:

Barring objections, I'll push the attached patch. I suppose we
could undo whatever dumbing-down was done in _create_recursive_target,
but is it worth troubling with?

Excellent, many thanks. I tried to get Make 3.80 built here, to no
avail. I have updated that to 3.81, but still haven't found a way past
the automake phase.

Anyway, I tried a revert of 1bd201214965 -- I ended up with the
attached. However, while a serial compile fails, parallel ones fail
randomly, and apparently because two submakes compete in building
libpq.a and each deletes the other's file. What I think this is saying
is that the 3.80-induced wording of that function limits concurrency of
the generated recursive rules, which prevents the problem from
occurring; and if we were to fix that bug we would probably end up with
more concurrency.

Here's the bottom of the 'make -j8' log:

rm -f libpq.a
ranlib libpq.a
ranlib: 'libpq.a': No such file
make[5]: *** [/pgsql/source/master/src/Makefile.shlib:261: libpq.a] Error 1
make[5]: *** Waiting for unfinished jobs....
ar crs libpq.a fe-auth-scram.o fe-connect.o fe-exec.o fe-lobj.o fe-misc.o fe-print.o fe-protocol3.o fe-secure.o fe-trace.o legacy-pqsignal.o libpq-events.o pqexpbuffer.o fe-auth.o fe-secure-common.o fe-secure-openssl.o
ranlib libpq.a
touch libpq.a
rm -f libpq.so.5
ln -s libpq.so.5.16 libpq.so.5
rm -f libpq.so
ln -s libpq.so.5.16 libpq.so
touch libpq-refs-stamp
rm -f libpq.so.5
ln -s libpq.so.5.16 libpq.so.5
rm -f libpq.so
ln -s libpq.so.5.16 libpq.so
rm -f libpq.so.5
ln -s libpq.so.5.16 libpq.so.5
touch libpq-refs-stamp
rm -f libpq.so
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2 -pthread -D_REENTRANT -D_THREAD_SAFE -fPIC -shared -Wl,-soname,libecpg.so.6 -Wl,--version-script=exports.list -o libecpg.so.6.16 connect.o data.o descriptor.o error.o execute.o memory.o misc.o prepare.o sqlda.o typename.o -L../../../../src/port -L../../../../src/common -L../pgtypeslib -lpgtypes -L../../../../src/common -lpgcommon_shlib -L../../../../src/port -lpgport_shlib -L../../../../src/interfaces/libpq -lpq -L/usr/lib/llvm-11/lib -Wl,--as-needed -Wl,-rpath,'/pgsql/install/master/lib',--enable-new-dtags -lm
ln -s libpq.so.5.16 libpq.so
rm -f libecpg.a
make[4]: *** [../../../../src/Makefile.global:618: submake-libpq] Error 2
ar crs libecpg.a connect.o data.o descriptor.o error.o execute.o memory.o misc.o prepare.o sqlda.o typename.o
make[3]: *** [Makefile:17: all-ecpglib-recursive] Error 2
make[3]: *** Waiting for unfinished jobs....
ranlib libecpg.a
touch libecpg.a
rm -f libecpg.so.6
ln -s libecpg.so.6.16 libecpg.so.6
rm -f libecpg.so
ln -s libecpg.so.6.16 libecpg.so
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2 -pthread -D_REENTRANT -D_THREAD_SAFE -fPIC -shared -Wl,-soname,libecpg_compat.so.3 -Wl,--version-script=exports.list -o libecpg_compat.so.3.16 informix.o -L../../../../src/port -L../../../../src/common -L../ecpglib -lecpg -L../pgtypeslib -lpgtypes -L../../../../src/common -lpgcommon_shlib -L../../../../src/port -lpgport_shlib -L../../../../src/interfaces/libpq -lpq -L/usr/lib/llvm-11/lib -Wl,--as-needed -Wl,-rpath,'/pgsql/install/master/lib',--enable-new-dtags -lm
rm -f libecpg_compat.a
ar crs libecpg_compat.a informix.o
ranlib libecpg_compat.a
touch libecpg_compat.a
rm -f libecpg_compat.so.3
ln -s libecpg_compat.so.3.16 libecpg_compat.so.3
rm -f libecpg_compat.so
ln -s libecpg_compat.so.3.16 libecpg_compat.so
make[2]: *** [Makefile:17: all-ecpg-recursive] Error 2
make[1]: *** [Makefile:42: all-interfaces-recursive] Error 2
make: *** [GNUmakefile:11: all-src-recursive] Error 2

--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Los dioses no protegen a los insensatos. Éstos reciben protección de
otros insensatos mejor dotados" (Luis Wu, Mundo Anillo)

On 2022-Jul-25, Alvaro Herrera wrote:

Anyway, I tried a revert of 1bd201214965 -- I ended up with the
attached.

--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/

Ah, I found what the actual problem is: we have sprinkled a few
dependencies on "...-recurse" throught the tree, but the patch I posted
yesterday changes the manufactured target as "-recursive", as it was
prior to 1bd201214965; so essentially these manually added dependencies
all became silent no-ops.

With this version I keep the target name as -recurse, and at least the
ecpg<->libpq problem is no more AFAICT.

--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/

Hi,

On 2022-07-22 19:50:02 -0400, Tom Lane wrote:

I wrote:

So it'll work in 3.81 (released 2006) and later, but not 3.80.

Confirmed that things are fine with 3.81.

Thanks for looking into this Alvaro, Andrew, Justin, Tom - I was on
vacation...

Greetings,

Andres Freund

On 2022-Jul-26, Alvaro Herrera wrote:

With this version I keep the target name as -recurse, and at least the
ecpg<->libpq problem is no more AFAICT.

... but I think we're missing some more dependencies, because if I
remove everything (beyond make clean), then a "make -j8 world-bin"
fails, per Cirrus.
https://cirrus-ci.com/build/6305635338813440
With that, I'm going to set this aside for the time being. If somebody
wants to play with these Makefile rules, be my guest. It sounds like
there's some compile time gains to be had, but it may require some
fiddling and it's not clear to me if the move to Meson is going to make
this moot.

Running it locally, I get this:

[...]
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2 -DFRONTEND -I. -I/pgsql/source/master/src/common -I../../src/include -I/pgsql/source/master/src/include -D_GNU_SOURCE -DVAL_CC="\"gcc\"" -DVAL_CPPFLAGS="\"-D_GNU_SOURCE\"" -DVAL_CFLAGS="\"-Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2\"" -DVAL_CFLAGS_SL="\"-fPIC\"" -DVAL_LDFLAGS="\"-L/usr/lib/llvm-11/lib -Wl,--as-needed -Wl,-rpath,'/pgsql/install/master/lib',--enable-new-dtags\"" -DVAL_LDFLAGS_EX="\"\"" -DVAL_LDFLAGS_SL="\"\"" -DVAL_LIBS="\"-lpgcommon -lpgport -llz4 -lssl -lcrypto -lz -lreadline -lpthread -lrt -ldl -lm \"" -c -o hashfn.o /pgsql/source/master/src/common/hashfn.c -MMD -MP -MF .deps/hashfn.Po
[...]
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2 -DFRONTEND -I. -I/pgsql/source/master/src/common -I../../src/include -I/pgsql/source/master/src/include -D_GNU_SOURCE -DVAL_CC="\"gcc\"" -DVAL_CPPFLAGS="\"-D_GNU_SOURCE\"" -DVAL_CFLAGS="\"-Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2\"" -DVAL_CFLAGS_SL="\"-fPIC\"" -DVAL_LDFLAGS="\"-L/usr/lib/llvm-11/lib -Wl,--as-needed -Wl,-rpath,'/pgsql/install/master/lib',--enable-new-dtags\"" -DVAL_LDFLAGS_EX="\"\"" -DVAL_LDFLAGS_SL="\"\"" -DVAL_LIBS="\"-lpgcommon -lpgport -llz4 -lssl -lcrypto -lz -lreadline -lpthread -lrt -ldl -lm \"" -c -o relpath.o /pgsql/source/master/src/common/relpath.c -MMD -MP -MF .deps/relpath.Po
[...]
In file included from /pgsql/source/master/src/include/postgres.h:47,
from /pgsql/source/master/src/common/hashfn.c:24:
/pgsql/source/master/src/include/utils/elog.h:75:10: fatal error: utils/errcodes.h: No existe el fichero o el directorio
75 | #include "utils/errcodes.h"
| ^~~~~~~~~~~~~~~~~~
compilation terminated.
[...]
make[2]: *** [../../src/Makefile.global:945: hashfn.o] Error 1
make[2]: *** Se espera a que terminen otras tareas....
/pgsql/source/master/src/common/relpath.c:21:10: fatal error: catalog/pg_tablespace_d.h: No existe el fichero o el directorio
21 | #include "catalog/pg_tablespace_d.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [../../src/Makefile.global:945: relpath.o] Error 1
make[2]: se sale del directorio '/home/alvherre/Code/pgsql-build/master/src/common'
make[1]: *** [Makefile:42: all-common-recurse] Error 2
make[1]: se sale del directorio '/home/alvherre/Code/pgsql-build/master/src'
make: *** [GNUmakefile:21: world-bin-src-recurse] Error 2

--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"I must say, I am absolutely impressed with what pgsql's implementation of
VALUES allows me to do. It's kind of ridiculous how much "work" goes away in
my code. Too bad I can't do this at work (Oracle 8/9)." (Tom Allison)
http://archives.postgresql.org/pgsql-general/2007-06/msg00016.php