pgsql: Build all Flex files standalone

Crake fails with

headerscheck:
Sep 04 01:56:01 In file included from /tmp/headerscheck.UNgmKy/test.c:2:
Sep 04 01:56:01
/home/andrew/bf/root/HEAD/pgsql.build/../pgsql/src/backend/utils/adt/jsonpath_internal.h:26:10:
fatal error: jsonpath_gram.h: No such file or directory
Sep 04 01:56:01 26 | #include "jsonpath_gram.h"
Sep 04 01:56:01 | ^~~~~~~~~~~~~~~~~
Sep 04 01:56:01 compilation terminated.
Sep 04 01:56:14 make: *** [GNUmakefile:138: headerscheck] Error 1
cpluspluscheck:
Sep 04 01:57:19 In file included from /tmp/cpluspluscheck.HhRV5k/test.cpp:3:
Sep 04 01:57:19
/home/andrew/bf/root/HEAD/pgsql.build/../pgsql/src/backend/utils/adt/jsonpath_internal.h:26:10:
fatal error: jsonpath_gram.h: No such file or directory
Sep 04 01:57:19 26 | #include "jsonpath_gram.h"
Sep 04 01:57:19 | ^~~~~~~~~~~~~~~~~

I don't get the same in an already-built tree. I also don't see an
exception for gramparse.h, which I imagine would have the same issue?

--
John Naylor
EDB: http://www.enterprisedb.com

The second failure is on drongo (MSVC):

"c:\\prog\\bf\\root\\HEAD\\pgsql.build\\pgsql.sln" (default target) (1) ->
"c:\\prog\\bf\\root\\HEAD\\pgsql.build\\isolationtester.vcxproj"
(default target) (13) ->
(Link target) ->
specparse.obj : error LNK2019: unresolved external symbol spec_yylex
referenced in function spec_yyparse
[c:\\prog\\bf\\root\\HEAD\\pgsql.build\\isolationtester.vcxproj]
specparse.obj : error LNK2019: unresolved external symbol
spec_yyerror referenced in function spec_yyparse
[c:\\prog\\bf\\root\\HEAD\\pgsql.build\\isolationtester.vcxproj]
.\\Release\\isolationtester\\isolationtester.exe : fatal error
LNK1120: 2 unresolved externals
[c:\\prog\\bf\\root\\HEAD\\pgsql.build\\isolationtester.vcxproj]

For this, Mkvcbuild.pm specifies specparse.c explicitly, so maybe
something like the following is needed:

--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -362,6 +362,7 @@ sub mkvcbuild
        $isolation_tester->AddFile('src/test/isolation/specparse.y');
        $isolation_tester->AddFile('src/test/isolation/specscanner.l');
        $isolation_tester->AddFile('src/test/isolation/specparse.c');
+       $isolation_tester->AddFile('src/test/isolation/specscanner.c');
        $isolation_tester->AddIncludeDir('src/test/isolation');

--
John Naylor
EDB: http://www.enterprisedb.com

On Sun, Sep 4, 2022 at 4:36 PM John Naylor <john.naylor@enterprisedb.com> wrote:

For this, Mkvcbuild.pm specifies specparse.c explicitly, so maybe
something like the following is needed:

--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -362,6 +362,7 @@ sub mkvcbuild
$isolation_tester->AddFile('src/test/isolation/specparse.y');
$isolation_tester->AddFile('src/test/isolation/specscanner.l');
$isolation_tester->AddFile('src/test/isolation/specparse.c');
+       $isolation_tester->AddFile('src/test/isolation/specscanner.c');
$isolation_tester->AddIncludeDir('src/test/isolation');

This gets CI green (tip o' the hat to Thomas Munro for off-list advice
on getting that working), so pushed.

--
John Naylor
EDB: http://www.enterprisedb.com

On 2022-09-04 Su 02:19, John Naylor wrote:

Crake fails with

headerscheck:
Sep 04 01:56:01 In file included from /tmp/headerscheck.UNgmKy/test.c:2:
Sep 04 01:56:01
/home/andrew/bf/root/HEAD/pgsql.build/../pgsql/src/backend/utils/adt/jsonpath_internal.h:26:10:
fatal error: jsonpath_gram.h: No such file or directory
Sep 04 01:56:01 26 | #include "jsonpath_gram.h"
Sep 04 01:56:01 | ^~~~~~~~~~~~~~~~~
Sep 04 01:56:01 compilation terminated.
Sep 04 01:56:14 make: *** [GNUmakefile:138: headerscheck] Error 1
cpluspluscheck:
Sep 04 01:57:19 In file included from /tmp/cpluspluscheck.HhRV5k/test.cpp:3:
Sep 04 01:57:19
/home/andrew/bf/root/HEAD/pgsql.build/../pgsql/src/backend/utils/adt/jsonpath_internal.h:26:10:
fatal error: jsonpath_gram.h: No such file or directory
Sep 04 01:57:19 26 | #include "jsonpath_gram.h"
Sep 04 01:57:19 | ^~~~~~~~~~~~~~~~~

I don't get the same in an already-built tree. I also don't see an
exception for gramparse.h, which I imagine would have the same issue?

Probably happens because crake does vpath builds. I think you're going
to have to set up EXTRAINCLUDES for it - see near the bottom of the
headerscheck script. Possibly something like

    src/backend/utils/adt/*)

    EXTRAINCLUDES="-I $builddir/src/backend/utils/adt" ;;

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 2022-09-04 18:04:13 +0700, John Naylor wrote:

On Sun, Sep 4, 2022 at 4:36 PM John Naylor <john.naylor@enterprisedb.com> wrote:

For this, Mkvcbuild.pm specifies specparse.c explicitly, so maybe
something like the following is needed:

--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -362,6 +362,7 @@ sub mkvcbuild
$isolation_tester->AddFile('src/test/isolation/specparse.y');
$isolation_tester->AddFile('src/test/isolation/specscanner.l');
$isolation_tester->AddFile('src/test/isolation/specparse.c');
+       $isolation_tester->AddFile('src/test/isolation/specscanner.c');
$isolation_tester->AddIncludeDir('src/test/isolation');

This gets CI green (tip o' the hat to Thomas Munro for off-list advice
on getting that working), so pushed.

It doesn't seem to have gotten all of CI green, cpluspluscheck fails:
https://cirrus-ci.com/task/6561938334285824

I think this just needs a copy the logic you added to headerscheck also to
cpluspluscheck (and yes, we should merge these files one of these days).

Greetings,

Andres Freund

Hi,

On 2022-09-04 13:19:56 +0700, John Naylor wrote:

Crake fails with

headerscheck:
Sep 04 01:56:01 In file included from /tmp/headerscheck.UNgmKy/test.c:2:
Sep 04 01:56:01
/home/andrew/bf/root/HEAD/pgsql.build/../pgsql/src/backend/utils/adt/jsonpath_internal.h:26:10:
fatal error: jsonpath_gram.h: No such file or directory
Sep 04 01:56:01 26 | #include "jsonpath_gram.h"
Sep 04 01:56:01 | ^~~~~~~~~~~~~~~~~
Sep 04 01:56:01 compilation terminated.
Sep 04 01:56:14 make: *** [GNUmakefile:138: headerscheck] Error 1
cpluspluscheck:
Sep 04 01:57:19 In file included from /tmp/cpluspluscheck.HhRV5k/test.cpp:3:
Sep 04 01:57:19
/home/andrew/bf/root/HEAD/pgsql.build/../pgsql/src/backend/utils/adt/jsonpath_internal.h:26:10:
fatal error: jsonpath_gram.h: No such file or directory
Sep 04 01:57:19 26 | #include "jsonpath_gram.h"
Sep 04 01:57:19 | ^~~~~~~~~~~~~~~~~

I don't get the same in an already-built tree. I also don't see an
exception for gramparse.h, which I imagine would have the same issue?

It doesn't happen for gramparse.h because gram.h is copied to
src/include/parser (see src/backend/Makefile), and included as parser/gram.h.

I think this could be addressed by always adding something like
-iquote $builddir/$(dirname "$f") -iquote $srcdir/$(dirname "$f") \
to cpluspluscheck/headerscheck.

Greetings,

Andres Freund

On Sun, Sep 4, 2022 at 8:34 PM Andrew Dunstan <andrew@dunslane.net> wrote:

Probably happens because crake does vpath builds. I think you're going
to have to set up EXTRAINCLUDES for it - see near the bottom of the
headerscheck script. Possibly something like

src/backend/utils/adt/*)

EXTRAINCLUDES="-I $builddir/src/backend/utils/adt" ;;

Okay, I'll look at this and the cpp header check within 24 hours.

--
John Naylor
EDB: http://www.enterprisedb.com

On Sun, Sep 4, 2022 at 8:34 PM Andrew Dunstan <andrew@dunslane.net> wrote:

On 2022-09-04 Su 02:19, John Naylor wrote:

Crake fails with

headerscheck:
Sep 04 01:56:01 In file included from /tmp/headerscheck.UNgmKy/test.c:2:
Sep 04 01:56:01
/home/andrew/bf/root/HEAD/pgsql.build/../pgsql/src/backend/utils/adt/jsonpath_internal.h:26:10:
fatal error: jsonpath_gram.h: No such file or directory
Sep 04 01:56:01 26 | #include "jsonpath_gram.h"
Sep 04 01:56:01 | ^~~~~~~~~~~~~~~~~
Sep 04 01:56:01 compilation terminated.
Sep 04 01:56:14 make: *** [GNUmakefile:138: headerscheck] Error 1
cpluspluscheck:
Sep 04 01:57:19 In file included from /tmp/cpluspluscheck.HhRV5k/test.cpp:3:
Sep 04 01:57:19
/home/andrew/bf/root/HEAD/pgsql.build/../pgsql/src/backend/utils/adt/jsonpath_internal.h:26:10:
fatal error: jsonpath_gram.h: No such file or directory
Sep 04 01:57:19 26 | #include "jsonpath_gram.h"
Sep 04 01:57:19 | ^~~~~~~~~~~~~~~~~

I don't get the same in an already-built tree. I also don't see an
exception for gramparse.h, which I imagine would have the same issue?

Probably happens because crake does vpath builds. I think you're going
to have to set up EXTRAINCLUDES for it - see near the bottom of the
headerscheck script. Possibly something like

src/backend/utils/adt/*)

EXTRAINCLUDES="-I $builddir/src/backend/utils/adt" ;;

I just tried a vpath build and the failing check doesn't reproduce for
me there. Any ideas?

--
John Naylor
EDB: http://www.enterprisedb.com

Hi,

On September 5, 2022 8:33:37 PM PDT, John Naylor <john.naylor@enterprisedb.com> wrote:

On Sun, Sep 4, 2022 at 8:34 PM Andrew Dunstan <andrew@dunslane.net> wrote:

On 2022-09-04 Su 02:19, John Naylor wrote:

Crake fails with

headerscheck:
Sep 04 01:56:01 In file included from /tmp/headerscheck.UNgmKy/test.c:2:
Sep 04 01:56:01
/home/andrew/bf/root/HEAD/pgsql.build/../pgsql/src/backend/utils/adt/jsonpath_internal.h:26:10:
fatal error: jsonpath_gram.h: No such file or directory
Sep 04 01:56:01 26 | #include "jsonpath_gram.h"
Sep 04 01:56:01 | ^~~~~~~~~~~~~~~~~
Sep 04 01:56:01 compilation terminated.
Sep 04 01:56:14 make: *** [GNUmakefile:138: headerscheck] Error 1
cpluspluscheck:
Sep 04 01:57:19 In file included from /tmp/cpluspluscheck.HhRV5k/test.cpp:3:
Sep 04 01:57:19
/home/andrew/bf/root/HEAD/pgsql.build/../pgsql/src/backend/utils/adt/jsonpath_internal.h:26:10:
fatal error: jsonpath_gram.h: No such file or directory
Sep 04 01:57:19 26 | #include "jsonpath_gram.h"
Sep 04 01:57:19 | ^~~~~~~~~~~~~~~~~

I don't get the same in an already-built tree. I also don't see an
exception for gramparse.h, which I imagine would have the same issue?

Probably happens because crake does vpath builds. I think you're going
to have to set up EXTRAINCLUDES for it - see near the bottom of the
headerscheck script. Possibly something like

src/backend/utils/adt/*)

EXTRAINCLUDES="-I $builddir/src/backend/utils/adt" ;;

I just tried a vpath build and the failing check doesn't reproduce for
me there. Any ideas?

Presumably it only happens if you previously triggered the parsers to be generated.

Andres

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

On Tue, Sep 6, 2022 at 10:54 AM Andres Freund <andres@anarazel.de> wrote:

I just tried a vpath build and the failing check doesn't reproduce for
me there. Any ideas?

Presumably it only happens if you previously triggered the parsers to be generated.

If you attempt to run the check before building, the first thing you get is

fatal error: utils/errcodes.h: No such file or directory

so not sure what you mean.

--
John Naylor
EDB: http://www.enterprisedb.com

John Naylor <john.naylor@enterprisedb.com> writes:

On Tue, Sep 6, 2022 at 10:54 AM Andres Freund <andres@anarazel.de> wrote:

Presumably it only happens if you previously triggered the parsers to be generated.

If you attempt to run the check before building, the first thing you get is
fatal error: utils/errcodes.h: No such file or directory
so not sure what you mean.

crake is still unhappy:

Sep 06 00:09:55 In file included from /tmp/headerscheck.AwLxya/test.c:2:
Sep 06 00:09:55 /home/andrew/bf/root/HEAD/pgsql.build/../pgsql/src/backend/utils/adt/jsonpath_internal.h:26:10: fatal error: jsonpath_gram.h: No such file or directory
Sep 06 00:09:55 26 | #include "jsonpath_gram.h"
Sep 06 00:09:55 | ^~~~~~~~~~~~~~~~~

I wonder if that file is not getting installed in a directory that
this test expects to search.

regards, tom lane

On Tue, Sep 6, 2022 at 11:49 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

crake is still unhappy:

Sep 06 00:09:55 In file included from /tmp/headerscheck.AwLxya/test.c:2:
Sep 06 00:09:55 /home/andrew/bf/root/HEAD/pgsql.build/../pgsql/src/backend/utils/adt/jsonpath_internal.h:26:10: fatal error: jsonpath_gram.h: No such file or directory
Sep 06 00:09:55 26 | #include "jsonpath_gram.h"
Sep 06 00:09:55 | ^~~~~~~~~~~~~~~~~

I wonder if that file is not getting installed in a directory that
this test expects to search.

There are two proposed fixes upthread, which I will copy below, but if
I can't reproduce locally, I don't have much confidence in which one
is the way to go to get crake green again:

Andrew:

Probably happens because crake does vpath builds. I think you're going
to have to set up EXTRAINCLUDES for it - see near the bottom of the
headerscheck script. Possibly something like

src/backend/utils/adt/*)

EXTRAINCLUDES="-I $builddir/src/backend/utils/adt" ;;

Andres:

It doesn't happen for gramparse.h because gram.h is copied to
src/include/parser (see src/backend/Makefile), and included as parser/gram.h.

I think this could be addressed by always adding something like
-iquote $builddir/$(dirname "$f") -iquote $srcdir/$(dirname "$f") \
to cpluspluscheck/headerscheck.

--
John Naylor
EDB: http://www.enterprisedb.com

I see why I couldn't reproduce: I neglected to add jsonpath_gram.h to
the backend's maintainer-clean so it failed to be deleted. Working on
the fix now.

--
John Naylor
EDB: http://www.enterprisedb.com