Can't compile PG 17 (master) from git under Msys2 autoconf

Hi Regina,

On 2024-Mar-27, Regina Obe wrote:

The error is

rm -f '../../src/include/storage/lwlocknames.h'
cp -pR ../../backend/storage/lmgr/lwlocknames.h
'../../src/include/storage/lwlocknames.h'
cp: cannot stat '../../backend/storage/lmgr/lwlocknames.h': No such file or
directory
make[1]: *** [Makefile:143: ../../src/include/storage/lwlocknames.h] Error 1
make[1]: Leaving directory '/projects/postgresql/postgresql-git/src/backend'
make: *** [../../src/Makefile.global:382: submake-generated-headers] Error 2

Hmm, I changed these rules again in commit da952b415f44, maybe your
problem is with that one? I wonder if changing the references to
../include/storage/lwlocklist.h to
$(topdir)/src/include/storage/lwlocklist.h (and similar things in
src/backend/storage/lmgr/Makefile) would fix it.

Do you run configure in the source directory or a separate build one?

--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"If it is not right, do not do it.
If it is not true, do not say it." (Marcus Aurelius, Meditations)

Hi Regina,

On 2024-Mar-27, Regina Obe wrote:

The error is

rm -f '../../src/include/storage/lwlocknames.h'
cp -pR ../../backend/storage/lmgr/lwlocknames.h
'../../src/include/storage/lwlocknames.h'
cp: cannot stat '../../backend/storage/lmgr/lwlocknames.h': No such
file or directory
make[1]: *** [Makefile:143: ../../src/include/storage/lwlocknames.h]
Error 1
make[1]: Leaving directory '/projects/postgresql/postgresql-

git/src/backend'

make: *** [../../src/Makefile.global:382: submake-generated-headers]
Error 2

Hmm, I changed these rules again in commit da952b415f44, maybe your
problem is with that one? I wonder if changing the references to
../include/storage/lwlocklist.h to $(topdir)/src/include/storage/lwlocklist.h
(and similar things in
src/backend/storage/lmgr/Makefile) would fix it.

Do you run configure in the source directory or a separate build one?

--
Álvaro Herrera Breisgau, Deutschland —
https://www.EnterpriseDB.com/
"If it is not right, do not do it.
If it is not true, do not say it." (Marcus Aurelius, Meditations)

I run in the source directory, but tried doing in a separate build directory and ran into the same issue.

Let me look at that commit and get back to you if that makes a difference.

Thanks,
Regina

Hi Regina,

On 2024-Mar-27, Regina Obe wrote:

The error is

rm -f '../../src/include/storage/lwlocknames.h'
cp -pR ../../backend/storage/lmgr/lwlocknames.h
'../../src/include/storage/lwlocknames.h'
cp: cannot stat '../../backend/storage/lmgr/lwlocknames.h': No such
file or directory
make[1]: *** [Makefile:143: ../../src/include/storage/lwlocknames.h]
Error 1
make[1]: Leaving directory '/projects/postgresql/postgresql-

git/src/backend'

make: *** [../../src/Makefile.global:382: submake-generated-headers]
Error 2

Hmm, I changed these rules again in commit da952b415f44, maybe your
problem is with that one? I wonder if changing the references to
../include/storage/lwlocklist.h to $(topdir)/src/include/storage/lwlocklist.h
(and similar things in
src/backend/storage/lmgr/Makefile) would fix it.

Do you run configure in the source directory or a separate build one?

--
Álvaro Herrera Breisgau, Deutschland —
https://www.EnterpriseDB.com/
"If it is not right, do not do it.
If it is not true, do not say it." (Marcus Aurelius, Meditations)

I tried the change

../include/storage/lwlocklist.h to $(top_builddir)/src/include/storage/lwlocklist.h

I assume you meant that instead of $(topdir)

But nah that made no difference. Your change was already in my patched version so isn't causing any issues.

But as stated, rolling back this change in src/backend/Makefile in 0a475f01a4a (November 6th commit) makes it work for me.

$(top_builddir)/src/include/storage/lwlocknames.h: storage/lmgr/lwlocknames.h
-   prereqdir=`cd '$(dir $<)' >/dev/null && pwd` && \
-     cd '$(dir $@)' && rm -f $(notdir $@) && \
-     $(LN_S) "$$prereqdir/$(notdir $<)" .
+   rm -f '$@'
+   $(LN_S) ../../backend/$< '$@'

 $(top_builddir)/src/include/utils/wait_event_types.h: utils/activity/wait_event_types.h
-   prereqdir=`cd '$(dir $<)' >/dev/null && pwd` && \
-     cd '$(dir $@)' && rm -f $(notdir $@) && \
-     $(LN_S) "$$prereqdir/$(notdir $<)" .
+   rm -f '$@'
+   $(LN_S) ../../backend/$< '$@'

Hi Regina,

On 2024-Mar-27, Regina Obe wrote:

The error is

rm -f '../../src/include/storage/lwlocknames.h'
cp -pR ../../backend/storage/lmgr/lwlocknames.h
'../../src/include/storage/lwlocknames.h'
cp: cannot stat '../../backend/storage/lmgr/lwlocknames.h': No such
file or directory
make[1]: *** [Makefile:143: ../../src/include/storage/lwlocknames.h]
Error 1
make[1]: Leaving directory '/projects/postgresql/postgresql-

git/src/backend'

make: *** [../../src/Makefile.global:382: submake-generated-headers]
Error 2

Hmm, I changed these rules again in commit da952b415f44, maybe your
problem is with that one? I wonder if changing the references to
../include/storage/lwlocklist.h to
$(topdir)/src/include/storage/lwlocklist.h
(and similar things in
src/backend/storage/lmgr/Makefile) would fix it.

Do you run configure in the source directory or a separate build one?

--
Álvaro Herrera Breisgau, Deutschland —
https://www.EnterpriseDB.com/
"If it is not right, do not do it.
If it is not true, do not say it." (Marcus Aurelius, Meditations)

I tried the change

../include/storage/lwlocklist.h to
$(top_builddir)/src/include/storage/lwlocklist.h

I assume you meant that instead of $(topdir)

But nah that made no difference. Your change was already in my patched
version so isn't causing any issues.

I think I got something not too far off from what's there now that works under my msys2 setup again. This is partly using your idea of using $(top_builddir) to qualify the path but in the LN_S section that is causing me grief.
This seems to work okay building in tree and out of tree.
By changing these lines in src/backend/Makefile

$(top_builddir)/src/include/storage/lwlocknames.h: storage/lmgr/lwlocknames.h
        rm -f '$@'
-       $(LN_S) ../../backend/$< '$@'
+       $(LN_S) $(top_builddir)/src/backend/$< '$@'

 $(top_builddir)/src/include/utils/wait_event_types.h: utils/activity/wait_event_types.h
        rm -f '$@'
-       $(LN_S) ../../backend/$< '$@'
+       $(LN_S) $(top_builddir)/src/backend/$< '$@'

I've also attached as a patch.

Thanks,
Regina

On 2024-Apr-04, Regina Obe wrote:

I think I got something not too far off from what's there now that works under my msys2 setup again. This is partly using your idea of using $(top_builddir) to qualify the path but in the LN_S section that is causing me grief.
This seems to work okay building in tree and out of tree.
By changing these lines in src/backend/Makefile

$(top_builddir)/src/include/storage/lwlocknames.h: storage/lmgr/lwlocknames.h
rm -f '$@'
-       $(LN_S) ../../backend/$< '$@'
+       $(LN_S) $(top_builddir)/src/backend/$< '$@'

$(top_builddir)/src/include/utils/wait_event_types.h: utils/activity/wait_event_types.h
rm -f '$@'
-       $(LN_S) ../../backend/$< '$@'
+       $(LN_S) $(top_builddir)/src/backend/$< '$@'

I've also attached as a patch.

Hmm, that's quite strange ... it certainly doesn't work for me. Maybe
the LN_S utility is resolving the symlink at creation time, rather than
letting it be a reference to be resolved later. Apparently, the only Msys2 buildfarm animal is now using Meson,
so we don't have any representative animal for your situation.

What does LN_S do for you anyway? Looking at
https://stackoverflow.com/questions/61594025/symlink-in-msys2-copy-or-hard-link
it sounds like this would work if the MSYS environment variable was set
to winsymlinks (or maybe not. I don't know if a "windows shortcut" would
be usable in this case.)

--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Nadie está tan esclavizado como el que se cree libre no siéndolo" (Goethe)

I think I got something not too far off from what's there now that works

under my msys2 setup again. This is partly using your idea of using
$(top_builddir) to qualify the path but in the LN_S section that is causing me
grief.

This seems to work okay building in tree and out of tree.
By changing these lines in src/backend/Makefile

$(top_builddir)/src/include/storage/lwlocknames.h:

storage/lmgr/lwlocknames.h

rm -f '$@'
-       $(LN_S) ../../backend/$< '$@'
+       $(LN_S) $(top_builddir)/src/backend/$< '$@'

$(top_builddir)/src/include/utils/wait_event_types.h:

utils/activity/wait_event_types.h

rm -f '$@'
-       $(LN_S) ../../backend/$< '$@'
+       $(LN_S) $(top_builddir)/src/backend/$< '$@'

I've also attached as a patch.

Hmm, that's quite strange ... it certainly doesn't work for me. Maybe the
LN_S utility is resolving the symlink at creation time, rather than letting it be a
reference to be resolved later. Apparently, the only Msys2 buildfarm animal is
now using Meson, so we don't have any representative animal for your
situation.

What does LN_S do for you anyway? Looking at
https://stackoverflow.com/questions/61594025/symlink-in-msys2-copy-or-
hard-link
it sounds like this would work if the MSYS environment variable was set to
winsymlinks (or maybe not. I don't know if a "windows shortcut" would be
usable in this case.)

--
Álvaro Herrera PostgreSQL Developer —
https://www.EnterpriseDB.com/
"Nadie está tan esclavizado como el que se cree libre no siéndolo" (Goethe)

I think it ends up doing a copy thus the copy error in my log failures which don't exist anywhere in the Makefil

cp -pR ../../backend/storage/lmgr/lwlocknames.h

Sorry for not checking on a linux system. I was thinking I should have done that first.

Thanks,
Regina

On 2024-Apr-05, Regina Obe wrote:

I think it ends up doing a copy thus the copy error in my log failures which don't exist anywhere in the Makefil

cp -pR ../../backend/storage/lmgr/lwlocknames.h

Sorry for not checking on a linux system. I was thinking I should have done that first.

Ah yeah, that's per configure:

if ln -s conf$$.file conf$$ 2>/dev/null; then
as_ln_s='ln -s'
# ... but there are two gotchas:
# 1) On MSYS, both `ln -s file dir' and `ln file dir' fail.
# 2) DJGPP < 2.04 has no symlinks; `ln -s' creates a wrapper executable.
# In both cases, we have to default to `cp -pR'.
ln -s conf$$.file conf$$.dir 2>/dev/null && test ! -f conf$$.exe ||
as_ln_s='cp -pR'

I guess we need to patch the rule so that the LN_S is called so that
it'd resolve correctly in both cases. I guess the easy option is to go
back to the original recipe and update the comment to indicate that we
do it to placate Msys2. Here it is with the old comment:

-# The point of the prereqdir incantation in some of the rules below is to
-# force the symlink to use an absolute path rather than a relative path.
-# For headers which are generated by make distprep, the actual header within
-# src/backend will be in the source tree, while the symlink in src/include
-# will be in the build tree, so a simple ../.. reference won't work.
-# For headers generated during regular builds, we prefer a relative symlink.

$(top_builddir)/src/include/storage/lwlocknames.h: storage/lmgr/lwlocknames.h
- prereqdir=`cd '$(dir $<)' >/dev/null && pwd` && \
- cd '$(dir $@)' && rm -f $(notdir $@) && \
- $(LN_S) "$$prereqdir/$(notdir $<)" .

Maybe it's possible to make this simpler, as it looks overly baroque,
and we don't really need absolute paths anyway -- we just need the path
resolved at the right time.

--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/

I think it ends up doing a copy thus the copy error in my log failures
which don't exist anywhere in the Makefil

cp -pR ../../backend/storage/lmgr/lwlocknames.h

Sorry for not checking on a linux system. I was thinking I should have done

that first.

Ah yeah, that's per configure:

if ln -s conf$$.file conf$$ 2>/dev/null; then
as_ln_s='ln -s'
# ... but there are two gotchas:
# 1) On MSYS, both `ln -s file dir' and `ln file dir' fail.
# 2) DJGPP < 2.04 has no symlinks; `ln -s' creates a wrapper executable.
# In both cases, we have to default to `cp -pR'.
ln -s conf$$.file conf$$.dir 2>/dev/null && test ! -f conf$$.exe ||
as_ln_s='cp -pR'

I guess we need to patch the rule so that the LN_S is called so that it'd resolve
correctly in both cases. I guess the easy option is to go back to the original
recipe and update the comment to indicate that we do it to placate Msys2.
Here it is with the old comment:

-# The point of the prereqdir incantation in some of the rules below is to
-# force the symlink to use an absolute path rather than a relative path.
-# For headers which are generated by make distprep, the actual header
within
-# src/backend will be in the source tree, while the symlink in src/include
-# will be in the build tree, so a simple ../.. reference won't work.
-# For headers generated during regular builds, we prefer a relative symlink.

$(top_builddir)/src/include/storage/lwlocknames.h:
storage/lmgr/lwlocknames.h
- prereqdir=`cd '$(dir $<)' >/dev/null && pwd` && \
- cd '$(dir $@)' && rm -f $(notdir $@) && \
- $(LN_S) "$$prereqdir/$(notdir $<)" .

Maybe it's possible to make this simpler, as it looks overly baroque, and we
don't really need absolute paths anyway -- we just need the path resolved at
the right time.

--
Álvaro Herrera Breisgau, Deutschland —
https://www.EnterpriseDB.com/

Yah I was thinking it was removed cause
no one could figure out why it was so complicated and decided to make it more readable.