PSA: Autoconf has risen from the dead

On Sun, Jan 23, 2022, at 17:29, Tom Lane wrote:

While chasing something else, I was surprised to learn that the
Autoconf project has started to make releases again. There are
2.70 (2020-12-08) and 2.71 (2021-01-28) versions available at
https://ftp.gnu.org/gnu/autoconf/

Right now, I'm not sure we care; there seems to be more
enthusiasm for switching to meson. But if that idea falls
through, we should update to a newer autoconf release.

Speaking of autoconf,

I don't have much experience in this area, but I noted there is
an AC_CACHE_SAVE feature to speed up rerunning ./configure,
necessary when it stops with an error due to some missing dependency.

Is there a good reason why AC_CACHE_SAVE is not used?

/Joel

"Joel Jacobson" <joel@compiler.org> writes:

I don't have much experience in this area, but I noted there is
an AC_CACHE_SAVE feature to speed up rerunning ./configure,
necessary when it stops with an error due to some missing dependency.
Is there a good reason why AC_CACHE_SAVE is not used?

Dunno ... it looks like that adds cycles to non-error cases,
which seems like optimizing for the wrong thing.

In any case, right at the moment is probably a bad time to be
working on improvements for configure per se. We can come
back to this if the meson idea crashes and burns.

regards, tom lane

On 23.01.22 17:29, Tom Lane wrote:

While chasing something else, I was surprised to learn that the
Autoconf project has started to make releases again. There are
2.70 (2020-12-08) and 2.71 (2021-01-28) versions available at
https://ftp.gnu.org/gnu/autoconf/

I have patches ready for this at
https://github.com/petere/postgresql/tree/autoconf-updates.

My thinking was to wait until Autoconf 2.71 has trickled down into the
OS versions that developers are likely to use. To survey that, I'm tracking

https://packages.debian.org/sid/autoconf [in testing]
https://packages.ubuntu.com/search?keywords=autoconf [in jammy, will be
22.04 LTS]
https://src.fedoraproject.org/rpms/autoconf [in Fedora 36, planned
2022-04-19]
https://formulae.brew.sh/formula/autoconf [done]

Currently, I think early PG16 might be good time to do this update.

Hi,

On 2022-01-23 11:29:17 -0500, Tom Lane wrote:

Right now, I'm not sure we care; there seems to be more
enthusiasm for switching to meson. But if that idea falls
through, we should update to a newer autoconf release.

Depending on the number of portability fixes in those releases the
backbranches could be reason enough to move to a newer autoconf, even if we
get to meson in HEAD? Of course only if there's more things fixed than
broken...

Greetings,

Andres Freund

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

I have patches ready for this at
https://github.com/petere/postgresql/tree/autoconf-updates.
My thinking was to wait until Autoconf 2.71 has trickled down into the
OS versions that developers are likely to use.

I find that kind of irrelevant, because we expect people to install
autoconf from source anyway to avoid distro-specific behavior.
I suppose that waiting for it to get out into the wild might be good
from the standpoint of being sure it's bug-free, though.

Do these versions fix any bugs that affect us (i.e., that we've
not already created workarounds for)?

regards, tom lane

On 24.01.22 15:14, Tom Lane wrote:

Do these versions fix any bugs that affect us (i.e., that we've
not already created workarounds for)?

The only thing that could be of interest is that the workaround we are
carrying in config/check_decls.m4 was originally upstreamed by Noah, but
was then later partially reverted and replaced by a different solution.
Further explanation is here:

https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=ec90049dfcf4538750e61d675d885157fa5ca7f8

I don't think it has affected us in practice, though.

On 24.01.22 09:11, Peter Eisentraut wrote:

On 23.01.22 17:29, Tom Lane wrote:

While chasing something else, I was surprised to learn that the
Autoconf project has started to make releases again.  There are
2.70 (2020-12-08) and 2.71 (2021-01-28) versions available at
https://ftp.gnu.org/gnu/autoconf/

I have patches ready for this at
https://github.com/petere/postgresql/tree/autoconf-updates.

I have updated this for 16devel and registered it in the commit fest.

To summarize:

- Autoconf 2.71 has been out for 1.5 years.
- It is available in many recently updated OSs.
- It allows us to throw away several workarounds.

Also:

- The created configure appears to be a bit faster, especially in the
cached case.
- It supports checks for C11 features, which is something we might want
to consider in the fullness of time.

Hence:

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

To summarize:
- Autoconf 2.71 has been out for 1.5 years.
- It is available in many recently updated OSs.
- It allows us to throw away several workarounds.
Hence:

Currently, I think early PG16 might be good time to do this update.

In preparation for reviewing this, I tried to install autoconf 2.71
from source locally. All went well on my RHEL8 workstation, but
autoconf's testsuite falls over rather badly on my macOS laptop [1]https://lists.gnu.org/archive/html/bug-autoconf/2022-07/msg00000.html.
It fails differently on another Mac where I have a MacPorts
installation at the head of the search path [2]https://lists.gnu.org/archive/html/bug-autoconf/2022-07/msg00001.html.

After sending the requested reports, I tried scanning the bug-autoconf
archives, and found a similar report that was answered thus [3]https://lists.gnu.org/archive/html/bug-autoconf/2022-04/msg00002.html:

I *think* this is the same problem as https://savannah.gnu.org/support/?110492
: current Autoconf doesn't work correctly with the (rather old) version of GNU
M4 that ships with MacOS. Please try installing a current version of GNU M4 in
your PATH and then retry the build and testsuite.

So that explains part of it: most of the failures are down to using
Apple's hoary m4 instead of the one from MacPorts. We could usefully
warn about that in our own docs, perhaps. But there's still these
scary failures:

509: AC_CHECK_HEADER_STDBOOL FAILED (acheaders.at:9)
514: AC_HEADER_STDBOOL FAILED (acheaders.at:14)

The generated autoconf program builds the same output files as you have
in your patch, and running the configure script gives the correct answer
from AC_HEADER_STDBOOL, so I'm not sure what these test failures are
unhappy about. Still, this is not a good look for a mainstream
development platform. I wonder if we ought to wait for a fix.

regards, tom lane

[1]: https://lists.gnu.org/archive/html/bug-autoconf/2022-07/msg00000.html
[2]: https://lists.gnu.org/archive/html/bug-autoconf/2022-07/msg00001.html
[3]: https://lists.gnu.org/archive/html/bug-autoconf/2022-04/msg00002.html

I wrote:

So that explains part of it: most of the failures are down to using
Apple's hoary m4 instead of the one from MacPorts. We could usefully
warn about that in our own docs, perhaps.

Hmm. I have just spent a very frustrating hour trying, and failing,
to build any version of GNU m4 from source on either RHEL8 or current
macOS. I don't quite understand why: neither the RPM specfile nor
the MacPorts recipe for their respective m4 packages seem to contain
any special hacks, so that it looks like the usual "configure; make;
make check; make install" procedure ought to work fine. But it doesn't.
I hit build failures (apparently because the source code is far too much
in bed with nonstandard aspects of libc), or get an executable that
SIGABRT's instantly, or if it doesn't do that it still fails some
self-tests. With the latest 1.4.19 on macOS, the configure script
hangs up, for crissakes.

I am now feeling *very* hesitant about doing anything where we might
be effectively asking people to build m4 for themselves.

On the whole, I'm questioning the value of messing with our autoconf
infrastructure at this stage. We did agree at PGCon that we'd keep
it going for a couple years more, but it's not real clear to me why
we can't limp along with 2.69 until we decide to drop it.

regards, tom lane

On Sat, Jul 2, 2022 at 1:42 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

On the whole, I'm questioning the value of messing with our autoconf
infrastructure at this stage. We did agree at PGCon that we'd keep
it going for a couple years more, but it's not real clear to me why
we can't limp along with 2.69 until we decide to drop it.

If building it on macOS is going to be annoying, then -1 from me for
upgrading to a new version until that's resolved.

Hmm, I also don't know how annoying it's going to be to get the new
ninja/meson stuff working on macOS ... I really hope someone puts a
good set of directions on the wiki or in the documentation or
someplace.

--
Robert Haas
EDB: http://www.enterprisedb.com

Robert Haas <robertmhaas@gmail.com> writes:

Hmm, I also don't know how annoying it's going to be to get the new
ninja/meson stuff working on macOS ... I really hope someone puts a
good set of directions on the wiki or in the documentation or
someplace.

If you use MacPorts it's just "install those packages", and I imagine
the same for Homebrew. I've not tried build-from-source on modern
platforms.

One thing I think we lack data on is whether we're going to need a
policy similar to everyone-must-use-exactly-this-autoconf-version.
If we do, that will greatly raise the importance of building from
source.

regards, tom lane

Hi,

On 2022-07-03 10:50:49 -0400, Tom Lane wrote:

Robert Haas <robertmhaas@gmail.com> writes:

Hmm, I also don't know how annoying it's going to be to get the new
ninja/meson stuff working on macOS ... I really hope someone puts a
good set of directions on the wiki or in the documentation or
someplace.

Yea, I guess I should start a documentation section...

I've only used homebrew on mac, but with that it should be something along the
lines of

brew install meson
meson setup --buildtype debug -Dcassert=true build-directory
cd build-directory
ninja

of course if you want to build against some dependencies and / or run tap
tests, you need to do something similar to what you have to do for
configure. I.e.
- install perl modules [1]https://github.com/anarazel/postgres/blob/meson/.cirrus.yml#L638
- tell the build about location of homebrew [2]https://github.com/anarazel/postgres/blob/meson/.cirrus.yml#L742

If you use MacPorts it's just "install those packages", and I imagine
the same for Homebrew. I've not tried build-from-source on modern
platforms.

I've done some semi automated testing (to be turned fully automatic) across
meson versions that didn't so far show any need for that. We do require a
certain minimum version of meson (indicated in the top-level meson.build,
raises an error if not met), which in turn requires a minimum version of ninja
(also errors).

The windows build with msbuild is slower on older versions of meson that are
unproblematic on other platforms. But given you're not going to install an
outdated meson from $package-manager there, I don't think it's worth worrying
about.

Greetings,

Andres Freund

[1]: https://github.com/anarazel/postgres/blob/meson/.cirrus.yml#L638
[2]: https://github.com/anarazel/postgres/blob/meson/.cirrus.yml#L742

On Sun, Jul 3, 2022 at 1:17 PM Andres Freund <andres@anarazel.de> wrote:

Yea, I guess I should start a documentation section...

I've only used homebrew on mac, but with that it should be something along the
lines of

brew install meson
meson setup --buildtype debug -Dcassert=true build-directory
cd build-directory
ninja

of course if you want to build against some dependencies and / or run tap
tests, you need to do something similar to what you have to do for
configure. I.e.
- install perl modules [1]
- tell the build about location of homebrew [2]

Since I'm a macports user I hope at some point we'll have directions
for that as well as for homebrew.

--
Robert Haas
EDB: http://www.enterprisedb.com

Hi,

On 2022-07-05 14:42:03 -0400, Robert Haas wrote:

On Sun, Jul 3, 2022 at 1:17 PM Andres Freund <andres@anarazel.de> wrote:

Yea, I guess I should start a documentation section...

I've only used homebrew on mac, but with that it should be something along the
lines of

brew install meson
meson setup --buildtype debug -Dcassert=true build-directory
cd build-directory
ninja

of course if you want to build against some dependencies and / or run tap
tests, you need to do something similar to what you have to do for
configure. I.e.
- install perl modules [1]
- tell the build about location of homebrew [2]

Since I'm a macports user I hope at some point we'll have directions
for that as well as for homebrew.

I am not a normal mac user, it looks hard to run macos in a VM, and I'm not
sure it's wise to mix macports and homebrew on my test box. So I don't want to
test it myself.

But it looks like it's just
sudo port install meson

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:

On 2022-07-05 14:42:03 -0400, Robert Haas wrote:

Since I'm a macports user I hope at some point we'll have directions
for that as well as for homebrew.

But it looks like it's just
sudo port install meson

Yeah, that's what I did to install it locally. The ninja package
has some weird name (ninja-build or some such), but you don't have
to remember that because installing meson is enough to pull it in.

I dunno anything about the other steps Andres mentioned, but
presumably they're independent of where you got meson from.

regards, tom lane

On Tue, Jul 5, 2022 at 2:52 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andres Freund <andres@anarazel.de> writes:

On 2022-07-05 14:42:03 -0400, Robert Haas wrote:

Since I'm a macports user I hope at some point we'll have directions
for that as well as for homebrew.

But it looks like it's just
sudo port install meson

Yeah, that's what I did to install it locally. The ninja package
has some weird name (ninja-build or some such), but you don't have
to remember that because installing meson is enough to pull it in.

I dunno anything about the other steps Andres mentioned, but
presumably they're independent of where you got meson from.

That seems simple enough that even I can handle it!

--
Robert Haas
EDB: http://www.enterprisedb.com

Hi,

On 2022-07-05 14:52:05 -0400, Tom Lane wrote:

I dunno anything about the other steps Andres mentioned, but
presumably they're independent of where you got meson from.

Yea. They might not be independent of where you get other dependencies from
though. Does macports install headers / libraries into a path that's found by
default? Or does one have to pass --with-includes / --with-libs to configure
and set PKG_CONFIG_PATH, like with homebrew?

Except that with meson doing PKG_CONFIG_PATH should suffice for most (all?)
dependencies on macos, and that the syntax for with-includes/libs is a bit
different (-Dextra_include_dirs=... and -Dextra_lib_dirs=...) and that
optionally one can use a parameter (--pkg-config-path) instead of
PKG_CONFIG_PATH, that part shouldn't really differ from what's neccesary
for configure.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:

Yea. They might not be independent of where you get other dependencies from
though. Does macports install headers / libraries into a path that's found by
default? Or does one have to pass --with-includes / --with-libs to configure
and set PKG_CONFIG_PATH, like with homebrew?

What are you expecting to need PKG_CONFIG_PATH for? Or more precisely,
why would meson/ninja create any new need for that that doesn't exist
in the autoconf case?

regards, tom lane

Hi,

On 2022-07-05 15:06:31 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

Yea. They might not be independent of where you get other dependencies from
though. Does macports install headers / libraries into a path that's found by
default? Or does one have to pass --with-includes / --with-libs to configure
and set PKG_CONFIG_PATH, like with homebrew?

What are you expecting to need PKG_CONFIG_PATH for? Or more precisely,
why would meson/ninja create any new need for that that doesn't exist
in the autoconf case?

It's just used in more cases than before, with fallback to non-pkg-config in
most cases. I think all dependencies besides perl can use pkg-config. So all
that changes compared to AC is that you might not need to pass extra
include/lib paths for some dependencies that needed it before, if you set/pass
PKG_CONFIG_PATH.

Greetings,

Andres Freund

On Tue, Jul 5, 2022 at 3:02 PM Andres Freund <andres@anarazel.de> wrote:

Yea. They might not be independent of where you get other dependencies from
though. Does macports install headers / libraries into a path that's found by
default? Or does one have to pass --with-includes / --with-libs to configure
and set PKG_CONFIG_PATH, like with homebrew?

My configure switches include: --with-libraries=/opt/local/lib
--with-includes=/opt/local/include

I don't do anything with PKG_CONFIG_PATH.

--
Robert Haas
EDB: http://www.enterprisedb.com

... anyway, to get back to the main point of this thread:

The Autoconf developers were pretty responsive to my bug reports,
and after some back-and-forth we determined that:

1. The minimum GNU m4 version for modern autoconf is 1.4.8; this
is directly traceable to intentional behavioral changes in that
version, so it's a pretty hard requirement. They've updated their
own configure script to enforce that minimum.

2. The macOS-specific problems I saw with the STDBOOL tests are
resolved by the attached patch, which should also appear in 2.72.
Since AC_HEADER_STDBOOL appears to work correctly in our usage
anyway, this is only important if you're the kind of person who
likes to see 100% pass from a tool's own self-tests before you
install it.

So as far as autoconf itself is concerned, we could probably move
forward, perhaps after waiting for 2.72. The difficulty here is the
prospect that some people might find themselves having to install a
newer GNU m4, because GNU m4 is a hot mess. Many post-1.4.8 versions
flat out don't compile on $your-favorite-platform [1]https://lists.gnu.org/archive/html/bug-autoconf/2022-07/msg00004.html, and many
others contain a showstopper bug (that's rejected by a runtime test in
autoconf's configure, independently of the min-version test) [2]https://lists.gnu.org/archive/html/bug-autoconf/2022-07/msg00006.html.
If you don't have a pretty recent m4 available from a package manager,
you might be in for a lot of hair-pulling.

The flip side of that is that probably nobody really needs to
update the configure script on non-mainstream platforms, so
maybe this wouldn't matter to us too much in practice.

On the whole though, my feeling is that autoconf 2.71 doesn't
offer enough to us to justify possibly causing substantial pain
for a few developers. I recommend setting this project aside
for now. We can always reconsider if the situation changes.

regards, tom lane

[1]: https://lists.gnu.org/archive/html/bug-autoconf/2022-07/msg00004.html
[2]: https://lists.gnu.org/archive/html/bug-autoconf/2022-07/msg00006.html

On 16.07.22 17:26, Tom Lane wrote:

On the whole though, my feeling is that autoconf 2.71 doesn't
offer enough to us to justify possibly causing substantial pain
for a few developers. I recommend setting this project aside
for now. We can always reconsider if the situation changes.

Ok, let's do that.