Is a modern build system acceptable for older platforms

About CMake:
We can use 3.9 version very well, it has all features for us, at least for
my postgres_cmake branch and I have the experience to introduce features to
cmake special for postgres build.
Also, cmake very easily build even for Solaris or AIX (on my webpage I have
examples to build postgres with cmake on this systems).

But, I totally agree with this topic, we can't keep same support matrix and
can't keep 100% same behavior and a "build" interface. Maybe, behavior
should be the second question.
In my opinion, I made too much now without the answer to this
important question.

2018-04-19 14:30 GMT+09:00 Catalin Iacob <iacobcatalin@gmail.com>:

The above is all about getting the build system to work at all. If that
isn't a showstopper there's a subsequent discussion to be had about older
platforms where one could get the build system to work but convenient
packages are missing. For example not even RHEL7 has any Python3 packages
in the base system (it does in Software Collections though) which means
some extra hoops on getting meson running there. And RHEL5 is in an even
worse spot as it has no Software Collections, who knows if Python 3 builds
on it out of the box etc.

I would expect that a new version of software should not target versions of
platform that are end of full support. Per
https://access.redhat.com/support/policy/updates/errata currently only
RHEL7 is at Full Support, and RHEL5 is already past Product Retirement. I
would say it's fine to support these at already released branches, but
limiting .

PostGIS has several forks that move it towards CMake (five-year-old ticket
https://trac.osgeo.org/postgis/ticket/2362, forks
https://github.com/nextgis-borsch/postgis,
https://github.com/mloskot/postgis/tree/cmake-build) - none of these are
upstream mostly because there's an expectation to match the Postgres build
system. If Postgres moved to CMake (there are already CMake-enabled forks
available for people who) then I expect PostGIS to quickly catch up.

A lot of libraries PostGIS depends on are already built using CMake, so if
the platform has recent PostGIS it has CMake available somehow.

Darafei Praliaskouski,
GIS Engineer / Juno Minsk

=?UTF-8?Q?Darafei_=22Kom=D1=8Fpa=22_Praliaskouski?= <me@komzpa.net> writes:

The above is all about getting the build system to work at all. If that
isn't a showstopper there's a subsequent discussion to be had about older
platforms where one could get the build system to work but convenient
packages are missing. ...

I would expect that a new version of software should not target versions of
platform that are end of full support.

The other side of that argument is that allowing a build system we haven't
even adopted yet to dictate which platforms we can support is definitely
letting the tail wag the dog.

My gut reaction to Catalin's list is that requiring C+11 is a pretty
darn high bar to clear for older platforms. I have a positive impression
of python's portability, so requiring a recent python version might not
be too awful ... but then requiring ninja pretty much tosses away the
advantage again. So, while in principle you could probably get these
toolchains going on an old platform, the reality is that moving to either
will amount to "we're desupporting everything that wasn't released in
this decade". That's a pretty big shift from the project's traditional
mindset. It's possible that our users wouldn't care; I don't know.
But to me it's a significant minus that we'd have to set against whatever
pluses are claimed for a move.

regards, tom lane

My gut reaction to Catalin's list is that requiring C+11 is a pretty
darn high bar to clear for older platforms.

It's only for latest version and we can support version 3.9 with C++98 I
think at least 5 years.
3.9.6 was realease in November 10, 2017 .

That's a pretty big shift from the project's traditional

mindset.

Sure, but I think time to time it should be happen.

But to me it's a significant minus that we'd have to set against whatever

pluses are claimed for a move.

It's obvious minuses but I still can't understand your position on this
question.

Regards

On Thu, Apr 19, 2018 at 10:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

The other side of that argument is that allowing a build system we haven't
even adopted yet to dictate which platforms we can support is definitely
letting the tail wag the dog.

My gut reaction to Catalin's list is that requiring C+11 is a pretty
darn high bar to clear for older platforms. I have a positive impression
of python's portability, so requiring a recent python version might not
be too awful ... but then requiring ninja pretty much tosses away the
advantage again. So, while in principle you could probably get these
toolchains going on an old platform, the reality is that moving to either
will amount to "we're desupporting everything that wasn't released in
this decade". That's a pretty big shift from the project's traditional
mindset. It's possible that our users wouldn't care; I don't know.
But to me it's a significant minus that we'd have to set against whatever
pluses are claimed for a move.

Yeah, I agree. I am not deathly opposed to moving, but I'd like to be
convinced that we're going to get real advantages from such a move,
and so far I'm not. The arguments thus far advanced for moving boil
down to (1) the current system is kind of old and creaky, which is
true but which I'm not sure is by itself a compelling argument for
changing anything, and (2) it might make things easier on Windows,
which could be a sufficiently good reason but I don't think I've seen
anyone explain exactly how much easier it will make things and in what
ways.

I think it's inevitable that a move like this will create some
disruption -- developers will need to install and learn new tools,
buildfarm members will need updating, and there will be some bugs.
None of that is a blocker, but the gains need to outweigh those
disadvantages, and we can't judge whether they do without a clear
explanation of what the gains will be.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

(2) it might make things easier on Windows,
which could be a sufficiently good reason but I don't think I've seen
anyone explain exactly how much easier it will make things and in what
ways.

1. You can remove tools/msvc folder because all your build rules will be
universal. (cmake build now have much fewer lines of code)
2. You can forget about terminal in Windows (for windows guys it's
important)
3. You can normally check environment on Windows, right now we have
hardcoded headers and many options. Configure process will be same on all
platforms.
4. You can generate not only GNU Make or MSVC project, you also can make
Xcode projects, Ninja or NMake for build under MSVC Make. For Windows, you
also can easily change MSVC to Clang it's not hardcoded at all.
5. With CMake you have an easy way to build extra modules (plugins), I have
already working prototype for windows PGXS. A plugin should just include
.cmake file generated with Postgres build.
Example:
https://github.com/stalkerg/postgres_cmake/blob/cmake/contrib/adminpack/CMakeLists.txt
If PGXS is True it's mean we build module outside postgres.

But in my opinion, you should just try CMake to figure out all benefits.

we can't judge whether they do without a clear explanation of what the

gains will be

I think it's not that thing what easy to explain. Main benefits not in unix
console area and C language...

On 27/04/18 19:10, Yuriy Zhuravlev wrote:

1. You can remove tools/msvc folder because all your build rules will
be universal. (cmake build now have much fewer lines of code)
2. You can forget about terminal in Windows (for windows guys it's
important)
3. You can normally check environment on Windows, right now we have
hardcoded headers and many options. Configure process will be same on
all platforms.
4. You can generate not only GNU Make or MSVC project, you also can
make Xcode projects, Ninja or NMake for build under MSVC Make. For
Windows, you also can easily change MSVC to Clang it's not hardcoded
at all.
5. With CMake you have an easy way to build extra modules (plugins), I
have already working prototype for windows PGXS.  A plugin should just
include .cmake file generated with Postgres build.
Example:
https://github.com/stalkerg/postgres_cmake/blob/cmake/contrib/adminpack/CMakeLists.txt
If PGXS is True it's mean we build module outside postgres.

But in my opinion, you should just try CMake to figure out all benefits.

I note that Mysql (yeah I know, we don't love 'em greatly, but their
product niche is similar to ours) and Ceph (ok it is a distributed
storage system but still a highly popular open src product) have
switched to using cmake (relatively) recently. Both these projects were
using autoconf etc related builds previously and seem to be doing just
fine with cmake.

regards
Mark

On 27.04.2018 10:45, Mark Kirkwood wrote:

I note that Mysql (yeah I know, we don't love 'em greatly, but their
product niche is similar to ours) and Ceph (ok it is a distributed
storage system but still a highly popular open src product) have
switched to using cmake (relatively) recently. Both these projects were
using autoconf etc related builds previously and seem to be doing just
fine with cmake.

I lived through that transition at MySQL, and later at SkySQL/MariaDB

Windows builds have been using CMake since somewhere in the MySQL 5.0
series at least already.

For a while autotools and CMake build systems coexisted side by side,
until everything was unified to use CMake only in the 5.5 series which
became "GA" in 2010, so "(relatively) recently" is rather relative.

Having to maintain two different build systems, and keep them in sync,
obviously wasn't a good thing to do in the long run and CMake (plus
CPack and friends) has proven itself to be "good enough" for quite
a while.

There are several things that autotools provide out of the box that
I still miss with CMake.

The most important one being "make distcheck" to check that creating
a source distribution tarball, unpacking it, doing an "out of source"
build with it that does not modify the actual source tree, runs test
and applies some release best practice sanity checks by e.g. checking
whether the ChangeLog file looks up to date.

As far as I can tell there's no CMake equivalent to that at all,
which is especially "funny" as CMake used to advertise their preference
for out-of-source builds as an advantage over autotools. We often enough
ended up with builds writing to the source directory tree instead of the
build tree over the years.

Makefiles generated by automake are more feature rich in general,
which is understandable as its the only backend it has to support.

Some of the CMake choices there are just weird though, like their
refusal to have "make uninstall" supported out of the box.

Some may also think that getting rid of a mix of bash, m4, and Makefile
code snippets may be a good thing (which in itself is true), but CMake
replaces this with its own home grown language that's not used anywhere
else, and which comes with only a very rudimentary lex/yacc parser,
leading to several semi-consistent function argument parsing hacks.

The bundled pacakge libs that come with CMake made some builds more
easy, but this part didn't seem to be seeing much love anymore last
time I looked. Meanwhile the Autoconf Macro Archive has at least partly
closed that gap.

Also last time I looked CMake hat nothing really comparable to
autotools submodules (bundled sub-projects that come with their
own autotools infrastructure and could be built standalone, but
in a bunlded context will inherit all "configure" settings from
the top level invocation).

There was also some weird thing about CMake changing shared library
default locations in already built binaries on the fly on "make
install", so that they work both in the install and build context,
e.g. for running tests before installing. Autotools handle this
by building for the install context right away, and setting up
wrapper scripts that set up load paths for libs in the build
context for pre-install testing. In this particular case I don't
really trust either approach, so that one's a tie.

What else?

CMakes more aggressive caching behavior can be confusing, but
then again that's really just a matter of preference.

It's command line argument parsing and help output is inferior
to autotools configure, all project specific options have to
start with a -D, and help output for these is strictly alphabetical,
while with autoconf you can group related options in help output,
and on modern Linux distributions there's good tab completion
for them, too.

cmake-gui is advertised to solve much of this, but it still
suffers from the alphabetic listing problem.

I could probably continue with this brain dump forever, but in the
end it comes down to:

There's no real alternative when you have to support windows, and
it is "good enough" on Unix, so that maintaining CMake and autotool
build setups in parallel isn't really justified in the long run"

PS: when you actually watch a full MariaDB CMake

--
Hartmut, former MySQL guy now working for MariaDB

Makefiles generated by automake are more feature rich in general,
which is understandable as its the only backend it has to support.

The main problem here - Postrges do not use automake at all!
Postgres it's autoconf + handmade GNU Make files + perl script for
generating old MSVC project from this Makefiles.

"make distcheck"

CMake have no this bad concept, in my opinion, if you want to make the
project you should have a full build environment. (but I don't want to
argue about it here)

Also, as I wrote before, CMake it's not equivalent of GNU Make or Autoconf,
many your reasons based on that fact what CMake, is not a build system it's
more like project generation system.
And anyway, you have no option if you want to support Windows without pain
and much more hacks ways.

On 27 April 2018 at 15:10, Yuriy Zhuravlev <stalkerg@gmail.com> wrote:

1. You can remove tools/msvc folder because all your build rules will be
universal. (cmake build now have much fewer lines of code)

Which is nice, but not actually a major day to day impact.

2. You can forget about terminal in Windows (for windows guys it's
important)

OK, but it's not really important for the PostgreSQL project, IMO.
Also, most people working on PostgreSQL are probably less bothered by
the terminal.

3. You can normally check environment on Windows, right now we have
hardcoded headers and many options. Configure process will be same on all
platforms.

Again, nice, but does that solve a real current problem?

4. You can generate not only GNU Make or MSVC project, you also can make
Xcode projects, Ninja or NMake for build under MSVC Make. For Windows, you
also can easily change MSVC to Clang it's not hardcoded at all.

Yeah, that's nice, but again, what're the real world benefits?

5. With CMake you have an easy way to build extra modules (plugins), I have
already working prototype for windows PGXS. A plugin should just include
.cmake file generated with Postgres build.

Yep. FWIW, I already use CMake for some PostgreSQL extensions because
of PGXS limitations and Windows support. I won't say I'm a big fan,
the documentation is a bit stale and it has some weird quirks and
limitations, but compared to autohell it's pure magic.

But in my opinion, you should just try CMake to figure out all benefits.

I use it fairly regularly. I'd never use autotools for any project I
was starting myself.

But that doesn't mean converting the whole postgres project is a good idea.

I'd do it, personally. But it's not just up to me. I've yet to hear
something that's compelling to a team who still set Perl 5.8.8 as the
minimum version and support SunOS. You'll need a compelling argument
that it's worth the pain and distruption.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 28.04.2018 05:27, Yuriy Zhuravlev wrote:

"make distcheck"

CMake have no this bad concept, in my opinion, if you want to make the
project you should have a full build environment. (but I don't want to
argue about it here)

this is not about having a working build environment, it is about having
a fully working and correct source tarball before distributing it as a
new release.

What "make distcheck" does is an end-to-end check of the configuration
and build process itself.

So what it does is:

* create a source tarball, putting the version number given in
configure.ac into the tarball name

* unpack the source tarball in a temporary directory, make
everything read-only

* create a build directory, run "configure" in there

* build with "make"

* run project test suite with "make check"

* run "make install" into a tmp directory, then "make uninstall",
to check that installation works, and that uninstall removes
everything again

* run "make distclean" to check that cleanup really works as expected

So things spotted by this are e.g.

* missing files that didn't end up in the src tarball ("works on my
computer")

* files being created in srcdir instead of builddir during build

* installed files missed by uninstall (ok, CMake developers have a
strong opinion about "make uninstall anyway)

* generated files that are not properly getting cleaned up

* ... more things ***

Especially the srcdir vs builddir is one I'm missing very much with
CMake, it happened several times that such problems have slipped
through in MySQL and MariaDB releases, and I've seen it in other
projects using CMake, too

--
hartmut

Which is nice

OK

Again, nice

Yeah, that's nice

I already use CMake

I'd do it, personally

I suppose we have no one silver bullet reason to change autoconf+make to
CMake but it's cumulative impression.

Also, I suppose this thread started to resolve at least one small question.
It's should be something like voting, but I see here only a few people and
your and Tom's
answers very strange, you can't say definitely yes or no and you more
thinking about another people who not exist here.

----
Some specific anwers:

Again, nice, but does that solve a real current problem?

This is main reason why PGXS on windows not exist, also it's solve problems
with differents between MSVC versions and future releases.

Yeah, that's nice, but again, what're the real world benefits?

it's just convinient, for easy work with Xcode for example.

Also, most people working on PostgreSQL are probably less bothered by

the terminal.

It's another one reson why windows users and students don't want to hack
Postgres.

Yep. FWIW, I already use CMake for some PostgreSQL extensions because

of PGXS limitations and Windows support.

And now, it can be problem if your postgres build by mingw or llvm for
example. (even under MSVC you can use clang now)
Enviroment start change quickly and it's too much efforts to support all
this by yourself.

this is not about having a working build environment, it is about having
a fully working and correct source tarball before distributing it as a
new release.

Sorry, I did not understand correctly it before.
I suppose it's not big problem especial if you have CI and tests farm.
And anyway in Postgres distcheck is handmade code and you can make the same
script for CMake too:
https://github.com/stalkerg/postgres_cmake/blob/cmake/GNUmakefile.in#L111

and it's not working for windows. ;)
You should forget about Postgres as common Autotools project.

On Fri, Apr 27, 2018 at 5:46 AM, Hartmut Holzgraefe
<hartmut.holzgraefe@gmail.com> wrote:

I could probably continue with this brain dump forever, ...

I found your brain dump an interesting read, and I have to say that it
leaves me rather uninspired about making a change. It sounds to me
like if we change, some things will be better and others will not be
as good. The good news is that if we decide to change, it sounds like
we won't be a lot worse off than we are today. The bad news is that
it doesn't sound like we'll be a lot better off, either.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2018-05-01 12:19:28 -0400, Robert Haas wrote:

On Fri, Apr 27, 2018 at 5:46 AM, Hartmut Holzgraefe
<hartmut.holzgraefe@gmail.com> wrote:

I could probably continue with this brain dump forever, ...

I found your brain dump an interesting read, and I have to say that it
leaves me rather uninspired about making a change. It sounds to me
like if we change, some things will be better and others will not be
as good. The good news is that if we decide to change, it sounds like
we won't be a lot worse off than we are today. The bad news is that
it doesn't sound like we'll be a lot better off, either.

How is being able to build extensions on windows reasonably not an
improvement? It's really hard to build pgxs like stuff on windows right
now. Also not having to maintain a fair amount of visual studio project
generation code? And getting faster builds that don't suffer from weird
parallelism issues because dependencies can't be expressed properly in
parallel make? ...

It seems fair to argue that it's not worth the pain to get there, but
how it'd not be an improvement to be there I really don't get.

Greetings,

Andres Freund

I'd like to add my 2c that, as a user who has to support postgres running
on some fairly old systems, changing to a modern build mechanism (with all
the resultant dependency hell that it would likely introduce) would be
likely to cause me much grief.

At the moment I can still log in to the old RH Shrike box I keep
specifically for building for older systems (it does admittedly have a more
recent gcc, but even building that was a trial) and build Postgres from
source. Unless I've misunderstood I strongly doubt that would still be the
case with the changes being discussed here.

Geoff

On Tue, 1 May 2018 at 17:19, Robert Haas <robertmhaas@gmail.com> wrote:

On Tue, May 1, 2018 at 12:31 PM, Andres Freund <andres@anarazel.de> wrote:

How is being able to build extensions on windows reasonably not an
improvement? It's really hard to build pgxs like stuff on windows right
now. Also not having to maintain a fair amount of visual studio project
generation code? And getting faster builds that don't suffer from weird
parallelism issues because dependencies can't be expressed properly in
parallel make? ...

Sure, those are notable advantages. Thanks for articulating them so clearly.

It seems fair to argue that it's not worth the pain to get there, but
how it'd not be an improvement to be there I really don't get.

Well that's probably because you understand cmake. I don't.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Andres Freund <andres@anarazel.de> writes:

On 2018-05-01 12:19:28 -0400, Robert Haas wrote:

I found your brain dump an interesting read, and I have to say that it
leaves me rather uninspired about making a change. It sounds to me
like if we change, some things will be better and others will not be
as good. The good news is that if we decide to change, it sounds like
we won't be a lot worse off than we are today. The bad news is that
it doesn't sound like we'll be a lot better off, either.

How is being able to build extensions on windows reasonably not an
improvement?

That indeed would be an improvement, but maybe we could fix that specific
pain point without having to throw away twenty years worth of work?

The amount of accumulated knowledge we've got in the existing build system
is slightly staggering ... so I'm afraid that moving to a different one
would involve a lot of expensive re-invention of portability hacks.

Of course, blowing off support for any platform not released in the
last five years would cut down on the number of such hacks that we'd
need to reinvent. But that's not a tradeoff I especially like either.

regards, tom lane

On Tue, May 1, 2018 at 12:46 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andres Freund <andres@anarazel.de> writes:

On 2018-05-01 12:19:28 -0400, Robert Haas wrote:

I found your brain dump an interesting read, and I have to say that it
leaves me rather uninspired about making a change. It sounds to me
like if we change, some things will be better and others will not be
as good. The good news is that if we decide to change, it sounds like
we won't be a lot worse off than we are today. The bad news is that
it doesn't sound like we'll be a lot better off, either.

How is being able to build extensions on windows reasonably not an
improvement?

That indeed would be an improvement, but maybe we could fix that specific
pain point without having to throw away twenty years worth of work?

Indeed. It's possibly today to use CMake without a huge amount of
difficulty to build extensions out of tree against MSVC-built
postgres. This was more or less the topic of my talk on Ottawa last
year, based on some excellent work by Craig Ringer. To my certain
knowledge this is being used successfully today. Testing is a
different story, but building is a nut that's more or less been
cracked.

There is also the point that EDB, according to my understanding, is
considering moving back, or has perhaps already moved back, to
Msys/Mingw-64 based builds, due to the runtime hell that MSVC can get
you into. And we know perfectly well how to build extensions out of
tree against such a build. You do it just like on Unix.

The amount of accumulated knowledge we've got in the existing build system
is slightly staggering ... so I'm afraid that moving to a different one
would involve a lot of expensive re-invention of portability hacks.

Of course, blowing off support for any platform not released in the
last five years would cut down on the number of such hacks that we'd
need to reinvent. But that's not a tradeoff I especially like either.

No, me either.

CMake is hardly a bed of roses, either, BTW.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services