Time to drop plpython2?

Andres Freund <andres@anarazel.de> writes:

To me it seems time to drop plpython2 support. Supporting plpython2
until ~7 years after python2 is EOL is already quite long... It'd be one
thing if it were completely painless, but imo it's not.

7 years? Oh, you're envisioning dropping plpython2 in HEAD but keeping
it alive in the back branches. Yeah, it's a bit hard to justify
continuing support in HEAD.

regards, tom lane

Hi,

On 2021-10-31 14:49:22 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

To me it seems time to drop plpython2 support. Supporting plpython2
until ~7 years after python2 is EOL is already quite long... It'd be one
thing if it were completely painless, but imo it's not.

7 years? Oh, you're envisioning dropping plpython2 in HEAD but keeping
it alive in the back branches.

Yea. It's annoying to support python2, but I don't think it's severe
enough to consider dropping support in released branches. We might get
to that point for the newer released versions, but for now...

Yeah, it's a bit hard to justify continuing support in HEAD.

Cool.

Greetings,

Andres Freund

On Mon, Nov 1, 2021 at 8:05 AM Andres Freund <andres@anarazel.de> wrote:

Yeah, it's a bit hard to justify continuing support in HEAD.

+1, it's dropping out of distros, it'd be unsupportable without
unreasonable effort.

On 31.10.21 19:45, Andres Freund wrote:

To me it seems time to drop plpython2 support. Supporting plpython2
until ~7 years after python2 is EOL is already quite long... It'd be one
thing if it were completely painless, but imo it's not.

I was about to list better plpython2/3 support (including the regex
replacements for the regress tests) as a TODO for the meson proposal,
but it doesn't really seem worth investing in that...

Usually, we have dropped support for older Python versions when
continued support would be a burden for some ongoing development
project. This would certainly count as one of those, I think.

I see you have posted a patch for this in the meson thread
(/messages/by-id/attachment/127770/v5-0003-plpython-Drop-support-python2.patch).
Here is my review of that.

I would change the search order in configure from

PGAC_PATH_PROGS(PYTHON, [python python3 python2])

to

PGAC_PATH_PROGS(PYTHON, [python3 python])

This makes sure you don't accidentally pick up a "python" binary that
points to Python 2. This would otherwise immediately generate lots of
build failures on older platforms that still have Python 2 around.

You left two FIXME sections in plpython.h. AFAICT, we can make the
substitutions corresponding to those #define's in the source code and
remove those blocks.

src/pl/plpython/expected/README should be updated for the removed files.

Some documentation updates are needed. I see possibly relevant text in
the following files:

hstore.sgml
install-windows.sgml
installation.sgml
json.sgml
libpq.sgml
plpython.sgml
ref/comment.sgml (examples)
ref/create_transform.sgml (examples)
ref/drop_transform.sgml (examples)
standalone-profile.xsl

src/tools/msvc/ has lots of Python-related content.

More stuff to clean up:

contrib/hstore_plpython/.gitignore
contrib/jsonb_plpython/.gitignore
contrib/ltree_plpython/.gitignore
src/pl/plpython/.gitignore

Finally, morally related, there is some Python 2/3 compat code in
contrib/unaccent/generate_unaccent_rules.py that could be removed.
Also, arguably, change the shebang line in that script.

Hi,

On 2021-11-04 19:58:54 +0100, Peter Eisentraut wrote:

I see you have posted a patch for this in the meson thread (/messages/by-id/attachment/127770/v5-0003-plpython-Drop-support-python2.patch).

Yea, I was planning to post that here after a bit more polish. I mostly wanted
to get rid of the gross gross hack I did for the transmutation of the
regression test files.

Here is my review of that.

I would change the search order in configure from

PGAC_PATH_PROGS(PYTHON, [python python3 python2])

to

PGAC_PATH_PROGS(PYTHON, [python3 python])

This makes sure you don't accidentally pick up a "python" binary that points
to Python 2. This would otherwise immediately generate lots of build
failures on older platforms that still have Python 2 around.

Yes, we should do that, at least for now. I did see build failures that
required me to specify the python version to avoid issues around it.

You left two FIXME sections in plpython.h. AFAICT, we can make the
substitutions corresponding to those #define's in the source code and remove
those blocks.

Yea, it shouldn't be hard. Just required more time than I had to send it out
before Nov 1st ;)

With meson I'd do a version: '>= 3' or such, to filter out a bare 'python'
being python2, but I don't think there's an equally trivial way to do that
with autoconf.

Finally, morally related, there is some Python 2/3 compat code in
contrib/unaccent/generate_unaccent_rules.py that could be removed. Also,
arguably, change the shebang line in that script.

Hm. So far the python used for plpython and python for code generation etc is
independent. I don't know if plpython actually can be cross-compiled, but if
so, they'd have to be independent. Otherwise I'd say we should just invoke
contrib/unaccent/generate_unaccent_rules.py with a python chosen by
configure/meson, rather than relying on a shebang that can't be adjusted
without modifying source code.

Another thing I wondered about is what we want to do with the extension
names. Do we want to leave it named plpython3u? Do we want to have a plpython
that depends on plpython3u?

I'd be inclined to just keep it at plpython3u for now, but there's an argument
that going for plpython would be better long term: Presumably there will be
python 4 at some point - but I'd expect that to not be a breaking release,
given the disaster that python 3 is. Making a non-versioned name better?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:

Another thing I wondered about is what we want to do with the extension
names. Do we want to leave it named plpython3u? Do we want to have a plpython
that depends on plpython3u?

I think we want to keep plpython3u. Maybe we can point plpythonu
at that, but I'm concerned about the potential for user confusion.
In particular, I think there's a nonzero probability that someone
will choose to maintain plpython2u/plpythonu outside of core,
just because they still don't want to migrate their Python code.

I'd be inclined to just keep it at plpython3u for now, but there's an argument
that going for plpython would be better long term: Presumably there will be
python 4 at some point - but I'd expect that to not be a breaking release,
given the disaster that python 3 is. Making a non-versioned name better?

Meh. If there is a python 4, I'd expect it to be named that way precisely
because it *is* a breaking release. Why would we set ourselves up for
a repeat of this mess?

regards, tom lane

On 04.11.21 20:54, Andres Freund wrote:

Finally, morally related, there is some Python 2/3 compat code in
contrib/unaccent/generate_unaccent_rules.py that could be removed. Also,
arguably, change the shebang line in that script.

Hm. So far the python used for plpython and python for code generation etc is
independent. I don't know if plpython actually can be cross-compiled, but if
so, they'd have to be independent. Otherwise I'd say we should just invoke
contrib/unaccent/generate_unaccent_rules.py with a python chosen by
configure/meson, rather than relying on a shebang that can't be adjusted
without modifying source code.

We don't rely on the shebang for running this (see Makefile). I just
see some trend of people changing shebang lines as a sort of signal,
"this script has abandoned Python 2". It's not very important.

Another thing I wondered about is what we want to do with the extension
names. Do we want to leave it named plpython3u? Do we want to have a plpython
that depends on plpython3u?

I would tend to mirror what the Python community does with
/usr/bin/python. Right now, most people are removing /usr/bin/python
and just provide /usr/bin/python3. This would be analogous to what your
patch does. Some time in the future, they may add /usr/bin/python back,
and then we could do the same, meaning add a plpythonu that depends on
plpython3u.

... btw, there's a fairly critical gating factor for any plan to drop
python2 support: the buildfarm. I just counted, and there are exactly
as many members running python 2.x as 3.x (49 apiece), not counting
Windows machines that aren't running configure. We can't commit
something that's going to make half the buildfarm go red.

(It's likely that some fraction of them do already have python3 installed,
in which case the search order change Peter recommended would be enough to
fix it. But I'm sure not all do.)

This ties into the business about converting the build system to meson,
as that also requires python 3 --- with, IIUC, a higher minimum version
than we might otherwise need. I'm disinclined to cause two separate
flag days for buildfarm owners, so what I now think is we ought to put
this idea on the shelf until we've finished that conversion or decided
we're not gonna do it. We need to identify exactly what needs to be
installed before we start pestering the owners.

regards, tom lane

Hi,

On 2021-11-14 21:24:31 -0500, Tom Lane wrote:

... btw, there's a fairly critical gating factor for any plan to drop
python2 support: the buildfarm. I just counted, and there are exactly
as many members running python 2.x as 3.x (49 apiece), not counting
Windows machines that aren't running configure. We can't commit
something that's going to make half the buildfarm go red.

(It's likely that some fraction of them do already have python3 installed,
in which case the search order change Peter recommended would be enough to
fix it. But I'm sure not all do.)

How about committing the order change alone? That seems like something
warranted completely in isolation? Afterwards we can see how many run what and
go from there?

This ties into the business about converting the build system to meson,
as that also requires python 3 --- with, IIUC, a higher minimum version
than we might otherwise need. I'm disinclined to cause two separate
flag days for buildfarm owners, so what I now think is we ought to put
this idea on the shelf until we've finished that conversion or decided
we're not gonna do it. We need to identify exactly what needs to be
installed before we start pestering the owners.

Yea, that's true.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:

On 2021-11-14 21:24:31 -0500, Tom Lane wrote:

(It's likely that some fraction of them do already have python3 installed,
in which case the search order change Peter recommended would be enough to
fix it. But I'm sure not all do.)

How about committing the order change alone? That seems like something
warranted completely in isolation? Afterwards we can see how many run what and
go from there?

I don't think that's warranted. The existing design is that we let
the user say which python is "python", and I do not think we should
change that in advance of actually dropping python2 support.

I was wondering about simply probing to see if python3 exists (and if
so, what version it is exactly), as an additional configure test that
doesn't hook into anything. That would give us some information without
suddenly changing what is being tested.

regards, tom lane

Hi,

On 2021-11-15 12:19:51 -0500, Tom Lane wrote:

I don't think that's warranted. The existing design is that we let
the user say which python is "python", and I do not think we should
change that in advance of actually dropping python2 support.

Hm. I think it'd be ok, given that python 2 is well past EOL. But I also see
your point.

I was wondering about simply probing to see if python3 exists (and if
so, what version it is exactly), as an additional configure test that
doesn't hook into anything. That would give us some information without
suddenly changing what is being tested.

But this is probably a good compromise. Were you thinking of doing a proper
autoconf test or just putting something like python3 --version || true in
configure?

I guess it'd be easiest to interpret if we output the current PYTHON version
and, iff < 3.0, also output the 'python3' version?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:

On 2021-11-15 12:19:51 -0500, Tom Lane wrote:

I was wondering about simply probing to see if python3 exists (and if
so, what version it is exactly), as an additional configure test that
doesn't hook into anything. That would give us some information without
suddenly changing what is being tested.

But this is probably a good compromise. Were you thinking of doing a proper
autoconf test or just putting something like python3 --version || true in
configure?

Hm, I had in mind an actual configure test; but since it's just a
temporary exploratory measure, shortcuts are fine. However, I'm
not sure that what you suggest would result in capturing anything
in config.log.

I guess it'd be easiest to interpret if we output the current PYTHON version
and, iff < 3.0, also output the 'python3' version?

We'll already know if PYTHON is set, at least so far as the buildfarm
is concerned.

regards, tom lane

On 15.11.21 19:18, Tom Lane wrote:

Andres Freund<andres@anarazel.de> writes:

On 2021-11-15 12:19:51 -0500, Tom Lane wrote:

I was wondering about simply probing to see if python3 exists (and if
so, what version it is exactly), as an additional configure test that
doesn't hook into anything. That would give us some information without
suddenly changing what is being tested.

But this is probably a good compromise. Were you thinking of doing a proper
autoconf test or just putting something like python3 --version || true in
configure?

Hm, I had in mind an actual configure test; but since it's just a
temporary exploratory measure, shortcuts are fine. However, I'm
not sure that what you suggest would result in capturing anything
in config.log.

I'm not sure this is really going to end up moving things forward.

I think we should just write to the build farm owners, we plan to drop
python2 support in, say, 60 days, please update your setup to use
python3 or disable python support.

If we add this test first, then all we're going to learn is probably
that 60% of those who are currently using python2 don't have python3
installed, and then we're still going to have to send that above email.

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

If we add this test first, then all we're going to learn is probably
that 60% of those who are currently using python2 don't have python3
installed, and then we're still going to have to send that above email.

I don't know what fraction don't have python3 installed, and it doesn't
matter much, since it's unlikely that it's either 0% or 100%. What
I hoped to learn was, of those who *do* have some python3 installed,
which version it is. That might inform our thoughts about where to
set the minimum python3 version.

Relevant data points:

* Our docs claim the minimum 3.x version for pl/python itself is 3.1,
but this is unbacked by any testing; the oldest 3.x in the buildfarm
is 3.4.3 (three such animals).

* Meson only promises support back to python 3.6, but if that's
accurate it's going to be a problem for us, because there are lots
of live LTS distributions with older python3 (RHEL7, Solaris 11.3,
AIX 7.2 for starters). I've been planning to do some testing and
see if meson will run under python 3.4 or 3.5.

But this probably belongs on the meson thread, since that looks like
it will be a much stronger constraint than pl/python is.

regards, tom lane

Hi,

Continuing the discussion from /messages/by-id/2146739.1637004415@sss.pgh.pa.us

On 2021-11-15 14:26:55 -0500, Tom Lane wrote:

* Meson only promises support back to python 3.6, but if that's
accurate it's going to be a problem for us, because there are lots
of live LTS distributions with older python3 (RHEL7, Solaris 11.3,
AIX 7.2 for starters). I've been planning to do some testing and
see if meson will run under python 3.4 or 3.5.

Slightly older versions, which do work to build postgres with the proposed
patchset, run on python3 3.5. I don't think it's likely we could make the
versions that only required 3.4 work reasonably.

Is RHEL7 really an issue? I only have Centos 7 around, but that has python
3.6.

I don't know much about AIX, but according to https://repology.org/project/python/versions
the AIX toolbox has 3.7.11. I don't know enough about AIX to know whether
there's other sources of python3, bison, etc that are common. On the AIX
system I have access to they all seem to be symlinked to /opt/freeware, which
I understand is that toolbox stuff?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:

On 2021-11-15 14:26:55 -0500, Tom Lane wrote:

* Meson only promises support back to python 3.6, but if that's
accurate it's going to be a problem for us, because there are lots
of live LTS distributions with older python3 (RHEL7, Solaris 11.3,
AIX 7.2 for starters). I've been planning to do some testing and
see if meson will run under python 3.4 or 3.5.

Slightly older versions, which do work to build postgres with the proposed
patchset, run on python3 3.5. I don't think it's likely we could make the
versions that only required 3.4 work reasonably.

OK, thanks for the datapoint.

Is RHEL7 really an issue? I only have Centos 7 around, but that has python
3.6.

The info I checked said that RHEL7 originally shipped with 3.3.
I'm not sure that Red Hat would've outright replaced that, but they
do have a notion of add-on "software collections", and I'm certain
that they would have provided newer pythons via that mechanism.
So it should only be a big issue for someone who didn't want to install
an add-on collection. I suppose the overlap of that group with the
people who will want to put PG 15+ on that platform is probably nil.

I don't know much about AIX, but according to https://repology.org/project/python/versions
the AIX toolbox has 3.7.11. I don't know enough about AIX to know whether
there's other sources of python3, bison, etc that are common. On the AIX
system I have access to they all seem to be symlinked to /opt/freeware, which
I understand is that toolbox stuff?

Hmm, I was basing that on (a) what I can see installed on gcc119,
which is 3.5.1, and (b) AIX 7.2's 2015 release date, which matches up
pretty well with python 3.5. As with RHEL, it's entirely likely that
IBM has made newer releases available as add-ons, but there's no
guarantee that any given installation would have that.

Solaris is a bit worse, since they shipped 11.3 a shade earlier,
with python 3.4.3. No idea about add-on conventions on that
platform.

There's room to argue that all three of these will be out of
primary support before PG15 ships, so maybe we don't need to
worry about whether we can build with their default toolsets.
Still, it's a tradeoff I'd rather not make.

regards, tom lane

On Mon, Nov 15, 2021 at 3:30 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

The info I checked said that RHEL7 originally shipped with 3.3.
I'm not sure that Red Hat would've outright replaced that, but they
do have a notion of add-on "software collections", and I'm certain
that they would have provided newer pythons via that mechanism.
So it should only be a big issue for someone who didn't want to install
an add-on collection. I suppose the overlap of that group with the
people who will want to put PG 15+ on that platform is probably nil.

It'd only be an issue if they want to compile from source, right?
We're not speaking of changing the runtime prerequisites, IIUC.

I think it's really important that we continue to run on all of the
supported Linux distributions and even some recently-out-of-support
ones if they are popular. My experience is that many people stay on
whatever version of RHEL they've got for quite a long time. I think
we'll be doing ourselves a disservice if such people find that they
can't easily upgrade to newer versions of PostgreSQL without
installing a bunch of new software and/or upgrading software they've
already got on the machine.

On the other hand, the class of users that I'm thinking about does not
typically install anything from source, so they are not directly
impacted by build prerequisites.

--
Robert Haas
EDB: http://www.enterprisedb.com

Hi,

On 2021-11-15 15:30:02 -0500, Tom Lane wrote:

Is RHEL7 really an issue? I only have Centos 7 around, but that has python
3.6.

The info I checked said that RHEL7 originally shipped with 3.3.
I'm not sure that Red Hat would've outright replaced that, but they
do have a notion of add-on "software collections", and I'm certain
that they would have provided newer pythons via that mechanism.
So it should only be a big issue for someone who didn't want to install
an add-on collection. I suppose the overlap of that group with the
people who will want to put PG 15+ on that platform is probably nil.

3.6 appears to be in the 'update' repository, which I think is separate from
the add-on collection? It appears to be enabled by default in centos7, not
sure how much that means.

Name : python3
Version : 3.6.8
Release : 18.el7
Architecture: x86_64
Size : 39896
Packager : CentOS BuildSystem <http://bugs.centos.org&gt;
Group : Unspecified
URL : https://www.python.org/
Repository : updates
Summary : Interpreter of the Python programming language
Source : python3-3.6.8-18.el7.src.rpm

Hmm, I was basing that on (a) what I can see installed on gcc119,
which is 3.5.1, and (b) AIX 7.2's 2015 release date, which matches up
pretty well with python 3.5. As with RHEL, it's entirely likely that
IBM has made newer releases available as add-ons, but there's no
guarantee that any given installation would have that.

I think AIX itself only ships with python 2.7, and all python3 versions are
from toolbox. But I don't actually know.

Solaris is a bit worse, since they shipped 11.3 a shade earlier,
with python 3.4.3. No idea about add-on conventions on that
platform.

Based solely on some person in meson's irc channel, it appears that they added
a newer python3 sometime in 11.4's lifetime:

alanc:
Solaris 11.4 currently ships Python 3.7 & 3.9
as the maintainer of the meson package for Solaris, I deal with the
lack of newer python versions on older Solaris releases by only packaging
meson for Solaris 11.4, and telling anyone who wants it for older releases to
upgrade their OS, or build all the dependencies themselves
...
or it's 11.4.0 from 2018, and not updated with the latest updates (which are
only available for support customers right now unfortunately) - 3.7 was added
in 11.4.12
https://docs.oracle.com/cd/E37838_01/html/E99082/bundledfreeware.html
...

There's room to argue that all three of these will be out of
primary support before PG15 ships, so maybe we don't need to
worry about whether we can build with their default toolsets.
Still, it's a tradeoff I'd rather not make.

There's two other paths worth mentioning:
- muon is a WIP alternative implementation of meson in plain C99. Doesn't yet
have enough feature coverage.
- pyinstaller generates an executable from the meson python code, and supports
solaris and aix

Greetings,

Andres Freund