What's our minimum supported Python version?

On 19 Apr 2025, at 7:17 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Our fine manual claims the answer to $SUBJECT is 3.2.

However, there is room to doubt that our code still actually
works with 3.2, because the oldest version I can find being
tested by the buildfarm is topminnow's 3.4.2. The next oldest
is shelduck's 3.4.10, and everything else has 3.5.2 or newer.
topminnow hasn't reported in a couple months, so it may
actually be dead.

The reason I bring this up is that I found out the hard way
that src/test/modules/oauth_validator fails on RHEL8, because
its oauth_server.py script is not compatible with the 3.6.8
version of Python supplied by this distro. (There are 22
buildfarm animals running 3.6.8, presumably mostly also Red
Hat-derived platforms, so I'm not going to apologize for
my workstation being a little long in the tooth.)

I think we need to do some combination of moving our
minimum-supported-version goalposts forward, making sure that
whatever we claim is the minimum Python version is actually
being tested in the buildfarm, and fixing oauth_server.py
so that it works on that version.

I am not familiar enough with the Python landscape to
have an informed opinion about what the minimum supported
version should be. But I'll present this data scraped
from the buildfarm about how many animals are running what
(counting animals that have reported since 2025-01-01):

Count Version

1 3.4.2
1 3.4.10
2 3.5.2
3 3.5.3
2 3.6.5
22 3.6.8
3 3.6.9
5 3.6.15
1 3.7.1
2 3.7.3
1 3.7.10
2 3.7.16
1 3.8.8
4 3.8.10
15 3.9.2
6 3.9.16
6 3.9.18
1 3.9.19
2 3.9.20
13 3.9.21
6 3.10.x
18 3.11.x
21 3.12.x
22 3.13.x

160 total

Thoughts?

regards, tom lane

From an Python ecosystem perspective,
3.9 is the usual minimum that people use in CI matrices nowdays.
So if it was up to me, that’s what I’d choose.

Mabe 3.7 if I was too liberal, but <3.6 and older it’s probably too old to bother.
Then again, I’m not a buildfarm power user, so my 2 cents are worth a bit less.

Florents Tselai <florents.tselai@gmail.com> writes:

On 19 Apr 2025, at 7:17 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I think we need to do some combination of moving our
minimum-supported-version goalposts forward, making sure that
whatever we claim is the minimum Python version is actually
being tested in the buildfarm, and fixing oauth_server.py
so that it works on that version.

From an Python ecosystem perspective,
3.9 is the usual minimum that people use in CI matrices nowdays.
So if it was up to me, that’s what I’d choose.

Per these numbers, that would be cutting off 31% of the buildfarm,
including a lot of still-in-support distros such as RHEL8.
So I would say that's not likely to be our choice.

regards, tom lane

On 19 Apr 2025, at 18:17, Tom Lane <tgl@sss.pgh.pa.us> wrote:

The reason I bring this up is that I found out the hard way
that src/test/modules/oauth_validator fails on RHEL8, because
its oauth_server.py script is not compatible with the 3.6.8
version of Python supplied by this distro.

Do you have the error message/log for the failure handy?

--
Daniel Gustafsson

Daniel Gustafsson <daniel@yesql.se> writes:

On 19 Apr 2025, at 18:17, Tom Lane <tgl@sss.pgh.pa.us> wrote:
The reason I bring this up is that I found out the hard way
that src/test/modules/oauth_validator fails on RHEL8, because
its oauth_server.py script is not compatible with the 3.6.8
version of Python supplied by this distro.

Do you have the error message/log for the failure handy?

The first problem is that this Python version seems not to
like assignments embedded in if statements:

File "t/oauth_server.py", line 319
if err := self._get_param("error_code", None):
^
SyntaxError: invalid syntax

I was able to work around that with the attached quick hack.
But then I get

Traceback (most recent call last):
File "t/oauth_server.py", line 19, in <module>
class OAuthHandler(http.server.BaseHTTPRequestHandler):
File "t/oauth_server.py", line 26, in OAuthHandler
JsonObject = dict[str, object] # TypeAlias is not available until 3.10
TypeError: 'type' object is not subscriptable

which I have no idea how to work around.

regards, tom lane

On Tue, Apr 22, 2025 at 8:05 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

The first problem is that this Python version seems not to
like assignments embedded in if statements: [...]

I was able to work around that with the attached quick hack.
But then I get [...]

Thanks, I'll put a patch together. Sorry -- IIRC, both of those
features are probably too new for me to have used, regardless of what
we decide is the minimum version for 18.

As for picking a version... 3.6 will have been EOL for almost three
years by the time 18 releases. It seems like we would drop it happily,
were it not for RHEL8. But RHEL also supports side-by-side Python,
right? In general, I'd be more than willing to try plumbing that
through the new tests, if it lets us drop end-of-life versions for new
code.

(I don't know if we'll need to drop 3.6 for PG18, specifically, but I
also think we shouldn't plan to support 3.6 for the full RHEL8
lifetime.)

Thanks,
--Jacob

Jacob Champion <jacob.champion@enterprisedb.com> writes:

As for picking a version... 3.6 will have been EOL for almost three
years by the time 18 releases. It seems like we would drop it happily,
were it not for RHEL8.

Agreed, but RHEL8 is out there and I don't think we can just drop
support for it. I'm also not excited by the idea that an incidental
test script gets to dictate what the cutoff is.

I do think we should stop claiming that Python 3.2 will work.
(Maybe it does, but we don't know that.) I see that the configure
script only checks for Python >= 3, and it doesn't look like the
meson scripts check explicitly at all, although there's a comment
saying that our meson version cutoff is intended to allow working
with Python 3.5.

Maybe it's sufficient to make a documentation change here, and
say we support Python >= 3.5? I'd be okay with saying 3.6.8
too, on the grounds that if anything older fails to work we'd
almost certainly just say "too bad". But RHEL8 is widespread
enough that I think we need to keep making the effort for 3.6.8.

regards, tom lane

On Tue, Apr 22, 2025 at 9:04 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

I'm also not excited by the idea that an incidental
test script gets to dictate what the cutoff is.

I agree, and I don't intend for the script to dictate that.

Maybe it's sufficient to make a documentation change here, and
say we support Python >= 3.5? I'd be okay with saying 3.6.8
too, on the grounds that if anything older fails to work we'd
almost certainly just say "too bad".

I'm definitely on board with pulling the minimum up, as high as we
think we can get away with. If you're comfortable with 3.6(.8), I'd
say let's start there.

But RHEL8 is widespread
enough that I think we need to keep making the effort for 3.6.8.

Even if we can get side-by-side versions working? RHEL8 has a bunch of
newer Python 3 versions available, according to the docs. And
virtualenvs can do a lot of lifting for us in practice.

Thanks,
--Jacob

Jacob Champion <jacob.champion@enterprisedb.com> writes:

On Tue, Apr 22, 2025 at 9:04 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

But RHEL8 is widespread
enough that I think we need to keep making the effort for 3.6.8.

Even if we can get side-by-side versions working?

That would be off-loading our problem onto the users (and the
buildfarm owners). If we really had little other choice,
we could go there; but it doesn't sound like this is that
hard to work around.

regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> writes:

Florents Tselai <florents.tselai@gmail.com> writes:

On 19 Apr 2025, at 7:17 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I think we need to do some combination of moving our
minimum-supported-version goalposts forward, making sure that
whatever we claim is the minimum Python version is actually
being tested in the buildfarm, and fixing oauth_server.py
so that it works on that version.

From an Python ecosystem perspective,
3.9 is the usual minimum that people use in CI matrices nowdays.
So if it was up to me, that’s what I’d choose.

Per these numbers, that would be cutting off 31% of the buildfarm,
including a lot of still-in-support distros such as RHEL8.
So I would say that's not likely to be our choice.

regards, tom lane

Also from a python world perspective, we really don't want to run EOL
versions. Besides the security reasons, bugs in thirdy-party
dependencies usually are not fixed and the whole system seems to rot
very quickly.

Of course, this does not happen if one does not rely on third-party
deps. Even in this case, I would say that it is wise to support a
relatively small set because the built-in libs will vary over versions,
and a wide range of versions will cause more troubles like this one.

The oldest non EOL version is 3.9 right now. My suggestion is to follow
the official supported releases. I'm not familiar with the buildfarm,
but I know that python3.6 does not come installed by default in RHEL8,
so instead of installing this version we can simply install 3.9, 3.11 or
3.12.

In a persistent workstation, I believe the following should do the
trick (tested on a 8.5 box):
sudo dnf install python3.12
sudo dnf remove python36

BTW, we are used to create a virtualenv for everything, but that is another topic.

Best regards,
Renan Fonseca

On Tue, Apr 22, 2025 at 8:29 AM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:

On Tue, Apr 22, 2025 at 8:05 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

The first problem is that this Python version seems not to
like assignments embedded in if statements: [...]

I was able to work around that with the attached quick hack.
But then I get [...]

Thanks, I'll put a patch together. Sorry -- IIRC, both of those
features are probably too new for me to have used, regardless of what
we decide is the minimum version for 18.

Splitting this off into its own $SUBJECT.

I coupled oauth_server to new Python stuff without realizing it, sorry:
- urllib's reclassification of `~` as a safe URL character (3.7)
- walrus operator `:=` (3.8)
- dict[type, type] annotations (3.9)
- str.removeprefix/suffix() (3.9)

Attached patch gets rid of all that.

This is tested against Python 3.6.15 (3.6 went EOL at the end of
2021). I'm working on getting Rocky 8 installed locally to test
against. If it's decided we want to support downwards to 3.5, I will
test there too (but I hope we don't; see parent thread).

Thanks,
--Jacob

On Tue, Apr 22, 2025 at 12:09 PM Renan Alves Fonseca
<renanfonseca@gmail.com> wrote:

The oldest non EOL version is 3.9 right now. My suggestion is to follow
the official supported releases.

I think this policy is too aggressive. Many operating systems we
support are going to ship Python versions past their EOL date (and
keep them supported for a long while with security patches). There's a
middle ground to be found, IMO.

--Jacob

Jacob Champion <jacob.champion@enterprisedb.com> writes:

This is tested against Python 3.6.15 (3.6 went EOL at the end of
2021). I'm working on getting Rocky 8 installed locally to test
against. If it's decided we want to support downwards to 3.5, I will
test there too (but I hope we don't; see parent thread).

Thanks! I confirm this works for me on RHEL8 with 3.6.8.

regards, tom lane

Jacob Champion <jacob.champion@enterprisedb.com> writes:

On Tue, Apr 22, 2025 at 12:09 PM Renan Alves Fonseca
<renanfonseca@gmail.com> wrote:

The oldest non EOL version is 3.9 right now. My suggestion is to follow
the official supported releases.

I think this policy is too aggressive. Many operating systems we
support are going to ship Python versions past their EOL date (and
keep them supported for a long while with security patches).

Yeah, that. The distros that are still shipping older Pythons
are LTS distros where part of the deal is that the vendor
is supposed to back-patch security fixes, even if the upstream
has moved on. Maybe a particular vendor is falling down on that
job, but it's not for us to rate their performance. So I don't
buy that "but security!" is a good argument here.

(Full disclosure: I worked for Red Hat for more than a decade,
and still preferentially use their distros. So I've pretty well
bought into that model.)

regards, tom lane

On Tue, Apr 22, 2025 at 12:36 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Thanks! I confirm this works for me on RHEL8 with 3.6.8.

Great, thanks for the quick review!

I'm considering committing this myself. Do you mind if it takes a day
or so to get this in, while I quadruple-check everything, or are you
blocked on this?

--Jacob

Jacob Champion <jacob.champion@enterprisedb.com> writes:

I'm considering committing this myself.

Please do.

Do you mind if it takes a day
or so to get this in, while I quadruple-check everything, or are you
blocked on this?

I'm not in a hurry. Push when you're ready.

regards, tom lane

tl;dr I think requiring support of Python 3.9 for PG18 would probably
be reasonable and save us a bunch of maintenance burden over the next
5 years.

On Tue, 22 Apr 2025 at 21:41, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Yeah, that. The distros that are still shipping older Pythons
are LTS distros where part of the deal is that the vendor
is supposed to back-patch security fixes, even if the upstream
has moved on. Maybe a particular vendor is falling down on that
job, but it's not for us to rate their performance. So I don't
buy that "but security!" is a good argument here.

I totally agree that "security" is not the discussion that we should
be having here. I think the Python version decision (or really any
minimum version decision) should be based on some analysis of costs to
us for maintaining support vs benefits to the users.

I'm pretty sure most users of RHEL expect most modern software not to
compile/work completely effortlessly on their distros without some
effort on their part or on the part of RHEL packagers. That's kinda
what you're signing up for if you choose a distro like that.

It might very well be that supporting Python 3.6 does not require many
code changes. But it's so old that installing it on most of the recent
OSes is not trivial, e.g. uv does not support installing it[1]https://github.com/astral-sh/uv/issues/9833. By
saying we support it, it means that all developers need to go through
serious hoops to be able to test that their python scripts work
correctly on their own machines. And given our own support policy,
we'd probably have to do that for 5 more years (unless we want to
change supported python versions in minor PG releases, which I highly
doubt we want).

So that leaves the question, how much are we actually helping users by
spending all that effort on keeping support for old Python versions?
Based on the Red Hat docs, it's pretty easy to install newer Python3
versions on RHEL8. Depending on the X in RHEL 8.X you can install
different Python3 versions from the official repos[2]https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/installing-and-using-dynamic-programming-languages_configuring-basic-system-settings#assembly_installing-and-using-python_installing-and-using-dynamic-programming-languages. And based on
the official RHEL EOL policy[3]https://access.redhat.com/support/policy/updates/errata#RHEL8_Planning_Guide, only RHEL 8.4, 8.6, 8.8 and 8.10 are
still getting some level of support. With 8.4 and 8.6 only getting the
final level of support called "Update Services for SAP Solutions", and
with 8.8 reaching that same final support level before PG18 is
released (and 8.4 becoming unsupported before PG18).

Based on that information (only taking into account only RHEL8, not
other distros). It seems like we could at least bump the minimum
Python support to Python 3.9, which is the newest python that's
installable in RHEL 8.4 and 8.6. Anyone that's running a RHEL version
that's still supported by Red Hat can then simply do "yum install
python39" and run the tests (assuming we set up meson/autoconf to
correctly detect the "python3.9" binary).

I'd even be curious how much resistance we'd get from RHEL users if
we'd bump the Python requirement for PG18 to Python 3.12, which is the
newest version that RHEL 8.10 supports. RHEL 8.10 is the only RHEL 8
version that is getting regular support from Red Hat.

(Full disclosure: I've used close to EOL versions of RHEL7
professionally, and I hated every second it. I constantly felt like I
was walking through a software museum. Having to work around bugs that
had been fixed for ages in upstream)

Looking a little further than RHEL:
- Ubuntu 20.04 can easily install Python3.9 from the official package
repos. Currently this is still supported by Cristoph his pgdg repos,
but looks like that support might be removed before PG18 given it's
transition to "Expanded Security Maintenance" next month.
- Ubuntu 22.04 can easily install Python3.11 from the official package
repos. Still has full upstream support for 2 years.
- Debian Bullseye (EOL in 1.5 years) can only install Python3.9 from
the official package repos.
- Debian Bookworm (EOL in 3 years) has Pytho
- Python 3.8 itself is declared EOL by upstream
- Python 3.10 will be declared EOL by upstream around the release of PG19

Based on all this info, I think that it sounds quite reasonable to
start requiring Python3.9 for PG18. And for PG19 we could probably
start requiring Python3.11

[1]: https://github.com/astral-sh/uv/issues/9833
[2]: https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/installing-and-using-dynamic-programming-languages_configuring-basic-system-settings#assembly_installing-and-using-python_installing-and-using-dynamic-programming-languages
[3]: https://access.redhat.com/support/policy/updates/errata#RHEL8_Planning_Guide

P.S. I CC-ed Devrim and Christoph, for insights into this from the
packagers perspective.

Re: Jelte Fennema-Nio

- Ubuntu 20.04 can easily install Python3.9 from the official package
repos. Currently this is still supported by Cristoph his pgdg repos,
but looks like that support might be removed before PG18 given it's
transition to "Expanded Security Maintenance" next month.

20.04 (focal) is already too old for PG18 because of llvm version
woes, so we are already not building it there. (I forgot what the
precise problem was, but I figured focal would be EOL before the PG18
release anyway.)

- Ubuntu 22.04 can easily install Python3.11 from the official package
repos. Still has full upstream support for 2 years.
- Debian Bullseye (EOL in 1.5 years) can only install Python3.9 from
the official package repos.
- Debian Bookworm (EOL in 3 years) has Pytho

^ n 3.11

- Python 3.8 itself is declared EOL by upstream
- Python 3.10 will be declared EOL by upstream around the release of PG19

Based on all this info, I think that it sounds quite reasonable to
start requiring Python3.9 for PG18. And for PG19 we could probably
start requiring Python3.11

Ack. Bullseye LTS will be supported until August 2026, so very close
to the PG19 release, but it should work out not to support PG19 there.

https://wiki.debian.org/LTS/

Christoph

Jacob Champion <jacob.champion@enterprisedb.com> writes:

This is tested against Python 3.6.15 (3.6 went EOL at the end of
2021). I'm working on getting Rocky 8 installed locally to test
against. If it's decided we want to support downwards to 3.5, I will
test there too (but I hope we don't; see parent thread).

I did a quick test on 3.5. It seems to work after removing
f-strings... At this point, I would be really happy with 3.6.

On Tue, Apr 22, 2025 at 2:28 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:

I'm pretty sure most users of RHEL expect most modern software not to
compile/work completely effortlessly on their distros without some
effort on their part or on the part of RHEL packagers. That's kinda
what you're signing up for if you choose a distro like that.

This is the core of my position, too. I think it's reasonable to
support Python versions for some time after they go EOL, but I don't
think that we need to treat "users who want bleeding-edge Postgres
with long-EOL dependencies" as something to cater to, at the expense
of the committer testing matrix.

Note that Meson itself has updated to Python 3.7 as a minimum version
(as it now warns you, loudly).

Thanks,
--Jacob