Replace uses of deprecated Python module distutils.sysconfig

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

With Python 3.10, configure spits out warnings about the module
distutils.sysconfig being deprecated and scheduled for removal in Python
3.12:

Bleah.

This patch changes the uses in configure to use the module sysconfig
instead. The logic stays the same. (It's basically the same module but
as its own top-level module.)
Note that sysconfig exists since Python 2.7, so this moves the minimum
required version up from Python 2.6.

That's surely no problem in HEAD, but as you say, it is an issue for
the older branches. How difficult would it be to teach configure to
try both ways, or adapt based on its python version check?

I suggest leaving the backbranches alone for now. At the moment, we
don't even know whether additional changes will be required for 3.12
(and 3.11) support, so the overall impact isn't known yet. In a few
months, we will probably know more about this.

Agreed, this is a moving target so we shouldn't be too concerned
about it yet.

regards, tom lane

On 02.12.21 19:22, Tom Lane wrote:

That's surely no problem in HEAD, but as you say, it is an issue for
the older branches. How difficult would it be to teach configure to
try both ways, or adapt based on its python version check?

I think it wouldn't be unreasonable to do that. I'll look into it.

On 02.12.21 08:20, Peter Eisentraut wrote:

Buildfarm impact:

gaur and prariedog use Python 2.6 and would need to be upgraded.

Tom, are you planning to update the Python version on these build farm
members? I realize these are very slow machines and this might take
some time; I'm just wondering if this had registered.

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

On 02.12.21 08:20, Peter Eisentraut wrote:

Buildfarm impact:
gaur and prariedog use Python 2.6 and would need to be upgraded.

Tom, are you planning to update the Python version on these build farm
members? I realize these are very slow machines and this might take
some time; I'm just wondering if this had registered.

I can do that when it becomes necessary. I've got one eye on the meson
conversion discussion, which will kill those two animals altogether;
so it seems possible that updating their Pythons now would just be
wasted effort depending on what lands first.

regards, tom lane

On 09.12.21 14:31, Tom Lane wrote:

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

On 02.12.21 08:20, Peter Eisentraut wrote:

Buildfarm impact:
gaur and prariedog use Python 2.6 and would need to be upgraded.

Tom, are you planning to update the Python version on these build farm
members? I realize these are very slow machines and this might take
some time; I'm just wondering if this had registered.

I can do that when it becomes necessary. I've got one eye on the meson
conversion discussion, which will kill those two animals altogether;
so it seems possible that updating their Pythons now would just be
wasted effort depending on what lands first.

I saw that the Python installations on gaur and prairiedog had been
updated, so I committed this patch. As the buildfarm shows, various
platforms have problems with this, in particular because they point to
the wrong place for the include directory. AFAICT, in most cases this
appears to have been fixed in more recent editions of those platforms
(e.g., Debian unstable members pass but older releases don't), so at
least the approach was apparently not wrong in principle. But
obviously, this leaves us in a mess. I will revert this patch in a bit,
after gathering a few more hours of data.

Also, considering the failure on prairiedog, I do see now on
<https://docs.python.org/3/library/sysconfig.html&gt; that the sysconfig
module is "New in version 3.2". I had interpreted the fact that it
exists in version 2.7 that that includes all higher versions, but
obviously there were multiple branches involved, so that was a mistaken
assumption.

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

Also, considering the failure on prairiedog, I do see now on
<https://docs.python.org/3/library/sysconfig.html&gt; that the sysconfig
module is "New in version 3.2". I had interpreted the fact that it
exists in version 2.7 that that includes all higher versions, but
obviously there were multiple branches involved, so that was a mistaken
assumption.

Hm. I installed 3.1 because we claim support for that. I don't mind
updating to 3.2 (as long as we adjust the docs to match), but it seems
kinda moot unless you figure out a solution for the include-path
issue. I see that platforms as recent as Debian 10 are failing,
so I don't think we can dismiss that as not needing fixing.

regards, tom lane

On 18.01.22 16:24, Tom Lane wrote:

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

Also, considering the failure on prairiedog, I do see now on
<https://docs.python.org/3/library/sysconfig.html&gt; that the sysconfig
module is "New in version 3.2". I had interpreted the fact that it
exists in version 2.7 that that includes all higher versions, but
obviously there were multiple branches involved, so that was a mistaken
assumption.

Hm. I installed 3.1 because we claim support for that. I don't mind
updating to 3.2 (as long as we adjust the docs to match), but it seems
kinda moot unless you figure out a solution for the include-path
issue. I see that platforms as recent as Debian 10 are failing,
so I don't think we can dismiss that as not needing fixing.

I have reverted this for now.

I don't have a clear idea how to fix this in the long run. We would
perhaps need to determine at which points the various platforms had
fixed this issue in their Python installations and select between the
old and the new approach based on that. Seems messy.

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

I don't have a clear idea how to fix this in the long run. We would
perhaps need to determine at which points the various platforms had
fixed this issue in their Python installations and select between the
old and the new approach based on that. Seems messy.

Are we sure it's an issue within Python, rather than something we
could dodge by invoking sysconfig differently? It's hard to believe
that sysconfig could be totally unfit for the purpose of finding out
the include path and would remain so for multiple years.

regards, tom lane

I wrote:

Are we sure it's an issue within Python, rather than something we
could dodge by invoking sysconfig differently? It's hard to believe
that sysconfig could be totally unfit for the purpose of finding out
the include path and would remain so for multiple years.

I dug up a Debian 9 image and found that I could reproduce the problem
against its python2 (2.7.13) installation, but not its python3 (3.5.3):

$ python2 -m sysconfig | grep include
include = "/usr/local/include/python2.7"
platinclude = "/usr/local/include/python2.7"
...
$ python3 -m sysconfig | grep include
include = "/usr/include/python3.5m"
platinclude = "/usr/include/python3.5m"
...

Looking at the buildfarm animals that failed this way, 10 out of 11
are using python 2.x. The lone exception is Andrew's prion. I wonder
if there is something unusual about its python3 installation.

Anyway, based on these results, we might have better luck switching to
sysconfig after we start forcing python3. I'm tempted to resurrect the
idea of changing configure's probe order to "python3 python python2"
in the meantime, just so we can see how much of the buildfarm is ready
for that.

regards, tom lane

I wrote:

Anyway, based on these results, we might have better luck switching to
sysconfig after we start forcing python3.

On the other hand, that answer is not back-patchable, and we surely
need a back-patchable fix, because people will try to build the
back branches against newer pythons.

Based on the buildfarm results so far, the problem can be described
as "some installations say /usr/local when they should have said /usr".
I experimented with the attached delta patch and it fixes the problem
on my Debian 9 image. (I don't know Python, so there may be a better
way to do this.) We'd have to also bump the minimum 3.x version to
3.2, but that seems very unlikely to bother anyone.

regards, tom lane

On 19.01.22 01:18, Tom Lane wrote:

Anyway, based on these results, we might have better luck switching to
sysconfig after we start forcing python3. I'm tempted to resurrect the
idea of changing configure's probe order to "python3 python python2"
in the meantime, just so we can see how much of the buildfarm is ready
for that.

This seems sensible in any case, given that we have quasi-committed to
enforcing Python 3 soon.

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

On 19.01.22 01:18, Tom Lane wrote:

... I'm tempted to resurrect the
idea of changing configure's probe order to "python3 python python2"
in the meantime, just so we can see how much of the buildfarm is ready
for that.

This seems sensible in any case, given that we have quasi-committed to
enforcing Python 3 soon.

Done. (I couldn't find any equivalent logic in the MSVC build scripts
though; is there something I missed?)

regards, tom lane

On Wed, Jan 19, 2022 at 9:40 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Done. (I couldn't find any equivalent logic in the MSVC build scripts
though; is there something I missed?)

MSVC will use the path configured in src\tools\msvc\config.pl $config->{"python"},

there is no ambiguity.

Regards,

Juan José Santamaría Flecha

I wrote:

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

On 19.01.22 01:18, Tom Lane wrote:

... I'm tempted to resurrect the
idea of changing configure's probe order to "python3 python python2"
in the meantime, just so we can see how much of the buildfarm is ready
for that.

This seems sensible in any case, given that we have quasi-committed to
enforcing Python 3 soon.

Done.

The early returns are not great: we have about half a dozen machines
so far that are finding python3, and reporting sane-looking Python
include paths, but not finding Python.h. They're all Linux-oid
machines, so I suppose what is going on is that they have the base
python3 package installed but not python3-dev or local equivalent.

I want to leave that patch in place long enough so we can get a
fairly full survey of which machines are OK and which are not,
but I suppose I'll have to revert it tomorrow or so. We did
promise the owners a month to adjust their configurations.

regards, tom lane

=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <juanjo.santamaria@gmail.com> writes:

On Wed, Jan 19, 2022 at 9:40 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Done. (I couldn't find any equivalent logic in the MSVC build scripts
though; is there something I missed?)

MSVC will use the path configured in src\tools\msvc\config.pl $config->{"python"},
there is no ambiguity.

Ah, right. We have only three active Windows animals that are building
with python, and all three are using 3.something, so that side of the
house seems to be ready to go.

regards, tom lane

On 1/18/22 19:18, Tom Lane wrote:

I wrote:

Are we sure it's an issue within Python, rather than something we
could dodge by invoking sysconfig differently? It's hard to believe
that sysconfig could be totally unfit for the purpose of finding out
the include path and would remain so for multiple years.

I dug up a Debian 9 image and found that I could reproduce the problem
against its python2 (2.7.13) installation, but not its python3 (3.5.3):

$ python2 -m sysconfig | grep include
include = "/usr/local/include/python2.7"
platinclude = "/usr/local/include/python2.7"
...
$ python3 -m sysconfig | grep include
include = "/usr/include/python3.5m"
platinclude = "/usr/include/python3.5m"
...

Looking at the buildfarm animals that failed this way, 10 out of 11
are using python 2.x. The lone exception is Andrew's prion. I wonder
if there is something unusual about its python3 installation.

It's an Amazon Linux instance, and using their packages, which seem a
bit odd (there's nothing in /usr/local/include). Maybe we should be
looking at INCLUEPY?

[ec2-user@ip-172-31-22-42 bf]$ python3 -m sysconfig | grep include
    include = "/usr/local/include/python3.6m"
    platinclude = "/usr/local/include/python3.6m"
    CONFIG_ARGS = "'--build=x86_64-redhat-linux-gnu'
'--host=x86_64-redhat-linux-gnu' '--target=x86_64-amazon-linux-gnu'
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
'--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64'
'--libexecdir=/usr/libexec' '--localstatedir=/var'
'--sharedstatedir=/var/lib' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--enable-ipv6' '--enable-shared'
'--with-computed-gotos=yes' '--with-dbmliborder=gdbm:ndbm:bdb'
'--with-system-expat' '--with-system-ffi'
'--enable-loadable-sqlite-extensions' '--with-dtrace' '--with-valgrind'
'--without-ensurepip' '--enable-optimizations'
'build_alias=x86_64-redhat-linux-gnu'
'host_alias=x86_64-redhat-linux-gnu'
'target_alias=x86_64-amazon-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC
-fwrapv  ' 'LDFLAGS= -g  ' 'CPPFLAGS= '
'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'"
    CONFINCLUDEDIR = "/usr/include"
    CONFINCLUDEPY = "/usr/include/python3.6m"
    INCLDIRSTOMAKE = "/usr/include  /usr/include/python3.6m"
    INCLUDEDIR = "/usr/include"
    INCLUDEPY = "/usr/include/python3.6m"

I have upgraded it to python 3.8, but got similar results.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

I wrote:

The early returns are not great: we have about half a dozen machines
so far that are finding python3, and reporting sane-looking Python
include paths, but not finding Python.h. They're all Linux-oid
machines, so I suppose what is going on is that they have the base
python3 package installed but not python3-dev or local equivalent.

I want to leave that patch in place long enough so we can get a
fairly full survey of which machines are OK and which are not,
but I suppose I'll have to revert it tomorrow or so. We did
promise the owners a month to adjust their configurations.

I have now reverted that patch, but I think this was a highly
worthwhile bit of reconnaissance. It identified 18 animals
that had incomplete python3 installations (versus only 13
that definitely or possibly lack python3 altogether). Their
owners most likely thought they were already good to go for the
changeover, so without this experiment we'd have had a whole lot
of buildfarm red when the real change is made.

I've notified the owners of these results.

regards, tom lane

I wrote:

Based on the buildfarm results so far, the problem can be described
as "some installations say /usr/local when they should have said /usr".
I experimented with the attached delta patch and it fixes the problem
on my Debian 9 image. (I don't know Python, so there may be a better
way to do this.) We'd have to also bump the minimum 3.x version to
3.2, but that seems very unlikely to bother anyone.

I did a little more digging into this. The python2 package on
my Deb9 (actually Raspbian) system says it is 2.7.13, but
/usr/lib/python2.7/sysconfig.py is different from what I find in
a virgin Python 2.7.13 tarball, as per attached diff. I conclude
that somebody at Debian decided that Python should live under
/usr/local, and changed sysconfig.py to match, but then failed
to adjust the actual install scripts to agree, because there is
certainly nothing installed under /usr/local. (I don't know
enough about Debian packaging to find the smoking gun though;
what apt-get claims is the source package contains no trace of
this diff.) There's no sign of comparable changes in
/usr/lib/python3.5/sysconfig.py on the same machine, either.

So I think this can fairly be characterized as brain-dead packaging
error, and we should just hack around it as per my previous patch.

In other news, I switched prairiedog and gaur to python 3.2.

regards, tom lane

Hi,

On 2022-01-23 16:06:21 -0500, Tom Lane wrote:

I wrote:

Based on the buildfarm results so far, the problem can be described
as "some installations say /usr/local when they should have said /usr".
I experimented with the attached delta patch and it fixes the problem
on my Debian 9 image. (I don't know Python, so there may be a better
way to do this.) We'd have to also bump the minimum 3.x version to
3.2, but that seems very unlikely to bother anyone.

I did a little more digging into this. The python2 package on
my Deb9 (actually Raspbian) system says it is 2.7.13, but
/usr/lib/python2.7/sysconfig.py is different from what I find in
a virgin Python 2.7.13 tarball, as per attached diff. I conclude
that somebody at Debian decided that Python should live under
/usr/local, and changed sysconfig.py to match, but then failed
to adjust the actual install scripts to agree, because there is
certainly nothing installed under /usr/local. (I don't know
enough about Debian packaging to find the smoking gun though;
what apt-get claims is the source package contains no trace of
this diff.) There's no sign of comparable changes in
/usr/lib/python3.5/sysconfig.py on the same machine, either.

+    'posix_local': {
+        'stdlib': '{base}/lib/python{py_version_short}',
+        'platstdlib': '{platbase}/lib/python{py_version_short}',
+        'purelib': '{base}/local/lib/python{py_version_short}/dist-packages',
+        'platlib': '{platbase}/local/lib/python{py_version_short}/dist-packages',
+        'include': '{base}/local/include/python{py_version_short}',
+        'platinclude': '{platbase}/local/include/python{py_version_short}',
+        'scripts': '{base}/local/bin',
+        'data': '{base}/local',
+        },
+    'deb_system': {
+        'stdlib': '{base}/lib/python{py_version_short}',
+        'platstdlib': '{platbase}/lib/python{py_version_short}',
+        'purelib': '{base}/lib/python{py_version_short}/dist-packages',
+        'platlib': '{platbase}/lib/python{py_version_short}/dist-packages',
+        'include': '{base}/include/python{py_version_short}',
+        'platinclude': '{platbase}/include/python{py_version_short}',
+        'scripts': '{base}/bin',
+        'data': '{base}',
+        },
'posix_home': {

Hm. It seems the intent of the different paths you show is that we can specify
which type of path we want. The one to locally installed extensions, or the
distribution ones. So we'd have to specify the scheme to get the other include
path?

andres@awork3:~$ python2
Python 2.7.18 (default, Sep 24 2021, 09:39:51)
[GCC 10.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import sysconfig
sysconfig.get_path('include')

'/usr/local/include/python2.7'

sysconfig.get_path('include', 'posix_prefix')

'/usr/include/python2.7'

sysconfig.get_path('include', 'posix_local')

'/usr/local/include/python2.7'

So it seems we could do something like

sysconfig.get_path('include', 'posix_prefix' if os.name == 'posix' else os.name)

or

scheme = sysconfig._get_default_scheme()
# Work around Debian / Ubuntu returning paths not useful for finding python headers
if scheme == 'posix_local':
scheme = 'posix_prefix'
sysconfig.get_path('include', scheme = scheme)

Greetings,

Andres Freund