Resolving the python 2 -> python 3 mess
We've had multiple previous discussions of $SUBJECT (eg [1]/messages/by-id/5351890.TdMePpdHBD@nb.usersys.redhat.com[2]/messages/by-id/CAKmB1PGDAy9mXxSTqUchYEi4iJAA6NKVj4P5BtAzvQ9wSDUwJw@mail.gmail.com),
without any resolution of what to do exactly. Thinking about this
some more, I had an idea that I don't think has been discussed.
To wit:
1. On platforms where Python 2.x is still supported, recommend that
packagers continue to build both plpython2 and plpython3, same as now.
2. On platforms where Python 2.x is no longer supported, transparently
map plpythonu and plpython2u to plpython3u. "Transparent" meaning that
dump/reload or pg_upgrade of existing plpythonu/plpython2u functions
will work, but when you run them, what you actually get is Python 3.x.
For existing functions that don't use any obsolete Python syntax
(which one would hope is a pretty large percentage), this is a
zero-effort conversion for users. If a function does use obsolete
constructs, it will get a parse failure when executed, and the user
will have to update it to Python 3 syntax. I propose that we make
that case reasonably painless by providing the conversion script
I posted in [3]/messages/by-id/11546.1566584867@sss.pgh.pa.us (or another one if somebody's got a better one),
bundled as a separately-installable extension.
A possible gotcha in this approach is if there are any python 2/3
incompatibilities that would not manifest as syntax errors or
obvious runtime errors, but would allow old code to execute and
silently do the wrong thing. One would hope that the Python crowd
weren't dumb enough to do that, but I don't know whether it's true.
If there are nasty cases like that, maybe what we have to do is allow
plpythonu/plpython2u functions to be dumped and reloaded into a
python-3-only install, but refuse to execute them until they've
been converted.
In either case, to allow dump/reload or pg_upgrade to work without
ugly hacks, what we need to do is provide a stub version of
plpython2.so. (The extension definitions that sit on top of it
then don't need to change.) The stub would either redirect calls
to plpython3.so if we prefer that approach, or throw errors if we
prefer that approach. I envision adding a configure option that
enables build and install of this stub library while doing a
plpython3 build; packagers not planning to build a "real" plpython2
should ask for the stub instead.
The end result given the first approach is that "plpythonu" and
"plpython2u" and "plpython3u" all work and mean the same thing.
Over some long time period we might want to deprecate and remove
the "plpython2u" alias, but there would be no hurry about it.
The work involved in making this happen seems fairly minimal, and
practical to get done in time for PG 13. Perhaps there'd even be
a case for back-patching it, though I'm not going to advocate for
that here.
Thoughts?
regards, tom lane
[1]: /messages/by-id/5351890.TdMePpdHBD@nb.usersys.redhat.com
[2]: /messages/by-id/CAKmB1PGDAy9mXxSTqUchYEi4iJAA6NKVj4P5BtAzvQ9wSDUwJw@mail.gmail.com
[3]: /messages/by-id/11546.1566584867@sss.pgh.pa.us
Hi Tom,
I really like the "stub .so" idea, but feel pretty uncomfortable for the
"transparent" upgrade. Response inlined.
On Mon, Feb 17, 2020 at 8:49 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
2. On platforms where Python 2.x is no longer supported, transparently
map plpythonu and plpython2u to plpython3u. "Transparent" meaning that
dump/reload or pg_upgrade of existing plpythonu/plpython2u functions
will work, but when you run them, what you actually get is Python 3.x.
It's fair enough that plpythonu changes its meaning, people who really
want the stability should explicitly use plpython2u.
For existing functions that don't use any obsolete Python syntax
(which one would hope is a pretty large percentage), this is a
zero-effort conversion for users. If a function does use obsolete
constructs, it will get a parse failure when executed, and the user
will have to update it to Python 3 syntax. I propose that we make
that case reasonably painless by providing the conversion script
I posted in [3] (or another one if somebody's got a better one),
bundled as a separately-installable extension.A possible gotcha in this approach is if there are any python 2/3
incompatibilities that would not manifest as syntax errors or
obvious runtime errors, but would allow old code to execute and
silently do the wrong thing. One would hope that the Python crowd
weren't dumb enough to do that, but I don't know whether it's true.
If there are nasty cases like that, maybe what we have to do is allow
plpythonu/plpython2u functions to be dumped and reloaded into a
python-3-only install, but refuse to execute them until they've
been converted.
"True division", one of the very first (2011, awww) few breaking changes
introduced in Python 3 [1]https://www.python.org/dev/peps/pep-0238/, comes to mind. While it's not the worst
incompatibilities between Python 2 and 3, it's bad enough to give pause
to the notion that a successful parsing implies successful conversion.
[1]: https://www.python.org/dev/peps/pep-0238/
Cheers,
Jesse
Jesse Zhang <sbjesse@gmail.com> writes:
I really like the "stub .so" idea, but feel pretty uncomfortable for the
"transparent" upgrade. Response inlined.
Fair enough, but ...
2. On platforms where Python 2.x is no longer supported, transparently
map plpythonu and plpython2u to plpython3u. "Transparent" meaning that
dump/reload or pg_upgrade of existing plpythonu/plpython2u functions
will work, but when you run them, what you actually get is Python 3.x.
It's fair enough that plpythonu changes its meaning, people who really
want the stability should explicitly use plpython2u.
Yeah, but then what do you want to do with functions declared plpython2u?
Have them fail even if they'd work fine under Python 3? Doesn't really
seem like that's helping anyone.
A possible gotcha in this approach is if there are any python 2/3
incompatibilities that would not manifest as syntax errors or
obvious runtime errors, but would allow old code to execute and
silently do the wrong thing.
"True division", one of the very first (2011, awww) few breaking changes
introduced in Python 3 [1], comes to mind. While it's not the worst
incompatibilities between Python 2 and 3, it's bad enough to give pause
to the notion that a successful parsing implies successful conversion.
Hm. I agree that's kind of nasty, because 2to3 doesn't fix it AFAICT
(and, likely, there is no way to do so that doesn't include solving
the halting problem). However, it's not clear to me why forcing users
to do a conversion is going to help them any with that, precisely
because the automated conversion won't fix it. They're going to have
to find such issues the hard way whenever they move to Python 3, no
matter what we do.
regards, tom lane
A possible gotcha in this approach is if there are any python 2/3
incompatibilities that would not manifest as syntax errors or
obvious runtime errors, but would allow old code to execute and
silently do the wrong thing. One would hope that the Python crowd
weren't dumb enough to do that, but I don't know whether it's true.
If there are nasty cases like that, maybe what we have to do is allow
plpythonu/plpython2u functions to be dumped and reloaded into a
python-3-only install, but refuse to execute them until they've
been converted.
Unfortunately, I think there are cases like that. The shift to Unicode as
the default string means that some functions that used to return a `str`
now return a `bytes` (I know of this in the hashlib and base64 modules, but
probably also in URL request data and others), and to use a `bytes` in
string manipulation you have to first explicitly convert it to some string
encoding. So things like a function that wraps around a python crypto
library would be the exact places where those was-str-now-bytes functions
would be used.
Corey Huinker <corey.huinker@gmail.com> writes:
A possible gotcha in this approach is if there are any python 2/3
incompatibilities that would not manifest as syntax errors or
obvious runtime errors, but would allow old code to execute and
silently do the wrong thing.
Unfortunately, I think there are cases like that. The shift to Unicode as
the default string means that some functions that used to return a `str`
now return a `bytes` (I know of this in the hashlib and base64 modules, but
probably also in URL request data and others), and to use a `bytes` in
string manipulation you have to first explicitly convert it to some string
encoding. So things like a function that wraps around a python crypto
library would be the exact places where those was-str-now-bytes functions
would be used.
So, as with Jesse's example, what I'm wondering is whether or not 2to3
will fix that for you (or even flag it). The basic difference between
the two alternatives I suggested is whether we force people to put their
python function through that converter before we'll even try to run it.
Subtleties that 2to3 doesn't catch seem like non-reasons to insist on
applying it.
regards, tom lane
So, as with Jesse's example, what I'm wondering is whether or not 2to3
will fix that for you (or even flag it). The basic difference between
the two alternatives I suggested is whether we force people to put their
python function through that converter before we'll even try to run it.
Subtleties that 2to3 doesn't catch seem like non-reasons to insist on
applying it.
The 2018 vintage of 2to3 didn't catch it.
It's not firsthand knowledge, but I just watched a nearby team have some
production issues where one library couldn't fetch b'http://foo.org' so I'm
guessing 2to3 still doesn't catch those things, or they stopped using it.
After thinking about this awhile longer, I'm starting to believe
we should do some of each. That is, the stub replacement for
plpython2.so should redirect "plpythonu" functions to plpython3.so,
but throw errors for "plpython2u" functions. This is not because
of any technical difference between plpythonu and plpython2u ---
up to now, there wasn't any --- but because it seems like users
would be expecting that if they've read what we have said in
https://www.postgresql.org/docs/current/plpython-python23.html
Admittedly, what it says there is that plpythonu might become
Python 3 in some "distant" future release, not next year.
But at least there's a direct line between that documentation
and this behavior.
So attached is a pair of draft patches that do it like that.
0001 creates an extension with two conversion functions, based
on the script I showed in the other thread. Almost independently
of that, 0002 provides code to generate a stub version of
plpython2.so that behaves as stated above. 0002 is incomplete,
because I haven't looked into what is needed in the MSVC build
scripts. Maybe we could create some regression tests, too.
But I think these are potentially committable with those additions,
if people approve of this approach.
regards, tom lane
On 2020-02-19 05:39, Tom Lane wrote:
After thinking about this awhile longer, I'm starting to believe
we should do some of each. That is, the stub replacement for
plpython2.so should redirect "plpythonu" functions to plpython3.so,
but throw errors for "plpython2u" functions.
I'm not sure these complications are worth it. They don't match
anything that is done in other Python 2/3 porting schemes. I think
there should just be an option "plpython is: {2|3|don't build it at
all}". Then packagers can match this to what their plan for
/usr/bin/python* is -- which appears to be different everywhere.
Your scheme appears to center around the assumption that people will
want to port their functions at the same time as not building plpython2u
anymore. This would defeat testing functions before and after in the
same installation. I think the decisions of what plpythonu points to
and which variants are built at all should be separate.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
Your scheme appears to center around the assumption that people will
want to port their functions at the same time as not building plpython2u
anymore.
Not really; use of the proposed porting infrastructure is the same whether
plpython2u still works or not. You end up with functions that are labeled
plpython3u, so what bare "plpythonu" means is not a factor.
It is true that as this patch is written, switching of plpythonu to
point at Python 3 rather than 2 is coupled to disabling plpython2u.
If we'd have gotten this done a year or two ago, I'd have made it more
complex to allow more separation there. But events have passed us by:
the info we are getting from packagers is that Python 2 is getting
dropped *this year*, not in some distant future. So I think that allowing
the plpythonu redefinition to be separate is no longer of any great value,
and not worth extra complication for. People are just going to be
shipping v13 with both things changed in any case.
If we wanted to do something to help people port their functions in
advance of the big changeover, the thing to do would be to back-patch
the proposed convert_python3 extension into existing branches.
regards, tom lane
Here's an updated pair of patches that attempt to fix the MSVC
scripts (pretty blindly) and provide a very simple regression test.
I'm not too sure whether the regression test will really prove
workable or not: for starters, it'll fail if "2to3" isn't available
in the PATH. Perhaps there's reason to object to even trying to
test that, on security grounds.
I set up the MSVC scripts to default to building the stub extension.
I don't know if we really want to commit it that way, but the idea
for the moment is to try to get the cfbot to test it on Windows.
regards, tom lane
I wrote:
Here's an updated pair of patches that attempt to fix the MSVC
scripts (pretty blindly) and provide a very simple regression test.
A little *too* blindly, evidently. Try again ...
regards, tom lane
I wrote:
I set up the MSVC scripts to default to building the stub extension.
I don't know if we really want to commit it that way, but the idea
for the moment is to try to get the cfbot to test it on Windows.
No joy there --- now that I look closer, it seems the cfbot doesn't
build any of the external-language PLs on Windows. I'll have to
wait for some reviewer to try it.
regards, tom lane
On 2/25/20 5:06 PM, Tom Lane wrote:
I wrote:
I set up the MSVC scripts to default to building the stub extension.
I don't know if we really want to commit it that way, but the idea
for the moment is to try to get the cfbot to test it on Windows.No joy there --- now that I look closer, it seems the cfbot doesn't
build any of the external-language PLs on Windows. I'll have to
wait for some reviewer to try it.
What are the requirements for testing? bowerbird builds with python 2.7,
although I guess I should really try to upgrade it 3.x.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
On 2/25/20 5:06 PM, Tom Lane wrote:
No joy there --- now that I look closer, it seems the cfbot doesn't
build any of the external-language PLs on Windows. I'll have to
wait for some reviewer to try it.
What are the requirements for testing? bowerbird builds with python 2.7,
although I guess I should really try to upgrade it 3.x.
Has to be python 3, unfortunately; the patch has no effect on a
python 2 build.
regards, tom lane
On 2/25/20 7:08 PM, Tom Lane wrote:
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
On 2/25/20 5:06 PM, Tom Lane wrote:
No joy there --- now that I look closer, it seems the cfbot doesn't
build any of the external-language PLs on Windows. I'll have to
wait for some reviewer to try it.What are the requirements for testing? bowerbird builds with python 2.7,
although I guess I should really try to upgrade it 3.x.Has to be python 3, unfortunately; the patch has no effect on a
python 2 build.
Yeah, I have python3 working on drongo, I'll test there.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2/25/20 8:24 PM, Andrew Dunstan wrote:
On 2/25/20 7:08 PM, Tom Lane wrote:
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
On 2/25/20 5:06 PM, Tom Lane wrote:
No joy there --- now that I look closer, it seems the cfbot doesn't
build any of the external-language PLs on Windows. I'll have to
wait for some reviewer to try it.What are the requirements for testing? bowerbird builds with python 2.7,
although I guess I should really try to upgrade it 3.x.Has to be python 3, unfortunately; the patch has no effect on a
python 2 build.Yeah, I have python3 working on drongo, I'll test there.
It's almost there, you need to add something like this to Mkvcbuild.pm:
if ($solution->{options}->{python2_stub})
{
my $plpython2_stub =
$solution->AddProject('plpython2', 'dll', 'PLs',
'src/pl/stub_plpython2');
$plpython2_stub->AddReference($postgres);
}
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2/26/20 2:47 AM, Andrew Dunstan wrote:
On 2/25/20 8:24 PM, Andrew Dunstan wrote:
On 2/25/20 7:08 PM, Tom Lane wrote:
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
On 2/25/20 5:06 PM, Tom Lane wrote:
No joy there --- now that I look closer, it seems the cfbot doesn't
build any of the external-language PLs on Windows. I'll have to
wait for some reviewer to try it.What are the requirements for testing? bowerbird builds with python 2.7,
although I guess I should really try to upgrade it 3.x.Has to be python 3, unfortunately; the patch has no effect on a
python 2 build.Yeah, I have python3 working on drongo, I'll test there.
It's almost there, you need to add something like this to Mkvcbuild.pm:
if ($solution->{options}->{python2_stub})
{
my $plpython2_stub =
$solution->AddProject('plpython2', 'dll', 'PLs',
'src/pl/stub_plpython2');
$plpython2_stub->AddReference($postgres);
}
However, when it get to testing contrib it complains like this:
============================================================
Checking hstore_plpython
C:/prog/bf/root/HEAD/pgsql/Release/pg_regress/pg_regress
--bindir=C:/prog/bf/root/HEAD/pgsql/Release/psql
--dbname=contrib_regression --load-ex
tension=hstore --load-extension=plpythonu
--load-extension=hstore_plpythonu hstore_plpython
(using postmaster on localhost, default port)
============== dropping database "contrib_regression" ==============
DROP DATABASE
============== creating database "contrib_regression" ==============
CREATE DATABASE
ALTER DATABASE
============== installing hstore ==============
CREATE EXTENSION
============== installing plpythonu ==============
CREATE EXTENSION
============== installing hstore_plpythonu ==============
ERROR: could not access file "$libdir/hstore_plpython2": No such file
or directory
command failed: "C:/prog/bf/root/HEAD/pgsql/Release/psql/psql" -X -c
"CREATE EXTENSION IF NOT EXISTS \"hstore_plpythonu\"" "contrib_regression"
So there's a bit more work to do.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2/26/20 3:17 AM, Andrew Dunstan wrote:
On 2/26/20 2:47 AM, Andrew Dunstan wrote:
On 2/25/20 8:24 PM, Andrew Dunstan wrote:
On 2/25/20 7:08 PM, Tom Lane wrote:
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
On 2/25/20 5:06 PM, Tom Lane wrote:
No joy there --- now that I look closer, it seems the cfbot doesn't
build any of the external-language PLs on Windows. I'll have to
wait for some reviewer to try it.What are the requirements for testing? bowerbird builds with python 2.7,
although I guess I should really try to upgrade it 3.x.Has to be python 3, unfortunately; the patch has no effect on a
python 2 build.Yeah, I have python3 working on drongo, I'll test there.
It's almost there, you need to add something like this to Mkvcbuild.pm:
if ($solution->{options}->{python2_stub})
{
my $plpython2_stub =
$solution->AddProject('plpython2', 'dll', 'PLs',
'src/pl/stub_plpython2');
$plpython2_stub->AddReference($postgres);
}However, when it get to testing contrib it complains like this:
============================================================
Checking hstore_plpython
C:/prog/bf/root/HEAD/pgsql/Release/pg_regress/pg_regress
--bindir=C:/prog/bf/root/HEAD/pgsql/Release/psql
--dbname=contrib_regression --load-ex
tension=hstore --load-extension=plpythonu
--load-extension=hstore_plpythonu hstore_plpython
(using postmaster on localhost, default port)
============== dropping database "contrib_regression" ==============
DROP DATABASE
============== creating database "contrib_regression" ==============
CREATE DATABASE
ALTER DATABASE
============== installing hstore ==============
CREATE EXTENSION
============== installing plpythonu ==============
CREATE EXTENSION
============== installing hstore_plpythonu ==============
ERROR: could not access file "$libdir/hstore_plpython2": No such file
or directory
command failed: "C:/prog/bf/root/HEAD/pgsql/Release/psql/psql" -X -c
"CREATE EXTENSION IF NOT EXISTS \"hstore_plpythonu\"" "contrib_regression"So there's a bit more work to do.
This seems to fix it.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
py3-fix-vcregress.patchtext/x-patch; charset=UTF-8; name=py3-fix-vcregress.patchDownload+6-6
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
This seems to fix it.
OK, so we need that *and* the AddProject addition you mentioned?
regards, tom lane
On Thu, Feb 27, 2020 at 1:33 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
This seems to fix it.
OK, so we need that *and* the AddProject addition you mentioned?
Yes, the first one builds it, the second one fixes the tests to run correctly.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
