Python 3.1 support

Peter Eisentraut <peter_e@gmx.net> writes:

Here's the patch to support Python >=3.1 with PL/Python. The
compatibility code is mostly in line with the usual 2->3 C porting
practice and is documented inline.

There was considerable debate earlier about whether we wanted to treat
Python 3 as a separate PL so it could be available in parallel with
plpython 2, because of the user-level coding incompatibilities. It
looks like this patch simply ignores that problem. What is going to
happen to plpython functions that depend on 2.x behavior?

regards, tom lane

On Nov 12, 2009, at 12:54 PM, Peter Eisentraut wrote:

Here's the patch to support Python >=3.1 with PL/Python.

:\

I was hoping to be able to use Python 3 to draw a clear distinction between plpython and the would-be "plpython3" that I've been working on. I understand that you're not in favor of a brand new implementation for Python 3. Despite my dislike for that position(well, it would seem to be in opposition to my initiative, so naturally =), I don't entirely disagree with your rationale[wrt doing things more incrementally]. For me, plpython has never been what I would call a pleasure to use, and many of the gripes that I have with it are, IMO, entrenched far enough into the implementation that any efforts to change it would(should? =) cause unacceptable breakage in user applications(?). Well, as far as additional Python interfaces are concerned, a lot of redundant functionality, but that's not even the half of it.

[I was hoping to get to a status message this weekend,
but it seems like I should follow-up here. =]

So here's where I'm at:
--
Mostly documentation improvements since I last pinged -hackers.
Still, *sigh*, filling in documentation and fighting bugs as I go.
Currently resolving a bug instantiating MD arrays from nested lists.
Once I'm "finished" with the docs, I'm going to start looking for refcount leaks.
No major additions or changes are planned, but I have been making some minor additions as I write more docs.

Overview/Features:
http://wiki.postgresql.org/wiki/WIP:plpython3
Documentation:
http://python.projects.postgresql.org/pldocs/plpython3.html
git repo[see the plpython3 branch]:
http://git.postgresql.org/gitweb?p=plpython3.git;a=summary

Most of the documented interfaces have tests. I only have two platforms at my disposal, so I do fear that this will not "just work" on all of PG's supported platforms. Specifically, I've ran the tests on freebsd/amd64 and Mac10.6/intel(of course 10.5 as well for some earlier revisions). [err, actually, it's been a while since I ran the tests on freebsd.]
--

plpython3 is turning out to be kinda beefy(~974K diff[eh, there is some fluff in there]), and I can't say that I've seen much interest in it, so I can't really blame anyone if -hackers ends up taking a pass on it. (python3 is too far away for most folk to care? folk are content with plpython?)

eh, cheers, either way. =)

On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

Here's the patch to support Python >=3.1 with PL/Python. The
compatibility code is mostly in line with the usual 2->3 C porting
practice and is documented inline.

There was considerable debate earlier about whether we wanted to treat
Python 3 as a separate PL so it could be available in parallel with
plpython 2, because of the user-level coding incompatibilities. It
looks like this patch simply ignores that problem.

Exactly how to package that is something to be determined by the
packagers, and we can give them the support they need. But first you
need code that works with Python 3, which is what this patch does.

What is going to happen to plpython functions that depend on 2.x behavior?

The porting path from 2 to 3 is pretty well established. You first port
to 2.6, then remove all the old features, then move to 3.x. This is not
something we have to reinvent. The only significant differences that
you can't use in 2.6 without future imports are unicode literals and the
print function, both of which are not in common use in PL/Python.

On tor, 2009-11-12 at 18:42 -0700, James Pye wrote:

For me, plpython has never been what I would call a pleasure to use,
and many of the gripes that I have with it are, IMO, entrenched far
enough into the implementation that any efforts to change it
would(should? =) cause unacceptable breakage in user applications(?).

Has this list of gripes ever been brought up and discussed in this
forum?

On Nov 13, 2009, at 4:47 AM, Peter Eisentraut wrote:

Has this list of gripes ever been brought up and discussed in this
forum?

Some are TODOs, so in part by other people. Some were briefly touched on in the recent past discussions(around the time that I announced the WIP). Native typing vs conversion, function fragments vs function modules.
I don't think native typing has seen any actual discussion, but I do recall mentioning it as something that I wanted to do(implicitly griped?).

...

There is a difference in the situation from the discussion before. Prior, it was, "I would like to implement a new PL for Python 3 with these features", and now, it is, "I have implemented a new PL for Python 3 with these features".
Simply, -hackers can choose among moving forward with Python 3 support in plpython or going with "plpython3" or even both, I suppose(?). Naturally, I'm biased toward something that involves plpython3, so I don't think I can(should?) be of much help to -hackers as a Python & PG user in any such decision. Of course, excepting the provision of justifications for my implementation/design choices...

I would really love to see some input from Python users.

I certainly don't want to waste time trying to get something into pgsql that Python users don't want.

[here's a gripe that I haven't brought up as I think it is a matter of taste]

I find (plpython) trigger functions to be a bit odd. I think it's the way in which manipulation/suppression decisions are made in BEFORE ROW triggers(return "OK", "SKIP", etc).. [label this as opinion at this point as I have yet to be able to nail down what, specifically, is "wrong" or un-pythonic about them.]

Also, having distinct entry points to handle trigger events helps reduce potential errors by forcing the user to explicitly state the events that the trigger function can handle. Currently, in plpython, users *should* do sanity checks.

Function modules opened the door for implementing this in a natural way, multiple functions(entry points) in the function module.

http://python.projects.postgresql.org/pldocs/plpython3-programming.html#PLPYTHON3-FUNCTIONS-TRIGGERS

On fre, 2009-11-13 at 11:27 -0700, James Pye wrote:

Some are TODOs, so in part by other people. Some were briefly touched
on in the recent past discussions(around the time that I announced the
WIP). Native typing vs conversion, function fragments vs function
modules.

I'm of course only one user, but these two features don't excite me at
all, and certainly not enough to go through the pain of dealing with a
second implementation.

On Nov 15, 2009, at 6:37 AM, Peter Eisentraut wrote:

but these two features don't excite me at all,

hrm.. at all?

I can see how function modules might look like a half-step backwards from function fragments at first, but the benefits of a *natural* initialization section (the module body) was enough to convince me. The added value on the PL developer's side was also compelling. Tracebacks were trivial to implement, and there is no need to munge the function's source. It seemed like a win all around...

AFA native typing is concerned, I think the flexibility and potential it offers is useful, no? Already, plpython3 provides properties on PG's datetime types to access the date_part()'s of the object.

OTOH, for folk who primarily use the PL to access functionality in Python modules(bindings), native typing may be of no direct utility as they will likely need to convert anyways. (If that's your common use-case, then the absence of interest in native typing is quite understandable.)

[looking at the PL/Python todo list..]

Excepting DB-API and trusted, I believe all the current PL/Python TODOs are fulfilled or N/A in plpython3... ugh, the docs are not yet complete, but I like to think of them as "better" anyways. :P

the pain of dealing with a second implementation.

What pain are you anticipating? Maintenance?

On sön, 2009-11-15 at 18:39 -0700, James Pye wrote:

I can see how function modules might look like a half-step backwards from function fragments at first, but the benefits of a *natural* initialization section (the module body) was enough to convince me. The added value on the PL developer's side was also compelling. Tracebacks were trivial to implement, and there is no need to munge the function's source. It seemed like a win all around...

The question is whether it helps the user, not the implementer. As far
as I can tell, it just creates more typing for no benefit whatsoever.
Also, it's inconsistent with normal Python script files and with other
PLs.

AFA native typing is concerned, I think the flexibility and potential it offers is useful, no? Already, plpython3 provides properties on PG's datetime types to access the date_part()'s of the object.

OTOH, for folk who primarily use the PL to access functionality in Python modules(bindings), native typing may be of no direct utility as they will likely need to convert anyways. (If that's your common use-case, then the absence of interest in native typing is quite understandable.)

Right, if I use PL/Python, I do it because I want to use Python. I
don't need another PostgreSQL implementation on top of Python. The
maintenance effort required to keep those two consistent aside.

Again, I'm only one user. But so far I haven't seen anyone else speak
up here, and clearly accepting this for inclusion will need nontrivial
convincing.

the pain of dealing with a second implementation.

What pain are you anticipating? Maintenance?

Right.

Again, I'm only one user.  But so far I haven't seen anyone else speak
up here, and clearly accepting this for inclusion will need nontrivial
convincing.

Well, FWIW, I am excited about better type integration.

Also, I am a little skeptical about this patch. I am sorry if this has
already been discussed, but would this mean that I need to choose
whether pl/python is built against Python 2.* or Python 3.*?

-Nathan

Nathan Boley <npboley@gmail.com> writes:

Also, I am a little skeptical about this patch. I am sorry if this has
already been discussed, but would this mean that I need to choose
whether pl/python is built against Python 2.* or Python 3.*?

Yes. That's exactly what I was complaining about upthread. I'm not
a Python user, but from what I can gather of the 2-to-3 changes,
having to choose one at package build time is going to be a disaster.

regards, tom lane

On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote:

Nathan Boley <npboley@gmail.com> writes:

Also, I am a little skeptical about this patch. I am sorry if this has
already been discussed, but would this mean that I need to choose
whether pl/python is built against Python 2.* or Python 3.*?

Yes. That's exactly what I was complaining about upthread. I'm not
a Python user, but from what I can gather of the 2-to-3 changes,
having to choose one at package build time is going to be a disaster.

Agreed. We really need to have a plpython and plpython3. Heck this would
play nicely too because we support backward compatibility but also
upward version differences.

Joshua D. Drake

regards, tom lane

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
If the world pushes look it in the eye and GRR. Then push back harder. - Salamander

"Joshua D. Drake" <jd@commandprompt.com> writes:

On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote:

Yes. That's exactly what I was complaining about upthread. I'm not
a Python user, but from what I can gather of the 2-to-3 changes,
having to choose one at package build time is going to be a disaster.

Agreed. We really need to have a plpython and plpython3.

Peter was concerned about duplicative maintenance effort, but what I
think this patch shows is that (at least for the near future) both
could be built from a single source file. What we need is configure
and makefile support to do that.

regards, tom lane

On Wed, 2009-11-18 at 12:28 -0500, Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote:

Yes. That's exactly what I was complaining about upthread. I'm not
a Python user, but from what I can gather of the 2-to-3 changes,
having to choose one at package build time is going to be a disaster.

Agreed. We really need to have a plpython and plpython3.

Peter was concerned about duplicative maintenance effort, but what I
think this patch shows is that (at least for the near future) both
could be built from a single source file. What we need is configure
and makefile support to do that.

Ahh, so we would have:

--enable-plpython2=/usr/bin/python2
--enable-plpython3=/usr/bin/python3

?

That seems reasonable if we can run both. Although I wonder if longer
term (2.x is going to be support a long time) we will end up with
frustration within the single source file trying to keep things
straight.

Joshua D. Drake

regards, tom lane

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
If the world pushes look it in the eye and GRR. Then push back harder. - Salamander

"Joshua D. Drake" <jd@commandprompt.com> writes:

On Wed, 2009-11-18 at 12:28 -0500, Tom Lane wrote:

Peter was concerned about duplicative maintenance effort, but what I
think this patch shows is that (at least for the near future) both
could be built from a single source file.

That seems reasonable if we can run both. Although I wonder if longer
term (2.x is going to be support a long time) we will end up with
frustration within the single source file trying to keep things
straight.

Once it gets to the point where it's more trouble to keep them together
than not, we can split the source. But judging from this patch, a
single source file is the ticket for the moment.

regards, tom lane

Here's the patch to support Python >=3.1 with PL/Python.  The
compatibility code is mostly in line with the usual 2->3 C porting
practice and is documented inline.

I took a cursory look at this patch and, while the logic seems sound
and roughly in line with the suggested python porting procedure, I'm
not quite certain what this implies for potential future patches.

For instance, if I wanted to write a type converter for bytea -> the
python 3 byte type would the expectation be that I ensure that it
works in Python 2? Or is an ifdef that ignores it in the case of
Python 2 OK, and we can just put a note in the docs.

Also, how far back do we want to maintain 2.x compatibility? 2.0? If I
wanted to submit a patch that makes use of the list sort method, do I
need to ensure that it can either use the cmp arguments or a key
argument?

What if I wanted to implement a set returning function that made use
of an iterators next() method. Would I just put ifdefs around the code
or a preprocessor definition that defines NEXT as next() for Python
2.x and __next__() for 3.x?

I guess that my first impression is that Python broke compatibility
for a reason, and that either plpython can't evolve, or it will
quickly become impossible to maintain. That being said, I mostly buy
the maintenance arguments from the previous discussion, but if we want
to have plpython and plpython3, a bunch of defines and ifdefs does not
seem like the best way to do this.

Would a better approach be to maintain compatibility layer? ie
plython_compat.h/c
plython2.c
plython3.c

Then patches that apply to a python3 can be applied to plython3.c and
any changed function can be ripped out of plython_compat and moved
into plpython2.

I'm sorry to snipe from the sidelines like this. If we didn't expect
plpython to evolve then this patch seems like the correct approach,
but there is clearly some desire to expand plpython and following this
path seems like it will end in a much more painful split in the future
or a necessary rewrite.

If there is some consensus that this is the best approach, then I will
do a more comprehensive review.

-Nathan

On Nov 18, 2009, at 8:37 AM, Peter Eisentraut wrote:

The question is whether it helps the user, not the implementer.

Sure, but do you have a patch waiting to implement tracebacks?

I'd argue the reason it's never been done is due to the way procedures are currently managed in PL/Python. And *without some significant refactoring*, any patch fully implementing tracebacks is going to be a seriously ugly hack.

What helped the implementer here would help the user.

As far
as I can tell, it just creates more typing for no benefit whatsoever.

"def main(*args):" is annoying, but not entirely lamentable...
It's explicit, as well(no need to document munging that occurs behind the scenes).

Also, compare the cases where you need to cache some initialized data:

if 'key' not in SD:
...
SD['key'] = my_newly_initialized_data
...

With function modules, SD is not needed as you have your module globals to keep your locally cached data in:

...
data = my_newly_initialized_data

def main(*args):
...

Also, it's inconsistent with normal Python script files

Hinges on whether "normal" is actually normal.
I often use the __name__ convention in script files myself:

if __name__ == '__main__':
main(...)

That is, using that convention, the script can be import'd and used without executing the "script functionality". (It has proven to be very handy a few times now)

and with other PLs.

I don't understand why that's a significant enough interest to note.

I don't need another PostgreSQL implementation on top of Python.

Indeed, and I do understand that. That is, I have removed some features with that very thought in mind. (OTOH, I consider the date_part properties on datetime types to be special: too likely useful.)

[tho, "PostgreSQL implementation"? I think I understand what you were getting at, but..]

The maintenance effort required to keep those two consistent aside.

I don't think there are many consistency issues here.
What did you have in mind?

Again, I'm only one user. But so far I haven't seen anyone else speak up here, and clearly accepting this for inclusion will need nontrivial convincing.

Agreed. It would seem quite doomed.

At this point, I'm not going to try getting it into PG. (apparent futility and such)

On Nov 18, 2009, at 1:36 PM, James Pye wrote:

At this point, I'm not going to try getting it into PG. (apparent futility and such)

ugh, on second thought, I think I've written a bit too much code to stop now. I'm going to get plpython3 as far as I can and submit it to the next commitfest.

On ons, 2009-11-18 at 09:48 -0800, Joshua D. Drake wrote:

Although I wonder if longer
term (2.x is going to be support a long time) we will end up with
frustration within the single source file trying to keep things
straight.

There are five million Python modules with C code out there with the
same problem. Considerable effort has been put in by Python upstream to
make the effort manageable. No one in their right mind is going to
create two separate source files just because in the future the mythical
differences will be too big, when clearly the effort is going into a
direction to reduce the differences.

If you look into the source file, there is already special code for
Python 2.2, 2.3, 2.4, 2.5, 2.6, and now 3.1. The chunk for 3.1 is a bit
bigger, but only a bit, and well, that's why it's 3.x and not 2.x. No
one has ever suggested, we might need to support Python 2.2 for a long
time, let's create a separate source file.

I agree, there will probably need to be some configuration/build support
on top of this, but that's something we should work out independently of
how to manage the source file.

On ons, 2009-11-18 at 08:43 -0800, Nathan Boley wrote:

Again, I'm only one user. But so far I haven't seen anyone else speak
up here, and clearly accepting this for inclusion will need nontrivial
convincing.

Well, FWIW, I am excited about better type integration.

Let's clarify, as there are two different models being proposed here.
The first approach, which is currently implemented (and some patches
pending), is to convert a PostgreSQL type to the "nearest" Python type.
For example, text to string, int to int, array to list, timestamp to
datetime.datetime, etc. The other approach, which is what James Pye's
new implementation proposes (as I understand it), is to convert
PostgreSQL types into specially made Python objects, such as
Postgres.types.record or Postgres.types.timestamp.

Also, I am a little skeptical about this patch. I am sorry if this has
already been discussed, but would this mean that I need to choose
whether pl/python is built against Python 2.* or Python 3.*?

Yeah, see later discussion about how to resolve this. But I think in
practice, unless you use lots of print statements in your stored
procedures (?!?), this problem is exaggerated.