plpython3

On Nov 19, 2009, at 5:41 PM, James Pye wrote:

Here's my latest patch.

Fixed a lot of memory/reference leaks, added some minor features(mostly around Arrays), and filled in more documentation.

At this point, I don't have any more minor features in mind(save extending Postgres.notify when the payload patch hits), so I'm just doing finish work(improvements/clarifications to docs, message strings, and maybe some makefile work).

On Sun, Dec 13, 2009 at 5:02 PM, James Pye <lists@jwp.name> wrote:

On Nov 19, 2009, at 5:41 PM, James Pye wrote:

Here's my latest patch.

Fixed a lot of memory/reference leaks, added some minor features(mostly around Arrays), and filled in more documentation.

At this point, I don't have any more minor features in mind(save extending Postgres.notify when the payload patch hits), so I'm just doing finish work(improvements/clarifications to docs, message strings, and maybe some makefile work).

I'm almost afraid to write anything at all about this patch for fear
of being branded a nattering nabob of negativity (see other thread:
damage control mode) but hopefully if I'm full of it (or not) others
will write in and set me straight (or confirm my thinking, whichever
is appropriate). Anyhow, I started by reviewing the past threads on
this patch, to which the author helpfully provided links:

Past threads on the subject:

http://archives.postgresql.org/pgsql-hackers/2009-05/msg01376.php
http://archives.postgresql.org/pgsql-hackers/2009-07/msg01519.php
http://archives.postgresql.org/pgsql-hackers/2009-08/msg01505.php

I think we should be clear that this patch doesn't have a great deal
to do with Python 3, since Peter Eisentraut has already patched the
existing code to support Python 3. It is, rather, a reimplementation
of PL/python, and accordingly it ought to be called pl/newpython or
perhaps pl/pye-thon (sorry, couldn't resist). Peter Eisentraut has
made it pretty clear that he would prefer to see us maintain and
enhance the existing implementation rather than starting over, and
even if we did start over, it seems from the above threads that we'd
still need to maintain the existing code for quite a while (if not
forever).

So it seems to me that the threshold question for this patch is - do
we think it's a good idea to maintain two implementations of PL/python
in core?

...Robert

So it seems to me that the threshold question for this patch is - do
we think it's a good idea to maintain two implementations of PL/python
in core?

Not really, no. This is why we need PGAN ;-)

If the new implementation is *better* that the existing PL/python, I
could see eventually replacing it. It wouldn't be the first time that a
rewrite exceeded the original tool.

However, I'm not in a position to judge quality.

--Josh Berkus

On Tue, 2010-01-12 at 20:06 -0800, Josh Berkus wrote:

So it seems to me that the threshold question for this patch is - do
we think it's a good idea to maintain two implementations of PL/python
in core?

Not really, no. This is why we need PGAN ;-)

If the new implementation is *better* that the existing PL/python, I
could see eventually replacing it. It wouldn't be the first time that a
rewrite exceeded the original tool.

I think it is important to remember that the current version of
PL/python is pretty weak compared to its counter parts (Specifically
PL/Perl). If the new version, is adequately written to community
standards and increases PL/Python's capabilities we need to seriously
consider it.

If we can address any issues with this module, let's commit it as
Pl/pythonng3 or something.

Anyway, I am +1 on reviewing this patch for viability.

I would love to never touch plPerl for advanced procedures again.

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On ons, 2010-01-13 at 09:47 -0800, Joshua D. Drake wrote:

I think it is important to remember that the current version of
PL/python is pretty weak compared to its counter parts (Specifically
PL/Perl).

How so?

On Wed, 2010-01-13 at 19:53 +0200, Peter Eisentraut wrote:

On ons, 2010-01-13 at 09:47 -0800, Joshua D. Drake wrote:

I think it is important to remember that the current version of
PL/python is pretty weak compared to its counter parts (Specifically
PL/Perl).

How so?

O.k. you may have just called me on an unintentional bluff. My knowledge
of plpython is dated. I just tested some of the things (like in/out) and
they appear to work at least on 8.4).

My argument would be now, what is the benefit of the James Pye version
over our version. James can you illustrate succinctly why we should be
supporting a new version?

If there is, I am still all for it, but I am a python bigot.

Sincerely,

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

My argument would be now, what is the benefit of the James Pye version
over our version. James can you illustrate succinctly why we should be
supporting a new version?

If there is, I am still all for it, but I am a python bigot.

Yeah, it's just my viewpoint that we don't want 2 python procedural
languages in core. One should be in core, and one should go on
pgFoundry/PGAN. Which ever one is "better" by some clear definition
should go in core.

As I said, I have no opinion about whether Pye's PLpython should replace
the existing because I'm in no position to judge.

--Josh Berkus

On Wed, Jan 13, 2010 at 1:16 PM, Josh Berkus <josh@agliodbs.com> wrote:

My argument would be now, what is the benefit of the James Pye version
over our version. James can you illustrate succinctly why we should be
supporting a new version?

If there is, I am still all for it, but I am a python bigot.

Yeah, it's just my viewpoint that we don't want 2 python procedural
languages in core.  One should be in core, and one should go on
pgFoundry/PGAN.  Which ever one is "better" by some clear definition
should go in core.

That was my thinking also. There are a couple of reasons to think
that throwing over the current implementation for an new one may not
be the right thing to do.

1. It's not just a rewrite, it's an incompatible rewrite that will
present significant user-visible behavioral differences. So replacing
the current implementation wholesale would produce massive breakage
for anyone actually using PL/python in production.

2. Peter Eisentraut, who has put significant time into improving
PL/python for this release (see the commit logs), and who is the ONLY
committer to work on improving PL/python for this release, has made it
clear that he prefers the current implementation.

Given these two facts, it's hard for me to see how we could decide to
REMOVE the current implementation and replace it with the new one. So
the most we could do is maintain them side by side, and then you have
to ask, why? It will actually be much easier for James Pye to
maintain his code outside core, and if it turns out to be popular and
people come back to us and say "hey, why isn't this the default?" then
we can revisit the issue. Sure, his code won't get as much exposure
that way, but it's been posted to the mailing list several times now
over a period of 8 months and nobody has said "oh, wow, this is
great". The most we've gotten is several versions of this exchange:

Somebody: We should really consider this code, the current code isn't very good.
Peter: What's wrong with it?
Somebody: <describes some problem, usually fairly vaguely>
Peter: Why can't we fix that in the existing code base?
<thread ends>

I'm not saying a complete rewrite isn't the way to go - I'm just
saying that there isn't much evidence at this point that it's really
necessary, and I don't think we want to maintain two copies of
PL/python unless we're really sure that the new implementation is an
improvement.

...Robert

On Jan 13, 2010, at 11:08 AM, Joshua D. Drake wrote:

My argument would be now, what is the benefit of the James Pye version
over our version. James can you illustrate succinctly why we should be
supporting a new version?

Doing so, succinctly, is unfortunately difficult.
It is primarily a matter of comparing features, AFAICT. And, furthermore, some features may not be useful to some users.

It exposes additional functionality that should *not* be incrementally developed in plpython as it would break applications. This was the point of trying to move forward with it for Python 3.

Function Modules:
- Does away with the need for GD/SD (more natural Python environment).
- Allows tracebacks (tracebacks are useful, right?) to implemented easily.
- Does *not* expose a bastardized variant of the language by pretending that "modules/script files" can return and yield.
- Helps to promote the Python tenet of being explicit.

Native Typing:
- Provides PG type introspection not available in any other PL, AFAIK.
- Improves efficiency in some cases (conversion must be _explicitly_ called for)
- MD Array support.
- Composites are a sequence and a mapping.

Other features: http://wiki.postgresql.org/wiki/WIP:plpython3

Aside from function modules and native typing, many of plpython3's features could be implemented incrementally. However, I had a chance to sprint and they are available now in a new implementation. I did so, rather than improving plpython, because I believe that native typing and function modules are very useful.

I'm not sure this fulfills your request, but, hopefully, it's a start.

On Wed, 2010-01-13 at 13:06 -0700, James William Pye wrote:

On Jan 13, 2010, at 11:08 AM, Joshua D. Drake wrote:

My argument would be now, what is the benefit of the James Pye version
over our version. James can you illustrate succinctly why we should be
supporting a new version?

Doing so, succinctly, is unfortunately difficult.
It is primarily a matter of comparing features, AFAICT. And, furthermore, some features may not be useful to some users.

It exposes additional functionality that should *not* be incrementally developed in plpython as it would break applications. This was the point of trying to move forward with it for Python 3.

Function Modules:
- Does away with the need for GD/SD (more natural Python environment).
- Allows tracebacks (tracebacks are useful, right?) to implemented easily.
- Does *not* expose a bastardized variant of the language by pretending that "modules/script files" can return and yield.
- Helps to promote the Python tenet of being explicit.

Native Typing:
- Provides PG type introspection not available in any other PL, AFAIK.
- Improves efficiency in some cases (conversion must be _explicitly_ called for)
- MD Array support.
- Composites are a sequence and a mapping.

Other features: http://wiki.postgresql.org/wiki/WIP:plpython3

Aside from function modules and native typing, many of plpython3's features could be implemented incrementally. However, I had a chance to sprint and they are available now in a new implementation. I did so, rather than improving plpython, because I believe that native typing and function modules are very useful.

I'm not sure this fulfills your request, but, hopefully, it's a start.

It does actually. Now, hackers... as a Python guy I can say these things
are truly useful, to a Python programmer trying to use Python as a
procedural language with PostgreSQL.

What do we think?

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On ons, 2010-01-13 at 12:12 -0800, Joshua D. Drake wrote:

On Wed, 2010-01-13 at 13:06 -0700, James William Pye wrote:

Function Modules:
- Does away with the need for GD/SD (more natural Python environment).
- Allows tracebacks (tracebacks are useful, right?) to implemented easily.
- Does *not* expose a bastardized variant of the language by pretending that "modules/script files" can return and yield.
- Helps to promote the Python tenet of being explicit.

Native Typing:
- Provides PG type introspection not available in any other PL, AFAIK.
- Improves efficiency in some cases (conversion must be _explicitly_ called for)
- MD Array support.
- Composites are a sequence and a mapping.

Other features: http://wiki.postgresql.org/wiki/WIP:plpython3

Aside from function modules and native typing, many of plpython3's features could be implemented incrementally. However, I had a chance to sprint and they are available now in a new implementation. I did so, rather than improving plpython, because I believe that native typing and function modules are very useful.

I'm not sure this fulfills your request, but, hopefully, it's a start.

It does actually. Now, hackers... as a Python guy I can say these things
are truly useful, to a Python programmer trying to use Python as a
procedural language with PostgreSQL.

The problem I'm having with this discussion is that every time someone
asks what the supposed advantages of this new Python PL are, a feature
list like the above is dumped, 75% of which is subjective and tends to
use semi-buzzwords, such that then someone else who by his own admission
isn't completely up to date on things says, sure, that sounds great.
Who wouldn't like a "more natural Python environment", "native typing"
and "efficiency", and maybe even "explicitness"? The current PL/Python
also has, arguably, a more natural Python environment, native typing,
efficiency, and explicitness. So there you go. Now what?

On Wed, 2010-01-13 at 23:27 +0200, Peter Eisentraut wrote:

The problem I'm having with this discussion is that every time someone
asks what the supposed advantages of this new Python PL are, a feature
list like the above is dumped, 75% of which is subjective and tends to
use semi-buzzwords, such that then someone else who by his own admission
isn't completely up to date on things says, sure, that sounds great.
Who wouldn't like a "more natural Python environment", "native typing"
and "efficiency", and maybe even "explicitness"? The current PL/Python
also has, arguably, a more natural Python environment, native typing,
efficiency, and explicitness. So there you go. Now what?

Peter, are you interested in the benefit of the project, or your own
ego? Take it easy and let's see if we can be productive here.

I was just stating that James's explanation was good and looking for
more information from a reviewer. Secondly nobody (at least I am not) is
suggesting we dump the current PL/Python, so your hard work is not going
to be lost.

The only thing I am currently looking for is an objective review of the
patch based on the benefits it provides. I can tell you that if the Pye
patch provides stack trace capability and the current PL does not. That
right there is enough for me to push a +1 for review and possible
inclusion.

If you feel what James is saying is actually trite and more buzzword
compliant I welcome the opportunity to see a comparative of the current
status of PL/Python versus the Pye patch from you. If nothing else it is
a learning opportunity for all involved.

Sincerely,

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On ons, 2010-01-13 at 13:33 -0800, Joshua D. Drake wrote:

The only thing I am currently looking for is an objective review of the
patch based on the benefits it provides.

Right, but I was opining that such a vague feature listing is not
adequate for that.

I can tell you that if the Pye
patch provides stack trace capability and the current PL does not. That
right there is enough for me to push a +1 for review and possible
inclusion.

That is the case.

If you feel what James is saying is actually trite and more buzzword
compliant I welcome the opportunity to see a comparative of the current
status of PL/Python versus the Pye patch from you. If nothing else it is
a learning opportunity for all involved.

There was extensive discussion about some of the design decisions
upthread, so any reviewer should review those. It may end up being a
agree-to-disagree situation.

On Jan 13, 2010, at 2:27 PM, Peter Eisentraut wrote:

The problem I'm having with this discussion is that every time someone
asks what the supposed advantages of this new Python PL are, a feature
list like the above is dumped,

I agree that this is unfortunate, but how else can we to discuss the advantages? It boils down to comparing a couple feature lists, and *maybe* some implementation details. No?

75% of which is subjective and tends to use semi-buzzwords,

You say "semi-buzzwords", I say "names". I have to call "it" something.

such that then someone else who by his own admission
isn't completely up to date on things says, sure, that sounds great.

Which is why we need to get some more experienced Python users involved in this.

Well, even the mileage of inexperienced users is quite useful for detecting what level of obviousness has been achieved by the features, so I'm not trying to exclude anyone.

The current PL/Python also has, arguably, a more natural Python environment,

No, it doesn't. GD/SD are contrived in order to compensate for the very absence of that. Additionally, "modules / script files" don't return or yield.

native typing,

Okay, that's arguably subjective, but when I write "native typing", it's wrt PG, not Python's built-in types. If that wasn't the case, I wouldn't call it anything--save "[data] conversion" when necessary--as it wouldn't be much of a feature to clamor about.

efficiency,

Yes, as discussed in the thread before there are trade-offs here wrt how PG data is handled. It depends pretty heavily on how the parameters / query results are used.

Although, as stated before, the difference in efficiency can be rather significant in situations where conversion to built-in Python types is *not* desired.

and explicitness.

Hrm? What is explicit about munging the source code of the procedure to make it into a function body? Perhaps you could give some examples where plpython helps promote explicitness?

And sure, I understand that other PLs do this. That may be fine for those languages, but for Python it's not, IMO.

On Jan 13, 2010, at 12:15 PM, Robert Haas wrote:

1. It's not just a rewrite, it's an incompatible rewrite that will
present significant user-visible behavioral differences. So replacing
the current implementation wholesale would produce massive breakage
for anyone actually using PL/python in production.

Right. That was the point of trying to leverage Python 3 to make the distinction. Most people will need to update their functions if they are moving to Python 3. And for the larger chunks of code, the hard stuff, the amount of change required is likely significant already.

<snip some>

Given these two facts, it's hard for me to see how we could decide to
REMOVE the current implementation and replace it with the new one. So
the most we could do is maintain them side by side, and then you have
to ask, why?

My original hope was that plpython would be maintained for several years to come and when Python 3 started picking up steam, we would deprecate plpython. If people still wanted to use it, they could continue using the older version of PG and/or someone could continue to maintain plpython out of core for legacy support.

<snip/maintaining out of core>

Sure, his code won't get as much exposure that way,

=)

Try next to none. The existence of core's implementation makes competing *very* difficult, IMO. Thinking of something along the lines: "Why would I use/contribute to your implementation when core has one?" And, all I can say in response is, "Check out my features." Subsequently, they will probably weigh the added risk of choosing the loner's implementation and come to the conclusion that using core's would be safer in the long term. I can't fault that line of reasoning, so ISTM that it would be difficult to "sell".

but it's been posted to the mailing list several times now
over a period of 8 months and nobody has said "oh, wow, this is
great".

Yeah. :(

In the past, one person showed interest in function modules(Stuart, see the first WIP message), and two others showed interest in native typing(Nathan and Tino). Mr. Drake has also shown some interest in this thread.

But, yes, you are correct. There has been no "wow, this is great" message.

James William Pye wrote:

On Jan 13, 2010, at 2:27 PM, Peter Eisentraut wrote:

The problem I'm having with this discussion is that every time someone
asks what the supposed advantages of this new Python PL are, a feature
list like the above is dumped,

I agree that this is unfortunate, but how else can we to discuss the advantages? It boils down to comparing a couple feature lists, and *maybe* some implementation details. No?

Code samples. You're trying to unseat a well established incumbent here,
and you're not ever going to do that with documentation. Maybe your
plpython3 has some compelling features to it. I don't know, because even
with several thousand lines of basic Python code to my credit I cannot
understand a single one of the arguments you presented for why your
implementation is better--except agreeing that, yes, tracebacks are
useful And even on that one, I'm not going to take your word on the
superiority of your implementation. You're writing way over people's
heads here. (Doesn't help that your docs link at the bottom of
http://wiki.postgresql.org/wiki/WIP:plpython3 is broken either). If one
has to be a Python expert to understand your position, you've already lost.

Python code is easy to read though. If you'd said "here's a great
example of how Function Modules are an improvement over what you can do
with the current pl/python," that would be infinitely more useful than
the list of language trivia related to them. You should be aiming to put
Peter on the spot to respond to claims you make like "you can't do this
easily with the current implementation" after showing an elegant bit of
code.

One of the things I'm increasingly frustrated by (and don't take this
personally, this is a general comment coming more from the last CF
rather than something I mean to single you out for) is how many patch
submissions we get that don't have *compelling* examples showing their
value. Have a better programming approach to something? Show me the old
way and how the new way is better. Performance improvement? Provide a
complete, self-contained example showing how to demonstrate it. New type
of feature? Cut and paste a whole session showing how it's used, with
every single command you typed after initdb.

Basically, if a reviewer can't confirm your patch is doing something
useful in five minutes and be excited that they've watched something
interesting happen, you're decreasing the chances that your patch will
ever go anywhere dramatically. I hope that everyone submitting patches
reads http://www.depesz.com/ at least once in a while. One of the things
I really enjoy about his blog is how he shows complete working examples
of so many patches. To pick a standout recent entry,
http://www.depesz.com/index.php/2010/01/03/waiting-for-8-5-exclusion-constraints/
takes "exclusion constraints"--a feature I didn't follow a bit of the
discussion about--and works through the whole feature with a series of
examples that, while still complicated, are completely self-contained
and possible to follow along until you understand how it all fits
together. Patch submitters should consider it a goal to make life that
easy for the reviewer stuck with checking their patch out.

--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com

* Greg Smith <greg@2ndquadrant.com> [100114 02:17]:

One of the things I'm increasingly frustrated by (and don't take this
personally, this is a general comment coming more from the last CF
rather than something I mean to single you out for) is how many patch
submissions we get that don't have *compelling* examples showing their
value. Have a better programming approach to something? Show me the old
way and how the new way is better. Performance improvement? Provide a
complete, self-contained example showing how to demonstrate it. New type
of feature? Cut and paste a whole session showing how it's used, with
every single command you typed after initdb.

Wow, I can't agree more... I've seen *so* many patches fly by that don't
mean *anything* to me with the description sent to -hackers, until I
find out what they actually do, or could do, until I find it in my RSS
reader, via:

I hope that everyone submitting patches
reads http://www.depesz.com/ at least once in a while. One of the things
I really enjoy about his blog is how he shows complete working examples
of so many patches. To pick a standout recent entry,
http://www.depesz.com/index.php/2010/01/03/waiting-for-8-5-exclusion-constraints/
takes "exclusion constraints"--a feature I didn't follow a bit of the
discussion about--and works through the whole feature with a series of
examples that, while still complicated, are completely self-contained
and possible to follow along until you understand how it all fits
together.

<what he said>++

Patch submitters should consider it a goal to make life that
easy for the reviewer stuck with checking their patch out.

Yes, submitters, please specifically try to make Hubert's life easier,
because we *all* will appreciate it...

--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.

On Jan 14, 2010, at 12:17 AM, Greg Smith wrote:

Code samples.

Okay.

I don't know, because even with several thousand lines of basic Python code to my credit I cannot understand a single one of the arguments you presented for why your implementation is better--except agreeing that, yes, tracebacks are useful

And even on that one, I'm not going to take your word on the superiority of your implementation.

Sure, that's what review is about. No?

You're writing way over people's heads here.

Okay. I guess I hoped the documentation would help clarify a lot of this, and make the advantages self-evident.

On that:

(Doesn't help that your docs link at the bottom of http://wiki.postgresql.org/wiki/WIP:plpython3 is broken either).

Ouch. Thanks, that's fixed now. Please take a look again:

http://python.projects.postgresql.org/pldocs/plpython3.html

If one has to be a Python expert to understand your position, you've already lost.

Function modules should be pretty obvious. "native typing" is a bit more difficult as a solid understanding of PG's type system is fairly important for a firm grasp.

Python code is easy to read though. If you'd said "here's a great example of how Function Modules are an improvement over what you can do with the current pl/python," that would be infinitely more useful than the list of language trivia related to them. You should be aiming to put Peter on the spot to respond to claims you make like "you can't do this easily with the current implementation" after showing an elegant bit of code.

Okay. So, some examples would help. The documentation is back up, so please be sure to look at the numerous examples provided therein. In addition to that, I'll try to get some contrasting examples posted as a follow-up to an earlier message. "In plpython you do X whereas in plpython3 you do Y."

Thanks.

On Thu, 2010-01-14 at 05:39 -0700, James William Pye wrote:

Python code is easy to read though. If you'd said "here's a great example of how Function Modules are an improvement over what you can do with the current pl/python," that would be infinitely more useful than the list of language trivia related to them. You should be aiming to put Peter on the spot to respond to claims you make like "you can't do this easily with the current implementation" after showing an elegant bit of code.

Okay. So, some examples would help. The documentation is back up, so please be sure to look at the numerous examples provided therein. In addition to that, I'll try to get some contrasting examples posted as a follow-up to an earlier message. "In plpython you do X whereas in plpython3 you do Y."

The documentation is very thorough, thank you. I am still a fan of
getting this reviewed for potential inclusion but I firmly agree with
what Greg has already said.

What I would (as a non hacker) would look for is:

(1) Generalized benchmarks between plpython(core) and plpython3u

I know a lot of these are subjective, but it is still good to see if
there are any curves or points that bring the performance of either to
light.

(2) Example of the traceback facility, I know it is silly but I don't
have time to actually download head, apply the patch and test this. This
type of thing, showing debugging facilities within the function would be
killer.

(3) A distinct real world comparison where the core plpython falls down
(if it does) against the plpython3u implementation

I can't speak to your code quality, that is going to have to be someone
else.

Sincerely,

Joshua D. Drake

Thanks.

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.