Confusion over Python drivers

Bruce,

http://wiki.postgresql.org/wiki/Python

The first one listed, Psycopg, is noted as "preferred libpq-based
driver", but the license is GPL. Isn't that a problem for many client
applications?

The licence of psycopg2 is a little more complicated; the "GPL" in that

summary just tries to sum it. The actual licence is: [from the LICENCE in
the source distribution]

---------------------------------------------
psycopg and the GPL
===================

psycopg is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version. See file COPYING for details.

As a special exception, specific permission is granted for the GPLed
code in this distribition to be linked to OpenSSL and PostgreSQL libpq
without invoking GPL clause 2(b).

Note that the GPL was chosen to avoid proprietary adapters based on
psycopg code. Using psycopg in a proprietary product (even bundling
psycopg with the proprietary product) is fine as long as:

1. psycopg is called from Python only using only the provided API
(i.e., no linking with C code and no C modules based on it); and

2. all the other points of the GPL are respected (you offer a copy
of psycopg's source code, and so on.)

Alternative licenses
====================

If you prefer you can use the Zope Database Adapter ZPsycopgDA (i.e.,
every file inside the ZPsycopgDA directory) user the ZPL license as
published on the Zope web site, http://www.zope.org/Resources/ZPL.

Also, the following BSD-like license applies (at your option) to the
files following the pattern psycopg/adapter*.{h,c} and
psycopg/microprotocol*.{h,c}:

Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:

1. The origin of this software must not be misrepresented; you must not
claim that you wrote the original software. If you use this
software in a product, an acknowledgment in the product documentation
would be appreciated but is not required.

2. Altered source versions must be plainly marked as such, and must not
be misrepresented as being the original software.

3. This notice may not be removed or altered from any source distribution.

psycopg is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

Proprietary licenses
====================

A non-exclusive license is available for companies that want to include
psycopg in their proprietary products without respecting the spirit of the
GPL. The price of the license is one day of development done by the author,
at the consulting fee he applies to his usual customers at the day of the
request.

---------------------------------------------

Harald

--
GHUM Harald Massa
persuadere et programmare
Harald Armin Massa
Spielberger Straße 49
70435 Stuttgart
0173/9409607
no fx, no carrier pigeon
-
%s is too gigantic of an industry to bend to the whims of reality

Massa, Harald Armin wrote:

Bruce,

http://wiki.postgresql.org/wiki/Python

The first one listed, Psycopg, is noted as "preferred libpq-based
driver", but the license is GPL. Isn't that a problem for many client
applications?

The licence of psycopg2 is a little more complicated; the "GPL" in that

summary just tries to sum it. The actual licence is: [from the LICENCE in
the source distribution]

Wow, that is super-confusing. I am dealing with an issue now that some
companies are concerned that individual names appear with our own BSD
copyrights in small files. That pales in comparison to this copyright.
I would never be able to recommend software with that complex a
copyright.

I think the community needs to look at other alternatives.

(I am amazed that not only are there a confusing number of drivers, but
there are a confusing number of copyright options for this single driver.)

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On fre, 2010-02-05 at 09:19 -0500, Bruce Momjian wrote:

While I realize experienced people can easily navigate this confusion,
I
am concerned about new Postgres adopters being very confused by this
and
it is hurting our database adoption in general.

What is really needed is for someone to take charge of one of these
projects and make a best-of-breed Python driver that can gain general
acceptance as our preferred driver. I feel Python is too important a
language to be left in this state.

The situation is unfortunate, but you might as well argue that too many
Linux desktops or Linux distributions confuse new users and hurt
adoption. These alternatives all exist for a reason, and it will be
difficult to get some of them to abandon their work with the aim of
reducing the overall "confusion".

On Fri, Feb 05, 2010 at 09:19:26AM -0500, Bruce Momjian wrote:

My son has brought to my attention that our current crop of Python
client libraries is inadequate/confusing. I took a look myself, and
asked on our IRC channel, and am now convinced this area needs
attention.

http://wiki.postgresql.org/wiki/Python
The Python-hosted PostgreSQL page has similar problems:
http://wiki.python.org/moin/PostgreSQL

Does Perl have a similar mess?

I don't think so.

The primary database interface is DBI and as far as I can see there's
only one DBI PostgreSQL driver: http://search.cpan.org/dist/DBD-Pg/

The only non-DBI interfaces I could find (by skimming the 384 results
for postgresql on search.cpan.org) were:
Postgres: http://search.cpan.org/dist/Postgres/ last updated in 1998.
Pg: http://search.cpan.org/dist/pgsql_perl5/ last updated in 2000.

Tim.

Peter Eisentraut wrote:

On fre, 2010-02-05 at 09:19 -0500, Bruce Momjian wrote:

While I realize experienced people can easily navigate this confusion,
I
am concerned about new Postgres adopters being very confused by this
and
it is hurting our database adoption in general.

What is really needed is for someone to take charge of one of these
projects and make a best-of-breed Python driver that can gain general
acceptance as our preferred driver. I feel Python is too important a
language to be left in this state.

The situation is unfortunate, but you might as well argue that too many
Linux desktops or Linux distributions confuse new users and hurt
adoption. These alternatives all exist for a reason, and it will be
difficult to get some of them to abandon their work with the aim of
reducing the overall "confusion".

Agreed, but can't we do a better job of steering new users and improving
the novice experience for the most popular of them, i.e. think Ubuntu.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Tim Bunce wrote:

On Fri, Feb 05, 2010 at 09:19:26AM -0500, Bruce Momjian wrote:

My son has brought to my attention that our current crop of Python
client libraries is inadequate/confusing. I took a look myself, and
asked on our IRC channel, and am now convinced this area needs
attention.

http://wiki.postgresql.org/wiki/Python
The Python-hosted PostgreSQL page has similar problems:
http://wiki.python.org/moin/PostgreSQL

Does Perl have a similar mess?

I don't think so.

The primary database interface is DBI and as far as I can see there's
only one DBI PostgreSQL driver: http://search.cpan.org/dist/DBD-Pg/

The only non-DBI interfaces I could find (by skimming the 384 results
for postgresql on search.cpan.org) were:
Postgres: http://search.cpan.org/dist/Postgres/ last updated in 1998.
Pg: http://search.cpan.org/dist/pgsql_perl5/ last updated in 2000.

Yes, that's what I thought, and am glad to here it.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On fre, 2010-02-05 at 14:45 +0000, Tim Bunce wrote:

Does Perl have a similar mess?

I don't think so.

The primary database interface is DBI and as far as I can see there's
only one DBI PostgreSQL driver: http://search.cpan.org/dist/DBD-Pg/

I think another difference is that the Perl DBI interface is very rich,
whereas the Python DB-API is quite minimal and almost forces people to
write (incompatible) extensions. The DB-SIG at Python that ought to
drive all this is also quite dead, possibly because everyone has moved
on to SQLAlchemy.

Peter Eisentraut wrote:

On fre, 2010-02-05 at 14:45 +0000, Tim Bunce wrote:

Does Perl have a similar mess?

I don't think so.

The primary database interface is DBI and as far as I can see there's
only one DBI PostgreSQL driver: http://search.cpan.org/dist/DBD-Pg/

I think another difference is that the Perl DBI interface is very rich,
whereas the Python DB-API is quite minimal and almost forces people to
write (incompatible) extensions. The DB-SIG at Python that ought to
drive all this is also quite dead, possibly because everyone has moved
on to SQLAlchemy.

I assumed it would be Perl that had confusion because of the Perl 6, but
obviously it is Python. The second PL/Python server-side language that
was proposed recently is taking on a whole new appearance for me. :-(

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

I think another difference is that the Perl DBI interface is very rich,
whereas the Python DB-API is quite minimal and almost forces people to
write (incompatible) extensions. The DB-SIG at Python that ought to
drive all this is also quite dead, possibly because everyone has moved
on to SQLAlchemy.

For people who use Python a lot, could I have a list of the deficiencies
in DBAPI? I've got my horse and lance ready.

Given that SQLAlchemy isn't for everyone, of course ... it couldn't be,
or Django would use it, no?

--Josh Berkus

The situation is unfortunate, but you might as well argue that too many
Linux desktops or Linux distributions confuse new users and hurt
adoption. These alternatives all exist for a reason, and it will be
difficult to get some of them to abandon their work with the aim of
reducing the overall "confusion".

I'm not as concerned about "confusion" as the fact that *all* of the
various Python drivers suck in different, and crippling, ways. I don't
care how many drivers we have, as long as we have at least one 1st-class
driver.

--Josh Berkus

Josh Berkus wrote:

The situation is unfortunate, but you might as well argue that too many
Linux desktops or Linux distributions confuse new users and hurt
adoption. These alternatives all exist for a reason, and it will be
difficult to get some of them to abandon their work with the aim of
reducing the overall "confusion".

I'm not as concerned about "confusion" as the fact that *all* of the
various Python drivers suck in different, and crippling, ways. I don't
care how many drivers we have, as long as we have at least one 1st-class
driver.

Agreed, and we can point newbies to that one great driver.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Feb 5, 2010, at 8:00 AM, Peter Eisentraut wrote:

I think another difference is that the Perl DBI interface is very rich,
whereas the Python DB-API is quite minimal and almost forces people to
write (incompatible) extensions.

Yep.

The DB-SIG at Python that ought to drive all this is also quite dead,

From reading messages that come into that list(not-so-lately), I've gotten the impression that they are content with DB-API 2. Aside from the TPC interfaces, the last round of activity that I know of was dbapi3[1 (which was a long while ago)].

possibly because everyone has moved on to SQLAlchemy.

Yeah. AFAICT, that is the general direction encouraged by DB-SIG. Write an abstraction on top of DB-API. SQLAlchemy, SQLObject, anysql, and I'm sure there are others.

[1]: http://wiki.python.org/moin/Aug2001DbApi3Strawman http://wiki.python.org/moin/DbApi3
http://wiki.python.org/moin/DbApi3

On Fri, 2010-02-05 at 09:38 -0500, Bruce Momjian wrote:

Wow, that is super-confusing.

Agreed. Standardization among licenses is useful, and I think it's
important to have a driver with a license that people already
understand.

Regards,
Jeff Davis

On Fri, 2010-02-05 at 10:35 -0800, Josh Berkus wrote:

I'm not as concerned about "confusion" as the fact that *all* of the
various Python drivers suck in different, and crippling, ways. I don't
care how many drivers we have, as long as we have at least one 1st-class
driver.

Absolutely.

And I would prefer that it can provide all (or nearly all) of the
capabilities of libpq. PyGreSQL apparently doesn't even offer
parameterized queries!

http://pygresql.org/pg.html

That was based on a quick glance at the document; I hope I'm mistaken.

Regards,
Jeff Davis

On Feb 5, 2010, at 11:34 AM, Josh Berkus wrote:

For people who use Python a lot, could I have a list of the deficiencies
in DBAPI? I've got my horse and lance ready.

Given that SQLAlchemy isn't for everyone, of course ... it couldn't be,
or Django would use it, no?

Here are some to start with:

- paramstyle
- No direct support of prepared statements
[they *tried* to compensate for this with "cached statements", but it's inadequate]
- Too many *optional* extensions (Cursor Method .__iter__() being rather notable)

http://www.python.org/dev/peps/pep-0249/

On 2/5/10, Josh Berkus <josh@agliodbs.com> wrote:

I think another difference is that the Perl DBI interface is very rich,
whereas the Python DB-API is quite minimal and almost forces people to
write (incompatible) extensions. The DB-SIG at Python that ought to
drive all this is also quite dead, possibly because everyone has moved
on to SQLAlchemy.

For people who use Python a lot, could I have a list of the deficiencies
in DBAPI? I've got my horse and lance ready.

I took quick glance on DBI and compared to that, DB-API does
not define specific api for:
- resultset cursors
- prepared plans
- db structure examination

OTOH, my guess is that DB-API authors assumed the first two
are used transparently by the driver, and I see no reason why
they cannot be, with current DB-API.

Last item is "rich" indeed, but seems slightly overengineered..

Now, none of the differences between drivers and current confusion
seem to be related to above points. "Hysterical Raisins" is only
good explanation about current situation.

The pg8000 / bpgsql seem to be toy projects, and anyway you dont
want to use pure-Python drivers in high-performance environments.
We are not talking about C#/java here.

py-postgresql seems to be more serious, but as it's python3 only
which makes it irrelevant today.

PyGreSQL is the oldest, older than DB-API, and so it's DB-API
interface seems an afterthought and is untested/underused - eg.
it does not support bytea.

Psycopg was the leader, especially in web-environments,
but it has non-obvious license and with dead website it does not
seem that attractive. Although it is well-maintained still.

Best path forward would be to talk with Psycopg guys about
license clarification/change.

I suspect GPL does not extend anyway to Python code that imports
it dynamically, and it does not seem to be their intention - they
even try to tell it in their current clarification, but it
is not as clear as it could be.

--
marko

Marko Kreen wrote:

Psycopg was the leader, especially in web-environments,
but it has non-obvious license and with dead website it does not
seem that attractive. Although it is well-maintained still.

Best path forward would be to talk with Psycopg guys about
license clarification/change.

I suspect GPL does not extend anyway to Python code that imports
it dynamically, and it does not seem to be their intention - they
even try to tell it in their current clarification, but it
is not as clear as it could be.

Yes, this is a good analysis and has some good suggestions for moving
forward. In summary, I don't think the current PG/Python driver
situation is helping PG or Python, and I believe there are enough people
who care about those two to get a group together to sort out a viable
direction and start working toward the goal of improving things.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:

While I realize experienced people can easily navigate this confusion...

No, that's the worst part--the more you know and the deeper you dig into
it, the more broken you realize the whole thing is. When one of the
best drivers (in some respects) has a web page that looks like this:
http://initd.org/ that doesn't seem so bad at first--but if you're
experienced, you know that the page has been in that disturbing state
since late 2006. You start digging into the driver mess, figure you
just need to learn how things fit together, but the hole keeps getting
bigger as you dig.

The issues here have already been identified: the Perl DBI is an
excellent spec, while the Python one is so weak everybody ends up
needing their own extensions to it. And then portability *even among
Python PostgreSQL drivers* goes out the window. If somebody built a
BSD/MIT licensed driver that replaces every useful feature of all the
forks, with no major problems, and a couple of major projects switched
over to it (think "Skype" level big), maybe this mess could get
resolved. I think it would take someone already familiar with the major
issues involved a couple of months of regular work to make any serious
progress on it.

--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com

Greg Smith wrote:

Bruce Momjian wrote:

While I realize experienced people can easily navigate this confusion...

No, that's the worst part--the more you know and the deeper you dig into
it, the more broken you realize the whole thing is. When one of the
best drivers (in some respects) has a web page that looks like this:
http://initd.org/ that doesn't seem so bad at first--but if you're
experienced, you know that the page has been in that disturbing state
since late 2006. You start digging into the driver mess, figure you
just need to learn how things fit together, but the hole keeps getting
bigger as you dig.

The issues here have already been identified: the Perl DBI is an
excellent spec, while the Python one is so weak everybody ends up
needing their own extensions to it. And then portability *even among
Python PostgreSQL drivers* goes out the window. If somebody built a
BSD/MIT licensed driver that replaces every useful feature of all the
forks, with no major problems, and a couple of major projects switched
over to it (think "Skype" level big), maybe this mess could get
resolved. I think it would take someone already familiar with the major
issues involved a couple of months of regular work to make any serious
progress on it.

Yes, it is going to take serious time, and it is going to take more than
one person to be involved, but I think there are enough people who care
that something serious can be done to improve its current state.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +