Debian readline/libedit breakage

On Thu, Feb 10, 2011 at 2:34 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

Hello,

Per:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=607109

It seems we may have a problem to consider. As far as I know, we are the
only major platform that supports libedit but our default is readline.
Unfortunately readline is not compatible with OpenSSL (apparently?)
licensing.

This seems that it may be a problem for us considering the pre-package
builds we do.

What does everyone think? Should we work on getting libedit up to snuff?

I have to admit, this change in debian packaging -- which I have
noticed, and not a little -- makes my hands angry. I considered
looking into the problem, but were I doing it, I would have considered
teaching psql to support NSS or GnuTLS as totally viable alternatives
to this problem, as to keep readline.

--
fdr

On 02/10/2011 05:34 PM, Joshua D. Drake wrote:

Hello,

Per:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=607109

It seems we may have a problem to consider. As far as I know, we are the
only major platform that supports libedit but our default is readline.
Unfortunately readline is not compatible with OpenSSL (apparently?)
licensing.

This seems that it may be a problem for us considering the pre-package
builds we do.

What does everyone think? Should we work on getting libedit up to snuff?

I'll be happy if you do, but why haven't I haven't noticed, say, RedHat
taking this line?

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

I'll be happy if you do, but why haven't I haven't noticed, say, RedHat
taking this line?

Less narrow-minded interpretation of GPL requirements, perhaps.
(And yes, we have real lawyers on staff considering these issues.)

libedit is a long way from being ready to replace readline,
much as one could wish it otherwise. If Debian want to shoot
themselves in the foot like that, we can't stop them, but neither
should we be devoting our project resources to fixing libedit.

(I have seen some noise recently on the Fedora lists about putting
work into libedit, so maybe something good will come of that.
I'm just not ready to define it as my/our problem.)

regards, tom lane

Excerpts from Joshua D. Drake's message of jue feb 10 19:34:31 -0300 2011:

Hello,

Per:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=607109

O, the joy of having people mess up with legal stuff that nobody cares
about creating endless work for everyone.

--
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

* Daniel Farina (drfarina@acm.org) wrote:

I have to admit, this change in debian packaging -- which I have
noticed, and not a little -- makes my hands angry. I considered
looking into the problem, but were I doing it, I would have considered
teaching psql to support NSS or GnuTLS as totally viable alternatives
to this problem, as to keep readline.

Supporting GnuTLS would be really nice.. That's how we addressed the
same issue w/ OpenLDAP (I was involved in that as a Debian
co-maintainer). GnuTLS has limitations too, but in the end, I find
those more palatable (and the GnuTLS maintainer is certainly willing to
work on improving it) than dropping readline. :/

THanks,

Stephen

On 02/10/2011 06:36 PM, Stephen Frost wrote:

* Daniel Farina (drfarina@acm.org) wrote:

I have to admit, this change in debian packaging -- which I have
noticed, and not a little -- makes my hands angry. I considered
looking into the problem, but were I doing it, I would have considered
teaching psql to support NSS or GnuTLS as totally viable alternatives
to this problem, as to keep readline.

Supporting GnuTLS would be really nice.. That's how we addressed the
same issue w/ OpenLDAP (I was involved in that as a Debian
co-maintainer). GnuTLS has limitations too, but in the end, I find
those more palatable (and the GnuTLS maintainer is certainly willing to
work on improving it) than dropping readline. :/

Strikes me as a lot of work to buy nothing much.

cheers

andrew

On Thu, Feb 10, 2011 at 06:04:46PM -0500, Tom Lane wrote:

Less narrow-minded interpretation of GPL requirements, perhaps.
(And yes, we have real lawyers on staff considering these issues.)

Is their opinion public/can be made public? This might possibly lead to
a re-evaluation of the situation by Debian.

If Debian want to shoot themselves in the foot like that, we can't
stop them

BTW, that change has been merged into Ubuntu and will be (as of now) in
the next Ubuntu release.

Michael

Michael Banck wrote:

On Thu, Feb 10, 2011 at 06:04:46PM -0500, Tom Lane wrote:

Less narrow-minded interpretation of GPL requirements, perhaps.
(And yes, we have real lawyers on staff considering these issues.)

Is their opinion public/can be made public? This might possibly lead to
a re-evaluation of the situation by Debian.

I doubt that. This is one of those situations where there is an
ideological position held by the FSF and Debian that's unlikely to
budge, but one that has limited testing in court. I believe that
RedHat's lawyers have assessed the business risk here and judged it not
sufficient to worry about. But their opinion on that isn't going to
sway anyone evaluating this primarily on free software principles.

I had to trace down the history here once before while working on
another project that couldn't link with readline; here's a timeline with
all the latest fun parts at the end:

http://clisp.cvs.sourceforge.net/viewvc/clisp/clisp/doc/Why-CLISP-is-under-GPL
: Early discussion with RMS about why linking with readline requires
code using it be GPL, and why "the user did the linking" and "I wrapped
it" aren't escape routes.

http://www.gnu.org/licenses/gpl-2.0.html : The GPLv2 includes the
following in section 3: "However, as a special exception, the source
code distributed need not include anything that is normally distributed
(in either source or binary form) with the major components (compiler,
kernel, and so on) of the operating system on which the executable runs,
unless that component itself accompanies the executable." This provides
some relief to people building software on their own, but when you're
the OS packager it doesn't help because you are providing the components
and the executables.

http://lists.debian.org/debian-legal/2002/10/msg00113.html : Discussion
of the exemption made for GPL libraries shipping with an OS, and an
early mention of "Debian['s] current hardline position on the
GPL+OpenSSL licensing issue"

http://bugs.ntp.org/show_bug.cgi?id=931 : ntp runs into readline
concerns. Pull quote: "What is less clear is the claim that the FSF
makes that any program written to even *use* the readline API is also a
derived work. This hasn't been tested in court yet, so its validity is
questionable. However, that is their claim."

http://people.gnome.org/~markmc/openssl-and-the-gpl.html : Why OpenSSL
is particularly troublesome here. This describes the now common
"OpenSSL exemption" as a suggested workaround for projects who can
modify their license terms.

http://lists.debian.org/debian-legal/2004/05/msg00595.html : Example of
adding an OpenSSL exemption

http://archives.postgresql.org/pgsql-patches/2006-05/msg00040.php : A
patch adding GnuTLS support is submitted to PostgreSQL. It's rejected
mainly because the code is so large/obtrusive. TODO item "Consider
GnuTLS if OpenSSL license becomes a problem" added. [Hint: it's now
become a problem]

http://archives.postgresql.org/pgsql-hackers/2006-12/msg01213.php : More
PostgreSQL discussion that predicts a collision with Debian policy is
coming. Concerns about the quality fo the GnuTLS API relative to the
feature set provided by OpenSSL are raised too, as impediments toward
switch away from OpenSSL.

http://archives.postgresql.org/pgsql-hackers/2006-12/msg01224.php : List
of GPL applications that use libpq in Debian.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=498857 : Python runs
into the readline+OpenSSL issue

http://redmine.ruby-lang.org/issues/show/2982 : Ruby runs into the
readline+OpenSSL issue

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=601754 : PostgreSQL
runs into the readline+OpenSSL issue on Debian Lenny. Note that this
bug being open means it's possible all these problems in Squeeze are
going to get backported to Lenny and break stable server installs all
over the world one day in our near future.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=603599 : Dupe of the
bug for 9.0+Squeeze. This is the one that was "fixed" by switching to
libedit.

Then we have the stream of bugs cascading out of that decision:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=605313 : Delete key
stopped working in psql
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=607109 : Cannot input
multibyte characters in psql
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=607907 : Missing
readline features
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608442 : Input of
non-ASCII characters broken
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=611918 : psql segfaults
in libedit

So where are we at?

-GNU libreadine is certainly never going to add an OpenSSL exemption
-If the OpenSSL project was going to switch to a reasonable license,
they'd have done it years ago
-There are many known and serious bugs/limitations in libedit relative
to libreadline
-Adding GnuTLS support to PostgreSQL would require solving several code
quality issues

Idealogically, I find the worst offendor here to be the OpenSSL
license. From a license purity perspective I'd like to see their
ridiculous requirements bypassed altogether by doing whatever is
necessary to get GnuTLS support working. But pragmatically, fixing the
bugs and adding features to libedit may be the easier route here.

--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

* Greg Smith (greg@2ndquadrant.com) wrote:

-GNU libreadine is certainly never going to add an OpenSSL exemption

I really wish they would, that's just them being obnoxious- it's already
LGPL, after all..

-If the OpenSSL project was going to switch to a reasonable license,
they'd have done it years ago

aiui, the problem here is actually a former OpenSSL hacker who has no
interest (and, in fact, a positive interest against) in changing the
OpenSSL licensing. Most of the current OpenSSL hackers don't have an
issue with the change (again, aiui).

-There are many known and serious bugs/limitations in libedit
relative to libreadline

Yes, which makes it suck. :(

-Adding GnuTLS support to PostgreSQL would require solving several
code quality issues

I'm curious about this, but I don't know that I've got time to dive into
it and solve it. :/

Idealogically, I find the worst offendor here to be the OpenSSL
license. From a license purity perspective I'd like to see their
ridiculous requirements bypassed altogether by doing whatever is
necessary to get GnuTLS support working. But pragmatically, fixing
the bugs and adding features to libedit may be the easier route
here.

That suprises me.. There are a ton of tools which work with GnuTLS
today, and hearing that it's got serious issues isn't good. :/

Thanks,

Stephen

On Fri, Feb 11, 2011 at 19:13, Stephen Frost <sfrost@snowman.net> wrote:

* Greg Smith (greg@2ndquadrant.com) wrote:

-Adding GnuTLS support to PostgreSQL would require solving several
code quality issues

I'm curious about this, but I don't know that I've got time to dive into
it and solve it. :/

Yeah, I'm curious about that one as well. A lot has happened since
this was last investigated, I believe.

We may also have a problem in that libpq exposes OpenSSL structs,
though. We actually return it as a void *, to make it possible to
change, but there's no API in libpq to tell you what it is...

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

* Magnus Hagander (magnus@hagander.net) wrote:

We may also have a problem in that libpq exposes OpenSSL structs,
though. We actually return it as a void *, to make it possible to
change, but there's no API in libpq to tell you what it is...

Ugh, yeah, that probably wasn't the best decision in the world.. :(

Stephen

On Fri, 2011-02-11 at 14:59 +0100, Michael Banck wrote:

On Thu, Feb 10, 2011 at 06:04:46PM -0500, Tom Lane wrote:

Less narrow-minded interpretation of GPL requirements, perhaps.
(And yes, we have real lawyers on staff considering these issues.)

Is their opinion public/can be made public? This might possibly lead to
a re-evaluation of the situation by Debian.

I certainly hope so. Although, what I question is... Did Debian seek
legal advice? Debian does have a corporation of which I am a director
for. Software in the Public Interest. I don't recall a legal request
coming through from the DPL?

If Debian want to shoot themselves in the foot like that, we can't
stop them

BTW, that change has been merged into Ubuntu and will be (as of now) in
the next Ubuntu release.

Yeah see, that is something that raises my red-alert bells. As popular
as Debian is, the "user" population is squarely in Ubuntu world and that
has some serious public implications as a whole.

Sincerely,

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering
http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt

Stephen Frost wrote:

-Adding GnuTLS support to PostgreSQL would require solving several
code quality issues

I'm curious about this, but I don't know that I've got time to dive into
it and solve it. :/

Note that the past discussion was on the difficulty of matching the
existing OpenSSL API using GnuTLS, which is apparently difficult to do.
I wasn't trying to suggest there were issues specificially with GnuTLS's
code quality. It's more that the APIs are just different enough that
it's not trivial to do a swap--which is surprising given how many people
have seemingly needed to do exactly this conversion. You'd think
there'd be a simple "OpenSSL-like" interface available for GnuTLS by now
or something.

--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
"PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books

On Fri, Feb 11, 2011 at 20:09, Greg Smith <greg@2ndquadrant.com> wrote:

Stephen Frost wrote:

-Adding GnuTLS support to PostgreSQL would require solving several
code quality issues

I'm curious about this, but I don't know that I've got time to dive into
it and solve it. :/

Note that the past discussion was on the difficulty of matching the existing
OpenSSL API using GnuTLS, which is apparently difficult to do.  I wasn't
trying to suggest there were issues specificially with GnuTLS's code
quality.  It's more that the APIs are just different enough that it's not
trivial to do a swap--which is surprising given how many people have
seemingly needed to do exactly this conversion.  You'd think there'd be a
simple "OpenSSL-like" interface available for GnuTLS by now or something.

There is one, but it's not complete - it will work for simple users,
though, AFAIK.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Fri, Feb 11, 2011 at 2:09 PM, Greg Smith <greg@2ndquadrant.com> wrote:

Note that the past discussion was on the difficulty of matching the existing
OpenSSL API using GnuTLS, which is apparently difficult to do.

I believe that the OpenSSL API is "make some function calls, and if it
works, then you're using it right; if not, copy some source code from
the examples and use the undocumented APIs that appear there to fix
whatever problem you're having."

At least, that's been my approach.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Excerpts from Greg Smith's message of vie feb 11 14:51:17 -0300 2011:

So where are we at?

-GNU libreadine is certainly never going to add an OpenSSL exemption
-If the OpenSSL project was going to switch to a reasonable license,
they'd have done it years ago
-There are many known and serious bugs/limitations in libedit relative
to libreadline
-Adding GnuTLS support to PostgreSQL would require solving several code
quality issues

Why do we have to involve the whole of PostgreSQL? Since the only piece
that links to libreadline is psql, perhaps we could fix this by having
only psql optionally use GnuTLS. (I don't know if you can make an
OpenSSL server talk to a GnuTLS client -- are these things supposed to
be interoperable?)

--
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Fri, Feb 11, 2011 at 11:49 AM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:

Why do we have to involve the whole of PostgreSQL?  Since the only piece
that links to libreadline is psql, perhaps we could fix this by having
only psql optionally use GnuTLS.  (I don't know if you can make an
OpenSSL server talk to a GnuTLS client -- are these things supposed to
be interoperable?)

I agree with this: barring shockingly convenient engineering details,
my plan was to just evaluate the option of doing this for the psql
client.

--
fdr

Don't forget that OpenSSL has a FIPS-140 compliant version, and FIPS-140 compliance is essential to many Federal users.

GnuTLS doesn't qualify.

If psql uses libreadline and libgnutls, does that mean psql will be distributed under the GPL in the future? Or Dual-licensed?

If I read the readline license right, applications that link to it must be GPL.

That's why we (EMC/Greenplum) switch to libedit, even though readline is nicer... We didn't want to ship part of our product as GPL