Should libedit be preferred to libreadline?

Nice analysis, but we can't hack configure like that. It has to be able
to be fully generated from its sources. I think the other source file
you would need to look at is config/programs.m4. (Not sure about quoting
$ac_popdir - why only that one?)

Also, I suspect we'd want to enable the libedit preference with a switch
rather than just force it, if we want to go this way.

cheers

andrew

Seneca Cunningham wrote:

It would certainly seem so on AIX.

In tracking down why postgres 8.x would segfault on AIX 5.3, it became
apparent to me that libreadline.a is a problematic library to link
against and that libedit.a is preferable (and for reasons other than
that readline is GPL while postgres is BSD-licensed).

With AIX 5, the easiest way to get a shared object is to pass "-bexpall"
to the linker. This results in all symbols being exported. The problem
with this is that the linker will export all of libreadline's and
libhistory's symbols. In the case of libreadline.so.4 (and .5) on AIX 5
this includes symbols like strncpy and memmove, but on .4, not memcpy.
This is likely because libc.a does not export them.

What results from this is that when postgres is linked against readline
on AIX, it gets these memory functions through readline instead of its
own code. When readline 4.3 is used (what IBM provides in their "AIX
Toolbox for Linux"), postgres is known to crash. These segfaults (if
postgres was compiled with gcc) have occurred on AIX 5.3ML3, AIX 5.3ML1,
and AIX 5.2ML7. With readline 5.0, postgres merely gets these functions
through the shared library memory segments instead of the user memory
segments[6].

While it is possible to build libreadline in a manner that doesn't
export strncpy, neither of the prebuilt readlines for AIX 5 that I
checked were both shared and did not export strncpy. IBM's readline[5]
exports strncpy, UCLA's readline[4] is static. Building a shared
readline that doesn't export strncpy requires creating export files for
libreadline and libhistory that only list the symbols that they are
supposed to export and editing the shared library Makefile to add the
exports flags to the appropriate linker calls.

Whatever strategy we might take, using readline on AIX requires
considerable trickery and hacking around with the build environments.
Simply put, it's ghastly.

On the other hand, the port of NetBSD's editline that I tried[1] works
without build-hackery to the library and has reasonable exports. The
only changes to postgres that I needed to make were confined to telling
the configure script to check for libedit before libreadline and adding
a test for histedit.h. The attached patch contains my modifications.

It is also possible to use a wrapper like rlwrap[2] instead of linking
postgres against libreadline or libedit.

[1] port of NetBSD's editline
http://www.thrysoee.dk/editline/
[2] rlwrap
http://utopia.knoware.nl/~hlub/uck/software/
[3] IBM Redbook "AIX 5L Porting Guide", section 9.2
http://www.redbooks.ibm.com/abstracts/sg246034.html?Open
http://www.redbooks.ibm.com/redbooks/pdfs/sg246034.pdf
[4] UCLA's readline package
http://aixpdslib.seas.ucla.edu/packages/readline.html
[5] IBM's readline package
http://www-03.ibm.com/servers/aix/products/aixos/linux/download.html
[6] IBM Redbook "Developing and Porting C and C++ Applications on AIX",
page 110
http://www.redbooks.ibm.com/abstracts/sg245674.html?Open
http://www.redbooks.ibm.com/redbooks/pdfs/sg245674.pdf

[patch snipped]

Andrew Dunstan <andrew@dunslane.net> writes:

Also, I suspect we'd want to enable the libedit preference with a switch
rather than just force it, if we want to go this way.

Quite. My recollection is that there are other platforms on which
readline works and libedit is broken. (Readline used to work just
fine even on AIX ;-))

regards, tom lane

With AIX 5, the easiest way to get a shared object is to pass

"-bexpall"

to the linker. This results in all symbols being exported.

Yes, that is another reason not to use this broken switch.
And last time I checked (AIX 4.3.3), -bexpall did not export all needed
symbols
(e.g. globals) from the backend eighter. And the counterpart -bimpall
did also not work.
Dynamic loading did not work without the .imp and .exp files :-(

Andreas

PS: I'd prefer if readline was only linked where it is needed, namely in
psql.

On Tue, Nov 22, 2005 at 10:07:15AM +0100, Zeugswetter Andreas DCP SD wrote:

PS: I'd prefer if readline was only linked where it is needed, namely in
psql.

The problem as stated is that people don't want to maintain lists of
libraries as needed by each program, so we link all of them.

Since it seems to always be the same few libraries that cause us
problems, maybe a simpler approach would be to, in the Makefile, use
filter-out to exclude libraries you *know* aren't needed. Like in the
Makefile we put:

postgres: $(OBJS)
$(CC) $(CFLAGS) $(LDFLAGS) $(export_dynamic) $^ $(filter-out -lreadline -ledit, $(LIBS)) -o $@

So instead of maintaining lists of what each binary needs, we can
maintain a few lists of what certain binaries *don't* need. Actually, I
think the backend is the only thing important enough to worry about
this, although if readline is exporting memcpy that it's quite possible
other binaries might be affected. Maybe create a BACKEND_LIBS which
contains a shorter list.

Even the GCC --as-needed flag can't save you from libs exporting
functions they shouldn't...

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

On Mon, Nov 21, 2005 at 07:50:48PM -0500, Andrew Dunstan wrote:

Nice analysis, but we can't hack configure like that. It has to be able
to be fully generated from its sources. I think the other source file
you would need to look at is config/programs.m4. (Not sure about quoting
$ac_popdir - why only that one?)

Also, I suspect we'd want to enable the libedit preference with a switch
rather than just force it, if we want to go this way.

BTW, we've run into issues with readline from a licensing standpoint. It
would be really nice if libedit was supported where practical (I suspect
most mainstream OSes support libedit) since it's BSD licensed.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Martijn van Oosterhout wrote:
-- Start of PGP signed section.

On Tue, Nov 22, 2005 at 10:07:15AM +0100, Zeugswetter Andreas DCP SD wrote:

PS: I'd prefer if readline was only linked where it is needed, namely in
psql.

The problem as stated is that people don't want to maintain lists of
libraries as needed by each program, so we link all of them.

Since it seems to always be the same few libraries that cause us
problems, maybe a simpler approach would be to, in the Makefile, use
filter-out to exclude libraries you *know* aren't needed. Like in the
Makefile we put:

postgres: $(OBJS)
$(CC) $(CFLAGS) $(LDFLAGS) $(export_dynamic) $^ $(filter-out -lreadline -ledit, $(LIBS)) -o $@

Agreed. I have no idea why this has not been done already.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Jim C. Nasby wrote:

On Mon, Nov 21, 2005 at 07:50:48PM -0500, Andrew Dunstan wrote:

Nice analysis, but we can't hack configure like that. It has to be able
to be fully generated from its sources. I think the other source file
you would need to look at is config/programs.m4. (Not sure about quoting
$ac_popdir - why only that one?)

Also, I suspect we'd want to enable the libedit preference with a switch
rather than just force it, if we want to go this way.

BTW, we've run into issues with readline from a licensing standpoint. It
would be really nice if libedit was supported where practical (I suspect
most mainstream OSes support libedit) since it's BSD licensed.

Why don't we have a libedit configure flag?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Why don't we have a libedit configure flag?

Well, I can code up a configure flag, but that doesn't mean that the
thing will compile at the end. :)

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:

Bruce Momjian wrote:

Why don't we have a libedit configure flag?

Well, I can code up a configure flag, but that doesn't mean that the
thing will compile at the end. :)

Attached is a patch which adds a flag to configure to prefer
BSD-licensed libedit:

--with-preference-bsd-libedit prefer libedit over readline

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

--with-preference-bsd-libedit prefer libedit over readline

Can't it just be --with-libedit? That seems awfully verbose,
particularly seeing that configure doesn't handle switch abbreviation.

The patch looks OK offhand, though I didn't try to test it.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

--with-preference-bsd-libedit prefer libedit over readline

Can't it just be --with-libedit? That seems awfully verbose,
particularly seeing that configure doesn't handle switch abbreviation.

The problem is that we need a clear way to say we don't want any line
editing. Right now we do it with --without-readline. Also, we already
test for libedit if we don't find readline. Would we stop doing that?
And if we do that, do we tell them they have to say --without-readline
too? And if we don't, how do we handle it? I am just confused how to
clean this up without making thing worse. I am looking for ideas.

I guess my point is do we enable looking for readline and libedit by
default, and if we do how do we specify with to test for first, and how
do we specify we want no line editing functionaliy?

The patch looks OK offhand, though I didn't try to test it.

I tested it and it does look for libedit first when defined, and matches
the patch posted by the AIX user, except it is a configure option.

Oh, one good thing is that the new configure 2.59 we are using throws an
error now for invalid user-supplied configure options, rather than
silently ignoring it like it used to.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Tom Lane wrote:

Can't it just be --with-libedit? That seems awfully verbose,
particularly seeing that configure doesn't handle switch abbreviation.

The problem is that we need a clear way to say we don't want any line
editing. Right now we do it with --without-readline. Also, we already
test for libedit if we don't find readline. Would we stop doing that?

Well, we could rename --without-readline to --without-editing, but
I think this would just break people's existing expectations without
adding much. I don't see a problem with documenting

--with-libedit prefer libedit over libreadline

and leaving the rest alone.

Oh, one good thing is that the new configure 2.59 we are using throws an
error now for invalid user-supplied configure options, rather than
silently ignoring it like it used to.

Really? I did "configure --with-bozo" and it didn't complain. It
does barf on "--bozo", but the autoconf boys have been insistent for
more than a decade that accepting --with-anything is a feature not
a bug. So I think --with-some-long-name is more user-unfriendly than
user-friendly.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Tom Lane wrote:

Can't it just be --with-libedit? That seems awfully verbose,
particularly seeing that configure doesn't handle switch abbreviation.

The problem is that we need a clear way to say we don't want any line
editing. Right now we do it with --without-readline. Also, we already
test for libedit if we don't find readline. Would we stop doing that?

Well, we could rename --without-readline to --without-editing, but
I think this would just break people's existing expectations without
adding much. I don't see a problem with documenting

--with-libedit prefer libedit over libreadline

and leaving the rest alone.

That seems confusing because you would assume the default,
--without-libedit, would not use libedit, but it does.

I trimmed it down to:

--with-bonjour build with Bonjour support
--with-openssl build with OpenSSL support
--with-prefer-libedit prefer libedit over readline
--without-readline do not use Readline
--without-zlib do not use Zlib

I did preference -> prefer and removed 'bsd'. I could name it
--with-libedit-first. Is that better?

Oh, one good thing is that the new configure 2.59 we are using throws an
error now for invalid user-supplied configure options, rather than
silently ignoring it like it used to.

Really? I did "configure --with-bozo" and it didn't complain. It
does barf on "--bozo", but the autoconf boys have been insistent for
more than a decade that accepting --with-anything is a feature not
a bug. So I think --with-some-long-name is more user-unfriendly than
user-friendly.

Oh, I see, if you do --blah, it complains, but you are right,
--with-blah doesn't complain. Boohoo.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

I trimmed it down to:
--with-prefer-libedit prefer libedit over readline

OK, I can live with that.

regards, tom lane

Bruce Momjian wrote:

I trimmed it down to:

--with-bonjour build with Bonjour support
--with-openssl build with OpenSSL support
--with-prefer-libedit prefer libedit over readline
--without-readline do not use Readline
--without-zlib do not use Zlib

I'm concerned that this still gives nondeterministic behavior. There's
no way to say, "I want readline, period" or "I want libedit, period".
I'd prefer simple --with-readline and --with-libedit, giving one turns
off the other, giving both is an error.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e@gmx.net> writes:

I'm concerned that this still gives nondeterministic behavior. There's
no way to say, "I want readline, period" or "I want libedit, period".
I'd prefer simple --with-readline and --with-libedit, giving one turns
off the other, giving both is an error.

OTOH that doesn't provide a way to express "I'll take either". Given
that I'll-take-either has so far satisfied 99.44% of users, getting rid
of it doesn't seem like the best plan.

It might be possible to set things up so that you can specify "I'll take
either" by writing both switches, and further that the order in which
you write the switches determines the preference --- though I'm not
entirely sure how to do the latter within the autoconf framework.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

I trimmed it down to:
--with-prefer-libedit prefer libedit over readline

OK, I can live with that.

I think it's ugly. Can't we just say --prefer-libedit ?

If must be a --with-foo flag, maybe --with-libedit-preferred or
--with-libedit-first would be better.

cheers

andrew

tgl@sss.pgh.pa.us (Tom Lane) writes:

Peter Eisentraut <peter_e@gmx.net> writes:

I'm concerned that this still gives nondeterministic behavior.
There's no way to say, "I want readline, period" or "I want
libedit, period". I'd prefer simple --with-readline and
--with-libedit, giving one turns off the other, giving both is an
error.

OTOH that doesn't provide a way to express "I'll take either".
Given that I'll-take-either has so far satisfied 99.44% of users,
getting rid of it doesn't seem like the best plan.

I'll bet that for well over 80% of those 99.44% (was this, by any
chance, part of the 80% in the infamous quote "80% of all statistics
quoted to prove a point are made up on the spot"??? :-)), that what
happens is that the satisfied users have taken a prepackaged copy of
PostgreSQL.

On my home installations, for instance, I'm satisfied with whatever
configuration Martin Pitt did when he built Debian packages for
PostgreSQL, and there are doubtless a lot of others being satisfied
identically.

Those that use .rpms that you manage for Red Hat, or that other
packagers manage for Mandriva, SuSE, FreeBSD Ports, and such, fall
into much the same category of "satisfaction" where a lot of the
99.44% are being satisfied by the choices of a set of on the order of
a dozen individuals that do packaging.

Those of us using packages, who are probably quite common, are a big
step indirected from this. We don't have a reason to prefer
determinism or nondeterminism in this matter; we'll get exactly one
choice, namely the choice that one or another of those ~ dozen people
make.

It might be possible to set things up so that you can specify "I'll take
either" by writing both switches, and further that the order in which
you write the switches determines the preference --- though I'm not
entirely sure how to do the latter within the autoconf framework.

I'll change hats; in my "overseeing binaries used at Afilias hat," my
vote would be with Peter, for determinism. I'm not particularly
interested in seeing psql "magically" configure itself to slightly
prefer one editing library over another; I'd be entirely happy with:

--with-readline
implying that GNU readline shall be used, and libedit shall not
--with-editline
implying that libedit shall be used, and GNU readline shall not

Supposing we were to change to this "deterministic semantic" for 8.2,
I don't see a grand problem, here. It seems likely to me that it
might confuse someone for all of 5 seconds when ./configure reports
back "Sorry, you don't have readline installed, so --with-readline
won't work!"

In contrast, the nondeterministic approach requires having extra knobs
to fiddle in order to prefer one thing to another. I'm not sure but
that "configure hints" are as unattractive as "optimizer hints" :-).

To my mind, giving BIG weight to the opinions of the relatively small
set of individuals that manage PostgreSQL packages for the popular
distributions of Linux and *BSD seems fairly appropriate.
--
let name="cbbrowne" and tld="ntlug.org" in String.concat "@" [name;tld];;
http://cbbrowne.com/info/advocacy.html
Rules of the Evil Overlord #25. "No matter how well it would perform,
I will never construct any sort of machinery which is completely
indestructible except for one small and virtually inaccessible
vulnerable spot." <http://www.eviloverlord.com/&gt;

Chris Browne <cbbrowne@acm.org> writes:

To my mind, giving BIG weight to the opinions of the relatively small
set of individuals that manage PostgreSQL packages for the popular
distributions of Linux and *BSD seems fairly appropriate.

The packagers are bright enough to adapt to whatever we do --- it's
the people who build their own from source that I'm worried about.
--with-readline has worked fine for libedit users for a long time,
and suddenly changing its semantics strikes me as a bad idea.

The other problem with the "let's be deterministic" argument is that
it rests on a fallacy, which is that configure can reliably tell the
difference between libreadline and libedit. Darwin, for example, goes
to some lengths to confuse matters.

(I think I'd actually be for the determinism point of view if it could
provide an #ifdef flag saying which library is in use --- then we could
fix the write_history return value problem we're seeing on Darwin ---
but I don't think we can do it short of a behavioral probe during
configure.)

regards, tom lane