Re: PG Patch (fwd) [OpenServer followup #1]

2nd followup from Kean.

LER

------------ Forwarded Message ------------
Date: Friday, July 18, 2003 23:43:55 -0700
From: Kean Johnston <jkj@sco.com>
To: Larry Rosenman <ler@lerctr.org>
Cc:
Subject: Re: PG Patch

Larry Rosenman wrote:

I got a question from the PG Core Team (Bruce Momjian) about the
rpathdir portion of
your patch.

Why can't it use libdir? Or can we wrap it in .if (port,=,sco) type
stuff?

Sorry I forgot to anwer that portion of the question. The only place that
used RPATHDIR *is* wrapped up in if port=sco. But why not use just libdir?
Well the rule for making shared libraries is shared across multiple
makefiles. Although I only set it for the main interface libraries, I had
originally set it for all the dynamically loadable modules too, and for
those, libdir isnt what you want, you want datadir or whatever its called
(I'm too lazy to go look now). So I needed variable the lower level
makefiles could specify that get used in the top level makefile.

Why do this at all? Security. Having shared libraries without full SONAME's
is a big security risk. There have been any number of huge explots based
around this. Point me at any Solaris machine <= 2.7, or any OSR5 system <
507 or any FreeBSD system <= 4.0 and I can get root with 1 tiny program
thats on all of them: xterm. It has long upset me, and I am done trying to
convince them, but libtool encourages the worst possible .so practices, and
may programs seem to have picked up those equally bad practices. There is
no need for futzing with ld.conf and the like if people take the time to
construct shared libraries propperly. Yes it can be a pain to bootstrap but
the reward is very well worth the effort it takes.

Suffice it to say that I believe that *EVERY* .so should have an absolute
SONAME. There are still a few I need to clean up in 507 but most of them
are correct. If you're not on the up-and-up with DT_RUNPATH, DT_RPATH and
SONAME ELF headers I suggest for light reading that you peruse the gABI.

Kean

---------- End Forwarded Message ----------

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

More on the shared lib stuff.

I'd LIKE to get a discussion of this (after just talking to Bruce on the
phone).

If I need to repost Kean's comments to -HACKERS, let me know.

LER

------------ Forwarded Message ------------
Date: Saturday, July 19, 2003 13:50:55 -0700
From: Kean Johnston <jkj@sco.com>
To: Larry Rosenman <ler@lerctr.org>
Cc:
Subject: Re: PG Patch

Larry Rosenman wrote:

BTW, you mentioned gAPI, what's the un-TLA'ing of that, and a WWW ref?

gABI - General Application Binary Interface.

http://www.sco.com/developers/gabi

Oh and this isnt a SCO-only thing, most UNIXes conform to it to varying
degrees.

Is the stuff you brought up the reason I always have to futz with
LD_LIBRARY_PATH for the libpq.so on UnixWare7?

Almost certainly. I havent looked at the UnixWare compile, but it too can
be fixed by using -h with a full pathname, or at least using -R correctly,
which is a workable but less preferable solution.

Kean

---------- End Forwarded Message ----------

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

------------ Forwarded Message ------------
Date: Saturday, July 19, 2003 16:10:36 -0700
From: Kean Johnston <jkj@sco.com>
To: Larry Rosenman <ler@lerctr.org>
Cc:
Subject: Re: PG Patch

I'm trying to get a discussion going, as Bruce wants to do it right for
ALL platforms or none. It probably WONT happen for 7.3.4, but WILL (If I
have my way) for 7.4.0.

Ok then let me explain the issue. You can forward this to Bruce since I
haven't heard from him yet.

As you know, the run-time link editor (RTLD) is responsible for loading an
ELF program, resolving its dependent libraries and symbols, setting things
up for _start, and calling it. There are a few ELF dynamic tags that come
into play. The ones we care about are: DT_SONAME
This is the name of the shared object that the RTLD will try to load.
DT_RPATH
Specifies the list of paths to search for dependencies (old way)
DT_RUNPATH
Specifies the list of paths to search for dependencies (new way)
DT_NEEDED
Lists the dependencies for this object

There is also one environment variable that is used at load time to resolve
dependencies, viz. LD_LIBRARY_PATH.

The gABI defines how and where these are used, but this is a basic summary.
I refer to the current object and the dependent object. The current object
is the entity which is having its dynamic section interpreted. This is the
executable or shared library that has a dependency that needs to be loaded
by the RTLD. The dependent object is the name of the actual dependency, and
comes from the DT_NEEDED list.

1) If the dependent object name contains a / use the name directly, with
no path searching.
2) Search along DT_RPATH if the current object doesnt have DT_RUNPATH
defined. DT_RPATH is a colon separated list of paths.
3) Search along LD_LIBRARY_PATH which is also a colon separated list of
paths. Only do this if the process does not have elevated (i.e setuid
or setgid) priveliges(*).
4) If the dependency still hasnt been met, search along DT_RUNPATH (if
defined for the current). DT_RUNPATH is a colon separated list of
path names.
5) If we still havent found it, look in the standard system places.
6) If we still havent resolved the dependency, bail.

(*) this is the kicker. There are *MANY* older systems out there that have
RTLD bugs that do not obey this rule. Consider the following. Most systems
have xterm. xterm is very frequently setuid root. All you need to do is run
dump -Lv on xterm to see if there are any shared libraries with no absolute
path names, or any of the dependencies of any of the libraries, and you can
get root like this. Let say, as is fairly common on older systems, that
libX11.so does not have a fully qualified path name in its DT_SONAME. When
xterm is linked, it will have a DT_NEEDED of libX11.so.5 or .6 or whatever,
without an absolute path. That means that it will use the searching
algorithms described above. All I need to do to get root is craft up
(fairly easily) a libX11.so.5 that has, in a call I know xterm will use
like XOpenDisplay, code that copies /bin/sh to somewhere and makes it
setuid root. Now I put that hacked libX11.so.5 in my home directory, set
LD_LIBRARY_PATH=$HOME, run xterm, and I've got a root shell.

This can all be so easily avoid by rule (1) above. Always hard-code your
libary names. Its a pain sometimes, to be sure, as I will describe below,
but it is completely unambiguous, its secure and it is quicker. Granted the
RTLD isnt that slow searching paths but hey, every bit counts.

Before going in to detail on the problems of using absolute path names
(there is always a catch, isn't there?), just a quick refresher on how
these various dynamic tags get set in an ELF object. This varies from
system to system but almost all system suse some subtle variation of the
following. AIX is a bit funky as I recall.

DT_RPATH is set if the link editor (ld) encounterd LD_RUN_PATH in the
environment at link edit time. Thus doing something like:
LD_RUN_PATH=/foo:/bar ld -o libfnoz.so blah.o
would set DT_RPATH to /foo:/bar.

DT_RUNPATH is set by the -R option on System-V-ish link editors, and by
-rpath with GNU-ish ones (I think, I am no GNU ld expert, please correct me
if I am wrong).

DT_SONAME is set by -h on System-V-ish link editors and by -soname on
GNU-ish ones.

DT_NEEDED is set by any ELF link editor based on the -l options or explicit
linkage against another shared object. It uses the DT_SONAME from the
dependency to put in the object's DT_NEEDED list.

While absolute path names are the way to fly, in general, they have their
drawbacks too. First, it can be a right royal pain to bootstrap things.
Consider this. You are building a program. As part of its build, it builds
a shared library, and link edits it with an absolute DT_SONAME. Later in
the build you link a program against it, and you want to use that program
in the build (perhaps executing it to produce some intermediate file or
whatever). If this is the very first time you are compiling the program and
library, then the shared library wont exist in its specified location, and
execution of the program will fail. So you have to wait for the build to
fail, then copy the just built library into its install location and
continue the build, possibly repeating this several times.

Another, sometimes more frustrating problem is encountered if you DO have
an older version of the library installed. Lets say the library was
/usr/lib/libfoo.so.2. You are recompiling your stuff, building a new
version of libfoo. It contains bug fixes, but is not sufficiently different
to warrant moving to libfoo.so.3. Now during your build you link a program
against -lfoo, and when you execute it, lo and behold, it runs, becuase
/usr/lib/libfoo.so.2 is already in place from an earlier install. But the
libfoo that the program is referencing is the buggy one, and it may make it
impossible to build your program, or may produce incorrect results.

So what we need is the ability to always reference the frehsly built
libraries while we are building the system, and to make sure that the final
installed ones ahve full path names and that executables have been
re-linked against them. This is possible, and fairly easy, but it does mean
that all programs and libraries need to be relinked at install time, and
they need to be done in the correct order. But its pretty straight-forward.

During build time, use LD_RUN_PATH or -R (or even -h with absolte path
names pointing into the build tree) and do the build. As you install, you
relink each shared library with -h and the final destination path name for
the library, making sure you relink all libraries in the correct order such
that all DT_NEEDED's have absolute path names. As you install each binary
that depends on these libraries, you also relink them before doing the
install.

libtool gets some, but not all of this right. However libtool has its own
drawbacks, not least of which is its compltely non-sensical version
numbering scheme which the docs go to great lengths to promote as an ideal
solution. Its not, its crap.

An even easier solution is one I have been thinking about a lot of late. It
would simplify the build and install procedures dramatically. I am thinking
of writing an open source tool called "somod". This will allow you to
change the ELF headers on already-installed ELF programs, adjusting the
DT_SONAME, DT_NEEDED, DT_RUNPATH and DT_RPATH variables as you see fit.
This would then simplify the build procedure by simply adding a step that
after installing a shared library or binary, you run somod on it to set
things up the way you want. That would be the least invasive way, and also
allow you to take remedial action on old programs you may not have the
source for, or even on mis-compiled or mis-produced vendor files.

Kean

---------- End Forwarded Message ----------

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

Larry Rosenman writes:

Why do this at all? Security. Having shared libraries without full SONAME's
is a big security risk. There have been any number of huge explots based
around this. Point me at any Solaris machine <= 2.7, or any OSR5 system <
507 or any FreeBSD system <= 4.0 and I can get root with 1 tiny program
thats on all of them: xterm. It has long upset me, and I am done trying to
convince them, but libtool encourages the worst possible .so practices, and
may programs seem to have picked up those equally bad practices. There is
no need for futzing with ld.conf and the like if people take the time to
construct shared libraries propperly. Yes it can be a pain to bootstrap but
the reward is very well worth the effort it takes.

These concerns might have some merit, but the solution could not possibly
be to only fix this on one platform, because the mechanisms are the same
everywhere. That said, it seems the universal practice is not to put full
sonames into shared libraries, so it seems better that our libraries
follow that practice. Otherwise it will be only a matter of time before
someone comes out of the wood and claims that libraries will full sonames
are a big whatever-else problem.

--
Peter Eisentraut peter_e@gmx.net

--On Wednesday, July 23, 2003 12:20:34 +0200 Peter Eisentraut
<peter_e@gmx.net> wrote:

Larry Rosenman writes:

Why do this at all? Security. Having shared libraries without full
SONAME's is a big security risk. There have been any number of huge
explots based around this. Point me at any Solaris machine <= 2.7, or
any OSR5 system < 507 or any FreeBSD system <= 4.0 and I can get root
with 1 tiny program thats on all of them: xterm. It has long upset me,
and I am done trying to convince them, but libtool encourages the worst
possible .so practices, and may programs seem to have picked up those
equally bad practices. There is no need for futzing with ld.conf and the
like if people take the time to construct shared libraries propperly.
Yes it can be a pain to bootstrap but the reward is very well worth the
effort it takes.

These concerns might have some merit, but the solution could not possibly
be to only fix this on one platform, because the mechanisms are the same
everywhere. That said, it seems the universal practice is not to put full
sonames into shared libraries, so it seems better that our libraries
follow that practice. Otherwise it will be only a matter of time before
someone comes out of the wood and claims that libraries will full sonames
are a big whatever-else problem.

Universal Practice does NOT equal Security and Usability.

Please consider what Kean is saying here.

Kean,
Please respond.

LER

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

To keep this on the list.

PG Core: I think Kean makes good points, and adding infrastructure to do it
with absolute pathnames in the shared libs would be a GOOD thing, and let
the
OS Specific maintainer(s) enable as their current or future practice
dictates.

LER

------------ Forwarded Message ------------
Date: Thursday, July 24, 2003 04:33:12 -0700
From: Kean Johnston <jkj@sco.com>
To: Larry Rosenman <ler@lerctr.org>
Cc: Peter Eisentraut <peter_e@gmx.net>
Subject: Re: [PATCHES] PG Patch (fwd) [openserver patch followup #2]

These concerns might have some merit, but the solution could not possibly
be to only fix this on one platform, because the mechanisms are the same

I was not trying to re-architect PostgreSQL's build system. I submitted a
patch for a specific OS that made it behave the way the vendor (us)
recommends you build things. If the PG folks dont want to accept the patch
thats really quite OK with me I will just apply it myself every time there
is a new release. I am not evangelizing for this to be a universal change,
but I thiunk that decling the OS patch becuase all othr OSes haven't done
the same thing is a wee bit harsh, but I have no emotional attachment to
this issue.

everywhere. That said, it seems the universal practice is not to put
full sonames into shared libraries, so it seems better that our libraries
follow that practice. Otherwise it will be only a matter of time before
someone comes out of the wood and claims that libraries will full sonames
are a big whatever-else problem.

I mean no offence when I say that that is an extremely weak argument. It
used to be universal practice that if you wanted a small pause in the
kernel you could do: for (spin = 0; spin < 100; spin++) ; And now
optimizers and faster CPUs and whatever make that plainly wrong. But that
aside I would also say that that position is wrong. libtool goes to some
considerable lengths to figure out how to hard-code paths into shared
libraries. It just rarely gets it right. Much of the "wisdom" about shared
libraries these days comes from folks reading libtool's info page. Most
people just dont care about the issue as long as it sorta-kinda works, so
they just accept what they read. But libtool does many many things
incorrectly, often in the name of expediency. Its not a bad program, it
just has a different design goal. But I digress. If Peter agrees in
principle that not having direct pathnames can be a problem then not at
least taking the time to investigate or analyze the impact becuase of some
potential future misunderstanding of the issue is a bit short-sighted. I
can hear the halls of Microsoft ringing with "hey lets not fix that bug,
someone in the future will complain about it if we do, or chips will get so
fast that people wont mind rebooting their OS every other mouse click" :)

Universal Practice does NOT equal Security and Usability.

How true ... just look at sendmail :)

*oops* ... was that my aloud voice? :)

Kean

---------- End Forwarded Message ----------

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

Larry Rosenman writes:

Universal Practice does NOT equal Security and Usability.

Please consider what Kean is saying here.

What Kean is saying is that your system is insecure if you have a setuid
executable that references shared libraries with nonabsolute sonames and
you have a system (an "older system") that contains a particular bug in
its run-time dynamic loader that it obeys LD_LIBRARY_PATH for setuid
executables. That is fairly common knowledge, and that's why
LD_LIBRARY_PATH is ignored for setuid executables on all properly
functioning operating systems.

If your system is broken in that particular way, upgrade your system or
don't use setuid programs at all. Those are the only sane choices. It is
not an acceptable choice to disable all valid uses of nonabsolute sonames
for all users, just because some users are running on broken systems with
obvious security flaws.

--
Peter Eisentraut peter_e@gmx.net

Finally I understand the issue, I think.

But wouldn't an ordinary user on SCO wanting to install a private copy of
Pg then have to hack the Makefiles to change/remove the abolute DT_SONAME?
If so, that seems to me to mandate that this not be in the vanilla
distribution. OS Vendors commonly make changes like this in software
versions they distribute - that's a different thing from putting it in the
standard distribution, ISTM.

The benefit Larry cites seems to me to be small - presumably his Makefiles
must include "-L /usr/local/pgsql/lib", so adding
"-R /usr/local/pgsql/lib" doesn't look like a big thing. Adding an Rpath
to executables to use libs in non-standard locations is very common,
surely?

Maybe this needs to be YACO (yet another configure option)

cheers

andrew

Larry wrote

--On Friday, July 25, 2003 09:37:04 +0200 Peter Eisentraut
<peter_e@gmx.net> wrote:

Larry Rosenman writes:

Universal Practice does NOT equal Security and Usability.

Please consider what Kean is saying here.

What Kean is saying is that your system is insecure if you have a setuid
executable that references shared libraries with nonabsolute sonames and
you have a system (an "older system") that contains a particular bug in
its run-time dynamic loader that it obeys LD_LIBRARY_PATH for setuid
executables. That is fairly common knowledge, and that's why
LD_LIBRARY_PATH is ignored for setuid executables on all properly
functioning operating systems.

If your system is broken in that particular way, upgrade your system or
don't use setuid programs at all. Those are the only sane choices. It is
not an acceptable choice to disable all valid uses of nonabsolute sonames
for all users, just because some users are running on broken systems with
obvious security flaws.

I disagree STRONGLY with what you are saying here. What harm does it do to
add the ABILITY for a port to use a ABSOLUTE DT_SONAME?

All the SYSTEM SUPPLIED .so's on UnixWare use an absolute DT_SONAME, and I
feel
that we should build libpq to supply same on UnixWare, and Kean suggests
that
the prefered, SCO recommended way on OpenServer is to do the same.

I belive that the issue is not broken systems, but broken practice.

LER

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

--On Friday, July 25, 2003 11:58:18 +0200 Peter Eisentraut
<peter_e@gmx.net> wrote:

Larry Rosenman writes:

I disagree STRONGLY with what you are saying here. What harm does it do
to add the ABILITY for a port to use a ABSOLUTE DT_SONAME?

We can discuss adding the ability, but I'm against enforcing it by
default.

I belive that the issue is not broken systems, but broken practice.

No, the issue is precisely that someone is proposing to break reasonable,
useful practice to accomodate broken systems. No one is claiming that
absolute sonames make the system more featureful or useful. In fact, it
was admitted that it would have the reverse effect. The only argument for
absolute sonames that was brought forth was that some older systems have
security holes that can be worked around in this manner.

For an example of ADDING to the usefulness, UnixWare has no ld.so.conf, or
ldconfig equivalent. For ALL my PostgreSQL apps, I either need to specify
-R/usr/local/pgsql/lib
on the EXECUTABLE build, or make sure there is a GLOBAL LD_LIBRARY_PATH
environment
variable set.

The absolute DT_SONAME will fix that issue on THIS platform, which is why
the ABILITY
of an INDIVIDUAL port to set an absolute DT_SONAME would be useful.

LER

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

Larry Rosenman writes:

I disagree STRONGLY with what you are saying here. What harm does it do to
add the ABILITY for a port to use a ABSOLUTE DT_SONAME?

We can discuss adding the ability, but I'm against enforcing it by
default.

I belive that the issue is not broken systems, but broken practice.

No, the issue is precisely that someone is proposing to break reasonable,
useful practice to accomodate broken systems. No one is claiming that
absolute sonames make the system more featureful or useful. In fact, it
was admitted that it would have the reverse effect. The only argument for
absolute sonames that was brought forth was that some older systems have
security holes that can be worked around in this manner.

--
Peter Eisentraut peter_e@gmx.net

--On Friday, July 25, 2003 03:28:55 -0500 Andrew Dunstan
<andrew@dunslane.net> wrote:

Finally I understand the issue, I think.

But wouldn't an ordinary user on SCO wanting to install a private copy of
Pg then have to hack the Makefiles to change/remove the abolute DT_SONAME?
If so, that seems to me to mandate that this not be in the vanilla
distribution. OS Vendors commonly make changes like this in software
versions they distribute - that's a different thing from putting it in the
standard distribution, ISTM.

The benefit Larry cites seems to me to be small - presumably his Makefiles
must include "-L /usr/local/pgsql/lib", so adding
"-R /usr/local/pgsql/lib" doesn't look like a big thing. Adding an Rpath
to executables to use libs in non-standard locations is very common,
surely?

except for things like PHP that create YASO (Yet Another Shared Object),
and do NOT
require the base EXECUTABLE to be recompiled, and use broken tools (e.g.
libtool) to
build the .SO.....

Maybe this needs to be YACO (yet another configure option)
the option proposed would Do the right thing with a private copy, as it
would make the
DT_SONAME for their private copy their path.

LER

]

cheers

andrew

Larry wrote

--On Friday, July 25, 2003 11:58:18 +0200 Peter Eisentraut
<peter_e@gmx.net> wrote:

Larry Rosenman writes:

I disagree STRONGLY with what you are saying here. What harm does it
do to add the ABILITY for a port to use a ABSOLUTE DT_SONAME?

We can discuss adding the ability, but I'm against enforcing it by
default.

I belive that the issue is not broken systems, but broken practice.

No, the issue is precisely that someone is proposing to break
reasonable, useful practice to accomodate broken systems. No one is
claiming that absolute sonames make the system more featureful or
useful. In fact, it was admitted that it would have the reverse
effect. The only argument for absolute sonames that was brought forth
was that some older systems have security holes that can be worked
around in this manner.

For an example of ADDING to the usefulness, UnixWare has no ld.so.conf,
or ldconfig equivalent. For ALL my PostgreSQL apps, I either need to
specify -R/usr/local/pgsql/lib
on the EXECUTABLE build, or make sure there is a GLOBAL LD_LIBRARY_PATH
environment
variable set.

The absolute DT_SONAME will fix that issue on THIS platform, which is
why the ABILITY
of an INDIVIDUAL port to set an absolute DT_SONAME would be useful.

LER

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

---------------------------(end of
broadcast)--------------------------- TIP 7: don't forget to increase
your free space map settings

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

Larry Rosenman wrote:

If your system is broken in that particular way, upgrade your system or
don't use setuid programs at all. Those are the only sane choices. It is
not an acceptable choice to disable all valid uses of nonabsolute sonames
for all users, just because some users are running on broken systems with
obvious security flaws.

I disagree STRONGLY with what you are saying here. What harm does it do to
add the ABILITY for a port to use a ABSOLUTE DT_SONAME?

All the SYSTEM SUPPLIED .so's on UnixWare use an absolute DT_SONAME, and I
feel
that we should build libpq to supply same on UnixWare, and Kean suggests
that
the prefered, SCO recommended way on OpenServer is to do the same.

I belive that the issue is not broken systems, but broken practice.

It is a broken system. Setuid shouldn't honor that environment
variable, because you are never going to be sure you catch all the
shared library file creations. Also, once we hard code it, you can't
move the library around later if you wish.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073