Relocatable installs

Bruce Momjian <pgman@candle.pha.pa.us> writes:

My idea is to write a /port function that uses various methods to find
the needed files. We could look in the relative location first, and if
the needed file is not found, look in the hardcoded directory.

I think a "search until you find something" approach would be a really
bad idea. Particularly on a machine with multiple PG versions installed
(and that has surely got to be a likely situation for people who are
wanting to move things around). It seems entirely too likely that you
would find the wrong version of some file.

So ISTM that the location in which a given installation looks for its
associated files should be completely predictable and *not* depend on
whether it finds something there.

I'm fine with offering an option to make that location be relative to
where the executable came from. But not with nondeterminism.

I think we should use the relative-path method *unless* the configure
command called out specific installation directories (that is, not
just --prefix but --datadir and/or related options). If you use one of
those then that absolute path should be used always, ie, you are
specifically asking for a nonrelocatable install and that's what you
should get.

regards, tom lane

On Sat, 8 May 2004, Bruce Momjian wrote:

Win32 is going to need relocatable installs, and Unix packagers have
asked for this too.

A relocatable install is one where you can do 'gmake install', tar up
the directory where you installed it, then untar it on to another
machine with the same operating system, but into a different directory
location.

For example, if you use the defaults for directory locations, PostgreSQL
will install into /usr/local/pgsql, and everything will be under that
directory --- bin, lib, include, share. (Not sure about etc, doc, and
man.)

However, right now, if you try to move /usr/local/pgsql to /var/pgsql,
the database will not work because there are hard-coded directory
dependencies in the binaries:

initdb has to find its *.bki files in /share
initdb has to find the postgres binary in /bin
pg_dumpall has to find pg_dump
postgres has to find shared objects the /lib
Win32 postgres has to find /share/timezone

Maybe there are more.

Is there a reason why we can't use a PGSQL_PREFIX ENV variable or
something like that? If not defined, use compiled in default?

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

Marc G. Fournier wrote:

However, right now, if you try to move /usr/local/pgsql to /var/pgsql,
the database will not work because there are hard-coded directory
dependencies in the binaries:

initdb has to find its *.bki files in /share
initdb has to find the postgres binary in /bin
pg_dumpall has to find pg_dump
postgres has to find shared objects the /lib
Win32 postgres has to find /share/timezone

Maybe there are more.

Is there a reason why we can't use a PGSQL_PREFIX ENV variable or
something like that? If not defined, use compiled in default?

Win32 isn't going to be able to define that. Also, why not make it
relocatable by default? We already know where the binary is.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

My idea is to write a /port function that uses various methods to find
the needed files. We could look in the relative location first, and if
the needed file is not found, look in the hardcoded directory.

I think a "search until you find something" approach would be a really
bad idea. Particularly on a machine with multiple PG versions installed
(and that has surely got to be a likely situation for people who are
wanting to move things around). It seems entirely too likely that you
would find the wrong version of some file.

So ISTM that the location in which a given installation looks for its
associated files should be completely predictable and *not* depend on
whether it finds something there.

I agree.

I'm fine with offering an option to make that location be relative to
where the executable came from. But not with nondeterminism.

I think we should use the relative-path method *unless* the configure
command called out specific installation directories (that is, not
just --prefix but --datadir and/or related options). If you use one of
those then that absolute path should be used always, ie, you are
specifically asking for a nonrelocatable install and that's what you
should get.

I think we are making this way too complicated in a quest for
flexibility that is of dubious value.

I think we could adopt a simple rule: if you configure it for relocation
(and I think you should have to do that explicitly) then all paths are
relative to the binary location. If not, then full hardcoded paths are
used. No exceptions.

Most people won't need this at all, I suspect - people who make binary
packages/installers for redistribution will find it a great boon.

cheers

andrew

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

My idea is to write a /port function that uses various methods to find
the needed files. We could look in the relative location first, and if
the needed file is not found, look in the hardcoded directory.

I think a "search until you find something" approach would be a really
bad idea. Particularly on a machine with multiple PG versions installed
(and that has surely got to be a likely situation for people who are
wanting to move things around). It seems entirely too likely that you
would find the wrong version of some file.

As I see it, we do two searches. One is for the path to our own binary,
and the second is a search for another binary or the /lib or /share
directories.

In initdb and postgres, we do searches by breaking up the $PATH value,
and testing for versions. In pg_dump, we just run system() (uses
default PATH), and then try hardcoded path if that fails, with no
version checking. This should all be consistent at least.

Then, once we find our own path, the next question is how to handle
looking for lib and share, and whether we look for a directory name, or
specific file, and can we check versions? Right now we do none of
these, I think. We just go for hard-coded.

So ISTM that the location in which a given installation looks for its
associated files should be completely predictable and *not* depend on
whether it finds something there.

Ah, got it. Makes sense. We do searches for finding our own path, but
you can argue that this predictable --- we know we are running so the
binary must be somewhere. :-) However, the version checking we do now
is a little non-predictable because if we find a binary of the wrong
version, we keep looking in the path.

I'm fine with offering an option to make that location be relative to
where the executable came from. But not with nondeterminism.

Ah, makes sense.

I think we should use the relative-path method *unless* the configure
command called out specific installation directories (that is, not
just --prefix but --datadir and/or related options). If you use one of
those then that absolute path should be used always, ie, you are
specifically asking for a nonrelocatable install and that's what you
should get.

That would make my job very easy. If they specify lib or share with a
unique path, it is hard to imagine how they would tar up the install
anyway because it would be usually mixed into their operating system
directories. Are others OK with this? You can still specify --prefix,
but if you override the other default directories, the program will use
those hard-coded ones rather than ones relative to your /bin.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Marc G. Fournier wrote:

Is there a reason why we can't use a PGSQL_PREFIX ENV variable or
something like that? If not defined, use compiled in default?

Win32 isn't going to be able to define that.

And this idea is even worse than the other one as far as destroying
the predictability of the search goes :-(.

ISTM that just about every place we have made the backend's behavior
dependent on environment variables, we have had cause to regret it.
It's too easy to screw up by launching the postmaster under a different
environment than you normally do.

regards, tom lane

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Ah, got it. Makes sense. We do searches for finding our own path, but
you can argue that this predictable --- we know we are running so the
binary must be somewhere. :-) However, the version checking we do now
is a little non-predictable because if we find a binary of the wrong
version, we keep looking in the path.

Hm? I see no version checks in FindExec. It looks to me like the code
is just trying to ensure that it finds the same file that the shell
would have found.

regards, tom lane

Andrew Dunstan wrote:

I think we should use the relative-path method *unless* the configure
command called out specific installation directories (that is, not
just --prefix but --datadir and/or related options). If you use one of
those then that absolute path should be used always, ie, you are
specifically asking for a nonrelocatable install and that's what you
should get.

I think we are making this way too complicated in a quest for
flexibility that is of dubious value.

I think we could adopt a simple rule: if you configure it for relocation
(and I think you should have to do that explicitly) then all paths are
relative to the binary location. If not, then full hardcoded paths are
used. No exceptions.

Most people won't need this at all, I suspect - people who make binary
packages/installers for redistribution will find it a great boon.

I think if we go for the plan outlined, we will not need a special
configure flag. (People might decide to move the install dir long after
they install it.) By default, everything sits under pgsql as pgsql/bin,
pgsql/lib, etc. I can't see how making it relative is going to bite us
unless folks move the binaries out of pgsql/bin. Is that common for
installs that don't specify a special bindir?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Andrew Dunstan <andrew@dunslane.net> writes:

Tom Lane wrote:

I think we should use the relative-path method *unless* the configure
command called out specific installation directories (that is, not
just --prefix but --datadir and/or related options).

I think we could adopt a simple rule: if you configure it for relocation
(and I think you should have to do that explicitly) then all paths are
relative to the binary location. If not, then full hardcoded paths are
used. No exceptions.

I think we're saying the same thing except for the question of whether
relative-path behavior has to be explicitly requested at configure time.

While I'm not dead set on it, I'm leaning to the idea that it's okay to
make relative-path the standard behavior. I cannot see any real serious
downsides to it. We have always bombed out if we are unable to locate
the executable, so it's not like that code isn't well-tested.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Ah, got it. Makes sense. We do searches for finding our own path, but
you can argue that this predictable --- we know we are running so the
binary must be somewhere. :-) However, the version checking we do now
is a little non-predictable because if we find a binary of the wrong
version, we keep looking in the path.

Hm? I see no version checks in FindExec. It looks to me like the code
is just trying to ensure that it finds the same file that the shell
would have found.

I thought it was in ValidateBinary, but now I don't see it. I must have
gotten it confused by pg_dumpall. And I see now that initdb only checks
the version and exits if it is wrong. It does not keep trying the path.

OK, we are better than I thought. :-)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Comments?

What's wrong with the way it's done by everybody else?

Have hardcoded paths (determined at configure time), and allow override
using a config file. Have a command line option for saying where the
config file should be.

For Windows, replace config file with "Registry". That is usually
hardcoded for (depending on whether you want it changeable per-user)
HKEY_LOCAL_MACHINE\Software\<company name>\<product name> (replace HKLM
with HKEY_LOCAL_USER if you want per-user config).

Shachar

--
Shachar Shemesh
Lingnu Open Source Consulting
http://www.lingnu.com/

Shachar Shemesh wrote:

Bruce Momjian wrote:

Comments?

What's wrong with the way it's done by everybody else?

Have hardcoded paths (determined at configure time), and allow override
using a config file. Have a command line option for saying where the
config file should be.

Where do we put the config file so it can be found? We can't assume
root privilege to write into /etc.

For Windows, replace config file with "Registry". That is usually
hardcoded for (depending on whether you want it changeable per-user)
HKEY_LOCAL_MACHINE\Software\<company name>\<product name> (replace HKLM
with HKEY_LOCAL_USER if you want per-user config).

Doesn't registry require admin privilege on the Windows box?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Shachar Shemesh wrote:

Bruce Momjian wrote:

Comments?

What's wrong with the way it's done by everybody else?

Have hardcoded paths (determined at configure time), and allow override
using a config file. Have a command line option for saying where the
config file should be.

Where do we put the config file so it can be found? We can't assume
root privilege to write into /etc.

Put it in /etc (or wherever else user said in configure). Allow a
command line parameter that says "run the server with this config file
instead". User can place override config file wherever she damn wishes.

For Windows, replace config file with "Registry". That is usually
hardcoded for (depending on whether you want it changeable per-user)
HKEY_LOCAL_MACHINE\Software\<company name>\<product name> (replace HKLM
with HKEY_LOCAL_USER if you want per-user config).

Doesn't registry require admin privilege on the Windows box?

HKLM does. HKCU doesn't. Actually, HKLM often doesn't either, but that's
a rant I won't start right now.

Again, you can use command-line arguments to override, if you want.
Alternatively, Postgres can look at HKCU, and if not found there, at
HKLM. This means that there can be a system-wide default, overrideable
per user by each user. The only reason I'm suggesting this (as it is
unlikely that for production installs more than one user will be running
postgres) is that editing user keys for other users is a task I'm not
sure many admins know how to do. It requires the use of the
not-well-known "regedt32" rather than "regedit", and involves loading
the correct file into the registry "hive". As such, placing stuff in
HKLM allows admins to easilly edit it. If that is not an issue, I'd say
"place it at HKCU".

Shachar

--
Shachar Shemesh
Lingnu Open Source Consulting
http://www.lingnu.com/

On Sun, 9 May 2004, Bruce Momjian wrote:

Shachar Shemesh wrote:

Bruce Momjian wrote:

Comments?

What's wrong with the way it's done by everybody else?

Have hardcoded paths (determined at configure time), and allow override
using a config file. Have a command line option for saying where the
config file should be.

Where do we put the config file so it can be found? We can't assume
root privilege to write into /etc.

The default location for the config file would be compiled into the
binary based on whatever settings were given when it was built.

If someone wants to use a different location after it is compiled,
they have to specify the config file path on the command line. For
starting up the database, that isn't a big deal because you normally
should be using some wrapper script anyway.

This is how the vast majority of other programs do it on unix.
Apache, mysql, BIND, the list goes on.

The downside to this is that it isn't as friendly for various
command line tools that people run, since they then have to specify
the path on the command line. One option to deal with this is to
allow compiling in a default path that is optionally relative to
the directory the binary is in, eg. "../foo/bar" would be relative
to where the binary is, while "/foo/bar" would be absolute. Then
there is a choice at build time; it could default to paths relative
to the binary and allow overriding that if the user compiling it wants
something different.

One key thing that this gives that the other proposals don't is the
ability to have a single, static tree of program files and run multiple
different instances of the database off them, each having their own
unique configuration and data trees. This also makes it easier to move
back and forth between different minor versions for testing or QA
just by tweaking a startup script to point to a different binary
tree, without having to copy data, config, or binaries around. In
this scenario, the above downside can be worked around by just
creating simple wrapper scripts.

We use this type of setup with products such as Apache to manage
having code for different projects or different versions of products
running out of different development or QA or production trees that
uses different versions of Apache, to ensure we are developing and
testing against the same version that will be in production, and
that we can manage upgrades on a project by project basis.

So, for example you could have:

/site/httpd/httpd-2.0.45
/site/httpd/httpd-2.0.45-2
/site/httpd/httpd-2.0.48

containing static copies of the binaries and libraries that are never
ever modified. All the config files and data live in specific projects,
then we just configure that project to use a specific version of Apache.
Multiple versions can live on a machine and each project can pick which
one it wants to use and easily upgrade to new ones, without having a
whole bunch of copies of the binaries.

While this is slightly less useful for postgresql, both because fewer
people want to be able to concurrenetly run multiple different instances
on the same box for testing purposes and because the database generally
has to be modified to work between non-minor version changes, it is
still something that is very useful in some environments and I would
love to have.

For Windows, replace config file with "Registry". That is usually
hardcoded for (depending on whether you want it changeable per-user)
HKEY_LOCAL_MACHINE\Software\<company name>\<product name> (replace HKLM
with HKEY_LOCAL_USER if you want per-user config).

Doesn't registry require admin privilege on the Windows box?

Not if it is in the per user part as Shachar suggested.

If it is done this way, however, it should be possible to override
the registry via command line flags to allow multiple installations
on the same box in different places.

It's rumoured that Marc Slemko once said:

For Windows, replace config file with "Registry". That is usually
hardcoded for (depending on whether you want it changeable per-user)
HKEY_LOCAL_MACHINE\Software\<company name>\<product name> (replace
HKLM with HKEY_LOCAL_USER if you want per-user config).

Doesn't registry require admin privilege on the Windows box?

Not if it is in the per user part as Shachar suggested.

Which is a problem in it's own right anyway. To properly write events to
the event log we need to register PostgreSQL as an event source and
specify a message DLL to use. As Magnus & I discussed the other day, in
our case we probably just want a single message of %s in the dll to allow
PostgreSQL to log anything it wants cleanly.
Anyway, the point is that to do that you need to write to
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\EventLog\<Appname>
(off the top of my head) otherwise, the messages logged in the event log
are pretty unreadable. Service installation will also require admin
privileges.
Regards, Dave

Dave Page wrote:

Anyway, the point is that to do that you need to write to
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\EventLog\<Appname>
(off the top of my head) otherwise, the messages logged in the event log
are pretty unreadable. Service installation will also require admin
privileges.
Regards, Dave

I don't think that's a problem at all:
1. Believe it or not, you can write to HKEY_CURRENT_USER even if you are
an admin! New and improved..... In fact, if you don't mind messing with
the "load hive" interfaces, you can even write to someone else's HKCU if
you are admin. It makes installs more complicated, but see 2 for why
this will not be necessary.
2. Original suggestion talked about looking up at HKCU, and then (if not
found) at HKEY_LOCAL_MACHINE
3. I doubt the same machine will require BOTH service and per-user
installation

Shachar

--
Shachar Shemesh
Lingnu Open Source Consulting
http://www.lingnu.com/

-----Original Message-----
From: Shachar Shemesh [mailto:psql@shemesh.biz]
Sent: 09 May 2004 10:20
To: Dave Page
Cc: marcs@znep.com; pgman@candle.pha.pa.us;
pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Relocatable installs

I don't think that's a problem at all:
1. Believe it or not, you can write to HKEY_CURRENT_USER even
if you are an admin! New and improved..... In fact, if you
don't mind messing with the "load hive" interfaces, you can
even write to someone else's HKCU if you are admin. It makes
installs more complicated, but see 2 for why this will not be
necessary.

I think you're missing what I was saying. To properly write to the event
log without getting messages like:

The description for Event ID ( 0 ) in Source ( btwdins ) cannot be
found. The local computer may not have the necessary registry
information or message DLL files to display messages from a remote
computer. You may be able to use the /AUXSOURCE= flag to retrieve this
description; see Help and Support for details. The following information
is part of the event: Service started.

you need to register an event source with a message dll under
HKEY_LOCAL_MACHINE. HKCU is not an option. This is not part of the
relocatable install issue I grant you - the mention of HKLM permissions
just made me think of it...

2. Original suggestion talked about looking up at HKCU, and
then (if not
found) at HKEY_LOCAL_MACHINE

Yes, that's fine for the relocatable install issues.

3. I doubt the same machine will require BOTH service and
per-user installation

No, but either may want to use the event log.

Regards, Dave.

Shachar Shemesh said:

Bruce Momjian wrote:

Comments?

What's wrong with the way it's done by everybody else?

Have hardcoded paths (determined at configure time), and allow override
using a config file. Have a command line option for saying where the
config file should be.

For Windows, replace config file with "Registry". That is usually
hardcoded for (depending on whether you want it changeable per-user)
HKEY_LOCAL_MACHINE\Software\<company name>\<product name> (replace HKLM
with HKEY_LOCAL_USER if you want per-user config).

IMNSHO this would not buy anything but would provide you with an extra
capacity for shooting yourself in the foot. Using what we have discussed
up to now this would be completely unnecessary - you could just move the
installation and it would work (modulo any necessary changes in
LD_LIBRARY_PATH ,PATH etc).

Perhaps you have not followed the discussion about this issue over the
last months, but a good part of it revolved around avoiding the very
dependencies you are suggesting.

"What everybody else does" doesn't carry much weight on its own with me.

cheers

andrew

Marc Slemko <marcs@znep.com> writes:

The downside to this is that it isn't as friendly for various
command line tools that people run, since they then have to specify
the path on the command line.

Exactly. Not only unfriendly, but quite error-prone, especially in
a multiple-install situation.

The fact that "everyone else does it this way" doesn't automatically
mean we should too. The fact of the matter is that most of those other
programs are not really designed to support multiple installs
conveniently. We've developed conventions that let us handle that,
and I don't wish to backslide.

The thing I like about the relative-path idea is that it actually
improves and extends our existing ability to support multiple
installs. I wonder whether we could even allow PGDATA to default
to a relative path (../data)?

regards, tom lane

Tom Lane wrote:

The thing I like about the relative-path idea is that it actually
improves and extends our existing ability to support multiple
installs. I wonder whether we could even allow PGDATA to default
to a relative path (../data)?

Doesn't work very well when you install into /usr/bin ...

If we determine the default data directory off the configure option
--localstatedir then we can simply use the same mechanisms that have
been discussed for determining all the other directories at run time
relative to the binaries.

Peter Eisentraut <peter_e@gmx.net> writes:

If we determine the default data directory off the configure option
--localstatedir then we can simply use the same mechanisms that have
been discussed for determining all the other directories at run time
relative to the binaries.

Agreed, we could let the default location of PGDATA work that way.
We still need the -D runtime option, since (unlike libdir and sharedir)
there could be multiple datadirs being run by a single set of install
files.

regards, tom lane

Bruce Momjian wrote:

Andrew Dunstan wrote:

I think we should use the relative-path method *unless* the configure
command called out specific installation directories (that is, not
just --prefix but --datadir and/or related options). If you use one of
those then that absolute path should be used always, ie, you are
specifically asking for a nonrelocatable install and that's what you
should get.

I think we are making this way too complicated in a quest for
flexibility that is of dubious value.

I think we could adopt a simple rule: if you configure it for relocation
(and I think you should have to do that explicitly) then all paths are
relative to the binary location. If not, then full hardcoded paths are
used. No exceptions.

Most people won't need this at all, I suspect - people who make binary
packages/installers for redistribution will find it a great boon.

I think if we go for the plan outlined, we will not need a special
configure flag. (People might decide to move the install dir long after
they install it.) By default, everything sits under pgsql as pgsql/bin,
pgsql/lib, etc. I can't see how making it relative is going to bite us
unless folks move the binaries out of pgsql/bin. Is that common for
installs that don't specify a special bindir?

Does that include a mechanism for -rpath?

Currently, if you have multiple installations of PostgreSQL on a server
and call ones psql or whatever explicitly, it is not loading another
ones libpq, but for sure the one belonging to its version. How does the
plan you're talking about cover this?

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Jan Wieck wrote:

I think if we go for the plan outlined, we will not need a special
configure flag. (People might decide to move the install dir long after
they install it.) By default, everything sits under pgsql as pgsql/bin,
pgsql/lib, etc. I can't see how making it relative is going to bite us
unless folks move the binaries out of pgsql/bin. Is that common for
installs that don't specify a special bindir?

Does that include a mechanism for -rpath?

Currently, if you have multiple installations of PostgreSQL on a server
and call ones psql or whatever explicitly, it is not loading another
ones libpq, but for sure the one belonging to its version. How does the
plan you're talking about cover this?

Someone asked about rpath, and I didn't deal with it. How many
platforms use rpath? I am not sure.

I assume folks are going to have to modify their ld.so.conf to point to
the proper library, or for non-root, set an environment variable like
LD_LIBRARY_PATH.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Jan Wieck wrote:

I think if we go for the plan outlined, we will not need a special
configure flag. (People might decide to move the install dir long after
they install it.) By default, everything sits under pgsql as pgsql/bin,
pgsql/lib, etc. I can't see how making it relative is going to bite us
unless folks move the binaries out of pgsql/bin. Is that common for
installs that don't specify a special bindir?

Does that include a mechanism for -rpath?

Currently, if you have multiple installations of PostgreSQL on a server
and call ones psql or whatever explicitly, it is not loading another
ones libpq, but for sure the one belonging to its version. How does the
plan you're talking about cover this?

Someone asked about rpath, and I didn't deal with it. How many
platforms use rpath? I am not sure.

I assume folks are going to have to modify their ld.so.conf to point to
the proper library, or for non-root, set an environment variable like
LD_LIBRARY_PATH.

You know how much trouble that causes? The existance of LD_LIBRARY_PATH
in your environment disables setuid() for security reasons on some
platforms. So one would have to wrap every PG related program into
equally named shell scripts or aliases to just set it for the program
call alone.

Relocatable installation means static linking of our tools against our
own libs. This does not mean static linking entirely, but at least
static linking against libpq.a.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Jan Wieck <JanWieck@Yahoo.com> writes:

You know how much trouble that causes? The existance of LD_LIBRARY_PATH in your
environment disables setuid() for security reasons on some platforms. So one
would have to wrap every PG related program into equally named shell scripts or
aliases to just set it for the program call alone.

Why would any pg tools need to be setuid?

But it's just wrong to have a binary that doesn't run unless you have
environment variables set up.

Relocatable installation means static linking of our tools against our own
libs. This does not mean static linking entirely, but at least static linking
against libpq.a.

The normal approach is to use rpath for relocatable installs.
That's what it's there for.
Static linking would work too but it's overkill.

But please don't use rpath for installs to standard paths like
/usr/{local,}/lib. That way lies madness.

--
greg

Jan Wieck wrote:

Bruce Momjian wrote:

Jan Wieck wrote:

I think if we go for the plan outlined, we will not need a special
configure flag. (People might decide to move the install dir long after
they install it.) By default, everything sits under pgsql as pgsql/bin,
pgsql/lib, etc. I can't see how making it relative is going to bite us
unless folks move the binaries out of pgsql/bin. Is that common for
installs that don't specify a special bindir?

Does that include a mechanism for -rpath?

Currently, if you have multiple installations of PostgreSQL on a server
and call ones psql or whatever explicitly, it is not loading another
ones libpq, but for sure the one belonging to its version. How does the
plan you're talking about cover this?

Someone asked about rpath, and I didn't deal with it. How many
platforms use rpath? I am not sure.

I assume folks are going to have to modify their ld.so.conf to point to
the proper library, or for non-root, set an environment variable like
LD_LIBRARY_PATH.

You know how much trouble that causes? The existence of LD_LIBRARY_PATH

Nope.

in your environment disables setuid() for security reasons on some
platforms. So one would have to wrap every PG related program into
equally named shell scripts or aliases to just set it for the program
call alone.

OK.

Relocatable installation means static linking of our tools against our
own libs. This does not mean static linking entirely, but at least
static linking against libpq.a.

Static linking of our binaries? Hmmm. Makes sense. We would need a
special flag for that. I can add it to the TODO.

Seems my testing was flawed because I didn't clean out my hard-coded
directory properly. I see now:

$ bin/initdb
bin/initdb: can't load library 'libpq.so.3'

and I see in my initdb link line:

-Wl,-rpath,/usr/local/pgsql/lib

So, right now, you can do relocatable installs, but you have to make
changes in ld.so.conf or your environment to allow it to find the shared
libraries. In the future, we can add a configure flag so everything is
linked statically.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Greg Stark wrote:

Jan Wieck <JanWieck@Yahoo.com> writes:

You know how much trouble that causes? The existance of LD_LIBRARY_PATH in your
environment disables setuid() for security reasons on some platforms. So one
would have to wrap every PG related program into equally named shell scripts or
aliases to just set it for the program call alone.

Why would any pg tools need to be setuid?

I assume he is saying that you would have problems having that set all
the time. You would need to set it only before running pg binaries,
which is a pain.

Relocatable installation means static linking of our tools against our own
libs. This does not mean static linking entirely, but at least static linking
against libpq.a.

The normal approach is to use rpath for relocatable installs.
That's what it's there for.
Static linking would work too but it's overkill.

The point is that we want someone to be able to do an install, and then
be able to move the pgsql directory. When we use rpath, the installed
binary has a hardcoded path for the libraries.

But please don't use rpath for installs to standard paths like
/usr/{local,}/lib. That way lies madness.

We don't. By default it is /usr/local/pgsql/lib, which isn't standard.

I see static linking as perhaps the only clean solution for relocatable
installs that require no user manipulation.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Greg Stark wrote:

Jan Wieck <JanWieck@Yahoo.com> writes:

You know how much trouble that causes? The existance of LD_LIBRARY_PATH in your
environment disables setuid() for security reasons on some platforms. So one
would have to wrap every PG related program into equally named shell scripts or
aliases to just set it for the program call alone.

Why would any pg tools need to be setuid?

That's got nothing to do with what I said. Setting LD_LIBRARY_PATH will
possibly affect everything else that needs setuid.

But it's just wrong to have a binary that doesn't run unless you have
environment variables set up.

Relocatable installation means static linking of our tools against our own
libs. This does not mean static linking entirely, but at least static linking
against libpq.a.

The normal approach is to use rpath for relocatable installs.
That's what it's there for.
Static linking would work too but it's overkill.

I don't mean complete static linking. But what's wrong with linking psql
and other PostgreSQL utilities against libpq.a instead of libpq.so?

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Bruce Momjian wrote:

Jan Wieck wrote:

Bruce Momjian wrote:

Jan Wieck wrote:

I think if we go for the plan outlined, we will not need a special
configure flag. (People might decide to move the install dir long after
they install it.) By default, everything sits under pgsql as pgsql/bin,
pgsql/lib, etc. I can't see how making it relative is going to bite us
unless folks move the binaries out of pgsql/bin. Is that common for
installs that don't specify a special bindir?

Does that include a mechanism for -rpath?

Currently, if you have multiple installations of PostgreSQL on a server
and call ones psql or whatever explicitly, it is not loading another
ones libpq, but for sure the one belonging to its version. How does the
plan you're talking about cover this?

Someone asked about rpath, and I didn't deal with it. How many
platforms use rpath? I am not sure.

I assume folks are going to have to modify their ld.so.conf to point to
the proper library, or for non-root, set an environment variable like
LD_LIBRARY_PATH.

You know how much trouble that causes? The existence of LD_LIBRARY_PATH

Nope.

in your environment disables setuid() for security reasons on some
platforms. So one would have to wrap every PG related program into
equally named shell scripts or aliases to just set it for the program
call alone.

OK.

Relocatable installation means static linking of our tools against our
own libs. This does not mean static linking entirely, but at least
static linking against libpq.a.

Static linking of our binaries? Hmmm. Makes sense. We would need a
special flag for that. I can add it to the TODO.

Seems my testing was flawed because I didn't clean out my hard-coded
directory properly. I see now:

$ bin/initdb
bin/initdb: can't load library 'libpq.so.3'

and I see in my initdb link line:

-Wl,-rpath,/usr/local/pgsql/lib

If I remore the whole -rpath thing, and remove the two -L options and
the -lpq and -lpgport, and add the libpq.a and libpgport.a explicitly to
the linker call, the psql executable on my Linux box grows from 421761
to 677682 bytes in size. It is still shared linked against libc, libz,
libreadline and a bunch of otheres, but all of them are in /lib or
/usr/lib, so they are standard or system libraries. It does not depend
on a libpq.so any more, and that is what we want.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Jan Wieck wrote:

Static linking of our binaries? Hmmm. Makes sense. We would need a
special flag for that. I can add it to the TODO.

Seems my testing was flawed because I didn't clean out my hard-coded
directory properly. I see now:

$ bin/initdb
bin/initdb: can't load library 'libpq.so.3'

and I see in my initdb link line:

-Wl,-rpath,/usr/local/pgsql/lib

If I remore the whole -rpath thing, and remove the two -L options and
the -lpq and -lpgport, and add the libpq.a and libpgport.a explicitly to
the linker call, the psql executable on my Linux box grows from 421761
to 677682 bytes in size. It is still shared linked against libc, libz,
libreadline and a bunch of otheres, but all of them are in /lib or
/usr/lib, so they are standard or system libraries. It does not depend
on a libpq.so any more, and that is what we want.

We already have --disable-rpath. Seems we would just need something to
use the *.a files.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Greg Stark wrote:

Jan Wieck <JanWieck@Yahoo.com> writes:

You know how much trouble that causes? The existance of LD_LIBRARY_PATH in your
environment disables setuid() for security reasons on some platforms. So one
would have to wrap every PG related program into equally named shell scripts or
aliases to just set it for the program call alone.

Why would any pg tools need to be setuid?

I assume he is saying that you would have problems having that set all
the time. You would need to set it only before running pg binaries,
which is a pain.

I've never heard of any system where that's the case. Normally the loader
simply ignores the LD_LIBRARY_PATH LD_PRELOAD etc. for setuid/setgid programs.

I feel weird here giving ammunition that LD_LIBRARY_PATH is happy while
simultaneously arguing it shouldn't be used though.

Relocatable installation means static linking of our tools against our own
libs. This does not mean static linking entirely, but at least static linking
against libpq.a.

The normal approach is to use rpath for relocatable installs.
That's what it's there for.
Static linking would work too but it's overkill.

The point is that we want someone to be able to do an install, and then
be able to move the pgsql directory. When we use rpath, the installed
binary has a hardcoded path for the libraries.

You do? That doesn't generally work. To do that you have to break all the unix
conventions about config files in /etc, platform independent files in /share,
modified data in /var, etc. I know postgres currently violates these
conventions, but most unix programs don't so anyone who has such an
expectation is in for a lot of surprises.

And long term I expect postgres to head in the direction of complying with
these conventions so distributions can stop kludging around the violations
with symlinks and wrapper scripts. Writing in more assumptions about the
non-standard installation will only create more pain down the road.

Generally I've only heard "relocatable" to refer to RPM packages that could be
installed to a non-standard prefix. For that you need an install script that
knows what magic has to be done. Not a directory that can be moved around
willy-nilly *after* installation.

But please don't use rpath for installs to standard paths like
/usr/{local,}/lib. That way lies madness.

We don't. By default it is /usr/local/pgsql/lib, which isn't standard.

Well that sucks, but that's for another day :)

As long as it is in a non-standard location then rpath really ought to be set.
Requiring every user of a program to set environment variables before it works
properly is lame. It engenders the kde/qt/oracle headaches of having to adjust
LD_LIBRARY_PATH for every user every time you install or update a package.

Of course rpath is evil, which is precisely why packages ought not be
installed in such non-standard locations...

--
greg

Jan Wieck wrote:

If I remore the whole -rpath thing, and remove the two -L options and
the -lpq and -lpgport, and add the libpq.a and libpgport.a explicitly
to the linker call, the psql executable on my Linux box grows from
421761 to 677682 bytes in size. It is still shared linked against
libc, libz, libreadline and a bunch of otheres, but all of them are in
/lib or /usr/lib, so they are standard or system libraries. It does
not depend on a libpq.so any more, and that is what we want.

One of the reasons I originally suggested an explicit 'relocatable'
config option was worry about the rpath thing. Maybe I should have
raised red flags a bit more ;-)

Static linking against libpq would have considerable advantages for the
Windows port (see recent discussion regarding library search paths on
W32 on the w32-hackers list).

cheers

andrew

Bruce Momjian wrote:

We already have --disable-rpath. Seems we would just need something
to use the *.a files.

I think it is perfectly sufficient to say that if you want a relocatable
install, don't use rpath. Static linking will lead to all other kinds
of madness.

Also, some platforms offer relative rpaths. (Maybe Solaris is the only
one.) We could make use of that.

Peter Eisentraut wrote:

Bruce Momjian wrote:

We already have --disable-rpath. Seems we would just need something
to use the *.a files.

I think it is perfectly sufficient to say that if you want a relocatable
install, don't use rpath. Static linking will lead to all other kinds
of madness.

OK, but if we don't use rpath, don't we have to do the ld.so.conf or
environment varaible usage so we can find our shared library. I assume
the big problem with rpath is that it might find the wrong version of
our library, right? Is there another downside to it being set?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Peter Eisentraut wrote:

Bruce Momjian wrote:

We already have --disable-rpath. Seems we would just need something
to use the *.a files.

I think it is perfectly sufficient to say that if you want a relocatable
install, don't use rpath. Static linking will lead to all other kinds
of madness.

OK, I added some more documentation about how to make relocatable
installs.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Peter Eisentraut wrote:

Bruce Momjian wrote:

We already have --disable-rpath. Seems we would just need something
to use the *.a files.

I think it is perfectly sufficient to say that if you want a relocatable
install, don't use rpath. Static linking will lead to all other kinds
of madness.

Boy, nobody was suggesting 100% static linking. What kind of madness are
you getting into if you link libpq.a into psql? There is something
between all or nothing, isn't there?

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Bruce Momjian wrote:

Peter Eisentraut wrote:

Bruce Momjian wrote:

We already have --disable-rpath. Seems we would just need something
to use the *.a files.

I think it is perfectly sufficient to say that if you want a relocatable
install, don't use rpath. Static linking will lead to all other kinds
of madness.

OK, but if we don't use rpath, don't we have to do the ld.so.conf or
environment varaible usage so we can find our shared library. I assume
the big problem with rpath is that it might find the wrong version of
our library, right? Is there another downside to it being set?

Exactly.

Suppose you have one of these silly RPM based Linux systems. One has to
install PostgreSQL from RPM's in order to satisfy all the dependencies
for whatever else you want. Now you want to install a relocatable
version of PostgreSQL somewhere else, because you don't want to mess up
the RPM installed one.

For sure the ld.so.conf is setup to find the RPM installed version of
libpq. That's why we use rpath, so that you can install PG somewhere
else and the /usr/local/pg75/bin/psql will find the libpq in
/usr/local/pg75/lib instead of /usr/lib. But that's currently a
configure time option and removing rpath altogether will let everything
find the old libpq.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Bruce Momjian wrote:

Peter Eisentraut wrote:

Bruce Momjian wrote:

We already have --disable-rpath. Seems we would just need
something to use the *.a files.

I think it is perfectly sufficient to say that if you want a
relocatable install, don't use rpath. Static linking will lead to
all other kinds of madness.

OK, I added some more documentation about how to make relocatable
installs.

The shared library path issues are explained in detail later. You
should refer there rather than listing only a few ideas that may or may
not work. We can't assume that people will have an enlightenment at
the mere appearance of the word LD_LIBRARY_PATH.

Jan Wieck wrote:

Boy, nobody was suggesting 100% static linking. What kind of madness
are you getting into if you link libpq.a into psql? There is
something between all or nothing, isn't there?

It's not going to be only psql and libpq. The next thing is, someone
wants to have a relocatable libpqxx, or a relocatable PHP, or a
relocatable MyCoolReplicationSystem. Before you know it, half the
world has the other half of the world statically linked. This can't be
the solution.

Peter Eisentraut wrote:

Bruce Momjian wrote:

Peter Eisentraut wrote:

Bruce Momjian wrote:

We already have --disable-rpath. Seems we would just need
something to use the *.a files.

I think it is perfectly sufficient to say that if you want a
relocatable install, don't use rpath. Static linking will lead to
all other kinds of madness.

OK, I added some more documentation about how to make relocatable
installs.

The shared library path issues are explained in detail later. You
should refer there rather than listing only a few ideas that may or may
not work. We can't assume that people will have an enlightenment at
the mere appearance of the word LD_LIBRARY_PATH.

Thanks. New text:

For relocatable installs, you might want to use
<filename>configure</filename>'s <literal>--disable-rpath</>
option. Also, you will need to tell the operating system how
to find the shared libraries.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Peter Eisentraut wrote:

Jan Wieck wrote:

Boy, nobody was suggesting 100% static linking. What kind of madness
are you getting into if you link libpq.a into psql? There is
something between all or nothing, isn't there?

It's not going to be only psql and libpq. The next thing is, someone
wants to have a relocatable libpqxx, or a relocatable PHP, or a
relocatable MyCoolReplicationSystem. Before you know it, half the
world has the other half of the world statically linked. This can't be
the solution.

So your opinion is that we need to just require some user configuration
changes to find the shared libs, right?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Peter Eisentraut wrote:

Jan Wieck wrote:

Boy, nobody was suggesting 100% static linking. What kind of madness
are you getting into if you link libpq.a into psql? There is
something between all or nothing, isn't there?

It's not going to be only psql and libpq. The next thing is, someone
wants to have a relocatable libpqxx, or a relocatable PHP, or a
relocatable MyCoolReplicationSystem. Before you know it, half the
world has the other half of the world statically linked. This can't be
the solution.

So your opinion is that we need to just require some user configuration
changes to find the shared libs, right?

Peter is mixing issues here. How another application finds the right
libpq.so, libpqxx.so or whatever interface lib after someone moved the
PostgreSQL lib directory somewhere else has absolutely nothing to do
with how we make sure that our binary utilities like initdb, psql and so
forth are protected against using the wrong one in a multi-version
relocatable environment.

And that some platforms support relative rpath is nice handwaving, but
it doesn't get us anywhere.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Tom Lane wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

If we determine the default data directory off the configure option
--localstatedir then we can simply use the same mechanisms that have
been discussed for determining all the other directories at run time
relative to the binaries.

Agreed, we could let the default location of PGDATA work that way.
We still need the -D runtime option, since (unlike libdir and sharedir)
there could be multiple datadirs being run by a single set of install
files.

Has there been any more thought on this? Do folks want PGDATA/-D to
default to the relocatable ../data, but it can be overridden? Also, if
you specify a data dir during configure, that is used instead.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073