Relocatable installs

Bruce Momjian <pgman@candle.pha.pa.us> writes:

My idea is to write a /port function that uses various methods to find
the needed files. We could look in the relative location first, and if
the needed file is not found, look in the hardcoded directory.

I think a "search until you find something" approach would be a really
bad idea. Particularly on a machine with multiple PG versions installed
(and that has surely got to be a likely situation for people who are
wanting to move things around). It seems entirely too likely that you
would find the wrong version of some file.

So ISTM that the location in which a given installation looks for its
associated files should be completely predictable and *not* depend on
whether it finds something there.

I'm fine with offering an option to make that location be relative to
where the executable came from. But not with nondeterminism.

I think we should use the relative-path method *unless* the configure
command called out specific installation directories (that is, not
just --prefix but --datadir and/or related options). If you use one of
those then that absolute path should be used always, ie, you are
specifically asking for a nonrelocatable install and that's what you
should get.

regards, tom lane

On Sat, 8 May 2004, Bruce Momjian wrote:

Win32 is going to need relocatable installs, and Unix packagers have
asked for this too.

A relocatable install is one where you can do 'gmake install', tar up
the directory where you installed it, then untar it on to another
machine with the same operating system, but into a different directory
location.

For example, if you use the defaults for directory locations, PostgreSQL
will install into /usr/local/pgsql, and everything will be under that
directory --- bin, lib, include, share. (Not sure about etc, doc, and
man.)

However, right now, if you try to move /usr/local/pgsql to /var/pgsql,
the database will not work because there are hard-coded directory
dependencies in the binaries:

initdb has to find its *.bki files in /share
initdb has to find the postgres binary in /bin
pg_dumpall has to find pg_dump
postgres has to find shared objects the /lib
Win32 postgres has to find /share/timezone

Maybe there are more.

Is there a reason why we can't use a PGSQL_PREFIX ENV variable or
something like that? If not defined, use compiled in default?

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

Marc G. Fournier wrote:

However, right now, if you try to move /usr/local/pgsql to /var/pgsql,
the database will not work because there are hard-coded directory
dependencies in the binaries:

initdb has to find its *.bki files in /share
initdb has to find the postgres binary in /bin
pg_dumpall has to find pg_dump
postgres has to find shared objects the /lib
Win32 postgres has to find /share/timezone

Maybe there are more.

Is there a reason why we can't use a PGSQL_PREFIX ENV variable or
something like that? If not defined, use compiled in default?

Win32 isn't going to be able to define that. Also, why not make it
relocatable by default? We already know where the binary is.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

My idea is to write a /port function that uses various methods to find
the needed files. We could look in the relative location first, and if
the needed file is not found, look in the hardcoded directory.

I think a "search until you find something" approach would be a really
bad idea. Particularly on a machine with multiple PG versions installed
(and that has surely got to be a likely situation for people who are
wanting to move things around). It seems entirely too likely that you
would find the wrong version of some file.

So ISTM that the location in which a given installation looks for its
associated files should be completely predictable and *not* depend on
whether it finds something there.

I agree.

I'm fine with offering an option to make that location be relative to
where the executable came from. But not with nondeterminism.

I think we should use the relative-path method *unless* the configure
command called out specific installation directories (that is, not
just --prefix but --datadir and/or related options). If you use one of
those then that absolute path should be used always, ie, you are
specifically asking for a nonrelocatable install and that's what you
should get.

I think we are making this way too complicated in a quest for
flexibility that is of dubious value.

I think we could adopt a simple rule: if you configure it for relocation
(and I think you should have to do that explicitly) then all paths are
relative to the binary location. If not, then full hardcoded paths are
used. No exceptions.

Most people won't need this at all, I suspect - people who make binary
packages/installers for redistribution will find it a great boon.

cheers

andrew

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

My idea is to write a /port function that uses various methods to find
the needed files. We could look in the relative location first, and if
the needed file is not found, look in the hardcoded directory.

I think a "search until you find something" approach would be a really
bad idea. Particularly on a machine with multiple PG versions installed
(and that has surely got to be a likely situation for people who are
wanting to move things around). It seems entirely too likely that you
would find the wrong version of some file.

As I see it, we do two searches. One is for the path to our own binary,
and the second is a search for another binary or the /lib or /share
directories.

In initdb and postgres, we do searches by breaking up the $PATH value,
and testing for versions. In pg_dump, we just run system() (uses
default PATH), and then try hardcoded path if that fails, with no
version checking. This should all be consistent at least.

Then, once we find our own path, the next question is how to handle
looking for lib and share, and whether we look for a directory name, or
specific file, and can we check versions? Right now we do none of
these, I think. We just go for hard-coded.

So ISTM that the location in which a given installation looks for its
associated files should be completely predictable and *not* depend on
whether it finds something there.

Ah, got it. Makes sense. We do searches for finding our own path, but
you can argue that this predictable --- we know we are running so the
binary must be somewhere. :-) However, the version checking we do now
is a little non-predictable because if we find a binary of the wrong
version, we keep looking in the path.

I'm fine with offering an option to make that location be relative to
where the executable came from. But not with nondeterminism.

Ah, makes sense.

I think we should use the relative-path method *unless* the configure
command called out specific installation directories (that is, not
just --prefix but --datadir and/or related options). If you use one of
those then that absolute path should be used always, ie, you are
specifically asking for a nonrelocatable install and that's what you
should get.

That would make my job very easy. If they specify lib or share with a
unique path, it is hard to imagine how they would tar up the install
anyway because it would be usually mixed into their operating system
directories. Are others OK with this? You can still specify --prefix,
but if you override the other default directories, the program will use
those hard-coded ones rather than ones relative to your /bin.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Marc G. Fournier wrote:

Is there a reason why we can't use a PGSQL_PREFIX ENV variable or
something like that? If not defined, use compiled in default?

Win32 isn't going to be able to define that.

And this idea is even worse than the other one as far as destroying
the predictability of the search goes :-(.

ISTM that just about every place we have made the backend's behavior
dependent on environment variables, we have had cause to regret it.
It's too easy to screw up by launching the postmaster under a different
environment than you normally do.

regards, tom lane

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Ah, got it. Makes sense. We do searches for finding our own path, but
you can argue that this predictable --- we know we are running so the
binary must be somewhere. :-) However, the version checking we do now
is a little non-predictable because if we find a binary of the wrong
version, we keep looking in the path.

Hm? I see no version checks in FindExec. It looks to me like the code
is just trying to ensure that it finds the same file that the shell
would have found.

regards, tom lane

Andrew Dunstan wrote:

I think we should use the relative-path method *unless* the configure
command called out specific installation directories (that is, not
just --prefix but --datadir and/or related options). If you use one of
those then that absolute path should be used always, ie, you are
specifically asking for a nonrelocatable install and that's what you
should get.

I think we are making this way too complicated in a quest for
flexibility that is of dubious value.

I think we could adopt a simple rule: if you configure it for relocation
(and I think you should have to do that explicitly) then all paths are
relative to the binary location. If not, then full hardcoded paths are
used. No exceptions.

Most people won't need this at all, I suspect - people who make binary
packages/installers for redistribution will find it a great boon.

I think if we go for the plan outlined, we will not need a special
configure flag. (People might decide to move the install dir long after
they install it.) By default, everything sits under pgsql as pgsql/bin,
pgsql/lib, etc. I can't see how making it relative is going to bite us
unless folks move the binaries out of pgsql/bin. Is that common for
installs that don't specify a special bindir?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Andrew Dunstan <andrew@dunslane.net> writes:

Tom Lane wrote:

I think we should use the relative-path method *unless* the configure
command called out specific installation directories (that is, not
just --prefix but --datadir and/or related options).

I think we could adopt a simple rule: if you configure it for relocation
(and I think you should have to do that explicitly) then all paths are
relative to the binary location. If not, then full hardcoded paths are
used. No exceptions.

I think we're saying the same thing except for the question of whether
relative-path behavior has to be explicitly requested at configure time.

While I'm not dead set on it, I'm leaning to the idea that it's okay to
make relative-path the standard behavior. I cannot see any real serious
downsides to it. We have always bombed out if we are unable to locate
the executable, so it's not like that code isn't well-tested.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Ah, got it. Makes sense. We do searches for finding our own path, but
you can argue that this predictable --- we know we are running so the
binary must be somewhere. :-) However, the version checking we do now
is a little non-predictable because if we find a binary of the wrong
version, we keep looking in the path.

Hm? I see no version checks in FindExec. It looks to me like the code
is just trying to ensure that it finds the same file that the shell
would have found.

I thought it was in ValidateBinary, but now I don't see it. I must have
gotten it confused by pg_dumpall. And I see now that initdb only checks
the version and exits if it is wrong. It does not keep trying the path.

OK, we are better than I thought. :-)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Comments?

What's wrong with the way it's done by everybody else?

Have hardcoded paths (determined at configure time), and allow override
using a config file. Have a command line option for saying where the
config file should be.

For Windows, replace config file with "Registry". That is usually
hardcoded for (depending on whether you want it changeable per-user)
HKEY_LOCAL_MACHINE\Software\<company name>\<product name> (replace HKLM
with HKEY_LOCAL_USER if you want per-user config).

Shachar

--
Shachar Shemesh
Lingnu Open Source Consulting
http://www.lingnu.com/

Shachar Shemesh wrote:

Bruce Momjian wrote:

Comments?

What's wrong with the way it's done by everybody else?

Have hardcoded paths (determined at configure time), and allow override
using a config file. Have a command line option for saying where the
config file should be.

Where do we put the config file so it can be found? We can't assume
root privilege to write into /etc.

For Windows, replace config file with "Registry". That is usually
hardcoded for (depending on whether you want it changeable per-user)
HKEY_LOCAL_MACHINE\Software\<company name>\<product name> (replace HKLM
with HKEY_LOCAL_USER if you want per-user config).

Doesn't registry require admin privilege on the Windows box?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Shachar Shemesh wrote:

Bruce Momjian wrote:

Comments?

What's wrong with the way it's done by everybody else?

Have hardcoded paths (determined at configure time), and allow override
using a config file. Have a command line option for saying where the
config file should be.

Where do we put the config file so it can be found? We can't assume
root privilege to write into /etc.

Put it in /etc (or wherever else user said in configure). Allow a
command line parameter that says "run the server with this config file
instead". User can place override config file wherever she damn wishes.

For Windows, replace config file with "Registry". That is usually
hardcoded for (depending on whether you want it changeable per-user)
HKEY_LOCAL_MACHINE\Software\<company name>\<product name> (replace HKLM
with HKEY_LOCAL_USER if you want per-user config).

Doesn't registry require admin privilege on the Windows box?

HKLM does. HKCU doesn't. Actually, HKLM often doesn't either, but that's
a rant I won't start right now.

Again, you can use command-line arguments to override, if you want.
Alternatively, Postgres can look at HKCU, and if not found there, at
HKLM. This means that there can be a system-wide default, overrideable
per user by each user. The only reason I'm suggesting this (as it is
unlikely that for production installs more than one user will be running
postgres) is that editing user keys for other users is a task I'm not
sure many admins know how to do. It requires the use of the
not-well-known "regedt32" rather than "regedit", and involves loading
the correct file into the registry "hive". As such, placing stuff in
HKLM allows admins to easilly edit it. If that is not an issue, I'd say
"place it at HKCU".

Shachar

--
Shachar Shemesh
Lingnu Open Source Consulting
http://www.lingnu.com/

On Sun, 9 May 2004, Bruce Momjian wrote:

Shachar Shemesh wrote:

Bruce Momjian wrote:

Comments?

What's wrong with the way it's done by everybody else?

Have hardcoded paths (determined at configure time), and allow override
using a config file. Have a command line option for saying where the
config file should be.

Where do we put the config file so it can be found? We can't assume
root privilege to write into /etc.

The default location for the config file would be compiled into the
binary based on whatever settings were given when it was built.

If someone wants to use a different location after it is compiled,
they have to specify the config file path on the command line. For
starting up the database, that isn't a big deal because you normally
should be using some wrapper script anyway.

This is how the vast majority of other programs do it on unix.
Apache, mysql, BIND, the list goes on.

The downside to this is that it isn't as friendly for various
command line tools that people run, since they then have to specify
the path on the command line. One option to deal with this is to
allow compiling in a default path that is optionally relative to
the directory the binary is in, eg. "../foo/bar" would be relative
to where the binary is, while "/foo/bar" would be absolute. Then
there is a choice at build time; it could default to paths relative
to the binary and allow overriding that if the user compiling it wants
something different.

One key thing that this gives that the other proposals don't is the
ability to have a single, static tree of program files and run multiple
different instances of the database off them, each having their own
unique configuration and data trees. This also makes it easier to move
back and forth between different minor versions for testing or QA
just by tweaking a startup script to point to a different binary
tree, without having to copy data, config, or binaries around. In
this scenario, the above downside can be worked around by just
creating simple wrapper scripts.

We use this type of setup with products such as Apache to manage
having code for different projects or different versions of products
running out of different development or QA or production trees that
uses different versions of Apache, to ensure we are developing and
testing against the same version that will be in production, and
that we can manage upgrades on a project by project basis.

So, for example you could have:

/site/httpd/httpd-2.0.45
/site/httpd/httpd-2.0.45-2
/site/httpd/httpd-2.0.48

containing static copies of the binaries and libraries that are never
ever modified. All the config files and data live in specific projects,
then we just configure that project to use a specific version of Apache.
Multiple versions can live on a machine and each project can pick which
one it wants to use and easily upgrade to new ones, without having a
whole bunch of copies of the binaries.

While this is slightly less useful for postgresql, both because fewer
people want to be able to concurrenetly run multiple different instances
on the same box for testing purposes and because the database generally
has to be modified to work between non-minor version changes, it is
still something that is very useful in some environments and I would
love to have.

For Windows, replace config file with "Registry". That is usually
hardcoded for (depending on whether you want it changeable per-user)
HKEY_LOCAL_MACHINE\Software\<company name>\<product name> (replace HKLM
with HKEY_LOCAL_USER if you want per-user config).

Doesn't registry require admin privilege on the Windows box?

Not if it is in the per user part as Shachar suggested.

If it is done this way, however, it should be possible to override
the registry via command line flags to allow multiple installations
on the same box in different places.

It's rumoured that Marc Slemko once said:

For Windows, replace config file with "Registry". That is usually
hardcoded for (depending on whether you want it changeable per-user)
HKEY_LOCAL_MACHINE\Software\<company name>\<product name> (replace
HKLM with HKEY_LOCAL_USER if you want per-user config).

Doesn't registry require admin privilege on the Windows box?

Not if it is in the per user part as Shachar suggested.

Which is a problem in it's own right anyway. To properly write events to
the event log we need to register PostgreSQL as an event source and
specify a message DLL to use. As Magnus & I discussed the other day, in
our case we probably just want a single message of %s in the dll to allow
PostgreSQL to log anything it wants cleanly.
Anyway, the point is that to do that you need to write to
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\EventLog\<Appname>
(off the top of my head) otherwise, the messages logged in the event log
are pretty unreadable. Service installation will also require admin
privileges.
Regards, Dave

Dave Page wrote:

Anyway, the point is that to do that you need to write to
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\EventLog\<Appname>
(off the top of my head) otherwise, the messages logged in the event log
are pretty unreadable. Service installation will also require admin
privileges.
Regards, Dave

I don't think that's a problem at all:
1. Believe it or not, you can write to HKEY_CURRENT_USER even if you are
an admin! New and improved..... In fact, if you don't mind messing with
the "load hive" interfaces, you can even write to someone else's HKCU if
you are admin. It makes installs more complicated, but see 2 for why
this will not be necessary.
2. Original suggestion talked about looking up at HKCU, and then (if not
found) at HKEY_LOCAL_MACHINE
3. I doubt the same machine will require BOTH service and per-user
installation

Shachar

--
Shachar Shemesh
Lingnu Open Source Consulting
http://www.lingnu.com/

-----Original Message-----
From: Shachar Shemesh [mailto:psql@shemesh.biz]
Sent: 09 May 2004 10:20
To: Dave Page
Cc: marcs@znep.com; pgman@candle.pha.pa.us;
pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Relocatable installs

I don't think that's a problem at all:
1. Believe it or not, you can write to HKEY_CURRENT_USER even
if you are an admin! New and improved..... In fact, if you
don't mind messing with the "load hive" interfaces, you can
even write to someone else's HKCU if you are admin. It makes
installs more complicated, but see 2 for why this will not be
necessary.

I think you're missing what I was saying. To properly write to the event
log without getting messages like:

The description for Event ID ( 0 ) in Source ( btwdins ) cannot be
found. The local computer may not have the necessary registry
information or message DLL files to display messages from a remote
computer. You may be able to use the /AUXSOURCE= flag to retrieve this
description; see Help and Support for details. The following information
is part of the event: Service started.

you need to register an event source with a message dll under
HKEY_LOCAL_MACHINE. HKCU is not an option. This is not part of the
relocatable install issue I grant you - the mention of HKLM permissions
just made me think of it...

2. Original suggestion talked about looking up at HKCU, and
then (if not
found) at HKEY_LOCAL_MACHINE

Yes, that's fine for the relocatable install issues.

3. I doubt the same machine will require BOTH service and
per-user installation

No, but either may want to use the event log.

Regards, Dave.

Shachar Shemesh said:

Bruce Momjian wrote:

Comments?

What's wrong with the way it's done by everybody else?

Have hardcoded paths (determined at configure time), and allow override
using a config file. Have a command line option for saying where the
config file should be.

For Windows, replace config file with "Registry". That is usually
hardcoded for (depending on whether you want it changeable per-user)
HKEY_LOCAL_MACHINE\Software\<company name>\<product name> (replace HKLM
with HKEY_LOCAL_USER if you want per-user config).

IMNSHO this would not buy anything but would provide you with an extra
capacity for shooting yourself in the foot. Using what we have discussed
up to now this would be completely unnecessary - you could just move the
installation and it would work (modulo any necessary changes in
LD_LIBRARY_PATH ,PATH etc).

Perhaps you have not followed the discussion about this issue over the
last months, but a good part of it revolved around avoiding the very
dependencies you are suggesting.

"What everybody else does" doesn't carry much weight on its own with me.

cheers

andrew

Marc Slemko <marcs@znep.com> writes:

The downside to this is that it isn't as friendly for various
command line tools that people run, since they then have to specify
the path on the command line.

Exactly. Not only unfriendly, but quite error-prone, especially in
a multiple-install situation.

The fact that "everyone else does it this way" doesn't automatically
mean we should too. The fact of the matter is that most of those other
programs are not really designed to support multiple installs
conveniently. We've developed conventions that let us handle that,
and I don't wish to backslide.

The thing I like about the relative-path idea is that it actually
improves and extends our existing ability to support multiple
installs. I wonder whether we could even allow PGDATA to default
to a relative path (../data)?

regards, tom lane