Configurable path to look up dynamic libraries

Peter Eisentraut <peter_e@gmx.net> writes:

About to be implemented, for your approval...
Variable name: dynamic_library_path
Permissions: superuser
Default value: empty string

This is of little value unless the default is intelligently chosen.
The default should be "$PGLIB", IMHO (inserted from configure's data).
Unless there is a usable default, we cannot start recommending that
people not use absolute paths in CREATE FUNCTION commands.

I do not believe that it's a good idea to allow the value to be changed
at runtime, either --- do you expect that backends will remove
already-loaded libraries in response to a change in the variable?
I think setting the path at postmaster start is necessary and sufficient.

Also, it'd be really nice if a platform-dependent suffix (.so, .sl,
etc) were automatically added to the given library name, if one's not
present already.

regards, tom lane

Tom Lane writes:

This is of little value unless the default is intelligently chosen.
The default should be "$PGLIB", IMHO (inserted from configure's data).

This default has little value as well. Users don't generally put their
loadable modules in the same directory as a PostgreSQL installation.
Maybe they do for general-purpose contrib-like stuff, but then they might
as well use an absolute path. (Remember that a PostgreSQL installation
could well be under /usr/lib; think of all the things that reside there
that we have no business with.)

This also ties in somewhat with the fact that we have no default for
PGDATA, on purpose. If we can have arbitrarily located data locations the
system should not have a hard-wired in default for libraries (which are
usually tied to particular databases in particular database clusters).

I do not believe that it's a good idea to allow the value to be changed
at runtime, either --- do you expect that backends will remove
already-loaded libraries in response to a change in the variable?

No, I would expect it to use the path for loading new libraries from then
on. People that use loadable libraries and C functions are superusers and
experienced enough to cope with this little (logical) fact. (Analogy:
When I change the PATH in my shell, the shell does not kill all processes
already running.)

The way I think this is most useful is in third-party provided
load_all_my_stuff.sql scripts, like:

set dynamic_library_path='/usr/local/foo/lib'; -- inserted by the package's build process

create function foo_xxx() ...

(Yes, you could do the same "inserted by package's build process" into
each of the create function's, but this way it's much cleaner.)

I also envision this to be used as part of dump/restore. pg_dump might
have an option "do not dump full path", and it would insert a 'set
dynamic_library_path'. This would work like the previous case, really.

Also think of a developer wanting to try out different sets of libraries
with a common load script.

If we make this parameter postmaster start only then we really don't gain
anything. We don't even gain the minimal expected convenience in pg_dump
because you would force all modules to reside in a certain place where
administrators would least like them to be.

Also, it'd be really nice if a platform-dependent suffix (.so, .sl,
etc) were automatically added to the given library name, if one's not
present already.

Yes.

--
Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter

Peter Eisentraut <peter_e@gmx.net> writes:

Tom Lane writes:

This is of little value unless the default is intelligently chosen.
The default should be "$PGLIB", IMHO (inserted from configure's data).

This default has little value as well. Users don't generally put their
loadable modules in the same directory as a PostgreSQL installation.

That's a sweeping statement with little to back it up. How do you know
that the usual procedure isn't to put things in $PGLIB? That's
certainly what all our contrib packages do. Even more to the point,
that's certainly where the PL call handler functions are. I will
consider this feature utterly without value unless it allows the
standard declaration of plpgsql_call_handler to become
installation-independent, viz
CREATE FUNCTION plpgsql_call_handler () RETURNS OPAQUE
AS 'plpgsql' LANGUAGE 'C';

This also ties in somewhat with the fact that we have no default for
PGDATA, on purpose. If we can have arbitrarily located data locations the
system should not have a hard-wired in default for libraries (which are
usually tied to particular databases in particular database clusters).

I'd be willing to accept a default path that points to somewhere under
$PGDATA, although I consider this rather less useful. Maybe we could
agree on a compromise two-entry default path: "$PGDATA/functions:$PGLIB"?
That would require some initdb-time shenanigans to set up, but if you
want to do it...

I also envision this to be used as part of dump/restore. pg_dump might
have an option "do not dump full path", and it would insert a 'set
dynamic_library_path'. This would work like the previous case, really.

What? What value would it have for pg_dump to do a set path operation?
The dump script would be unlikely to actually invoke any of the
functions it's loading. By the time anyone tries to use the functions,
they'd be running in a different backend with a different path setting,
namely the default for the installation.

If we make this parameter postmaster start only then we really don't gain
anything. We don't even gain the minimal expected convenience in pg_dump
because you would force all modules to reside in a certain place where
administrators would least like them to be.

I fail to follow this claim also. The point as far as I'm concerned is
that paths mentioned in CREATE FUNCTION ought to be relative to
someplace that's installation-dependent. That way, when you dump out
and reload a CREATE FUNCTION command, the declaration is still good,
you just have to have put a copy of the function's shlib in the right
place for the new installation.

regards, tom lane

Tom Lane writes:

The point as far as I'm concerned is that paths mentioned in CREATE
FUNCTION ought to be relative to someplace that's
installation-dependent. That way, when you dump out and reload a
CREATE FUNCTION command, the declaration is still good, you just have
to have put a copy of the function's shlib in the right place for the
new installation.

Okay, I'm convinced that $libdir can be a useful default. But given the
case where users might want to *add* his directory to the path he needs to
have knowledge of what the default path is. (Unfortunately we can't do
PATH=$PATH:xxx.) Perhaps it would be good to make the empty path
component equivalent to $libdir, e.g.,

'' default, search libdir
':/my/own' search libdir before my own
'/my/own:' search libdir after my own
'/my/own' don't seach libdir

But I think there are enough possibly useful applications for changing
this while the postmaster is running at no real harm.

--
Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter

Peter Eisentraut <peter_e@gmx.net> writes:

Perhaps it would be good to make the empty path
component equivalent to $libdir, e.g.,

Hmm, that would work, and also avoid having to figure out how to stuff
$PGLIB into postgresql.conf during initdb.

Sold as far as I'm concerned ...

regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> writes:

Peter Eisentraut <peter_e@gmx.net> writes:

Perhaps it would be good to make the empty path
component equivalent to $libdir, e.g.,

Hmm, that would work, and also avoid having to figure out how to stuff
$PGLIB into postgresql.conf during initdb.

While on the subject of postgresql conf... shouldn't it be in
sysconfdir instead of the database directory? And there's no switch to
the postmaster to tell it you've put it somewhere else either.

--
Trond Eivind Glomsr�d
Red Hat, Inc.

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

While on the subject of postgresql conf... shouldn't it be in
sysconfdir instead of the database directory?

No. That would (a) not allow different postmasters to have different
config files; (b) not allow a person to create an unprivileged
installation (assuming that sysconfdir is root-owned).

regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> writes:

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

While on the subject of postgresql conf... shouldn't it be in
sysconfdir instead of the database directory?

No. That would (a) not allow different postmasters to have different
config files;

You could search in a path... first sysconfdir, then datadir.

(b) not allow a person to create an unprivileged
installation (assuming that sysconfdir is root-owned).

Sysconfdir defaults to $prefix/etc, so that's not a problem.

--
Trond Eivind Glomsr�d
Red Hat, Inc.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 15 May 2001 14:44, Trond Eivind Glomsr�d wrote:

Tom Lane <tgl@sss.pgh.pa.us> writes:

Peter Eisentraut <peter_e@gmx.net> writes:

Perhaps it would be good to make the empty path
component equivalent to $libdir, e.g.,

Hmm, that would work, and also avoid having to figure out how to stuff
$PGLIB into postgresql.conf during initdb.

While on the subject of postgresql conf... shouldn't it be in
sysconfdir instead of the database directory? And there's no switch to
the postmaster to tell it you've put it somewhere else either.

While I understand and, to an extent, agree with this sentiment, it would be
unworkable at present unless the postgresql.conf file contained constructs
that differentiated between multiple datadirs. While the RPM currently
doesn't support that possibility (not that it will never support such a
possibility :-)), there are many who do use PostgreSQL with multiple
postmasters and datadirs.

I personally wouldn't mind a construct similar to that of a webserver that
supported multiple domain hosting -- you have a master config file or config
file section that is in a standard place, and you have either a separate
config file or config file section for each datadir -- in the case of the
multiple config files, the master would point to each one.

BUT, given the current mindset in the postgresql.conf file, keeping it with
the datadir is presently the only practical option.
- --
Lamar Owen
WGCR Internet Radio
1 Peter 4:11
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE7AXvy5kGGI8vV9eERAhS+AKDGhUchvGN5AEkBqE11wEq8xskrGwCgmIFs
FEDp+xn6e9rdVskMOlhtEKI=
=B0tI
-----END PGP SIGNATURE-----

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

You could search in a path... first sysconfdir, then datadir.

Surely the other way around.

regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> writes:

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

You could search in a path... first sysconfdir, then datadir.

Surely the other way around.

Which could work as well - or just a switch to postmaster to tell it
which file to use.
--
Trond Eivind Glomsr�d
Red Hat, Inc.

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

Tom Lane <tgl@sss.pgh.pa.us> writes:

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

You could search in a path... first sysconfdir, then datadir.

Surely the other way around.

Which could work as well - or just a switch to postmaster to tell it
which file to use.

I could live with a datadir-then-sysconfdir path search. (It should be
datadir first, since the sysconfdir file would serve as a system-wide
default for multiple postmasters.) Given that approach I see no real
need for a postmaster switch.

Possibly the same approach should apply to all the config files we
currently store in datadir?

There is a security issue here: stuff stored in datadir is not visible
to random other users on the machine (since datadir is mode 700), but
I would not expect sysconfdir to be mode 700. We'd need to think about
the implications of allowing Postgres config files to be world-visible.

regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> writes:

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

Tom Lane <tgl@sss.pgh.pa.us> writes:

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

You could search in a path... first sysconfdir, then datadir.

Surely the other way around.

Which could work as well - or just a switch to postmaster to tell it
which file to use.

I could live with a datadir-then-sysconfdir path search. (It should be
datadir first, since the sysconfdir file would serve as a system-wide
default for multiple postmasters.) Given that approach I see no real
need for a postmaster switch.

Possibly the same approach should apply to all the config files we
currently store in datadir?

There is a security issue here: stuff stored in datadir is not visible
to random other users on the machine (since datadir is mode 700), but
I would not expect sysconfdir to be mode 700.

It could be (the RPMs specify a sysconfdir of /etc/pgsql)

We'd need to think about the implications of allowing Postgres
config files to be world-visible.

The files doesn't need to be visible to others...

--
Trond Eivind Glomsr�d
Red Hat, Inc.

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

There is a security issue here: stuff stored in datadir is not visible
to random other users on the machine (since datadir is mode 700), but
I would not expect sysconfdir to be mode 700.

It could be (the RPMs specify a sysconfdir of /etc/pgsql)

The usual install procedure would probably leave sysconfdir owned by
root, if one likes to install in such a way that the binaries are owned
by root (ie make, su root, make install). I'd object to a setup that's
insecure for people who aren't using RPMs.

The real bottom line here, though, is that you haven't shown me any
positive reason to move the config files out of datadir. They're not
broken where they are; and arguably they *are* data.

regards, tom lane

Trond Eivind Glomsr�d writes:

Tom Lane <tgl@sss.pgh.pa.us> writes:

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

You could search in a path... first sysconfdir, then datadir.

Surely the other way around.

Which could work as well - or just a switch to postmaster to tell it
which file to use.

Might as well use a symlink in this case.

I could go for a solution that processed both files in order (possibly
even ${sysconfdir}/postgresql.conf, $PGDATA/postgresql.conf,
${sysconfdir}/postgresql.conf.fixed in order, � la PINE). It could be as
easy as adding two or three lines in postmaster.c. However, I'm afraid
users will interpret a file in $sysconfdir as something clients should
process as well, which is not part of this deal.

--
Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter

Tom Lane <tgl@sss.pgh.pa.us> writes:

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

There is a security issue here: stuff stored in datadir is not visible
to random other users on the machine (since datadir is mode 700), but
I would not expect sysconfdir to be mode 700.

It could be (the RPMs specify a sysconfdir of /etc/pgsql)

The usual install procedure would probably leave sysconfdir owned by
root, if one likes to install in such a way that the binaries are owned
by root (ie make, su root, make install). I'd object to a setup that's
insecure for people who aren't using RPMs.

So make the files unreadable, if so required.

The real bottom line here, though, is that you haven't shown me any
positive reason to move the config files out of datadir.

It conflicts with the FHS - and no, I don't consider configuration
files and data as an identical item.

--
Trond Eivind Glomsr�d
Red Hat, Inc.

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

The real bottom line here, though, is that you haven't shown me any
positive reason to move the config files out of datadir.

It conflicts with the FHS -

AFAIK, the FHS is not designed to support multiple instances of
unprivileged daemons. I'm not interested in forcing Postgres into
that straitjacket ...

regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> writes:

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

The real bottom line here, though, is that you haven't shown me any
positive reason to move the config files out of datadir.

It conflicts with the FHS -

AFAIK, the FHS is not designed to support multiple instances of
unprivileged daemons.

It's OK to support such files, what I don't like is _requiring_ them.

--
Trond Eivind Glomsr�d
Red Hat, Inc.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 15 May 2001 16:12, Tom Lane wrote:

teg@redhat.com (Trond Eivind =?iso-8859-1?q?Glomsr=F8d?=) writes:

The real bottom line here, though, is that you haven't shown me any
positive reason to move the config files out of datadir.

It conflicts with the FHS -

AFAIK, the FHS is not designed to support multiple instances of
unprivileged daemons.

Really? I've never read that into the FHS.

A '/etc/pgsql' directory can easily accomodate multiple config files, or a
single config file with multiple sections. It can just as easily have a
whole tree of stuff under it. And it can be owned by a non-privileged user.
It just cannot be installed into without root's help -- and I know that that
is the main objection here.

But I have a hard time understanding this all-or-nothing approach. So what
if we have a configure-time option to allow a more FHS-compliant approach?
It won't interfere with the 'traditional' /usr/local/pgsql or /opt/pgsql or
whatever way of doing things. Further, it won't uglify the code in the least
- -- it just allows more choice. Further, it won't take a hacker long to make
it work. It won't even touch 'real' backend code, either -- this is a
postmaster thingy.

What's the opposition about? We have all the configure options already for
many things that the 'traditional' postgresql user cares nothing about.

But, on a more pragmatic note, I am contemplating the ability for the RPM's
initscript to allow multiple postmasters -- even up to sane behavior in the
presence of postmasters that it didn't start -- with multiple datadirs, etc.

I don't want the RPM's to 'editorialize' any more than anyone else might --
but unless a more FHS-compliant approach is at least _allowed_ (NOT
mandated!), I guess there will need to be some editorializing in the
intscript as it will have to place its own config file somewhere in order to
make multiple postmasters happen.

But I don't want to go through that if I don't have to -- or if it's going to
happen anyway.

And, I know, currently the '-D' postmaster directive does, indirectly, point
to the location of postgresql.conf..... :-) I will be using that in the
initscript's logic if another option isn't done.

But, if I may editorialize a little myself, this is just indicative of a
'Fortress PostgreSQL' attitude that is easy to get into. 'We've always done
it this way' or 'This is the way PostgreSQL works' are pat answers to
anything from 'Why can't I more smoothly upgrade?' to 'Why does PostgreSQL
use non-SQL-standard case-folding?' to 'Why does everything go in
/usr/local/pgsql?' to 'Do I _really_ have to do an ASCII dump of my 100GB
database in order to go to the next major version?' to any number of other
FAQ's.

Just because we've always done it one way does not that one way correct make.

We're one component of a system -- and the PostgreSQL Group has done such a
good job of being platform agnostic that the platform and systems issues are
almost second-class citizens.

Well, gotta get off the soap box now, and get to work producing some code, I
guess. People are going to be expecting multiple postmaster support in the
RPMset soon. :-)
- --
Lamar Owen
WGCR Internet Radio
1 Peter 4:11
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE7AZ+15kGGI8vV9eERAu+MAJ9bk8mY8n1qIk8zKqWM1K188/530wCeJnwd
ZZDjAosFhRnTENBWJ+THju4=
=mPC9
-----END PGP SIGNATURE-----