Runtime SHAREDIR for testing CREATE EXTENSION

Sandro Santilli <strk@keybit.net> writes:

I'm trying to understand what options I have to test "CREATE EXTENSION"
w/out installing the extension files in their final destination.

There aren't any. Generally speaking, if you want to be testing an
extension, you should be doing it in a test installation anyway;
so I don't see what's the objection to sticking the files into the
installation's real SHAREDIR.

regards, tom lane

On Tue, Feb 21, 2012 at 10:21:17AM -0500, Tom Lane wrote:

Sandro Santilli <strk@keybit.net> writes:

I'm trying to understand what options I have to test "CREATE EXTENSION"
w/out installing the extension files in their final destination.

There aren't any. Generally speaking, if you want to be testing an
extension, you should be doing it in a test installation anyway;
so I don't see what's the objection to sticking the files into the
installation's real SHAREDIR.

We do a staged installation of the extension library and scripts
in an ad-hoc dir which does not match the built-in SHAREDIR of PostgreSQL.

It works fine for testing the extension directly sourcing the scripts.
It's only "CREATE EXTENSION" that doesn't play nicely with that setup.

--strk;

Sandro Santilli <strk@keybit.net> writes:

On Tue, Feb 21, 2012 at 10:21:17AM -0500, Tom Lane wrote:

Sandro Santilli <strk@keybit.net> writes:

I'm trying to understand what options I have to test "CREATE EXTENSION"
w/out installing the extension files in their final destination.

There aren't any. Generally speaking, if you want to be testing an
extension, you should be doing it in a test installation anyway;
so I don't see what's the objection to sticking the files into the
installation's real SHAREDIR.

We do a staged installation of the extension library and scripts
in an ad-hoc dir which does not match the built-in SHAREDIR of PostgreSQL.

It works fine for testing the extension directly sourcing the scripts.
It's only "CREATE EXTENSION" that doesn't play nicely with that setup.

Please see the "inline extension" thread where answers to your problem
have been discussed.

https://commitfest.postgresql.org/action/patch_view?id=746

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On Tue, Feb 21, 2012 at 10:34:42PM +0100, Dimitri Fontaine wrote:

Sandro Santilli <strk@keybit.net> writes:

On Tue, Feb 21, 2012 at 10:21:17AM -0500, Tom Lane wrote:

Sandro Santilli <strk@keybit.net> writes:

I'm trying to understand what options I have to test "CREATE EXTENSION"
w/out installing the extension files in their final destination.

There aren't any. Generally speaking, if you want to be testing an
extension, you should be doing it in a test installation anyway;
so I don't see what's the objection to sticking the files into the
installation's real SHAREDIR.

We do a staged installation of the extension library and scripts
in an ad-hoc dir which does not match the built-in SHAREDIR of PostgreSQL.

It works fine for testing the extension directly sourcing the scripts.
It's only "CREATE EXTENSION" that doesn't play nicely with that setup.

Please see the "inline extension" thread where answers to your problem
have been discussed.

https://commitfest.postgresql.org/action/patch_view?id=746

I'm not really looking for "inline extensions".
I do want to install the extension objects somewhere, just NOT
in the PostgreSQL builtin SHAREDIR but in an arbitrary staging
directory to use for QA the extension before distribution.

--strk;

,------o-.
| __/ | Delivering high quality PostGIS 2.0 !
| / 2.0 | http://strk.keybit.net
`-o------'

On Tue, Feb 21, 2012 at 1:34 PM, Dimitri Fontaine
<dimitri@2ndquadrant.fr> wrote:

Sandro Santilli <strk@keybit.net> writes:
Please see the "inline extension" thread where answers to your problem
have been discussed.

I'm pretty sure Sandro is hacking PostGIS, so inline extensions are of
no help here.

Can you tell us why alternative approaches that don't require adding a
knob (such as copying/symlinking of compiled artifacts) is such a huge
pain for you?

--
fdr

Re: Sandro Santilli 2012-02-22 <20120222101656.GB6125@gnash>

I'm not really looking for "inline extensions".
I do want to install the extension objects somewhere, just NOT
in the PostgreSQL builtin SHAREDIR but in an arbitrary staging
directory to use for QA the extension before distribution.

We have the same problem with testing extensions at build-time in the
Debian packages. The server's SHAREDIR /usr/share/postgresql/... is
only writable by root, while the build is running as buildd user, so
there is no way to do "create extension whatimbuildingrightnow" to be
able to run regression tests, even if this is a cluster I have just
created with initdb.

Christoph
--
cb@df7cb.de | http://www.df7cb.de/

Daniel Farina <daniel@heroku.com> writes:

I'm pretty sure Sandro is hacking PostGIS, so inline extensions are of
no help here.

Sorry about the sloppy mention, one of the things under discussion (but
out of scope for 9.2) in the inline extension thread is about where to
store the shared objects and extension files so that we have a real
per-database objects here, rather than per major version deployments
that happen to be able to impact several clusters and databases in
different and subtle ways.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On Fri, Feb 24, 2012 at 09:59:12AM +0100, Christoph Berg wrote:

Re: Sandro Santilli 2012-02-22 <20120222101656.GB6125@gnash>

I'm not really looking for "inline extensions".
I do want to install the extension objects somewhere, just NOT
in the PostgreSQL builtin SHAREDIR but in an arbitrary staging
directory to use for QA the extension before distribution.

We have the same problem with testing extensions at build-time in the
Debian packages. The server's SHAREDIR /usr/share/postgresql/... is
only writable by root, while the build is running as buildd user, so
there is no way to do "create extension whatimbuildingrightnow" to be
able to run regression tests, even if this is a cluster I have just
created with initdb.

Exactly. We don't initdb with PostGIS regression testing framework
but I've considered doing it for this specific case and it stroke me
that even then we couldn't control SHAREDIR.

--strk;

,------o-.
| __/ | Delivering high quality PostGIS 2.0 !
| / 2.0 | http://strk.keybit.net - http://vizzuality.com
`-o------'

On Thu, Feb 23, 2012 at 06:53:05PM -0800, Daniel Farina wrote:

On Tue, Feb 21, 2012 at 1:34 PM, Dimitri Fontaine
<dimitri@2ndquadrant.fr> wrote:

Sandro Santilli <strk@keybit.net> writes:
Please see the "inline extension" thread where answers to your problem
have been discussed.

I'm pretty sure Sandro is hacking PostGIS, so inline extensions are of
no help here.

Can you tell us why alternative approaches that don't require adding a
knob (such as copying/symlinking of compiled artifacts) is such a huge
pain for you?

It's not specifically a pain for me, but it breaks the
"tests can be run in a staged environment" idiom.

See Christoph Berg mail, that's exactly the same issue.

--strk;

,------o-.
| __/ | Delivering high quality PostGIS 2.0 !
| / 2.0 | http://strk.keybit.net - http://vizzuality.com
`-o------'

Christoph Berg <cb@df7cb.de> writes:

Re: Sandro Santilli 2012-02-22 <20120222101656.GB6125@gnash>

I'm not really looking for "inline extensions".
I do want to install the extension objects somewhere, just NOT
in the PostgreSQL builtin SHAREDIR but in an arbitrary staging
directory to use for QA the extension before distribution.

We have the same problem with testing extensions at build-time in the
Debian packages. The server's SHAREDIR /usr/share/postgresql/... is
only writable by root, while the build is running as buildd user, so
there is no way to do "create extension whatimbuildingrightnow" to be
able to run regression tests, even if this is a cluster I have just
created with initdb.

This seems like nonsense. If the build process has installed the
software, you surely have got permissions to write in that directory.
If you haven't installed the software, you need to do testing in a
temporary installation per "make check", and we are able to test
extensions that way too.

regards, tom lane

On fre, 2012-02-24 at 11:53 -0500, Tom Lane wrote:

We have the same problem with testing extensions at build-time in

the

Debian packages. The server's SHAREDIR /usr/share/postgresql/... is
only writable by root, while the build is running as buildd user, so
there is no way to do "create extension whatimbuildingrightnow" to

be

able to run regression tests, even if this is a cluster I have just
created with initdb.

This seems like nonsense. If the build process has installed the
software, you surely have got permissions to write in that directory.

The build process just installs the software in a fake root where it
will be wrapped up by the packaging software.

If you haven't installed the software, you need to do testing in a
temporary installation per "make check", and we are able to test
extensions that way too.

That looks like the right answer.

On fre, 2012-02-24 at 17:26 +0100, Sandro Santilli wrote:

We don't initdb with PostGIS regression testing framework
but I've considered doing it for this specific case and it stroke me
that even then we couldn't control SHAREDIR.

I would always create a new instance using initdb for test runs. It's
easy and avoids all these problems.

On Fri, Feb 24, 2012 at 10:21 AM, Peter Eisentraut <peter_e@gmx.net> wrote:

On fre, 2012-02-24 at 17:26 +0100, Sandro Santilli wrote:

We don't initdb with PostGIS regression testing framework
but I've considered doing it for this specific case and it stroke me
that even then we couldn't control SHAREDIR.

I would always create a new instance using initdb for test runs.  It's
easy and avoids all these problems.

Having been in this position once before in a different but similar
situation, there's one big caveat: initdb is *really* slow, so it is
really painful for people who write Postgres-linked code that is
compiled separately, whereby the initdb binary is stable. The
frustration is not unlike when I have to run ./configure, except I had
to do it all the time, every time.

A ccache-like solution has worked well for at least one person I know,
and wasn't hard to implement(?) for personal use.

But by hook or crook, a fresh database is the "right" thing for sure.

--
fdr

On Fri, Feb 24, 2012 at 3:31 PM, Daniel Farina <daniel@heroku.com> wrote:

Having been in this position once before in a different but similar
situation, there's one big caveat: initdb is *really* slow, so it is
really painful for people who write Postgres-linked code that is
compiled separately, whereby the initdb binary is stable.  The
frustration is not unlike when I have to run ./configure, except I had
to do it all the time, every time.

A ccache-like solution has worked well for at least one person I know,
and wasn't hard to implement(?) for personal use.

Being the person in question: I don't have the full context so I'm not
sure how useful this will be, but here's hoping it's not just line
noise. I'm probably not at liberty to share the original code, but it
was a fairly trivial shell script that did more or less the following:

1. Capture all arguments (to pass to initdb itself) except the path,
and figure out the path to the cluster (via looking for
-D/--pgdata/$PGDATA)
2. Create a hash of the concatenation of the arguments (again, sans
cluster path)
3. Create a "template" path based on a local cache directory and the
argument hash
4. a) If the directory with this path did not exist, run initdb with
-D set to this and the rest of the arguments as captured; then copy to
*actual* path
b) If it does exist, that's because you've run initdb with these
arguments before: just copy that to the cluster path you captured in
step (1) instead of starting fresh

This was not terribly robust (e.g., I remember that Dan had pointed
out several shortcomings), but none of these were practical issues for
me, and I never had a problem. There is some more work around
capturing the cluster path in the initdb output and mangling it back
from the template version into the original (so the output of the
cached initdb looks like output of a real initdb), but that's not a
big deal. The approach could probably be made solid enough for
general-purpose use.

On 02/24/2012 06:31 PM, Daniel Farina wrote:

On Fri, Feb 24, 2012 at 10:21 AM, Peter Eisentraut<peter_e@gmx.net> wrote:

On fre, 2012-02-24 at 17:26 +0100, Sandro Santilli wrote:

We don't initdb with PostGIS regression testing framework
but I've considered doing it for this specific case and it stroke me
that even then we couldn't control SHAREDIR.

I would always create a new instance using initdb for test runs. It's
easy and avoids all these problems.

Having been in this position once before in a different but similar
situation, there's one big caveat: initdb is *really* slow, so it is
really painful for people who write Postgres-linked code that is
compiled separately, whereby the initdb binary is stable. The
frustration is not unlike when I have to run ./configure, except I had
to do it all the time, every time.

Really? Here's what I just got on a severely under-resourced SL6 VM:

[andrew@xena ~]$ time /usr/pgsql-9.1/bin/initdb -A ident foo > /dev/null

real 0m1.441s
user 0m1.131s
sys 0m0.309s

1.5s doesn't seem terribly slow.

cheers

andrew

On Fri, Feb 24, 2012 at 6:31 PM, Andrew Dunstan <andrew@dunslane.net> wrote:

Really? Here's what I just got on a severely under-resourced SL6 VM:

1.5s doesn't seem terribly slow.

You are right. Come to think of it, I do seem to recall that initdb
got some speed improvements; these were in 8.3 days, I think.

Here are my numbers for comparison, in some various forms:

env time /usr/lib/postgresql/9.1/bin/initdb /tmp/foo
0.94user 0.28system 0:01.87elapsed 64%CPU (0avgtext+0avgdata 79920maxresident)k
14288inputs+85144outputs (36major+66622minor)pagefaults 0swaps

env time eatmydata /usr/lib/postgresql/9.1/bin/initdb /tmp/foo
0.90user 0.27system 0:01.26elapsed 92%CPU (0avgtext+0avgdata 79984maxresident)k
856inputs+85144outputs (0major+67202minor)pagefaults 0swaps

env time /usr/lib/postgresql/9.1/bin/initdb /dev/shm/blah
0.90user 0.19system 0:01.15elapsed 95%CPU (0avgtext+0avgdata 79984maxresident)k
0inputs+0outputs (0major+67192minor)pagefaults 0swaps

Retracted.

--
fdr

Re: Peter Eisentraut 2012-02-24 <1330107599.32452.15.camel@vanquo.pezone.net>

On fre, 2012-02-24 at 11:53 -0500, Tom Lane wrote:

We have the same problem with testing extensions at build-time in

the

Debian packages. The server's SHAREDIR /usr/share/postgresql/... is
only writable by root, while the build is running as buildd user, so
there is no way to do "create extension whatimbuildingrightnow" to

be

able to run regression tests, even if this is a cluster I have just
created with initdb.

This seems like nonsense. If the build process has installed the
software, you surely have got permissions to write in that directory.

The build process just installs the software in a fake root where it
will be wrapped up by the packaging software.

If you haven't installed the software, you need to do testing in a
temporary installation per "make check", and we are able to test
extensions that way too.

That looks like the right answer.

Well, I'm trying to invoke the extension's "make check" target at
extension build time. I do have a temporary installation I own
somehwere in my $HOME, but that is still trying to find extensions in
/usr/share/postgresql/9.1/extension/*.control, because I am using the
system's postgresql version. The build process is not running as root,
so I cannot do an install of the extension to its final location.
Still it would be nice to run regression tests. All that seems to be
missing is the ability to put

extension_control_path = /home/buildd/tmp/extension

into the postgresql.conf of the temporary PG installation, or some
other way like "CREATE EXTENSION foobar WITH CONTROL '/home/buildd/...'.

Christoph
--
cb@df7cb.de | http://www.df7cb.de/

On lör, 2012-02-25 at 14:21 +0100, Christoph Berg wrote:

Well, I'm trying to invoke the extension's "make check" target at
extension build time. I do have a temporary installation I own
somehwere in my $HOME, but that is still trying to find extensions in
/usr/share/postgresql/9.1/extension/*.control, because I am using the
system's postgresql version. The build process is not running as root,
so I cannot do an install of the extension to its final location.
Still it would be nice to run regression tests. All that seems to be
missing is the ability to put

extension_control_path = /home/buildd/tmp/extension

into the postgresql.conf of the temporary PG installation, or some
other way like "CREATE EXTENSION foobar WITH CONTROL
'/home/buildd/...'.

Yeah, of course, the extension path is not related to the data
directory. So we do need some kind of path setting, just like
dynamic_library_path.

On Sun, Feb 26, 2012 at 10:36 AM, Peter Eisentraut <peter_e@gmx.net> wrote:

On lör, 2012-02-25 at 14:21 +0100, Christoph Berg wrote:

Well, I'm trying to invoke the extension's "make check" target at
extension build time. I do have a temporary installation I own
somehwere in my $HOME, but that is still trying to find extensions in
/usr/share/postgresql/9.1/extension/*.control, because I am using the
system's postgresql version. The build process is not running as root,
so I cannot do an install of the extension to its final location.
Still it would be nice to run regression tests. All that seems to be
missing is the ability to put

extension_control_path = /home/buildd/tmp/extension

into the postgresql.conf of the temporary PG installation, or some
other way like "CREATE EXTENSION foobar WITH CONTROL
'/home/buildd/...'.

Yeah, of course, the extension path is not related to the data
directory.  So we do need some kind of path setting, just like
dynamic_library_path.

That logic seems sound to me, so +1.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company