Proposal: allow installation of any contrib module simultaneously with Postgres itself

Nikolay Samokhvalov wrote:

1. Change default behaviour of <MODULE_NAME>.sql file so it will be
installed in <MODULE_NAME> schema instead of "public" (e.g., "hstore"
schema will contain all hstore relations and functions).

That might be a good idea in any case.

2. Allow running configure with "--with-<MODULE_NAME>" (or
"--enable-<MODULE_NAME>") to include compilation of module's
libraries simultaneously with Postgres itself and including running
of module's registration SQLs (from that .sql files) simultaneously
with cluster creation (in other words, with inidb invocation -- this
will add "<MODULE_NAME>" schema to template0).

Build-time options will generally suffer from the problem that package
builders will turn them all on and then users are stuck with all
modules and then they're reallly not modular anymore.

But I think your general idea of making them "more default" is sound.
But it needs to be a run-time choice.

Maybe we really just need to call them "modules" instead of "contrib",
since users of Apache, PHP, or Linux will be familiar with that term.
(I don't know how this overlaps with SQL modules though.)

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

"Nikolay Samokhvalov" <samokhvalov@gmail.com> writes:

1. Change default behaviour of <MODULE_NAME>.sql file so it will be
installed in <MODULE_NAME> schema instead of "public" (e.g., "hstore"
schema will contain all hstore relations and functions).

This might be a good idea, but it's hardly transparent; it can be
counted on to break the applications of just about everyone using those
modules today.

2. Allow running configure with "--with-<MODULE_NAME>" (or
"--enable-<MODULE_NAME>") to include compilation of module's libraries
simultaneously with Postgres itself and including running of module's
registration SQLs (from that .sql files) simultaneously with cluster
creation (in other words, with inidb invocation -- this will add
"<MODULE_NAME>" schema to template0).

This will simplify the procedure of starting to use contrib modules
and will help to promote the modules themselves (and, as a result,
some PostgreSQL's advanced features). I think many projects have
similar behaviour with regard to their extensions. And ISPs will
install PostgreSQL with a bundle of useful and "trusted" extensions,
simply running "./configure --with-tsearch2 --with-hstore
--with-dblink" (actually, I hope that tsearch2 will be in core, but
this is really good example at the moment ;-) ) - like they do with
PHP, Apache and other software.

The problem with this proposal is that the ISPs aren't the ones running
configure --- these days, most people are running prebuilt packages
(RPMs or DEBs or what have you). So what you are hoping is that the
packagers will choose to do this and thereby force these modules into
the "standard" configuration for everybody using their packages. I'm
not sure that the packagers will change ... well maybe Gentoo will,
but not anyone with more conservative policies ... and I'm pretty sure
any who do will get push-back from people who still won't trust contrib.

regards, tom lane

The problem with this proposal is that the ISPs aren't the ones running
configure --- these days, most people are running prebuilt packages
(RPMs or DEBs or what have you). So what you are hoping is that the
packagers will choose to do this and thereby force these modules into
the "standard" configuration for everybody using their packages. I'm
not sure that the packagers will change ... well maybe Gentoo will,
but not anyone with more conservative policies ... and I'm pretty sure
any who do will get push-back from people who still won't trust contrib.

Well perhaps it is time to trim Contrib even further. E.g;

Why is ltree still in contrib? What prevents it from being in core?

Why is pgcrypto,pgstattuple and pg_freespacemap in contrib?

I would almost think it makes sense to have:

/modules - This is for things that are indeed modules they will be
installed by default but are not *internal* like sum() or generate_series()

/contrib - This is for things like adminpack, dblink, pgbench

Sincerely,

Joshua D. Drake

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Joshua D. Drake wrote:

The problem with this proposal is that the ISPs aren't the ones running
configure --- these days, most people are running prebuilt packages
(RPMs or DEBs or what have you). So what you are hoping is that the
packagers will choose to do this and thereby force these modules into
the "standard" configuration for everybody using their packages. I'm
not sure that the packagers will change ... well maybe Gentoo will,
but not anyone with more conservative policies ... and I'm pretty sure
any who do will get push-back from people who still won't trust contrib.

Well perhaps it is time to trim Contrib even further. E.g;

Why is ltree still in contrib? What prevents it from being in core?

not sure - ltree is quite useful but I'm not convinced it is really core
material

Why is pgcrypto,pgstattuple and pg_freespacemap in contrib?

I would like to see pgcrypto (or at least some of it's functionality) in
core.
pgstattuple is still being quite activily developed (just look at the
large changes/feature enhancements for 8.2 which seems like a sign of it
not being ready yet.
pg_freespacemap is iirc new as of 8.2 so it imho never was a candidate
for core.

Stefan

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:

I would like to see pgcrypto (or at least some of it's functionality) in
core.

I believe the reason we keep it separate is so that people can easily
make crypto-free versions of PG for use in countries where encryption
capability is considered subject to arms regulations. Not sure how
important that case really is today, but it was a big consideration
last time this was discussed.

regards, tom lane

Well perhaps it is time to trim Contrib even further. E.g;

Why is ltree still in contrib? What prevents it from being in core?

not sure - ltree is quite useful but I'm not convinced it is really core
material

Why?

Why is pgcrypto,pgstattuple and pg_freespacemap in contrib?

I would like to see pgcrypto (or at least some of it's functionality) in
core.
pgstattuple is still being quite activily developed (just look at the
large changes/feature enhancements for 8.2 which seems like a sign of it
not being ready yet.

Is PostgreSQL not under active development? Is it not receiving large
changes or feature enhancements?

pg_freespacemap is iirc new as of 8.2 so it imho never was a candidate
for core.

I think you are missing my point. I only picked contribs that seemed
logical to be in core. The point of the post was the discussion, not the
particular module.

Joshua D. Drake

Stefan

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Tom Lane wrote:

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:

I would like to see pgcrypto (or at least some of it's functionality) in
core.

I believe the reason we keep it separate is so that people can easily
make crypto-free versions of PG for use in countries where encryption
capability is considered subject to arms regulations. Not sure how
important that case really is today, but it was a big consideration
last time this was discussed.

That is a good question but in reality people who need to do so, can get
the source from a mirror that is outside the country that has the rules.

Sincerely,

Joshua D. Drake

regards, tom lane

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

"Joshua D. Drake" <jd@commandprompt.com> writes:

Tom Lane wrote:

I believe the reason we keep it separate is so that people can easily
make crypto-free versions of PG for use in countries where encryption
capability is considered subject to arms regulations. Not sure how
important that case really is today, but it was a big consideration
last time this was discussed.

That is a good question but in reality people who need to do so, can get
the source from a mirror that is outside the country that has the rules.

You miss the point. Everybody knows that those laws are not too hard
to circumvent if you are willing to break the law. The question is
how hard is it for someone to distribute Postgres into one of those
countries *without* breaking any local law. We won't be making things
better if we make it difficult or impossible to put together a
crypto-free PG distribution; we'll just be cutting off some potential
users.

As Peter keeps pointing out, this urge to push everything into core is
misguided anyway. The fact that you can build useful functionality as
extension modules is a strength of PG, not a weakness, and we should
build on that strength instead of running away from it. If you ask me
we should be working on these things:

* Better packaging support, eg make it easier to add/remove an extension
module and control how pg_dump deals with it. We talked about that
awhile back but nobody did anything with the ideas.

* Better documentation for the contrib modules; some of them are
reasonably well doc'd now, but many are not, and in almost all cases
it's only plain text not SGML.

* Better advertising, for instance make the contrib documentation
available on the website (which probably requires SGML conversion
to happen first...)

regards, tom lane

You miss the point. Everybody knows that those laws are not too hard
to circumvent if you are willing to break the law. The question is
how hard is it for someone to distribute Postgres into one of those
countries *without* breaking any local law. We won't be making things
better if we make it difficult or impossible to put together a
crypto-free PG distribution; we'll just be cutting off some potential
users.

O.k. that is certainly true.

As Peter keeps pointing out, this urge to push everything into core is
misguided anyway.

I do not believe we need to push everything into core. I do believe
there is a lot of stuff that is not in core, that could be and possibly
should be.

The fact that you can build useful functionality as
extension modules is a strength of PG, not a weakness, and we should
build on that strength instead of running away from it. If you ask me
we should be working on these things:

I am not suggesting it is a weakness in any way. I think its great. I am
just saying there appears to be several modules that really should be in
core that aren't.

* Better packaging support, eg make it easier to add/remove an extension
module and control how pg_dump deals with it. We talked about that
awhile back but nobody did anything with the ideas.

Yes that would be great.

* Better documentation for the contrib modules; some of them are
reasonably well doc'd now, but many are not, and in almost all cases
it's only plain text not SGML.

Agreed... more to the point, if it is in contrib it needs to be
documented in Docbok and it needs to have a section in the main docs.

* Better advertising, for instance make the contrib documentation
available on the website (which probably requires SGML conversion
to happen first...)

Agreed.

Sincerely,

Joshua D. Drake

regards, tom lane

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

On Thu, 25 Jan 2007, Joshua D. Drake wrote:

The problem with this proposal is that the ISPs aren't the ones running
configure --- these days, most people are running prebuilt packages
(RPMs or DEBs or what have you). So what you are hoping is that the
packagers will choose to do this and thereby force these modules into
the "standard" configuration for everybody using their packages. I'm
not sure that the packagers will change ... well maybe Gentoo will,
but not anyone with more conservative policies ... and I'm pretty sure
any who do will get push-back from people who still won't trust contrib.

Well perhaps it is time to trim Contrib even further. E.g;

Why is ltree still in contrib? What prevents it from being in core?

we have several requests to improve ltree, particularly, people want
to expand class of allowed symbols and configurable separator, which
is hard-coded right now. Also, we discussed GiN support for ltree.

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

"Joshua D. Drake" <jd@commandprompt.com> writes:

Oleg Bartunov wrote:

we have several requests to improve ltree, particularly, people want
to expand class of allowed symbols and configurable separator, which is
hard-coded right now. Also, we discussed GiN support for ltree.

O.k. but how does that prevent it from being in core?

Once it's in core the expectations for API stability get a lot higher ...

regards, tom lane

Joshua D. Drake wrote:

Tom Lane wrote:

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:

I would like to see pgcrypto (or at least some of it's
functionality) in core.

I believe the reason we keep it separate is so that people can
easily make crypto-free versions of PG for use in countries where
encryption capability is considered subject to arms regulations.
Not sure how important that case really is today, but it was a big
consideration last time this was discussed.

That is a good question but in reality people who need to do so, can
get the source from a mirror that is outside the country that has the
rules.

The point is that vendors (consultants, distributors, nice people) can
produce safe versions to distribute in or to places with crypto
regulations. But since PostgreSQL contains code that links with SSL
and Kerberos functionality, you need to patch the code anyway in order
to do that legally. But it's nice to keep these things constrained to
well-known places in the source anyway.

(This opinion was sponsored by real lawyers.)

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Oleg Bartunov wrote:

On Thu, 25 Jan 2007, Joshua D. Drake wrote:

The problem with this proposal is that the ISPs aren't the ones running
configure --- these days, most people are running prebuilt packages
(RPMs or DEBs or what have you). So what you are hoping is that the
packagers will choose to do this and thereby force these modules into
the "standard" configuration for everybody using their packages. I'm
not sure that the packagers will change ... well maybe Gentoo will,
but not anyone with more conservative policies ... and I'm pretty sure
any who do will get push-back from people who still won't trust contrib.

Well perhaps it is time to trim Contrib even further. E.g;

Why is ltree still in contrib? What prevents it from being in core?

we have several requests to improve ltree, particularly, people want
to expand class of allowed symbols and configurable separator, which is
hard-coded right now. Also, we discussed GiN support for ltree.

O.k. but how does that prevent it from being in core?

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

Oleg Bartunov wrote:

we have several requests to improve ltree, particularly, people want
to expand class of allowed symbols and configurable separator, which is
hard-coded right now. Also, we discussed GiN support for ltree.

O.k. but how does that prevent it from being in core?

Once it's in core the expectations for API stability get a lot higher ...

True but any API can be backward compatible for say a release while
offering the new functionality as well. I fully understand stability in
core. I certainly don't want any level of instability, even if just
syntactic.

Maybe there isn't a good answer.

Joshua D. Drake

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Joshua D. Drake wrote:

Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

Oleg Bartunov wrote:

we have several requests to improve ltree, particularly, people want
to expand class of allowed symbols and configurable separator, which is
hard-coded right now. Also, we discussed GiN support for ltree.

O.k. but how does that prevent it from being in core?

Once it's in core the expectations for API stability get a lot higher ...

True but any API can be backward compatible for say a release while
offering the new functionality as well. I fully understand stability in
core. I certainly don't want any level of instability, even if just
syntactic.

I don't think two releases from API change to API change is enough -
postgresql is running larger and larger databases by now and I expect
people to upgrade less often in the future (and iirc you already said
something along the lines of recommending such things on occasion to
your customers too).
So if you have things in core you really have to provide a stable API
over a number of releases or in practise - years.
And depending on how the API changed it might not even be possible to
stay backwards compatible so:

Maybe there isn't a good answer.

Indeed there might not be one ...

Stefan

Why is ltree still in contrib? What prevents it from being in core?

Nothing. But I don't see any advantage of placing it in core - it changes
nothing in SQL, API or feature. Moving tsearch2 into core allows to manage
configuration with nice SQL API, using SysCache, automatical rereading
dictionaries files etc.

IMHO, common problem for hstore/ltree/tsearch2 is a problem with public
namespace is dump/restore. Users usually dump the whole db and restore it in new
version. Even we make new version of contrib module with the same function, new
function/feature will not be accessible.

--
Teodor Sigaev E-mail: teodor@sigaev.ru
WWW: http://www.sigaev.ru/

FWIW:

* Better packaging support, eg make it easier to add/remove an extension
module and control how pg_dump deals with it. We talked about that
awhile back but nobody did anything with the ideas.

+1

* Better documentation for the contrib modules; some of them are
reasonably well doc'd now, but many are not, and in almost all cases
it's only plain text not SGML.

+1

* Better advertising, for instance make the contrib documentation
available on the website (which probably requires SGML conversion
to happen first...)

+1

RT

--
Richard Troy, Chief Scientist
Science Tools Corporation
510-924-1363 or 202-747-1263
rtroy@ScienceTools.com, http://ScienceTools.com/

I don't think two releases from API change to API change is enough -
postgresql is running larger and larger databases by now and I expect
people to upgrade less often in the future (and iirc you already said
something along the lines of recommending such things on occasion to
your customers too).
So if you have things in core you really have to provide a stable API
over a number of releases or in practise - years.

Two releases is approximately 3 years. With the exception of 8.3.

Joshua D. Drake

And depending on how the API changed it might not even be possible to
stay backwards compatible so:

Maybe there isn't a good answer.

Indeed there might not be one ...

Stefan

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

This might be a good idea, but it's hardly transparent; it can be
counted on to break the applications of just about everyone using those
modules today.

Hmm, can we make separate schema for all contib modules and include it in
default search_path? It will not touchs most users.

--
Teodor Sigaev E-mail: teodor@sigaev.ru
WWW: http://www.sigaev.ru/

Teodor Sigaev wrote:

Why is ltree still in contrib? What prevents it from being in core?

Nothing. But I don't see any advantage of placing it in core - it
changes nothing in SQL, API or feature. Moving tsearch2 into core allows
to manage configuration with nice SQL API, using SysCache, automatical
rereading dictionaries files etc.

The advantage is that *users* don't know ltree exists and when they find
out they term it, "User contributed","Non official","Unsupported"
because it is in contrib.

I recognize that this does play into Tom's extremely valid arguments on
modules/plugins and correctly documentation and advertising.

Sincerely,

Joshua D. Drake

IMHO, common problem for hstore/ltree/tsearch2 is a problem with public
namespace is dump/restore. Users usually dump the whole db and restore
it in new version. Even we make new version of contrib module with the
same function, new function/feature will not be accessible.

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Teodor Sigaev wrote:

This might be a good idea, but it's hardly transparent; it can be
counted on to break the applications of just about everyone using those
modules today.

Hmm, can we make separate schema for all contib modules and include it
in default search_path? It will not touchs most users.

Oh now that is interesting... something like a contrib schema that
everything is loaded into? That would would work nicely and removes a
lot of problems.

People can use the modules if they want, they don't have to enable the
functions but they aren't in the way of their app or db either.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Joshua D. Drake wrote:

Teodor Sigaev wrote:

This might be a good idea, but it's hardly transparent; it can be
counted on to break the applications of just about everyone using those
modules today.

Hmm, can we make separate schema for all contib modules and include it
in default search_path? It will not touchs most users.

Oh now that is interesting... something like a contrib schema that
everything is loaded into? That would would work nicely and removes a
lot of problems.

People can use the modules if they want, they don't have to enable the
functions but they aren't in the way of their app or db either.

I think I'd rather have one schema per module than one for all of
contrib, but maybe the addition to the search path makes a single schema
more sensible.

Another approach to backwards compatibility issue, perhaps, would have 2
install scripts per module, one that installs stuff in the public schema
for legacy purposes, and one that uses a dedicated schema.

cheers

andrew