Modifying and solidifying contrib

"Joshua D. Drake" <jd@commandprompt.com> writes:

With all the recent discussion on contrib modules etc.. I would like to
offer the following suggestion.

AFAICT you're proposing an entirely cosmetic reclassification of /contrib.
Aside from the difficulty of getting agreement on which ones should be
"in" and which "out", what does that really buy us? The thing that
would be really useful to work on is developing a concrete
representation of a "module" that pg_dump would understand, so that you
could e.g. tell it to omit btree_gist from a dump. It might be that
just segregating a contrib module into its own schema would be
sufficient, or maybe that wouldn't work well because of making people
need to deal with long search paths.

regards, tom lane

Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

With all the recent discussion on contrib modules etc.. I would like to
offer the following suggestion.

AFAICT you're proposing an entirely cosmetic reclassification of /contrib.

For the most part yes. Perception is reality and all. The exception
being that modules are installed by default.

Aside from the difficulty of getting agreement on which ones should be
"in" and which "out", what does that really buy us?

True and I am sure that people with more time to waste than I would like
to spend days creating a thread that is 500 responses long on why their
particular module should be a module or a contrib.

The thing that
would be really useful to work on is developing a concrete
representation of a "module" that pg_dump would understand, so that you
could e.g. tell it to omit btree_gist from a dump.

I am offering what I can.

It might be that
just segregating a contrib module into its own schema would be
sufficient, or maybe that wouldn't work well because of making people
need to deal with long search paths.

I do like the contrib schema idea would could easily be melded into this
proposal. I don't like the idea that all of contrib would automatically
be included which is one of the reasons I wanted to split this up.

Sincerely,

Joshua D. Drake

regards, tom lane

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

On Sat, Jan 27, 2007 at 08:59:47AM -0800, Joshua D. Drake wrote:

Hello,

With all the recent discussion on contrib modules etc.. I would like to
offer the following suggestion. I am willing to do a good portion of the
work myself and I can get it done before feature freeze. I will need
help with the global make file stuff however so that is one dependency.

Add directory /modules

Not so great. SQL:2003 has a special meaning for the word "module."

Cheers,
D
--
David Fetter <david@fetter.org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

David Fetter wrote:

On Sat, Jan 27, 2007 at 08:59:47AM -0800, Joshua D. Drake wrote:

Hello,

With all the recent discussion on contrib modules etc.. I would like to
offer the following suggestion. I am willing to do a good portion of the
work myself and I can get it done before feature freeze. I will need
help with the global make file stuff however so that is one dependency.

Add directory /modules

Not so great. SQL:2003 has a special meaning for the word "module."

Yeah I saw mention of that in another thread, but I really didn't like
the word plugins. Do you have another thought? Extensions?

Sincerely,

Joshua D. Drake

Cheers,
D

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

On Jan 28, 2007, at 11:25 , Joshua D. Drake wrote:

David Fetter wrote:

Not so great. SQL:2003 has a special meaning for the word "module."

Yeah I saw mention of that in another thread, but I really didn't like
the word plugins. Do you have another thought? Extensions?

"Extensions" would tie in nicely with its common use in the docs,
especially wrt pgxs:

PostgreSQL can be extended by the user in many ways ...
PostgreSQL also accepts escape string constants, which are an
extension to the SQL standard
To use the infrastructure for your extension ...
Here is an example that builds an extension module ...
They test standard SQL operations as well as the extended
capabilities of PostgreSQL.

The pgxs docs do use "module" as well, but as previously mentioned
module already has a particular meaning in the spec.

Michael Glaesemann
grzm seespotcode net

PostgreSQL can be extended by the user in many ways ...
PostgreSQL also accepts escape string constants, which are an extension
to the SQL standard
To use the infrastructure for your extension ...
Here is an example that builds an extension module ...
They test standard SQL operations as well as the extended capabilities
of PostgreSQL.

The pgxs docs do use "module" as well, but as previously mentioned
module already has a particular meaning in the spec.

So what are we thinking here? Along with my suggestion of extensions /
contrib that we modify initdb to load an extensions schema with all
extensions into template1?

Sincerely,

Joshua D. Drake

Michael Glaesemann
grzm seespotcode net

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

"Joshua D. Drake" <jd@commandprompt.com> writes:

So what are we thinking here? Along with my suggestion of extensions /
contrib that we modify initdb to load an extensions schema with all
extensions into template1?

No, I don't think so. If you do that it's effectively moving all that
stuff into core, especially if you haven't provided a way to turn it off.
I don't plan to hold still for any back-door avenues to suddenly turning
large parts of contrib into core code.

regards, tom lane

Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

So what are we thinking here? Along with my suggestion of extensions /
contrib that we modify initdb to load an extensions schema with all
extensions into template1?

No, I don't think so. If you do that it's effectively moving all that
stuff into core, especially if you haven't provided a way to turn it off.

O.k. any thoughts there? What if we didn't make the extensions schema
PUBLIC? Meaning that explicits rights would have to be given for the
extensions to be used by anyone but a super user?

Obviously the initdb switch could also be selective:

initdb --enable-extensions

?

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

On Sat, Jan 27, 2007 at 09:49:25PM -0800, Joshua D. Drake wrote:

Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

So what are we thinking here? Along with my suggestion of
extensions / contrib that we modify initdb to load an extensions
schema with all extensions into template1?

No, I don't think so. If you do that it's effectively moving all
that stuff into core, especially if you haven't provided a way to
turn it off.

O.k. any thoughts there? What if we didn't make the extensions
schema PUBLIC? Meaning that explicits rights would have to be given
for the extensions to be used by anyone but a super user?

Whether they're auto-installable or not, I'd vote for putting each one
in its own schema by default. That way, people can get an excellent
idea just by looking at what schemas exist what extensions are
installed in a given DB, and it's fairly straight-forward to remove
the thing simply by dropping the schema cascade.

Obviously the initdb switch could also be selective:

initdb --enable-extensions

If it were an initdb switch, I'd want to have something more like

--enable-extension=earthdistance

Cheers,
D
--
David Fetter <david@fetter.org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

On 1/27/07, Joshua D. Drake <jd@commandprompt.com> wrote:

Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

So what are we thinking here? Along with my suggestion of extensions /
contrib that we modify initdb to load an extensions schema with all
extensions into template1?

No, I don't think so. If you do that it's effectively moving all that
stuff into core, especially if you haven't provided a way to turn it

off.

O.k. any thoughts there? What if we didn't make the extensions schema
PUBLIC? Meaning that explicits rights would have to be given for the
extensions to be used by anyone but a super user?

Obviously the initdb switch could also be selective:

initdb --enable-extensions

?

Sincerely,

Joshua D. Drake

I've been listening in on this discussion and it is closely tied to an
extension that I would like to try. The only thing holding me back is that
I am still far to green on the source code to actually propose the work I
would like to do.

It makes sense the you would want to include the contrib into the install,
but I agree with Tom this isn't something you would want in template1.
plpgsql is not even in template1 IIRC. It makes more sense to compile the
libraries and have an activation module that can be represented in a
pg_dump. Maybe the best way to do this would be to place the SQL that
modifies system catalog into a compiled library and call this as a
function. The function call could then be placed into the pg_dump if
needed.

Here is where this ties into what I am working on proposing. I have several
custom C functions in a library that I need in a restore/build. It seems
like at least a couple times every year a new cluster is built and the Admin
forgets to compile the custom functions. He builds the DB and then
"something doesn't work".

The basic idea of my proposal would be to have a function that checks for
the existence of C libraries. The next piece I would need would be a
mechanism for psql to halt the restore SQL script if these checks failed.
Finally I would need to add a switch to tell pg_dump to do the checks first
thereby causing the restore to fail, hopefully with a nice message that
clues the admin that something is wrong.

Yes all of this work so I don't get called after hours yet here I am working
on the fix after hours. :-)

It seems to me like this might be useful in incorporating an
extension/module/whatever into Postgres.

--
Kevin Barnard

"Great Beauty, great strength, and great Riches,
are really and truly of no great Use;
a right Heart exceeds all." -- Benjamin Franklin

Joshua D. Drake wrote:

Add directory /modules

Moving stuff around in the source code probably won't matter much at all
since few users will actually look into the source code. What we need
is better presentation and accessibility of the installed items, for
example documentation and installation and upgrade handling.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Kevin Barnard wrote:

The basic idea of my proposal would be to have a function that checks
for the existence of C libraries. The next piece I would need would
be a mechanism for psql to halt the restore SQL script if these
checks failed. Finally I would need to add a switch to tell pg_dump
to do the checks first thereby causing the restore to fail, hopefully
with a nice message that clues the admin that something is wrong.

You already get this today. If you create a C function and the library
doesn't exist, you get an error, and if you use the right flags for
psql or pg_restore, the restoration aborts.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

David Fetter wrote:

On Sat, Jan 27, 2007 at 09:49:25PM -0800, Joshua D. Drake wrote:

Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

So what are we thinking here? Along with my suggestion of
extensions / contrib that we modify initdb to load an extensions
schema with all extensions into template1?

No, I don't think so. If you do that it's effectively moving all
that stuff into core, especially if you haven't provided a way to
turn it off.

O.k. any thoughts there? What if we didn't make the extensions
schema PUBLIC? Meaning that explicits rights would have to be given
for the extensions to be used by anyone but a super user?

Whether they're auto-installable or not, I'd vote for putting each one
in its own schema by default. That way, people can get an excellent
idea just by looking at what schemas exist what extensions are
installed in a given DB, and it's fairly straight-forward to remove
the thing simply by dropping the schema cascade.

Well to me that gets a little messy. I mean:

pg_catalog,public,<user schemas>,xml2,ltree (just to get a could
functions?) etc...

Obviously the initdb switch could also be selective:

initdb --enable-extensions

If it were an initdb switch, I'd want to have something more like

--enable-extension=earthdistance

And have to parse for each extension?

Sincerely,

Joshua D. Drake

Cheers,
D

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

On Sun, Jan 28, 2007 at 10:10:14AM -0800, Joshua D. Drake wrote:

David Fetter wrote:

On Sat, Jan 27, 2007 at 09:49:25PM -0800, Joshua D. Drake wrote:

Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

So what are we thinking here? Along with my suggestion of
extensions / contrib that we modify initdb to load an
extensions schema with all extensions into template1?

No, I don't think so. If you do that it's effectively moving
all that stuff into core, especially if you haven't provided a
way to turn it off.

O.k. any thoughts there? What if we didn't make the extensions
schema PUBLIC? Meaning that explicits rights would have to be
given for the extensions to be used by anyone but a super user?

Whether they're auto-installable or not, I'd vote for putting each
one in its own schema by default. That way, people can get an
excellent idea just by looking at what schemas exist what
extensions are installed in a given DB, and it's fairly
straight-forward to remove the thing simply by dropping the schema
cascade.

Well to me that gets a little messy. I mean:

pg_catalog,public,<user schemas>,xml2,ltree (just to get a could
functions?) etc...

Not as messy as trying to drop or re-create a package when there are
already 500 functions in the public schema.

Obviously the initdb switch could also be selective:

initdb --enable-extensions

If it were an initdb switch, I'd want to have something more like

--enable-extension=earthdistance

And have to parse for each extension?

I don't see this as a big problem.

Cheers,
D
--
David Fetter <david@fetter.org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

On Sun, Jan 28, 2007 at 04:52:27PM -0500, Andrew Dunstan wrote:

Michael Glaesemann wrote:

On Jan 28, 2007, at 11:25 , Joshua D. Drake wrote:

David Fetter wrote:

Not so great. SQL:2003 has a special meaning for the word "module."

Yeah I saw mention of that in another thread, but I really didn't like
the word plugins. Do you have another thought? Extensions?

"Extensions" would tie in nicely with its common use in the docs,
especially wrt pgxs:

I don't mind this term, BUT, what we need to get across is not just that
these are extensions, but that they are *standard* extensions, supplied
with PostgreSQL core code and supported by the PostgreSQL core team.
This would be analogous with, say, the standard perl modules (like
Exporter or IO::Handle) that come with the standard perl source
distribution. If we can get that idea across then we might lower the
resistance of people like hosting providers to loading them.

Integrating their docs into the standard PostgreSQL SGML (or XML,
should we go there) docs would go a long, long way toward helping with
this.

Cheers,
D
--
David Fetter <david@fetter.org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

Michael Glaesemann wrote:

On Jan 28, 2007, at 11:25 , Joshua D. Drake wrote:

David Fetter wrote:

Not so great. SQL:2003 has a special meaning for the word "module."

Yeah I saw mention of that in another thread, but I really didn't like
the word plugins. Do you have another thought? Extensions?

"Extensions" would tie in nicely with its common use in the docs,
especially wrt pgxs:

I don't mind this term, BUT, what we need to get across is not just that
these are extensions, but that they are *standard* extensions, supplied
with PostgreSQL core code and supported by the PostgreSQL core team.
This would be analogous with, say, the standard perl modules (like
Exporter or IO::Handle) that come with the standard perl source
distribution. If we can get that idea across then we might lower the
resistance of people like hosting providers to loading them.

cheers

andrew

Well to me that gets a little messy. I mean:

pg_catalog,public,<user schemas>,xml2,ltree (just to get a could
functions?) etc...

Not as messy as trying to drop or re-create a package when there are
already 500 functions in the public schema.

I am not sure I understand the correlation. I am not suggesting we
install anything into public. They would all go into a single additional
schema called extensions or some such.

Obviously the initdb switch could also be selective:

initdb --enable-extensions

If it were an initdb switch, I'd want to have something more like

--enable-extension=earthdistance

And have to parse for each extension?

I don't see this as a big problem.

Well I am not really interesting in this. Someone else is welcome to try
that.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

I don't mind this term, BUT, what we need to get across is not just that
these are extensions, but that they are *standard* extensions, supplied
with PostgreSQL core code and supported by the PostgreSQL core team.
This would be analogous with, say, the standard perl modules (like
Exporter or IO::Handle) that come with the standard perl source
distribution. If we can get that idea across then we might lower the
resistance of people like hosting providers to loading them.

Furthering the perception is reality rule perhaps the schema should be
called std_ext?

Integrating their docs into the standard PostgreSQL SGML (or XML,
should we go there) docs would go a long, long way toward helping with
this.

Agreed.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

On Sun, Jan 28, 2007 at 01:58:38PM -0800, Joshua D. Drake wrote:

Well to me that gets a little messy. I mean:

pg_catalog,public,<user schemas>,xml2,ltree (just to get a could
functions?) etc...

Not as messy as trying to drop or re-create a package when there
are already 500 functions in the public schema.

I am not sure I understand the correlation. I am not suggesting we
install anything into public. They would all go into a single
additional schema called extensions or some such.

I don't think "all or nothing" is a good way to do this. 500
functions in a schema called extensions isn't much more helpful than
500 in public. There's a reason namespaces were invented long ago,
and this is classic use case for same. :)

Obviously the initdb switch could also be selective:

initdb --enable-extensions

If it were an initdb switch, I'd want to have something more
like

--enable-extension=earthdistance

And have to parse for each extension?

I don't see this as a big problem.

Well I am not really interesting in this. Someone else is welcome to
try that.

It's really not hard, even for a C n00b like me. :)

The only trick here, and again it's not a huge one, is to modify the
postgresql.conf to have the correct default search_path.

Cheers,
D
--
David Fetter <david@fetter.org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!