Extensions, patch v16
Hi,
Please find attached revision 16 of the extension patch, with the
following additions over the previous one:
- added documentation as proposed by David Wheeler, you can browse it
online if reading SGML ain't your thing
http://pgsql.tapoueh.org/extensions/doc/html/extend-extension.html
- renamed pg_extensions system view to pg_available_extensions
- add ALTER EXTENSION SET SCHEMA command
so that I'm going to mark the patch for that as rejected, where it's
in fact deprecated: it turned out we already need it as part of the
main patch here
- add support for 'relocatable' boolean property in the control file,
as discussed on list
this controls what happens at create extension time, by doing a
relocation of the extension objects when the extension is relocatable
and the asked schema isn't the first of the search_path (you can't
relocate an object to a schema where it already is)
when the extension is not relocatable, the mechanism used is the
@extschema@ replacement in the script so that the user still has a
say, but at create time only
- adapt all contrib
the SET search_path TO public, which became TO @extschema@, is now
removed
2 contribs are not relocatable: adminpack installs its functions
directly into pg_catalog and earthdistance depends on cube.
As we said we won't support extension dependencies in the first cut,
there's no guard in the dependency recursion that would allow us not
to relocate cube objects when relocating earthdistance. The easiest
solution seemed to me to mark the earthdistance extension as not
relocatable.
- nothing is done for the psql commands \dx and \dx+, here's an idea:
\dx lists only installed extensions
\dx+ <extension> lists the objects, calling pg_extension_objects()
\dX lists available extensions (and installed too)
- we still depend on extension authors providing a control file. Do we
want to spend some efforts on trying to get rid of this file? I know
David desperately want to, but that's at the cost of making it much
harder to manage more than one extension in a single directory, for
once, and the Makefile mechanisms to make than happen (include a rule
depending on the presence of some variables, keep track of it for the
cleaning, etc) doesn't seem to me to worth it.
- this patch still includes the current version of pg_execute_from_file
patch, and the next thing I'm going to do is have it change
underneath us to remove some SQL visible functions that shouldn't
exists. Please don't focus on that part of the changes.
Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Attachments:
extension.v16.patch.gzapplication/octet-streamDownload+1-0
On Dec 9, 2010, at 12:34 PM, Dimitri Fontaine wrote:
- add support for 'relocatable' boolean property in the control file,
as discussed on listthis controls what happens at create extension time, by doing a
relocation of the extension objects when the extension is relocatable
and the asked schema isn't the first of the search_path (you can't
relocate an object to a schema where it already is)when the extension is not relocatable, the mechanism used is the
@extschema@ replacement in the script so that the user still has a
say, but at create time only
This still isn't ideal, but I think it's a big improvement. Thanks.
- nothing is done for the psql commands \dx and \dx+, here's an idea:
\dx lists only installed extensions
\dx+ <extension> lists the objects, calling pg_extension_objects()
\dX lists available extensions (and installed too)
+1 I think that's much more like existing psql commands.
- we still depend on extension authors providing a control file. Do we
want to spend some efforts on trying to get rid of this file? I know
David desperately want to, but that's at the cost of making it much
harder to manage more than one extension in a single directory, for
once, and the Makefile mechanisms to make than happen (include a rule
depending on the presence of some variables, keep track of it for the
cleaning, etc) doesn't seem to me to worth it.
I don't think it makes it any harder to manage multiple extension in a single directory because one can create the control file explicitly (or perhaps rely on .control.in for that), just as they do now. Everyone else can do less work.
So:
* If $extension.control.in exists, use that
* If it doesn't, generate $extension.control from the Makefile variables
* Always remove $extension.control in the `clean` targets
Best,
David
"David E. Wheeler" <david@kineticode.com> writes:
On Dec 9, 2010, at 12:34 PM, Dimitri Fontaine wrote:
- add support for 'relocatable' boolean property in the control file,
as discussed on listThis still isn't ideal, but I think it's a big improvement. Thanks.
Glad you like it :) If you see any way to manage that better, please do
tell. Just be sure to review the past 18 months of on-list discussion
about the topic before to go thinking extension vs search_path is easy
to solve, or even possible to solve.
\dx lists only installed extensions
\dx+ <extension> lists the objects, calling pg_extension_objects()
\dX lists available extensions (and installed too)+1 I think that's much more like existing psql commands.
Good, I'll have that in the next patch version, waiting for until your
review of the new one :)
So:
* If $extension.control.in exists, use that
* If it doesn't, generate $extension.control from the Makefile variables
What if $extension.control exists? Is it a byproduct of the .in file
from previous `make` run or a user file? What if we have both the .in
and the make variable because people are confused? Or both the make
variables and a .control and not .control.in? Etc...
* Always remove $extension.control in the `clean` targets
Hell no, as you can bypass the .in mechanism and provide directly the
.control file.
Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Dimitri Fontaine <dimitri@2ndQuadrant.fr> writes:
"David E. Wheeler" <david@kineticode.com> writes:
What if $extension.control exists? Is it a byproduct of the .in file
from previous `make` run or a user file? What if we have both the .in
and the make variable because people are confused? Or both the make
variables and a .control and not .control.in? Etc...
* Always remove $extension.control in the `clean` targets
Hell no, as you can bypass the .in mechanism and provide directly the
.control file.
Are there any actual remaining use-cases for that sed step? It's
certainly vestigial as far as the contrib modules are concerned:
it would be simpler and more readable to replace MODULE_PATHNAME with
$libdir in the sources. Unless somebody can point to a real-world
use-case, I'd just as soon get rid of the .in files altogether while
we're having this flag day.
regards, tom lane
Tom Lane <tgl@sss.pgh.pa.us> writes:
Are there any actual remaining use-cases for that sed step?
The goal here is to allow extension authors to maintain their version
number in the Makefile rather than in the Makefile and in the control
file separately. Having the same version number in more than one place
never eases maintenance.
Oh and in PostgreSQL sources cases, that would add like 36 spots where
to manually maintain our major version string. I'm not eager to do that.
Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Dimitri Fontaine <dimitri@2ndQuadrant.fr> writes:
Tom Lane <tgl@sss.pgh.pa.us> writes:
Are there any actual remaining use-cases for that sed step?
The goal here is to allow extension authors to maintain their version
number in the Makefile rather than in the Makefile and in the control
file separately. Having the same version number in more than one place
never eases maintenance.
Why is it in the makefile at all? If the makefile does need to know it,
why don't we have it scrape the number out of the control file? Or even
more to the point, since when do we need version numbers in extensions?
regards, tom lane
Tom Lane <tgl@sss.pgh.pa.us> writes:
Why is it in the makefile at all? If the makefile does need to know it,
why don't we have it scrape the number out of the control file? Or even
more to the point, since when do we need version numbers in extensions?
It's in the Makefile so that you find it in the control file later, then
in the extension catalog. We need the version number just because I'm
not able to name a single software that's not letting you know about its
version number once installed.
Well in fact I know about one, and I wish the situation would be quite
different there.
Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Dimitri Fontaine <dimitri@2ndQuadrant.fr> writes:
Tom Lane <tgl@sss.pgh.pa.us> writes:
Why is it in the makefile at all? If the makefile does need to know it,
why don't we have it scrape the number out of the control file? Or even
more to the point, since when do we need version numbers in extensions?
It's in the Makefile so that you find it in the control file later, then
in the extension catalog.
This doesn't answer my question of why it couldn't be done the other
way. Why does the makefile need to know it? If it does need to know
it, couldn't it get it out of the control file instead of vice versa?
We need the version number just because I'm
not able to name a single software that's not letting you know about its
version number once installed.
I'm not convinced that this is actually a requirement, or that doing it
this specific way is a good solution. In particular, keeping the
version number in the system catalogs seems pretty dubious. The common
method for upgrading an already-installed contrib module just involves
dropping in a new .so --- that's not going to change the system
catalogs. It would likely be better to keep the version ID inside the
.so file.
regards, tom lane
On Dec 10, 2010, at 12:26 AM, Dimitri Fontaine wrote:
What if $extension.control exists? Is it a byproduct of the .in file
from previous `make` run or a user file? What if we have both the .in
and the make variable because people are confused? Or both the make
variables and a .control and not .control.in? Etc...
There are ways to deal with those issue, I'm sure.
* Always remove $extension.control in the `clean` targets
Hell no, as you can bypass the .in mechanism and provide directly the
.control file.
I'm saying disallow the .control file, only allow the control.in file.
David
On Dec 10, 2010, at 7:32 AM, Tom Lane wrote:
Are there any actual remaining use-cases for that sed step? It's
certainly vestigial as far as the contrib modules are concerned:
it would be simpler and more readable to replace MODULE_PATHNAME with
$libdir in the sources. Unless somebody can point to a real-world
use-case, I'd just as soon get rid of the .in files altogether while
we're having this flag day.
I've made extensive use of them in pgTAP, but they don't depend on PGXS's doing its bit. So no. Unless we require control.in and not .control.
Best,
David
On Fri, Dec 10, 2010 at 12:30 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I'm not convinced that this is actually a requirement, or that doing it
this specific way is a good solution. In particular, keeping the
version number in the system catalogs seems pretty dubious. The common
method for upgrading an already-installed contrib module just involves
dropping in a new .so --- that's not going to change the system
catalogs. It would likely be better to keep the version ID inside the
.so file.
This is an interesting point. There are really two things here: the
.so version, and the version of the system catalog entries. For
example, imagine that an extension provides a single function, called
foo(). So we load up the .so and CREATE FUNCTION statement to match.
Later, the extension is so successful that the author writes a second
function, bar(). The new .so can (at least possibly) be used with the
old schema definitions, but the new schema definitions aren't
compatible with the old .so. The logical upgrade process is to swap
out the .so first, and then add update the catalog definitions.
On the other hand, if you were dropping a deprecated function, you'd
need to do the steps in reverse order.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
On Fri, Dec 10, 2010 at 12:30 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
... In particular, keeping the
version number in the system catalogs seems pretty dubious. �The common
method for upgrading an already-installed contrib module just involves
dropping in a new .so --- that's not going to change the system
catalogs. �It would likely be better to keep the version ID inside the
.so file.
This is an interesting point. There are really two things here: the
.so version, and the version of the system catalog entries.
True. Consider a situation like an RPM upgrade: it's going to drop in a
new .so version, *and nothing else*. It's pure fantasy to imagine that
the RPM script is going to find all your databases and execute some SQL
commands against them. Since a large number of bug-fix cases do require
only a .so update, not being able to track the .so version seems like
it's missing most of the argument for having version tracking at all.
(In the RPM case, the RPM infrastructure would be able to tell you
which version you had installed, so I'm not sold that PG needs to
duplicate that.)
regards, tom lane
On Dec 10, 2010, at 10:20 AM, Tom Lane wrote:
True. Consider a situation like an RPM upgrade: it's going to drop in a
new .so version, *and nothing else*. It's pure fantasy to imagine that
the RPM script is going to find all your databases and execute some SQL
commands against them. Since a large number of bug-fix cases do require
only a .so update, not being able to track the .so version seems like
it's missing most of the argument for having version tracking at all.
Sometimes there will be changes to the SQL, too. How does that work with CREATE EXTENSION? Do I install the upgrade, then run CREATE EXTENSION to get the latest SQL script to run? But then all the objects already exist…
Best,
David
Tom Lane <tgl@sss.pgh.pa.us> writes:
This doesn't answer my question of why it couldn't be done the other
way. Why does the makefile need to know it? If it does need to know
it, couldn't it get it out of the control file instead of vice versa?
Well the Makefile support is just a facility to fill in the control file
automatically for you, on the grounds that you're probably already
maintaining your version number in the Makefile. Or that it's easy to
get it there, as in:
EXTVERSION = $(shell dpkg-parsechangelog | awk -F '[:-]' '/^Version:/ { print substr($$2, 2) }')
That comes from a real world example that's yet to be adapted to being
an extension in 9.1, but still:
https://github.com/dimitri/pgfincore/blob/debian/Makefile
I'm not convinced that this is actually a requirement, or that doing it
this specific way is a good solution. In particular, keeping the
version number in the system catalogs seems pretty dubious. The common
method for upgrading an already-installed contrib module just involves
dropping in a new .so --- that's not going to change the system
catalogs. It would likely be better to keep the version ID inside the
.so file.
Upgrade are left for a future patch, did we decide. Still, it seems to
me that we will support some upgrade scripts so that author can decide
what to do knowing current and next version, and yes, knowing that the
module has already been taken care of by the OS-level packaging.
That means some extensions upgrades will break the database between the
OS-level package upgrade and the sql upgrade (support to come), but in
my experience that's seldom the case. And not by chance.
So in the case that only the module (.so) needs upgrading, we would
still provide for an upgrade path in the script / sql support so that
the version number has a chance of being upgraded too. As you say in
another mail, of course, the OS packaging system will not forcibly be
willing to care for that all by itself. I can imagine debian offering
the choice to the users and acting accordingly, though.
Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On Dec 10, 2010, at 11:28 AM, Dimitri Fontaine wrote:
Well the Makefile support is just a facility to fill in the control file
automatically for you, on the grounds that you're probably already
maintaining your version number in the Makefile. Or that it's easy to
get it there, as in:EXTVERSION = $(shell dpkg-parsechangelog | awk -F '[:-]' '/^Version:/ { print substr($$2, 2) }')
That comes from a real world example that's yet to be adapted to being
an extension in 9.1, but still:
I use that in pgTAP, too (line 23):
https://github.com/theory/pgtap/blob/master/Makefile
But I don't need core to support that. Frankly, if we're not going to generate the control file from Makefile variables, then I'd rather not have any control file Makefile variables at all.
Upgrade are left for a future patch, did we decide. Still, it seems to
me that we will support some upgrade scripts so that author can decide
what to do knowing current and next version, and yes, knowing that the
module has already been taken care of by the OS-level packaging.
Yeah, this will be needed ASAP.
Best,
David
"David E. Wheeler" <david@kineticode.com> writes:
On Dec 10, 2010, at 11:28 AM, Dimitri Fontaine wrote:
Upgrade are left for a future patch, did we decide. Still, it seems to
me that we will support some upgrade scripts so that author can decide
what to do knowing current and next version, and yes, knowing that the
module has already been taken care of by the OS-level packaging.
Yeah, this will be needed ASAP.
I don't mind if we don't have an implementation of upgrade cases in
hand. But we had better have a design in hand, to make sure what we're
doing now doesn't foreclose upgrade cases.
regards, tom lane
Dimitri Fontaine <dimitri@2ndQuadrant.fr> writes:
Tom Lane <tgl@sss.pgh.pa.us> writes:
This doesn't answer my question of why it couldn't be done the other
way. Why does the makefile need to know it? If it does need to know
it, couldn't it get it out of the control file instead of vice versa?
Well the Makefile support is just a facility to fill in the control file
automatically for you, on the grounds that you're probably already
maintaining your version number in the Makefile.
Why would you choose to maintain it in the Makefile? In most cases
makefiles are the least likely thing to be changing during a minor
update. I would think that the right place for it is in the C code
(if we're trying to version .so files) or the .sql file, if we're trying
to version the SQL objects. In particular, if the only reason the
makefile needs to know it is to inject it into the control file, it
seems completely silly to not just maintain it in the control file
instead.
regards, tom lane
On Dec 10, 2010, at 11:47 AM, Tom Lane wrote:
Why would you choose to maintain it in the Makefile? In most cases
makefiles are the least likely thing to be changing during a minor
update. I would think that the right place for it is in the C code
(if we're trying to version .so files) or the .sql file, if we're trying
to version the SQL objects. In particular, if the only reason the
makefile needs to know it is to inject it into the control file, it
seems completely silly to not just maintain it in the control file
instead.
+1
David
Tom Lane <tgl@sss.pgh.pa.us> writes:
Why would you choose to maintain it in the Makefile? In most cases
makefiles are the least likely thing to be changing during a minor
update.
I must have a packager skewed view of things here, but ok, point noted.
I would think that the right place for it is in the C code
(if we're trying to version .so files) or the .sql file, if we're trying
to version the SQL objects. In particular, if the only reason the
makefile needs to know it is to inject it into the control file, it
seems completely silly to not just maintain it in the control file
instead.
If we are after the a bare feature set, sure, the Makefile facility is
only about trying to be nice to the user. I accept that you think it's
counter productive rather than helping.
In the next patch, I will rename the control files to be straight
.control files, remove the implicit rule in the pgxs.mk file, remove the
note in the documentation and apply the following:
sed -i 's/EXTVERSION/9.1devel/' contrib/*/*.control
Or do we want contrib's specific version numbers that are not all the
same as the current PostgreSQL version number?
On to your question about the upgrade design, in order not to paint
ourselves into a corner. What I now have in mind is the following:
When there's an extension upgrade the user will have to install the new
files (.so, .sql, .control) and run an upgrade command in his databases:
ALTER EXTENSION pair UPGRADE;
The version we upgrade from is known from the catalog, the version we
upgrade to is read in the control file. So we are able to call the sql
script and offer a way for it to know about the versions. The simplest
way seems to be a new pair of functions:
pg_extension_upgrade() returns bool
pg_extension_versions() returns table(current text, next text)
Those are to be run only from the extension's script.
The first returns false when the user did CREATE EXTENSION and true when
the user did ALTER EXTENSION UPGRADE, which are the only two commands
that will run the script.
The second will return the versions we detailed above, and the
extension's author is free to compare them however he wants to and
decide what to do now. It's cool that we have DO blocks here, and
pg_execute_sql_file() to offer the same facility as \i for psql scripts.
Of course if calling the script succeeds, then the version number in the
pg_extension catalog is changed to the "next" one.
Now, it would be better if it were easy to compare version numbers, for
example with a -core datatype that handles that. Do we already want to
open this can of worms?
Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On 12/10/10 12:17 PM, Dimitri Fontaine wrote:
Or do we want contrib's specific version numbers that are not all the
same as the current PostgreSQL version number?
I think that each contrib needs its own version numbers. The reason
being that most minor updates don't touch contrib.
Also, once extensions and pgxn are operating full swing, I see contrib
going away anyway ...
--
-- Josh Berkus
PostgreSQL Experts Inc.
http://www.pgexperts.com
