rmgr hooks (v2)
Latest version of rmgr hooks patch for later review in current
commitfest.
(Minor update to CVS HEAD).
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Simon Riggs <simon@2ndQuadrant.com> wrote:
Latest version of rmgr hooks patch for later review in current
commitfest.
(Minor update to CVS HEAD).
It doesn't work on Window (EXEC_BACKEND platform) because
shared_preload_libraries are not loaded in startup process.
So, there are no timing to initialize rmgr_hook.
I have the same problem in contrib/pg_stat_statements.
The both patches requires the following modifications.
diff -cprN HEAD/src/backend/storage/lmgr/proc.c pg_stat_statements/src/backend/storage/lmgr/proc.c
*** HEAD/src/backend/storage/lmgr/proc.c Mon Nov 3 06:24:52 2008
--- pg_stat_statements/src/backend/storage/lmgr/proc.c Tue Dec 2 16:57:33 2008
*************** InitAuxiliaryProcess(void)
*** 381,386 ****
--- 381,390 ----
if (MyProc != NULL)
elog(ERROR, "you already exist");+ #ifdef EXEC_BACKEND
+ process_shared_preload_libraries();
+ #endif
+
/*
* We use the ProcStructLock to protect assignment and releasing of
* AuxiliaryProcs entries.Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center
On Thu, 2008-12-18 at 14:30 +0900, ITAGAKI Takahiro wrote:
Simon Riggs <simon@2ndQuadrant.com> wrote:
Latest version of rmgr hooks patch for later review in current
commitfest.
(Minor update to CVS HEAD).It doesn't work on Window (EXEC_BACKEND platform) because
shared_preload_libraries are not loaded in startup process.
So, there are no timing to initialize rmgr_hook.
Ah, thank you. How amusing that I should notice that in review but not
in my own patch.
I won't add this just yet, since it sounds like it will be added by your
patch. But I'll keep track of this just in case.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Wow, you are really shooting out a lot of good stuff today!
---------------------------------------------------------------------------
Simon Riggs wrote:
Latest version of rmgr hooks patch for later review in current
commitfest.(Minor update to CVS HEAD).
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
[ Attachment, skipping... ]
[ Attachment, skipping... ]
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Simon Riggs wrote:
Latest version of rmgr hooks patch for later review in current
commitfest.
I'd like to reject this patch.
I've read through all the related threads again, and I just still don't
see a convincing use case for it. I think that tools that let you
introspect and modify WAL files should be written as an external
toolkit, like pglesslog. The external indexam use case doesn't impress
me either, and Tom seems to agree
(http://archives.postgresql.org/message-id/24006.1221483138@sss.pgh.pa.us).
Plus there's the version incompatibility dangers. Although I think we
could put in some safeguards and live with it, it does open new
opportunities for confusion, so I'd rather not go there without a very
convincing use case.
Regarding the example plugin included, for debugging purposes you could
just compile with WAL_DEBUG, and the plugin to suppress actions for all
but one database is clearly not ready for any real work. It only
suppresses heapam records, replaying index updates and full-page-images
as usual, and it requires that you know the Oid of the database,
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
On Wed, 2009-01-21 at 14:05 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
Latest version of rmgr hooks patch for later review in current
commitfest.I'd like to reject this patch.
...
The external indexam use case doesn't impress me either, and Tom seems to agree
(http://archives.postgresql.org/message-id/24006.1221483138@sss.pgh.pa.us).Plus there's the version incompatibility dangers. Although I think we
could put in some safeguards and live with it, it does open new
opportunities for confusion, so I'd rather not go there without a very
convincing use case.
The original design of Postgres allowed pluggable index access methods,
but that capability has not been brought forward to allow for WAL. This
patch would bridge that gap.
Right now we've got a variety of index types that are *not* flourishing
(hash, bitmap, grouped). If we allow them to develop as separate
projects, then whenever they are ready they can be used with particular
releases. You may doubt the worth of those index types but preventing
other people from building them seems strange.
Why do we have 12+ pluggable languages, but we're not allowed to write
pluggable indexes? Whatever argument you put against it being "too hard"
or dangerous or whatever *also* applies to languages. Yet experience
shows pluggability has resulted in a variety of robust and useful
language types, some that might not have been predicted (PL/Proxy, PL/R
etc). They cover a variety of users and situations. Personally, I'd like
to enable people to come up with audio, video, bioinformatics datatypes
and indexes and I definitely don't want to limit the possibilities
there.
There is no danger here for Core, only opportunity. There *is* danger in
forcing new index designers to fit them into Core - look how unusable
hash indexes are. How can we allow that functionality to continue to
exist in Core and yet block the path by which we might reasonably
correct that?
You don't want pluggable indexes, don't use 'em. But that isn't an
argument against allowing the capability for others. That line of
thought would have led us to banning pluggable languages. We should
respect the roots of this project and look for ways to enable the
advancement of database technology, not limit it to only how far we can
currently see ahead through the fog.
Plus there's the version incompatibility dangers. Although I think we
could put in some safeguards and live with it, it does open new
opportunities for confusion, so I'd rather not go there without a
very
convincing use case.
There is danger in every plugin, so not a reasonable objection. Any
badly written external module can kill the database or smash data.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
The external indexam use case doesn't impress me either, and Tom seems to agree
(http://archives.postgresql.org/message-id/24006.1221483138@sss.pgh.pa.us).
Just for correctness - there is one external index http://www.cs.purdue.edu/spgist/
--
Teodor Sigaev E-mail: teodor@sigaev.ru
WWW: http://www.sigaev.ru/
On Wed, 2009-01-21 at 14:05 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
Latest version of rmgr hooks patch for later review in current
commitfest.I'd like to reject this patch.
...
I've read through all the related threads again, and I just still don't
see a convincing use case for it. I think that tools that let you
introspect and modify WAL files should be written as an external
toolkit, like pglesslog.
The only reasonable way to examine the contents of WAL files is with
reference to a copy of the catalog that wrote them, timed *exactly* in
synchronisation with the WAL stream.
If somebody issued
CREATE TABLE x
INSERT INTO x
DROP TABLE
then the only time you can reasonably look at the data from the insert
is while replaying that record. At no other time does the data have
certain meaning.
So you *must* replay catalog entries and recreate the original catalog
in exact synchronisation with reading WAL files. Recreating the catalog
can only be done by Postgres itself. It simply isn't practical to do
this all with an external tool, or even link in to replay somehow to
keep replay and the reading of the external file synchronised. If it
*was*, somebody would have done it already - some have already tried and
failed.
(I haven't suggested modifying WAL files, BTW, not sure where that came
from).
Regarding the example plugin included, for debugging purposes you could
just compile with WAL_DEBUG, and the plugin to suppress actions for all
but one database is clearly not ready for any real work. It only
suppresses heapam records, replaying index updates and full-page-images
as usual, and it requires that you know the Oid of the database,
They're minor examples, so don't reject the plugin patch because the
example of usage isn't as useful as it could be. I'm hardly likely to
invest lots of time in a plugin while the approach has not been agreed,
am I?
It is viable for us to filter WAL records in this way, and not very
viable any other way. It doesn't require you to know the Oid of the
database, cos you can look that up in the catalog (with hot standby).
The example plugin doesn't do that, but it could.
So two use cases: inspecting WAL and filtering records before applying
them are covered here. Pluggable indexes is another, and there are
others also, as discussed on the original patch.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
On Wed, 2009-01-21 at 16:25 +0300, Teodor Sigaev wrote:
The external indexam use case doesn't impress me either, and Tom seems to agree
(http://archives.postgresql.org/message-id/24006.1221483138@sss.pgh.pa.us).
Just for correctness - there is one external index http://www.cs.purdue.edu/spgist/
If there is one even when we don't allow them (!), just think how many
there will be if we did allow them...
The docs for the SP-GiST describe PostgreSQL as "highly extensible". I'd
like that to extend to allowing recoverable extensions also.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
On Wed, Jan 21, 2009 at 1:25 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
The only reasonable way to examine the contents of WAL files is with
reference to a copy of the catalog that wrote them, timed *exactly* in
synchronisation with the WAL stream.
This is a good point.
Regarding the example plugin included, for debugging purposes you could
just compile with WAL_DEBUG, and the plugin to suppress actions for all
but one database is clearly not ready for any real work. It only
suppresses heapam records, replaying index updates and full-page-images
as usual, and it requires that you know the Oid of the database,They're minor examples, so don't reject the plugin patch because the
example of usage isn't as useful as it could be. I'm hardly likely to
invest lots of time in a plugin while the approach has not been agreed,
am I?
Well for these two cases I think the question is would the be better
done from within the core instead of a plugin? And if they are better
done as a plugin are the advantages strong enough to outweigh the
downsides of a plugin. (This actually reinforces the point that doing
these things externally is not very realistic.)
I don't see much of an advantage for plugins instead of core features
for either of these two cases. And given how tightly bound to a
specific version and the WAL record formats of that version a plugin
will have are there any advantages? If a plugin will only work with a
particular version of Postgres and it needs access to internal include
files then what separation does it give? From a code structure point
of view it may as well be integrated, in which case anyone who
modifies the wal structures is more likely to keep the other features
up to date.
Moreover, for things like restoring a single database I think there
are further disadvantages. You would have to ensure that the records
you're skipping don't result in an incoherent database. That means
either doing a cold restore of just a single database. That could be
really cool, you could, for instance allow rolling back a single
database to a hot backup + PITR without even shutting down the rest of
the cluster. However for anything like this to work properly you have
to know what version of the data files were restored and what version
the rest of the database is at, etc. If it's a plugin I think you
don't have enough information or control of the overall state to
handle it.
The only advantage that remains, I think, is the real-world concern
that you can have proprietary plugins that add features to the
database for dealing with emergency situations. It also means people
can experiment with features without maintaining a fork. That's not a
trivial advantage at all. I could see that being quite useful. But on
balance, considering how critical backups and restores are I would
personally avoid experimenting in this area anyways.
--
greg
Simon Riggs <simon@2ndQuadrant.com> writes:
The original design of Postgres allowed pluggable index access methods,
but that capability has not been brought forward to allow for WAL. This
patch would bridge that gap.
Well I think what people do is what GIST did early on -- they just don't
support recoverability until they get merged into core.
Nonetheless this *would* be a worthwhile problem to put effort into solving. I
agree that there are lots of exotic index methods out there that it would be
good to be able to develop externally.
But to do that we need an abstract interface that doesn't depend on internal
data structures, not a generic plugin facility that allows the plugin to
hijack the whole system.
We need something more like indexams which provides a set of call points which
do specific functions, only get called when they're needed, and are expected
to only do the one thing they've been asked to do.
This could be a bit tricky since the catalog isn't available to the wal replay
system. We can't just store the info needed in the pg_indexam table. And it
has to span all the databases in the cluster in any case.
Perhaps this should be solved along with the "plugins" thread. Binary modules
could have some way to register their rmgr id so you could guarantee that
there aren't two plugins with conflicting rmgr ids or version mismatches.
--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's Slony Replication support!
On Wed, 2009-01-21 at 14:28 +0000, Greg Stark wrote:
The only advantage that remains, I think, is the real-world concern
that you can have proprietary plugins
How exactly is this plugin more likely to result in a proprietary plugin
than all of the other plugin types we have? Because I suggest it??
I find it quite amazing that anybody would think I proposed a patch
whose "only advantage" lay in commercial exploitation, implying that I
intend that. But at least you had the courage to write it, allowing me
to answer, so actually I'll say thank you for raising that point:
** I have no plans for selling software that has been enabled by this
patch. **
The plugin approach was suggested because it brings together so many use
cases in one and adds missing robustness to a case where we already have
extensibility. Extensibility is about doing things for specific
implementations *without* needing to patch Postgres, not just allowing
external projects to exist alongside.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
On Wed, 2009-01-21 at 14:57 +0000, Gregory Stark wrote:
But to do that we need an abstract interface that doesn't depend on
internal data structures, not a generic plugin facility that allows
the plugin to hijack the whole system.We need something more like indexams which provides a set of call
points which do specific functions, only get called when they're
needed, and are expected to only do the one thing they've been asked
to do.
Really this is just ridiculous scare-mongering. Hijack the whole system?
The patch takes special care to allow calls to the rmgr functions only
from the startup process. The APIs are exactly like the indexams and
*are* called only in specific ways, at specific times. At your earlier
request I put in filters to prevent WAL inserts for plugins that didn't
exist, ensuring that all WAL writes were crash recoverable.
You can already do all the weird stuff you like with index AMs, like
send emails to the Pope on every row insert. I can already create an
in-memory index for example. How exactly do the rmgr interface give more
power? The structure of the function pointers is identical to the
indexAM code...
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Simon Riggs <simon@2ndQuadrant.com> writes:
On Wed, 2009-01-21 at 14:28 +0000, Greg Stark wrote:
The only advantage that remains, I think, is the real-world concern
that you can have proprietary plugins** I have no plans for selling software that has been enabled by this
patch. **
Hm, I didn't specifically mean this. However I'm not sure why this would be
considered so prejudicial. The Postgres project isn't generally hostile to
commercial use and extensions. If there was something you *did* want to sell
based on this and you needed a clean, generally useful interface to do it then
I think it would be an argument in *favour* of providing it, not against.
But I meant more generally, that the real-world use case for a generic rmgr
plugin function is for providing interfaces for things which cannot -- for
whatever non-code-related reason -- be integrated in core. That is, from a
code point of view they would be best integrated in core. So either they're
not generally useful, not production quality, not license compatible, or
whatever.
The plugin approach was suggested because it brings together so many use
cases in one and adds missing robustness to a case where we already have
extensibility. Extensibility is about doing things for specific
implementations *without* needing to patch Postgres, not just allowing
external projects to exist alongside.
I think a generic plugin architecture is *too* many use cases. That is it's
too flexible and doesn't make any promises at all of what its intended to do.
As a result the system can't be sure it's calling the right method, can't
detect conflicts or catch errors. There's a sweet spot of abstraction where
the interface has to be specific enough to be useful but general enough to
cover all the use cases.
I'm not sure though, your comments in the other email make me think there
might be more to the patch that I had the impression was there. Will now go
read the patch and see if I was mistaken.
--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's RemoteDBA services!
Gregory Stark wrote:
But to do that we need an abstract interface that doesn't depend on internal
data structures, not a generic plugin facility that allows the plugin to
hijack the whole system.We need something more like indexams which provides a set of call points which
do specific functions, only get called when they're needed, and are expected
to only do the one thing they've been asked to do.
That's called GiST. ;-)
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
Simon Riggs wrote:
Right now we've got a variety of index types that are *not* flourishing
(hash, bitmap, grouped).
Hash indexam has been in core for ages, and yet no-one has bothered to
implement WAL logging. If I've understood correctly, it has been now
been revamped in 8.4 so that there's a performance use case to use it. I
wouldn't be surprised if someone (GSoC?) implements WAL logging for it
for 8.5.
Bitmap indexes required significant changes to the rest of the system,
the indexam API in particular.
By "grouped", I presume you mean my grouped index tuples patch, aka
clustered indexes. That too required changes to the indexam API, and
even if it didn't, I can guarantee that I wouldn't spend any more time
on it than I do now (= 0) if it was on pgfoundry.
If we allow them to develop as separate
projects, then whenever they are ready they can be used with particular
releases.
Developing a new indexam is not something you do over the weekend. It's
a long way from design to an implementation robust enough that anyone
cares about crash recovery. Short-circuiting the release cycle with a
plugin won't get you a production-ready indexam much sooner.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
On Wed, 2009-01-21 at 16:07 +0000, Gregory Stark wrote:
The plugin approach was suggested because it brings together so many
use cases in one and adds missing robustness to a case where we
already have extensibility. Extensibility is about doing things for
specific implementations *without* needing to patch Postgres, not just
allowing external projects to exist alongside.I think a generic plugin architecture is *too* many use cases. That is
it's too flexible and doesn't make any promises at all of what its
intended to do.
I agree. I don't see providing the plugin capability should prevent
provision of further features in this area. Indeed, I see it as a way of
encouraging people to write stuff for Postgres, which we then reel
slowly back into core, if it is robust enough and general purpose
enough. My model is PL/Proxy: the capability we will eventually gain in
Core will be because we gave solution designers a free hand to invent
and a free hand to overcome obstacles in months, not years. Solutions
now, better solutions later.
I'm not sure though, your comments in the other email make me think
there might be more to the patch that I had the impression was there.
Will now go read the patch and see if I was mistaken.
Thank you.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Simon Riggs wrote:
So you *must* replay catalog entries and recreate the original catalog
in exact synchronisation with reading WAL files. Recreating the catalog
can only be done by Postgres itself.
The startup process doesn't have a relcache, so this rmgr patch is
nowhere near enough to enable that. If I understood correctly, the hot
standby patch doesn't change that either.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas wrote:
Simon Riggs wrote:
Right now we've got a variety of index types that are *not* flourishing
(hash, bitmap, grouped).Hash indexam has been in core for ages, and yet no-one has bothered to
implement WAL logging. If I've understood correctly, it has been now
been revamped in 8.4 so that there's a performance use case to use it. I
wouldn't be surprised if someone (GSoC?) implements WAL logging for it
for 8.5.Bitmap indexes required significant changes to the rest of the system,
the indexam API in particular.By "grouped", I presume you mean my grouped index tuples patch, aka
clustered indexes. That too required changes to the indexam API, and
even if it didn't, I can guarantee that I wouldn't spend any more time
on it than I do now (= 0) if it was on pgfoundry.If we allow them to develop as separate
projects, then whenever they are ready they can be used with particular
releases.Developing a new indexam is not something you do over the weekend. It's
a long way from design to an implementation robust enough that anyone
cares about crash recovery. Short-circuiting the release cycle with a
plugin won't get you a production-ready indexam much sooner.
Agreed.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Simon Riggs wrote:
Why do we have 12+ pluggable languages, but we're not allowed to write
pluggable indexes? Whatever argument you put against it being "too hard"
or dangerous or whatever *also* applies to languages. Yet experience
shows pluggability has resulted in a variety of robust and useful
language types, some that might not have been predicted (PL/Proxy, PL/R
etc). They cover a variety of users and situations.
Languages are quite different. People already know language X, so they
want to use it for stored procedures too. Or they want to interface
other libraries or functionality available in language X. There's no
such argument with indexams. Also, PL handlers are not as tightly
integrated into the rest of the system, no need for low-level page
access, for example, which is why it's easier to have a generic
interface for them. There's also less issues with concurrency and
version-compatibility.
Personally, I'd like
to enable people to come up with audio, video, bioinformatics datatypes
and indexes and I definitely don't want to limit the possibilities
there.
Yeah, I'd like to see all those datatypes too. But I'd presume that
audio, video and bioinformatics indexing could all be implemented using
GiST. You don't want to write an indexam from scratch for every data type.
... - look how unusable
hash indexes are. How can we allow that functionality to continue to
exist in Core and yet block the path by which we might reasonably
correct that?
I don't see how ripping out hash indexes from core and pushing it into
an external module where it could use the rmgr plugin mechanism would
help to add WAL-logging to it. If someone wants to implement WAL-logging
for hash indexes, just do it, and send a patch.
You don't want pluggable indexes, don't use 'em. But that isn't an
argument against allowing the capability for others. That line of
thought would have led us to banning pluggable languages. We should
respect the roots of this project and look for ways to enable the
advancement of database technology, not limit it to only how far we can
currently see ahead through the fog.
This is an open source project. There's already a lot of people writing
their thesis and whatnot using PostgreSQL, having no problem modifying
the code as they see fit to try completely novel things. We're not
banning or blocking that. On the contrary, that's great! Anyone can
download the source code, modify it, and publish a patch. Others will
find the patch interesting and embrace it, or not. *That's* how this
project moves forward.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
