logical decoding documentation?
Where, if anywhere, is the current documentation for writing or using a
logical decoding output plugin consumer thingy?
I'm trying to find my way into it ...
src/backend/replication/logical/logical.c, which textually contains most
of the functions that appear to interact with the test_decoding module,
contains this in the header comment:
"""
The idea is that a consumer provides three callbacks, one to read WAL,
one to prepare a data write, and a final one for actually writing since
their implementation depends on the type of consumer. Check
logicalfunc.c for an example implementations of a fairly simple consumer
and a implementation of a WAL reading callback that's suitable for
simpler consumers.
"""
There is no file logicalfunc.c. And test_decoding actually uses five
callbacks, not three.
Is a consumer the same as a decoder?
test_decoding.c contains this:
/* These must be available to pg_dlsym() */
static void pg_decode_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt, bool is_init);
...
which is surely wrong.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Hi,
On 2014-03-11 15:57:39 -0400, Peter Eisentraut wrote:
Where, if anywhere, is the current documentation for writing or using a
logical decoding output plugin consumer thingy?
There's a pending patch for it. The corresponding commit is
http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=commit;h=5eeedd55b2d7e53b5fdcdab6a8e74bb666d75bcc
I welcome feedback about it. I've spent a fair bit of time immersed in
this stuff, and I am not really sure anymore what's understandable and
whatnot ;)
It's referencing pg_recvlogical which isn't committed yet (that's the
commit just before), that's why the docs weren't committed with the
feature itself.
src/backend/replication/logical/logical.c, which textually contains most
of the functions that appear to interact with the test_decoding module,
contains this in the header comment:"""
The idea is that a consumer provides three callbacks, one to read WAL,
one to prepare a data write, and a final one for actually writing since
their implementation depends on the type of consumer. Check
logicalfunc.c for an example implementations of a fairly simple consumer
and a implementation of a WAL reading callback that's suitable for
simpler consumers.
"""There is no file logicalfunc.c.
Hrmpf: There's a missing 's', it's logicalfuncs.c.
And test_decoding actually uses five callbacks, not three.
The callbacks logicalfuncs.c header comment is talking about adding a
new method to output data. Currently you can stream out changes via
walsender and via the SQL SRFs. But it might be interesting to
e.g. consume the changes in a bgworker, without going through either SQL
or walsender. To do that you need the three callbacks referenced above.
Is a consumer the same as a decoder?
A consumer is just the recipient of the changestream. I.e the walsender
that streams out the changestream, or the SRF that spills the data into
a tuplestore.
I don't think the term "decoder" is used anywhere, but if it is, it'd
be the output plugin.
test_decoding.c contains this:
/* These must be available to pg_dlsym() */
static void pg_decode_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt, bool is_init);
...which is surely wrong.
Hm, these days the comment should be above _PG_init() and
_PG_output_plugin_init(). That changed around several times...
Could you perhaps commit the attached patch fixing the issues you
mentioned?
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
minor-logical-decoding-comment-improvements.patchtext/x-patch; charset=us-asciiDownload
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index e356c7c..31aa012 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -33,6 +33,7 @@
PG_MODULE_MAGIC;
+/* These must be available to pg_dlsym() */
extern void _PG_init(void);
extern void _PG_output_plugin_init(OutputPluginCallbacks *cb);
@@ -43,7 +44,6 @@ typedef struct
bool include_timestamp;
} TestDecodingData;
-/* These must be available to pg_dlsym() */
static void pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init);
static void pg_decode_shutdown(LogicalDecodingContext *ctx);
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 13a22d4..04e407a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -12,14 +12,17 @@
* together provide logical decoding, primarily by providing so
* called LogicalDecodingContexts. The goal is to encapsulate most of the
* internal complexity for consumers of logical decoding, so they can
- * create and consume a changestream with a low amount of code.
+ * create and consume a changestream with a low amount of code. Builtin
+ * consumers are the walsender and SQL SRF interface, but it's possible to
+ * add further ones without changing core code, e.g. to consume changes in
+ * a bgworker.
*
* The idea is that a consumer provides three callbacks, one to read WAL,
* one to prepare a data write, and a final one for actually writing since
* their implementation depends on the type of consumer. Check
- * logicalfunc.c for an example implementations of a fairly simple consumer
+ * logicalfuncs.c for an example implementation of a fairly simple consumer
* and a implementation of a WAL reading callback that's suitable for
- * simpler consumers.
+ * simple consumers.
*-------------------------------------------------------------------------
*/
On Tue, Mar 11, 2014 at 4:16 PM, Andres Freund <andres@2ndquadrant.com> wrote:
Could you perhaps commit the attached patch fixing the issues you
mentioned?
I committed this.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers