"pgoutput" options missing on documentation
Started by Emre Hasegeliabout 2 years ago23 messages
I noticed that Logical Streaming Replication Protocol documentation
[1]: https://www.postgresql.org/docs/devel/protocol-logical-replication.html
patch is attached to fix it together with another small one to give a
nice error when "proto_version" parameter is not provided.
[1]: https://www.postgresql.org/docs/devel/protocol-logical-replication.html
Attachments:
v00-0001-pgoutput-Test-if-protocol_version-is-given.patchapplication/octet-stream; name=v00-0001-pgoutput-Test-if-protocol_version-is-given.patchDownload
From ae0f8ba0b886512f8e66e89ff794b5a28ded0b96 Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 13 Mar 2023 16:54:18 +0100
Subject: [PATCH v00 1/2] pgoutput: Test if protocol_version is given
---
src/backend/replication/pgoutput/pgoutput.c | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index f9ed1083df..f8aac22cfe 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -393,20 +393,30 @@ parse_output_parameters(List *options, PGOutputData *data)
else if (pg_strcasecmp(origin, LOGICALREP_ORIGIN_ANY) == 0)
data->publish_no_origin = false;
else
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+ /* Check required parameters */
+ if (!protocol_version_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("proto_version parameter missing")));
+ if (!publication_names_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("publication_names parameter missing")));
}
/*
* Initialize this plugin
*/
static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
PGOutputData *data = palloc0(sizeof(PGOutputData));
@@ -442,25 +452,20 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("client sent proto_version=%d but server only supports protocol %d or lower",
data->protocol_version, LOGICALREP_PROTO_MAX_VERSION_NUM)));
if (data->protocol_version < LOGICALREP_PROTO_MIN_VERSION_NUM)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("client sent proto_version=%d but server only supports protocol %d or higher",
data->protocol_version, LOGICALREP_PROTO_MIN_VERSION_NUM)));
- if (data->publication_names == NIL)
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("publication_names parameter missing")));
-
/*
* Decide whether to enable streaming. It is disabled by default, in
* which case we just update the flag in decoding context. Otherwise
* we only allow it with sufficient version of the protocol, and when
* the output plugin supports it.
*/
if (data->streaming == LOGICALREP_STREAM_OFF)
ctx->streaming = false;
else if (data->streaming == LOGICALREP_STREAM_ON &&
data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
--
2.39.3 (Apple Git-145)
v00-0002-pgoutput-Document-missing-options.patchapplication/octet-stream; name=v00-0002-pgoutput-Document-missing-options.patchDownload
From 7774783114e14d04c728bd0ad04a5286a4307bd2 Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH v00 2/2] pgoutput: Document missing options
---
doc/src/sgml/logical-replication.sgml | 3 +-
doc/src/sgml/protocol.sgml | 81 ++++++++++++++++++++++++++-
2 files changed, 81 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 1bb85e4ad9..e7fd1a38b2 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1655,21 +1655,22 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <literal>walsender</literal> and <literal>apply</literal>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index af3f016f74..d4c500adcc 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3075,26 +3075,32 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> supports extensible logical decoding
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ The standard logical decoding plugin (<literal>pgoutput</literal>) accepts
+ following parameters with <literal>START_REPLICATION</literal> command:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
Protocol version. Currently versions <literal>1</literal>, <literal>2</literal>,
<literal>3</literal>, and <literal>4</literal> are supported.
@@ -3120,20 +3126,91 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
(receive changes). The individual publication names are treated
as standard objects names and can be quoted the same as needed.
</para>
</listitem>
</varlistentry>
+
+<!-- Backpack through version 14. -->
+ <varlistentry>
+ <term>
+ binary
+ </term>
+ <listitem>
+ <para>
+ Boolean parameter to use the binary transfer mode. This is faster
+ than the text mode, but slightly less robust.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 14. -->
+ <varlistentry>
+ <term>
+ messages
+ </term>
+ <listitem>
+ <para>
+ Boolean parameter to enable sending the messages that are written
+ by <function>pg_logical_emit_message</function>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 14, but remove "parallel" part until version 16. -->
+ <varlistentry>
+ <term>
+ streaming
+ </term>
+ <listitem>
+ <para>
+ Boolean parameter to enable streaming of in-progress transactions.
+ It accepts an additional value "parallel" to enable sending extra
+ information with the "Stream Abort" messages to be used for
+ parallelisation. Minimum protocol version 2 is required to turn it on.
+ Minimum protocol version 4 is required for the "parallel" option.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 15. -->
+ <varlistentry>
+ <term>
+ two_phase
+ </term>
+ <listitem>
+ <para>
+ Boolean parameter to enable two-phase transactions. Minimum protocol
+ version 3 is required to turn it on.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 16. -->
+ <varlistentry>
+ <term>
+ origin
+ </term>
+ <listitem>
+ <para>
+ String parameter to send only changes by an origin. It also gets
+ the option "none" to send the changes that have no origin associated,
+ and the option "any" to send the changes regardless of their origin.
+ This can be used to avoid loops (infinite replication of the same data)
+ among replication nodes.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
<para>
The individual protocol messages are discussed in the following
--
2.39.3 (Apple Git-145)
Re: "pgoutput" options missing on documentation
On Wed, Dec 13, 2023 at 9:25 PM Emre Hasegeli <emre@hasegeli.com> wrote:
I noticed that Logical Streaming Replication Protocol documentation
[1] is missing the options added to "pgoutput" since version 14. A
patch is attached to fix it together with another small one to give a
nice error when "proto_version" parameter is not provided.
I agree that we missed updating the parameters of the Logical
Streaming Replication Protocol documentation. I haven't reviewed all
the details yet but one minor thing that caught my attention while
looking at your patch is that we can update the exact additional
information we started to send for streaming mode parallel. We should
update the following sentence: "It accepts an additional value
"parallel" to enable sending extra information with the "Stream Abort"
messages to be used for parallelisation."
--
With Regards,
Amit Kapila.
Re: "pgoutput" options missing on documentation
Hi, here are some initial review comments.
//////
Patch v00-0001
1.
+
+ /* Check required parameters */
+ if (!protocol_version_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("proto_version parameter missing")));
+ if (!publication_names_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("publication_names parameter missing")));To reduce translation efforts, perhaps it is better to arrange for
these to share a common message.
For example,
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
/* translator: %s is a pgoutput option */
errmsg("missing pgoutput option: %s", "proto_version"));
~
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
/* translator: %s is a pgoutput option */
errmsg("missing pgoutput option: %s", "publication_names"));
Also, I am unsure whether to call these "parameters" or "options" -- I
wanted to call them parameters like in the documentation, but every
other message in this function refers to "options", so in my example,
I mimicked the nearby code YMMV.
//////
Patch v00-0002
2.
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ The standard logical decoding plugin (<literal>pgoutput</literal>) accepts
+ following parameters with <literal>START_REPLICATION</literal> command:Since the previous line already said pgoutput is the standard decoding
plugin, maybe it's not necessary to repeat that.
SUGGESTION
Using the <literal>START_REPLICATION</literal> command,
<literal>pgoutput</literal>) accepts the following parameters:
~~~
3.
I noticed in the protocol message formats docs [1]https://www.postgresql.org/docs/current/protocol-logicalrep-message-formats.html that those messages
are grouped according to the protocol version that supports them.
Perhaps this page should be arranged similarly for these parameters?
For example, document the parameters in the order they were introduced.
SUGGESTION
-proto_version
...
-publication_names
...
-binary
...
-messages
...
Since protocol version 2:
-streaming (boolean)
...
Since protocol version 3:
-two_phase
...
Since protocol version 4:
-streaming (boolean/parallel)
...
-origin
...
~~~
4.
+ Boolean parameter to use the binary transfer mode. This is faster
+ than the text mode, but slightly less robustSUGGESTION
Boolean parameter to use binary transfer mode. Binary mode is faster
than the text mode but slightly less robust
======
[1]: https://www.postgresql.org/docs/current/protocol-logicalrep-message-formats.html
Kind Regards,
Peter Smith.
Fujitsu Australia
Re: "pgoutput" options missing on documentation
To reduce translation efforts, perhaps it is better to arrange for
these to share a common message.
Good idea. I've done so.
Also, I am unsure whether to call these "parameters" or "options" -- I
wanted to call them parameters like in the documentation, but every
other message in this function refers to "options", so in my example,
I mimicked the nearby code YMMV.
It looks like they are called "options" in most places. I changed the
documentation to be consistent too.
Since the previous line already said pgoutput is the standard decoding
plugin, maybe it's not necessary to repeat that.SUGGESTION
Using the <literal>START_REPLICATION</literal> command,
<literal>pgoutput</literal>) accepts the following parameters:
Changed.
3.
I noticed in the protocol message formats docs [1] that those messages
are grouped according to the protocol version that supports them.
Perhaps this page should be arranged similarly for these parameters?For example, document the parameters in the order they were introduced.
SUGGESTION
-proto_version
...
-publication_names
...
-binary
...
-messages
...Since protocol version 2:
-streaming (boolean)
...Since protocol version 3:
-two_phase
...Since protocol version 4:
-streaming (boolean/parallel)
...
-origin
This is not going to be correct because not all options do require a
protocol version. "origin" is added in version 16, but doesn't check
for any "proto_version". Perhaps we should fix this too.
4. + Boolean parameter to use the binary transfer mode. This is faster + than the text mode, but slightly less robustSUGGESTION
Boolean parameter to use binary transfer mode. Binary mode is faster
than the text mode but slightly less robust
Done.
Thanks for the review.
The new versions are attached.
Attachments:
v01-0001-pgoutput-Improve-error-messages-for-options.patchapplication/octet-stream; name=v01-0001-pgoutput-Improve-error-messages-for-options.patchDownload
From 77279f4b360aafa135bee43697b1cad262142225 Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 13 Mar 2023 16:54:18 +0100
Subject: [PATCH v01 1/2] pgoutput: Improve error messages for options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
src/backend/replication/pgoutput/pgoutput.c | 61 ++++++++++++++-------
1 file changed, 41 insertions(+), 20 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index f9ed1083df..1ff818af76 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -267,21 +267,21 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->stream_abort_cb = pgoutput_stream_abort;
cb->stream_commit_cb = pgoutput_stream_commit;
cb->stream_change_cb = pgoutput_change;
cb->stream_message_cb = pgoutput_message;
cb->stream_truncate_cb = pgoutput_truncate;
/* transaction streaming - two-phase commit */
cb->stream_prepare_cb = pgoutput_stream_prepare_txn;
}
static void
-parse_output_parameters(List *options, PGOutputData *data)
+parse_output_options(List *options, PGOutputData *data)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool binary_option_given = false;
bool messages_option_given = false;
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
@@ -289,30 +289,32 @@ parse_output_parameters(List *options, PGOutputData *data)
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
foreach(lc, options)
{
DefElem *defel = (DefElem *) lfirst(lc);
Assert(defel->arg == NULL || IsA(defel->arg, String));
- /* Check each param, whether or not we recognize it */
+ /* Check each option, whether or not we recognize it */
if (strcmp(defel->defname, "proto_version") == 0)
{
unsigned long parsed;
char *endptr;
if (protocol_version_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "proto_version")));
protocol_version_given = true;
errno = 0;
parsed = strtoul(strVal(defel->arg), &endptr, 10);
if (errno != 0 || *endptr != '\0')
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid proto_version")));
if (parsed > PG_UINT32_MAX)
@@ -321,92 +323,116 @@ parse_output_parameters(List *options, PGOutputData *data)
errmsg("proto_version \"%s\" out of range",
strVal(defel->arg))));
data->protocol_version = (uint32) parsed;
}
else if (strcmp(defel->defname, "publication_names") == 0)
{
if (publication_names_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "publication_names")));
publication_names_given = true;
if (!SplitIdentifierString(strVal(defel->arg), ',',
&data->publication_names))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
else if (strcmp(defel->defname, "binary") == 0)
{
if (binary_option_given)
ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "binary"));
binary_option_given = true;
data->binary = defGetBoolean(defel);
}
else if (strcmp(defel->defname, "messages") == 0)
{
if (messages_option_given)
ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "messages"));
messages_option_given = true;
data->messages = defGetBoolean(defel);
}
else if (strcmp(defel->defname, "streaming") == 0)
{
if (streaming_given)
ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "streaming"));
streaming_given = true;
data->streaming = defGetStreamingMode(defel);
}
else if (strcmp(defel->defname, "two_phase") == 0)
{
if (two_phase_option_given)
ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "two_phase"));
two_phase_option_given = true;
data->two_phase = defGetBoolean(defel);
}
else if (strcmp(defel->defname, "origin") == 0)
{
char *origin;
if (origin_option_given)
ereport(ERROR,
errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options"));
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "origin"));
origin_option_given = true;
origin = defGetString(defel);
if (pg_strcasecmp(origin, LOGICALREP_ORIGIN_NONE) == 0)
data->publish_no_origin = true;
else if (pg_strcasecmp(origin, LOGICALREP_ORIGIN_ANY) == 0)
data->publish_no_origin = false;
else
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+ /* Check required options */
+ if (!protocol_version_given)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ /* translator: %s is a pgoutput option */
+ errmsg("missing pgoutput option: %s", "proto_version"));
+ if (!publication_names_given)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ /* translator: %s is a pgoutput option */
+ errmsg("missing pgoutput option: %s", "publication_names"));
}
/*
* Initialize this plugin
*/
static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
PGOutputData *data = palloc0(sizeof(PGOutputData));
@@ -426,41 +452,36 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* This plugin uses binary protocol. */
opt->output_type = OUTPUT_PLUGIN_BINARY_OUTPUT;
/*
* This is replication start and not slot initialization.
*
* Parse and validate options passed by the client.
*/
if (!is_init)
{
- /* Parse the params and ERROR if we see any we don't recognize */
- parse_output_parameters(ctx->output_plugin_options, data);
+ /* Parse the options and ERROR if we see any we don't recognize */
+ parse_output_options(ctx->output_plugin_options, data);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_MAX_VERSION_NUM)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("client sent proto_version=%d but server only supports protocol %d or lower",
data->protocol_version, LOGICALREP_PROTO_MAX_VERSION_NUM)));
if (data->protocol_version < LOGICALREP_PROTO_MIN_VERSION_NUM)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("client sent proto_version=%d but server only supports protocol %d or higher",
data->protocol_version, LOGICALREP_PROTO_MIN_VERSION_NUM)));
- if (data->publication_names == NIL)
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("publication_names parameter missing")));
-
/*
* Decide whether to enable streaming. It is disabled by default, in
* which case we just update the flag in decoding context. Otherwise
* we only allow it with sufficient version of the protocol, and when
* the output plugin supports it.
*/
if (data->streaming == LOGICALREP_STREAM_OFF)
ctx->streaming = false;
else if (data->streaming == LOGICALREP_STREAM_ON &&
data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
--
2.39.3 (Apple Git-145)
v01-0002-pgoutput-Document-missing-options.patchapplication/octet-stream; name=v01-0002-pgoutput-Document-missing-options.patchDownload
From 55e949c8939898660929f31617a5ff27145780fd Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH v01 2/2] pgoutput: Document missing options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Backpatch-through: 14
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
doc/src/sgml/logical-replication.sgml | 3 +-
doc/src/sgml/protocol.sgml | 81 ++++++++++++++++++++++++++-
2 files changed, 81 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 1bb85e4ad9..e7fd1a38b2 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1655,21 +1655,22 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <literal>walsender</literal> and <literal>apply</literal>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index af3f016f74..76501a7da6 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -3075,26 +3075,32 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> supports extensible logical decoding
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal>) accepts the following options:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
Protocol version. Currently versions <literal>1</literal>, <literal>2</literal>,
<literal>3</literal>, and <literal>4</literal> are supported.
@@ -3120,20 +3126,91 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
(receive changes). The individual publication names are treated
as standard objects names and can be quoted the same as needed.
</para>
</listitem>
</varlistentry>
+
+<!-- Backpack through version 14. -->
+ <varlistentry>
+ <term>
+ binary
+ </term>
+ <listitem>
+ <para>
+ Boolean option to use binary transfer mode. Binary mode is faster
+ than the text mode but slightly less robust.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 14. -->
+ <varlistentry>
+ <term>
+ messages
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable sending the messages that are written
+ by <function>pg_logical_emit_message</function>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 14, but remove "parallel" part until version 16. -->
+ <varlistentry>
+ <term>
+ streaming
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable streaming of in-progress transactions.
+ It accepts an additional value "parallel" to enable sending extra
+ information with some messages to be used for parallelisation.
+ Minimum protocol version 2 is required to turn it on. Minimum protocol
+ version 4 is required for the "parallel" option.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 15. -->
+ <varlistentry>
+ <term>
+ two_phase
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable two-phase transactions. Minimum protocol
+ version 3 is required to turn it on.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 16. -->
+ <varlistentry>
+ <term>
+ origin
+ </term>
+ <listitem>
+ <para>
+ String option to send only changes by an origin. It also gets
+ the option "none" to send the changes that have no origin associated,
+ and the option "any" to send the changes regardless of their origin.
+ This can be used to avoid loops (infinite replication of the same data)
+ among replication nodes.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
<para>
The individual protocol messages are discussed in the following
--
2.39.3 (Apple Git-145)
Re: "pgoutput" options missing on documentation
I agree that we missed updating the parameters of the Logical
Streaming Replication Protocol documentation. I haven't reviewed all
the details yet but one minor thing that caught my attention while
looking at your patch is that we can update the exact additional
information we started to send for streaming mode parallel. We should
update the following sentence: "It accepts an additional value
"parallel" to enable sending extra information with the "Stream Abort"
messages to be used for parallelisation."
I changed this in the new version.
Thank you for picking this up.
Re: "pgoutput" options missing on documentation
Thanks for the update. Here are some more review comments for the v01* patches.
//////
Patch v00-0001
v01 modified the messages more than I was expecting, although what you
did looks fine to me.
~~~
1.
+ /* Check required options */
+ if (!protocol_version_given)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ /* translator: %s is a pgoutput option */
+ errmsg("missing pgoutput option: %s", "proto_version"));
+ if (!publication_names_given)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ /* translator: %s is a pgoutput option */
+ errmsg("missing pgoutput option: %s", "publication_names"));I saw that the original "publication_names" error was using
errcode(ERRCODE_INVALID_PARAMETER_VALUE), but TBH since there is no
option given at all I felt ERRCODE_SYNTAX_ERROR might be more
appropriate errcode for those 2 mandatory option errors.
//////
Patch v00-0002
2.
I saw that the chapter "55.4. Streaming Replication Protocol" [1]55.4 https://www.postgresql.org/docs/devel/protocol-replication.html
introduces "START_REPLICATION SLOT slot_name LOGICAL ..." command and
it says
---
option_name
The name of an option passed to the slot's logical decoding plugin.
---
Perhaps that part should now include a reference to your new information:
SUGGESTION
option_name
The name of an option passed to the slot's logical decoding plugin.
Please see <XXX (55.5.1)> for details about options that are accepted
by the standard (pgoutput) plugin.
~~~
3.
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal>) accepts the following options:Oops, you copied my typo. There is a spurious ')'.
~~~
4.
+<!-- Backpack through version 16. -->
+ <varlistentry>
+ <term>
+ origin
+ </term>
+ <listitem>
+ <para>
+ String option to send only changes by an origin. It also gets
+ the option "none" to send the changes that have no origin associated,
+ and the option "any" to send the changes regardless of their origin.
+ This can be used to avoid loops (infinite replication of the same data)
+ among replication nodes.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>AFAIK pgoutput only understands origin values "any" and "none" and
nothing else; I felt the "It also gets..." part implies it is more
flexible than it is.
SUGGESTION
Possible values are "none" (to only send the changes that have no
origin associated), or "any" (to send the changes regardless of their
origin).
~~~
5. Rearranging option details
SUGGESTION
-proto_version
...
-publication_names
...
-binary
...
-messages
...Since protocol version 2:
-streaming (boolean)
...Since protocol version 3:
-two_phase
...Since protocol version 4:
-streaming (boolean/parallel)
...
-originThis is not going to be correct because not all options do require a
protocol version. "origin" is added in version 16, but doesn't check
for any "proto_version". Perhaps we should fix this too.
OK, to deal with that can't you just include "origin" in the first
group which has no special protocol requirements?
SUGGESTION
-proto_version
-publication_names
-binary
-messages
-origin
Requires minimum protocol version 2:
-streaming (boolean)
Requires minimum protocol version 3:
-two_phase
Requires minimum protocol version 4:
-streaming (parallel)
======
[1]: 55.4 https://www.postgresql.org/docs/devel/protocol-replication.html
Kind Regards,
Peter Smith.
Fujitsu Australia
Re: "pgoutput" options missing on documentation
I saw that the original "publication_names" error was using
errcode(ERRCODE_INVALID_PARAMETER_VALUE), but TBH since there is no
option given at all I felt ERRCODE_SYNTAX_ERROR might be more
appropriate errcode for those 2 mandatory option errors.
It makes sense to me. Changed.
2.
I saw that the chapter "55.4. Streaming Replication Protocol" [1]
introduces "START_REPLICATION SLOT slot_name LOGICAL ..." command and
it says
---
option_name
The name of an option passed to the slot's logical decoding plugin.
---Perhaps that part should now include a reference to your new information:
SUGGESTION
option_name
The name of an option passed to the slot's logical decoding plugin.
Please see <XXX (55.5.1)> for details about options that are accepted
by the standard (pgoutput) plugin.
Good idea. Incorporated.
3. <para> - The logical replication <literal>START_REPLICATION</literal> command - accepts following parameters: + Using the <literal>START_REPLICATION</literal> command, + <literal>pgoutput</literal>) accepts the following options:Oops, you copied my typo. There is a spurious ')'.
Fixed.
4. +<!-- Backpack through version 16. --> + <varlistentry> + <term> + origin + </term> + <listitem> + <para> + String option to send only changes by an origin. It also gets + the option "none" to send the changes that have no origin associated, + and the option "any" to send the changes regardless of their origin. + This can be used to avoid loops (infinite replication of the same data) + among replication nodes. + </para> + </listitem> + </varlistentry> </variablelist>AFAIK pgoutput only understands origin values "any" and "none" and
nothing else; I felt the "It also gets..." part implies it is more
flexible than it is.SUGGESTION
Possible values are "none" (to only send the changes that have no
origin associated), or "any" (to send the changes regardless of their
origin).
Oh, it's not how I understood it. I think you are right. Changed.
OK, to deal with that can't you just include "origin" in the first
group which has no special protocol requirements?
I think it'd be confusing because the option is not available before
version 16. I think it should really check for the version number and
complain if it's less than 4.
SUGGESTION
-proto_version
-publication_names
-binary
-messages
-originRequires minimum protocol version 2:
-streaming (boolean)Requires minimum protocol version 3:
-two_phaseRequires minimum protocol version 4:
-streaming (parallel)
I am still not sure if this is any better. I don't like that
"streaming" appears twice, and I wouldn't know how to format this
nicely.
The new versions are attached.
I also added "Required." for "proto_version" and "publication_names".
Attachments:
v02-0001-pgoutput-Improve-error-messages-for-options.patchapplication/octet-stream; name=v02-0001-pgoutput-Improve-error-messages-for-options.patchDownload
From 7c00681b05302dc117e227192f68f4bf2a26de84 Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 13 Mar 2023 16:54:18 +0100
Subject: [PATCH v02 1/2] pgoutput: Improve error messages for options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
src/backend/replication/pgoutput/pgoutput.c | 65 ++++++++++++++-------
1 file changed, 43 insertions(+), 22 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index f9ed1083df..0370db972a 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -267,21 +267,21 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->stream_abort_cb = pgoutput_stream_abort;
cb->stream_commit_cb = pgoutput_stream_commit;
cb->stream_change_cb = pgoutput_change;
cb->stream_message_cb = pgoutput_message;
cb->stream_truncate_cb = pgoutput_truncate;
/* transaction streaming - two-phase commit */
cb->stream_prepare_cb = pgoutput_stream_prepare_txn;
}
static void
-parse_output_parameters(List *options, PGOutputData *data)
+parse_output_options(List *options, PGOutputData *data)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool binary_option_given = false;
bool messages_option_given = false;
bool streaming_given = false;
bool two_phase_option_given = false;
bool origin_option_given = false;
@@ -289,30 +289,32 @@ parse_output_parameters(List *options, PGOutputData *data)
data->streaming = LOGICALREP_STREAM_OFF;
data->messages = false;
data->two_phase = false;
foreach(lc, options)
{
DefElem *defel = (DefElem *) lfirst(lc);
Assert(defel->arg == NULL || IsA(defel->arg, String));
- /* Check each param, whether or not we recognize it */
+ /* Check each option, whether or not we recognize it */
if (strcmp(defel->defname, "proto_version") == 0)
{
unsigned long parsed;
char *endptr;
if (protocol_version_given)
ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "proto_version"));
protocol_version_given = true;
errno = 0;
parsed = strtoul(strVal(defel->arg), &endptr, 10);
if (errno != 0 || *endptr != '\0')
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid proto_version")));
if (parsed > PG_UINT32_MAX)
@@ -320,93 +322,117 @@ parse_output_parameters(List *options, PGOutputData *data)
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("proto_version \"%s\" out of range",
strVal(defel->arg))));
data->protocol_version = (uint32) parsed;
}
else if (strcmp(defel->defname, "publication_names") == 0)
{
if (publication_names_given)
ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "publication_names"));
publication_names_given = true;
if (!SplitIdentifierString(strVal(defel->arg), ',',
&data->publication_names))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
else if (strcmp(defel->defname, "binary") == 0)
{
if (binary_option_given)
ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "binary"));
binary_option_given = true;
data->binary = defGetBoolean(defel);
}
else if (strcmp(defel->defname, "messages") == 0)
{
if (messages_option_given)
ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "messages"));
messages_option_given = true;
data->messages = defGetBoolean(defel);
}
else if (strcmp(defel->defname, "streaming") == 0)
{
if (streaming_given)
ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "streaming"));
streaming_given = true;
data->streaming = defGetStreamingMode(defel);
}
else if (strcmp(defel->defname, "two_phase") == 0)
{
if (two_phase_option_given)
ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "two_phase"));
two_phase_option_given = true;
data->two_phase = defGetBoolean(defel);
}
else if (strcmp(defel->defname, "origin") == 0)
{
char *origin;
if (origin_option_given)
ereport(ERROR,
errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options"));
+ /* translator: %s is a pgoutput option */
+ errmsg("conflicting or redundant option: %s",
+ "origin"));
origin_option_given = true;
origin = defGetString(defel);
if (pg_strcasecmp(origin, LOGICALREP_ORIGIN_NONE) == 0)
data->publish_no_origin = true;
else if (pg_strcasecmp(origin, LOGICALREP_ORIGIN_ANY) == 0)
data->publish_no_origin = false;
else
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+ /* Check required options */
+ if (!protocol_version_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is a pgoutput option */
+ errmsg("missing pgoutput option: %s", "proto_version"));
+ if (!publication_names_given)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ /* translator: %s is a pgoutput option */
+ errmsg("missing pgoutput option: %s", "publication_names"));
}
/*
* Initialize this plugin
*/
static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
PGOutputData *data = palloc0(sizeof(PGOutputData));
@@ -426,41 +452,36 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* This plugin uses binary protocol. */
opt->output_type = OUTPUT_PLUGIN_BINARY_OUTPUT;
/*
* This is replication start and not slot initialization.
*
* Parse and validate options passed by the client.
*/
if (!is_init)
{
- /* Parse the params and ERROR if we see any we don't recognize */
- parse_output_parameters(ctx->output_plugin_options, data);
+ /* Parse the options and ERROR if we see any we don't recognize */
+ parse_output_options(ctx->output_plugin_options, data);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_MAX_VERSION_NUM)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("client sent proto_version=%d but server only supports protocol %d or lower",
data->protocol_version, LOGICALREP_PROTO_MAX_VERSION_NUM)));
if (data->protocol_version < LOGICALREP_PROTO_MIN_VERSION_NUM)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("client sent proto_version=%d but server only supports protocol %d or higher",
data->protocol_version, LOGICALREP_PROTO_MIN_VERSION_NUM)));
- if (data->publication_names == NIL)
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("publication_names parameter missing")));
-
/*
* Decide whether to enable streaming. It is disabled by default, in
* which case we just update the flag in decoding context. Otherwise
* we only allow it with sufficient version of the protocol, and when
* the output plugin supports it.
*/
if (data->streaming == LOGICALREP_STREAM_OFF)
ctx->streaming = false;
else if (data->streaming == LOGICALREP_STREAM_ON &&
data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
--
2.39.3 (Apple Git-145)
v02-0002-pgoutput-Document-missing-options.patchapplication/octet-stream; name=v02-0002-pgoutput-Document-missing-options.patchDownload
From 6265265b8fb23d5b2192c77b5773eea0f3b181fa Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH v02 2/2] pgoutput: Document missing options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 14
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
doc/src/sgml/logical-replication.sgml | 3 +-
doc/src/sgml/protocol.sgml | 92 +++++++++++++++++++++++++--
2 files changed, 88 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 1bb85e4ad9..e7fd1a38b2 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1655,21 +1655,22 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <literal>walsender</literal> and <literal>apply</literal>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index af3f016f74..6605e2aede 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2539,20 +2539,22 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
The WAL location to begin streaming at.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_name</replaceable></term>
<listitem>
<para>
The name of an option passed to the slot's logical decoding plugin.
+ See <xref linkend="protocol-logical-replication"/> for options that
+ are accepted by the standard (<literal>pgoutput</literal>) plugin.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_value</replaceable></term>
<listitem>
<para>
Optional value, in the form of a string constant, associated with the
specified option.
@@ -3075,36 +3077,43 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> supports extensible logical decoding
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal> accepts the following options:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
- Protocol version. Currently versions <literal>1</literal>, <literal>2</literal>,
- <literal>3</literal>, and <literal>4</literal> are supported.
+ Protocol version. Required. Currently versions <literal>1</literal>,
+ <literal>2</literal>, <literal>3</literal>, and <literal>4</literal>
+ are supported.
</para>
<para>
Version <literal>2</literal> is supported only for server version 14
and above, and it allows streaming of large in-progress transactions.
</para>
<para>
Version <literal>3</literal> is supported only for server version 15
and above, and it allows streaming of two-phase commits.
</para>
<para>
@@ -3115,22 +3124,93 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
<varlistentry>
<term>
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
- (receive changes). The individual publication names are treated
- as standard objects names and can be quoted the same as needed.
+ (receive changes). Required. The individual publication names are
+ treated as standard objects names and can be quoted the same as needed.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 14. -->
+ <varlistentry>
+ <term>
+ binary
+ </term>
+ <listitem>
+ <para>
+ Boolean option to use binary transfer mode. Binary mode is faster
+ than the text mode but slightly less robust.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 14. -->
+ <varlistentry>
+ <term>
+ messages
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable sending the messages that are written
+ by <function>pg_logical_emit_message</function>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 14, but remove "parallel" part until version 16. -->
+ <varlistentry>
+ <term>
+ streaming
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable streaming of in-progress transactions.
+ It accepts an additional value "parallel" to enable sending extra
+ information with some messages to be used for parallelisation.
+ Minimum protocol version 2 is required to turn it on. Minimum protocol
+ version 4 is required for the "parallel" option.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 15. -->
+ <varlistentry>
+ <term>
+ two_phase
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable two-phase transactions. Minimum protocol
+ version 3 is required to turn it on.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 16. -->
+ <varlistentry>
+ <term>
+ origin
+ </term>
+ <listitem>
+ <para>
+ Option to send changes by their origin. Possible values are "none"
+ to only send the changes that have no origin associated, or "any"
+ to send the changes regardless of their origin. This can be used
+ to avoid loops (infinite replication of the same data) among
+ replication nodes.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
--
2.39.3 (Apple Git-145)
Re: "pgoutput" options missing on documentation
Hi, I had a look at the latest v02 patches.
On Sat, Dec 16, 2023 at 12:36 AM Emre Hasegeli <emre@hasegeli.com> wrote:
OK, to deal with that can't you just include "origin" in the first
group which has no special protocol requirements?I think it'd be confusing because the option is not available before
version 16. I think it should really check for the version number and
complain if it's less than 4.
Hm. I don't think a proto_version check is required for "origin".
IIUC, the protocol version number indicates the message byte format.
It's needed so that those messages bytes can be read/written in the
same/compatible way. OTOH I thought the "origin" option has nothing
really to do with actual message formats on the wire; I think it works
just by filtering up-front to decide either to send the changes or not
send the changes. For example, so long as PostgreSQL >= v16, I expect
you could probably use "origin" with any proto_version you wanted.
SUGGESTION
-proto_version
-publication_names
-binary
-messages
-originRequires minimum protocol version 2:
-streaming (boolean)Requires minimum protocol version 3:
-two_phaseRequires minimum protocol version 4:
-streaming (parallel)I am still not sure if this is any better. I don't like that
"streaming" appears twice, and I wouldn't know how to format this
nicely.
I won't keep pushing to rearrange the docs. I think all the content is
OK anyway, so let's see if other people have any opinions on how the
new information is best presented.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
Re: "pgoutput" options missing on documentation
On Mon, Dec 18, 2023 at 11:35 AM Peter Smith <smithpb2250@gmail.com> wrote:
Hi, I had a look at the latest v02 patches.
On Sat, Dec 16, 2023 at 12:36 AM Emre Hasegeli <emre@hasegeli.com> wrote:
OK, to deal with that can't you just include "origin" in the first
group which has no special protocol requirements?I think it'd be confusing because the option is not available before
version 16. I think it should really check for the version number and
complain if it's less than 4.Hm. I don't think a proto_version check is required for "origin".
IIUC, the protocol version number indicates the message byte format.
It's needed so that those messages bytes can be read/written in the
same/compatible way. OTOH I thought the "origin" option has nothing
really to do with actual message formats on the wire; I think it works
just by filtering up-front to decide either to send the changes or not
send the changes. For example, so long as PostgreSQL >= v16, I expect
you could probably use "origin" with any proto_version you wanted.
But, I don't know how the user would be able to arrange for such a
mixture of PG/proto_version versions. because they do seem tightly
coupled for pgoutput.
e.g.
server_version = walrcv_server_version(LogRepWorkerWalRcvConn);
options.proto.logical.proto_version =
server_version >= 160000 ?
LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM :
server_version >= 150000 ? LOGICALREP_PROTO_TWOPHASE_VERSION_NUM :
server_version >= 140000 ? LOGICALREP_PROTO_STREAM_VERSION_NUM :
LOGICALREP_PROTO_VERSION_NUM;
======
Kind Regards,
Peter Smith.
Fujitsu Australia
Re: "pgoutput" options missing on documentation
On Fri, Dec 15, 2023 at 7:06 PM Emre Hasegeli <emre@hasegeli.com> wrote:
I saw that the original "publication_names" error was using
errcode(ERRCODE_INVALID_PARAMETER_VALUE), but TBH since there is no
option given at all I felt ERRCODE_SYNTAX_ERROR might be more
appropriate errcode for those 2 mandatory option errors.It makes sense to me. Changed.
I found the existing error code appropriate because for syntax
specification, either we need to mandate this at the grammar level or
at the API level. Also, I think we should give a message similar to an
existing message: "publication_names parameter missing". For example,
we can say, "proto_version parameter missing". BTW, I also don't like
the other changes parse_output_parameters() done in 0001, if we want
to improve all the similar messages there are other places in the code
as well, so we can separately make the case for the same.
--
With Regards,
Amit Kapila.
Re: "pgoutput" options missing on documentation
On Fri, Dec 15, 2023 at 7:06 PM Emre Hasegeli <emre@hasegeli.com> wrote:
SUGGESTION
-proto_version
-publication_names
-binary
-messages
-originRequires minimum protocol version 2:
-streaming (boolean)Requires minimum protocol version 3:
-two_phaseRequires minimum protocol version 4:
-streaming (parallel)I am still not sure if this is any better. I don't like that
"streaming" appears twice, and I wouldn't know how to format this
nicely.
The currently proposed way seems reasonable to me.
The new versions are attached.
I also added "Required." for "proto_version" and "publication_names".
Comma separated list of publication names for which to subscribe
- (receive changes). The individual publication names are treated
- as standard objects names and can be quoted the same as needed.
+ (receive changes). Required. The individual publication names areThis change (Required in between two sentences) looks slightly odd to
me. Can we instead extend the second line to something like: "This
parameter is required, and the individual publication names are ...".
Similarly we can adjust the proto_vesion explanation.
One minor comment:
====================
+ <para>
+ <productname>PostgreSQL</productname> supports extensible logical decoding
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>This sounds like we are supporting more than one logical decoding
plugin. Can we slightly rephrase it to something like:
"PostgreSQL</productname> supports extensible logical decoding plugin
<literal>pgoutput</literal> which is used for the built-in logical
replication as well."
--
With Regards,
Amit Kapila.
Re: "pgoutput" options missing on documentation
I found the existing error code appropriate because for syntax
specification, either we need to mandate this at the grammar level or
at the API level. Also, I think we should give a message similar to an
existing message: "publication_names parameter missing". For example,
we can say, "proto_version parameter missing". BTW, I also don't like
the other changes parse_output_parameters() done in 0001, if we want
to improve all the similar messages there are other places in the code
as well, so we can separately make the case for the same.
Okay, I am changing these back. I think we should keep the word
"option". It is used on other error messages.
Re: "pgoutput" options missing on documentation
This change (Required in between two sentences) looks slightly odd to
me. Can we instead extend the second line to something like: "This
parameter is required, and the individual publication names are ...".
Similarly we can adjust the proto_vesion explanation.
I don't think it's an improvement to join 2 independent sentences with
a comma. I expanded these by mentioning what is required.
This sounds like we are supporting more than one logical decoding
plugin. Can we slightly rephrase it to something like:
"PostgreSQL</productname> supports extensible logical decoding plugin
<literal>pgoutput</literal> which is used for the built-in logical
replication as well."
I understand the confusion. I reworded it and dropped "extensible".
The new versions are attached.
Attachments:
v03-0002-doc-Clarify-pgoutput-options.patchapplication/octet-stream; name=v03-0002-doc-Clarify-pgoutput-options.patchDownload
From 545a53c46a6a505c2ce536288ce65e1357534c24 Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH v03 2/2] doc: Clarify pgoutput options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 12
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
doc/src/sgml/logical-replication.sgml | 5 +-
doc/src/sgml/protocol.sgml | 90 +++++++++++++++++++++++++--
2 files changed, 89 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 1bb85e4ad9..ec2130669e 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -528,21 +528,21 @@ test_sub=# SELECT * FROM t3;
<sect2 id="logical-replication-subscription-examples-deferred-slot">
<title>Examples: Deferred Replication Slot Creation</title>
<para>
There are some cases (e.g.
<xref linkend="logical-replication-subscription-slot"/>) where, if the
remote replication slot was not created automatically, the user must create
it manually before the subscription can be activated. The steps to create
the slot and activate the subscription are shown in the following examples.
- These examples specify the standard logical decoding plugin
+ These examples specify the standard logical decoding output plugin
(<literal>pgoutput</literal>), which is what the built-in logical
replication uses.
</para>
<para>
First, create a publication for the examples to use.
<programlisting>
test_pub=# CREATE PUBLICATION pub1 FOR ALL TABLES;
CREATE PUBLICATION
</programlisting></para>
<para>
@@ -1655,21 +1655,22 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <literal>walsender</literal> and <literal>apply</literal>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding output plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index af3f016f74..0ea93f149b 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2538,21 +2538,24 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<para>
The WAL location to begin streaming at.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_name</replaceable></term>
<listitem>
<para>
- The name of an option passed to the slot's logical decoding plugin.
+ The name of an option passed to the slot's logical decoding output
+ plugin. See <xref linkend="protocol-logical-replication"/> for
+ options that are accepted by the standard (<literal>pgoutput</literal>)
+ plugin.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_value</replaceable></term>
<listitem>
<para>
Optional value, in the form of a string constant, associated with the
specified option.
@@ -3075,36 +3078,43 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> logical decoding supports output
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal> accepts the following options:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
Protocol version. Currently versions <literal>1</literal>, <literal>2</literal>,
- <literal>3</literal>, and <literal>4</literal> are supported.
+ <literal>3</literal>, and <literal>4</literal> are supported. A valid
+ version is required.
</para>
<para>
Version <literal>2</literal> is supported only for server version 14
and above, and it allows streaming of large in-progress transactions.
</para>
<para>
Version <literal>3</literal> is supported only for server version 15
and above, and it allows streaming of two-phase commits.
</para>
<para>
@@ -3117,20 +3127,92 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<varlistentry>
<term>
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
(receive changes). The individual publication names are treated
as standard objects names and can be quoted the same as needed.
+ At least one publication name is required.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 14. -->
+ <varlistentry>
+ <term>
+ binary
+ </term>
+ <listitem>
+ <para>
+ Boolean option to use binary transfer mode. Binary mode is faster
+ than the text mode but slightly less robust.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 14. -->
+ <varlistentry>
+ <term>
+ messages
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable sending the messages that are written
+ by <function>pg_logical_emit_message</function>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 14, but remove "parallel" part until version 16. -->
+ <varlistentry>
+ <term>
+ streaming
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable streaming of in-progress transactions.
+ It accepts an additional value "parallel" to enable sending extra
+ information with some messages to be used for parallelisation.
+ Minimum protocol version 2 is required to turn it on. Minimum protocol
+ version 4 is required for the "parallel" option.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 15. -->
+ <varlistentry>
+ <term>
+ two_phase
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable two-phase transactions. Minimum protocol
+ version 3 is required to turn it on.
+ </para>
+ </listitem>
+ </varlistentry>
+
+<!-- Backpack through version 16. -->
+ <varlistentry>
+ <term>
+ origin
+ </term>
+ <listitem>
+ <para>
+ Option to send changes by their origin. Possible values are "none"
+ to only send the changes that have no origin associated, or "any"
+ to send the changes regardless of their origin. This can be used
+ to avoid loops (infinite replication of the same data) among
+ replication nodes.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
--
2.39.3 (Apple Git-145)
v03-0001-pgoutput-Test-if-proto_version-is-given.patchapplication/octet-stream; name=v03-0001-pgoutput-Test-if-proto_version-is-given.patchDownload
From 57deda3c2a705b41ce5a1816f59819d5da4bc6fd Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 13 Mar 2023 16:54:18 +0100
Subject: [PATCH v03 1/2] pgoutput: Test if proto_version is given
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 12
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
src/backend/replication/pgoutput/pgoutput.c | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index f9ed1083df..25a95076cf 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -393,20 +393,30 @@ parse_output_parameters(List *options, PGOutputData *data)
else if (pg_strcasecmp(origin, LOGICALREP_ORIGIN_ANY) == 0)
data->publish_no_origin = false;
else
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized origin value: \"%s\"", origin));
}
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+ /* Check required options */
+ if (!protocol_version_given)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("proto_version option missing"));
+ if (!publication_names_given)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("publication_names option missing"));
}
/*
* Initialize this plugin
*/
static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
PGOutputData *data = palloc0(sizeof(PGOutputData));
@@ -442,25 +452,20 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("client sent proto_version=%d but server only supports protocol %d or lower",
data->protocol_version, LOGICALREP_PROTO_MAX_VERSION_NUM)));
if (data->protocol_version < LOGICALREP_PROTO_MIN_VERSION_NUM)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("client sent proto_version=%d but server only supports protocol %d or higher",
data->protocol_version, LOGICALREP_PROTO_MIN_VERSION_NUM)));
- if (data->publication_names == NIL)
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("publication_names parameter missing")));
-
/*
* Decide whether to enable streaming. It is disabled by default, in
* which case we just update the flag in decoding context. Otherwise
* we only allow it with sufficient version of the protocol, and when
* the output plugin supports it.
*/
if (data->streaming == LOGICALREP_STREAM_OFF)
ctx->streaming = false;
else if (data->streaming == LOGICALREP_STREAM_ON &&
data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
--
2.39.3 (Apple Git-145)
Re: "pgoutput" options missing on documentation
On Mon, Dec 18, 2023 at 1:08 PM Emre Hasegeli <emre@hasegeli.com> wrote:
I found the existing error code appropriate because for syntax
specification, either we need to mandate this at the grammar level or
at the API level. Also, I think we should give a message similar to an
existing message: "publication_names parameter missing". For example,
we can say, "proto_version parameter missing". BTW, I also don't like
the other changes parse_output_parameters() done in 0001, if we want
to improve all the similar messages there are other places in the code
as well, so we can separately make the case for the same.Okay, I am changing these back. I think we should keep the word
"option". It is used on other error messages.
Fair enough. I think we should push your first patch only in HEAD as
this is a minor improvement over the current behaviour. What do you
think?
--
With Regards,
Amit Kapila.
Re: "pgoutput" options missing on documentation
Fair enough. I think we should push your first patch only in HEAD as
this is a minor improvement over the current behaviour. What do you
think?
I agree.
Re: "pgoutput" options missing on documentation
On Tue, Dec 19, 2023 at 1:35 AM Emre Hasegeli <emre@hasegeli.com> wrote:
Fair enough. I think we should push your first patch only in HEAD as
this is a minor improvement over the current behaviour. What do you
think?I agree.
Patch 0001
AFAICT parse_output_parameters possible errors are never tested. For
example, there is no code coverage [1]code coverage -- https://coverage.postgresql.org/src/backend/replication/pgoutput/pgoutput.c.gcov.html touching any of these ereports.
IMO there should be some simple test cases -- I am happy to create
some tests if you agree they should exist.
~~~
While looking at the function parse_output_parameters() I noticed that
if an unrecognised option is passed the function emits an elog instead
of an ereport
------
test_pub=# SELECT * FROM pg_logical_slot_get_changes('test_slot_v1',
NULL, NULL, 'banana', '1');
2023-12-19 17:08:21.627 AEDT [8921] ERROR: unrecognized pgoutput option: banana
2023-12-19 17:08:21.627 AEDT [8921] CONTEXT: slot "test_slot_v1",
output plugin "pgoutput", in the startup callback
2023-12-19 17:08:21.627 AEDT [8921] STATEMENT: SELECT * FROM
pg_logical_slot_get_changes('test_slot_v1', NULL, NULL, 'banana',
'1');
ERROR: unrecognized pgoutput option: banana
CONTEXT: slot "test_slot_v1", output plugin "pgoutput", in the startup callback
------
But that doesn't seem right. AFAIK elog messages use errmsg_internal
so this message would not get translated.
PSA a patch to fix that.
======
[1]: code coverage -- https://coverage.postgresql.org/src/backend/replication/pgoutput/pgoutput.c.gcov.html
https://coverage.postgresql.org/src/backend/replication/pgoutput/pgoutput.c.gcov.html
Kind Regards,
Peter Smith.
Fujitsu Australia
Attachments:
parse_output_parameters.diffapplication/octet-stream; name=parse_output_parameters.diffDownload
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 25a9507..e160173 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -398,7 +398,9 @@ parse_output_parameters(List *options, PGOutputData *data)
errmsg("unrecognized origin value: \"%s\"", origin));
}
else
- elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("unrecognized pgoutput option: %s", defel->defname));
}
/* Check required options */
Re: "pgoutput" options missing on documentation
On Tue, Dec 19, 2023 at 12:07 PM Peter Smith <smithpb2250@gmail.com> wrote:
On Tue, Dec 19, 2023 at 1:35 AM Emre Hasegeli <emre@hasegeli.com> wrote:
Fair enough. I think we should push your first patch only in HEAD as
this is a minor improvement over the current behaviour. What do you
think?I agree.
Patch 0001
AFAICT parse_output_parameters possible errors are never tested. For
example, there is no code coverage [1] touching any of these ereports.IMO there should be some simple test cases -- I am happy to create
some tests if you agree they should exist.
I don't think having tests for all sorts of error checking will add
much value as compared to the overhead they bring.
~~~
While looking at the function parse_output_parameters() I noticed that
if an unrecognised option is passed the function emits an elog instead
of an ereport
We don't expect unrecognized option here and for such a thing, we use
elog in the code. See the similar usage in
parseCreateReplSlotOptions().
I think we should move to 0002 patch now. In that, I suggest preparing
separate back branch patches.
--
With Regards,
Amit Kapila.
Re: "pgoutput" options missing on documentation
On Tue, Dec 19, 2023 at 6:25 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Tue, Dec 19, 2023 at 12:07 PM Peter Smith <smithpb2250@gmail.com> wrote:
On Tue, Dec 19, 2023 at 1:35 AM Emre Hasegeli <emre@hasegeli.com> wrote:
Fair enough. I think we should push your first patch only in HEAD as
this is a minor improvement over the current behaviour. What do you
think?I agree.
Patch 0001
AFAICT parse_output_parameters possible errors are never tested. For
example, there is no code coverage [1] touching any of these ereports.IMO there should be some simple test cases -- I am happy to create
some tests if you agree they should exist.I don't think having tests for all sorts of error checking will add
much value as compared to the overhead they bring.~~~
While looking at the function parse_output_parameters() I noticed that
if an unrecognised option is passed the function emits an elog instead
of an ereportWe don't expect unrecognized option here and for such a thing, we use
elog in the code. See the similar usage in
parseCreateReplSlotOptions().
IIUC the untranslated elog should be used for internal/sanity errors,
debugging, or stuff that cannot happen under any normal circumstances.
While that may be the case for parseCreateReplSlotOptions() mentioned,
IMO the scenario in the parse_output_parameters() is very different,
because these options can come directly from user input so any user
typo can cause this error. Indeed, this is probably one of the more
likely reasons for getting any error in parse_output_parameters()
function. I thought any errors that can be easily caused by some user
actions ought to be translated.
For example, the user accidentally misspells 'proto_version':
test_pub=# SELECT * FROM pg_logical_slot_get_changes('test_slot_v1',
NULL, NULL, 'protocol_version', '1');
ERROR: unrecognized pgoutput option: protocol_version
CONTEXT: slot "test_slot_v1", output plugin "pgoutput", in the startup callback
======
Kind Regards,
Peter Smith.
Fujitsu Australia
Re: "pgoutput" options missing on documentation
On Tue, Dec 19, 2023 at 12:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
I think we should move to 0002 patch now. In that, I suggest preparing
separate back branch patches.
Emre, are you planning to share back-branch patches?
--
With Regards,
Amit Kapila.
Re: "pgoutput" options missing on documentation
We don't expect unrecognized option here and for such a thing, we use
elog in the code. See the similar usage in
parseCreateReplSlotOptions().
"pgoutput" is useful for a lot of applications other than our logical
replication subscriber. I think we should expect anything and handle
errors nicely.
I think we should move to 0002 patch now. In that, I suggest preparing
separate back branch patches.
They are attached.
Attachments:
v13-0001-doc-Clarify-pgoutput-options.patchapplication/octet-stream; name=v13-0001-doc-Clarify-pgoutput-options.patchDownload
From ec41ab8554c57c8adde4dc56f5b8441eca458bda Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH v13] doc: Clarify pgoutput options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 12
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
doc/src/sgml/logical-replication.sgml | 3 ++-
doc/src/sgml/protocol.sgml | 11 +++++++++--
2 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 1ac374ce40..77c7404568 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -435,21 +435,22 @@
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <quote>walsender</quote> and <quote>apply</quote>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding output plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 8670230d48..4c6984c718 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2856,26 +2856,32 @@ The commands accepted in replication mode are:
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> logical decoding supports output
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal> accepts the following options:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
Protocol version. Currently only version <literal>1</literal> is
supported.
@@ -2885,20 +2891,21 @@ The commands accepted in replication mode are:
<varlistentry>
<term>
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
(receive changes). The individual publication names are treated
as standard objects names and can be quoted the same as needed.
+ At least one publication name is required.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
--
2.39.3 (Apple Git-145)
master-0001-doc-Clarify-pgoutput-options.patchapplication/octet-stream; name=master-0001-doc-Clarify-pgoutput-options.patchDownload
From d1a4e0ea7eab1a46b2ce41af2539b790ff695e12 Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH] doc: Clarify pgoutput options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 12
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
doc/src/sgml/logical-replication.sgml | 5 +-
doc/src/sgml/protocol.sgml | 85 +++++++++++++++++++++++++--
2 files changed, 84 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 1bb85e4ad9..ec2130669e 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -528,21 +528,21 @@ test_sub=# SELECT * FROM t3;
<sect2 id="logical-replication-subscription-examples-deferred-slot">
<title>Examples: Deferred Replication Slot Creation</title>
<para>
There are some cases (e.g.
<xref linkend="logical-replication-subscription-slot"/>) where, if the
remote replication slot was not created automatically, the user must create
it manually before the subscription can be activated. The steps to create
the slot and activate the subscription are shown in the following examples.
- These examples specify the standard logical decoding plugin
+ These examples specify the standard logical decoding output plugin
(<literal>pgoutput</literal>), which is what the built-in logical
replication uses.
</para>
<para>
First, create a publication for the examples to use.
<programlisting>
test_pub=# CREATE PUBLICATION pub1 FOR ALL TABLES;
CREATE PUBLICATION
</programlisting></para>
<para>
@@ -1655,21 +1655,22 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <literal>walsender</literal> and <literal>apply</literal>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding output plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9a66918171..6c3e8a631d 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2538,21 +2538,24 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<para>
The WAL location to begin streaming at.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_name</replaceable></term>
<listitem>
<para>
- The name of an option passed to the slot's logical decoding plugin.
+ The name of an option passed to the slot's logical decoding output
+ plugin. See <xref linkend="protocol-logical-replication"/> for
+ options that are accepted by the standard (<literal>pgoutput</literal>)
+ plugin.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_value</replaceable></term>
<listitem>
<para>
Optional value, in the form of a string constant, associated with the
specified option.
@@ -3099,36 +3102,43 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> logical decoding supports output
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal> accepts the following options:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
Protocol version. Currently versions <literal>1</literal>, <literal>2</literal>,
- <literal>3</literal>, and <literal>4</literal> are supported.
+ <literal>3</literal>, and <literal>4</literal> are supported. A valid
+ version is required.
</para>
<para>
Version <literal>2</literal> is supported only for server version 14
and above, and it allows streaming of large in-progress transactions.
</para>
<para>
Version <literal>3</literal> is supported only for server version 15
and above, and it allows streaming of two-phase commits.
</para>
<para>
@@ -3141,20 +3151,87 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<varlistentry>
<term>
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
(receive changes). The individual publication names are treated
as standard objects names and can be quoted the same as needed.
+ At least one publication name is required.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ binary
+ </term>
+ <listitem>
+ <para>
+ Boolean option to use binary transfer mode. Binary mode is faster
+ than the text mode but slightly less robust.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ messages
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable sending the messages that are written
+ by <function>pg_logical_emit_message</function>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ streaming
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable streaming of in-progress transactions.
+ It accepts an additional value "parallel" to enable sending extra
+ information with some messages to be used for parallelisation.
+ Minimum protocol version 2 is required to turn it on. Minimum protocol
+ version 4 is required for the "parallel" option.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ two_phase
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable two-phase transactions. Minimum protocol
+ version 3 is required to turn it on.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ origin
+ </term>
+ <listitem>
+ <para>
+ Option to send changes by their origin. Possible values are "none"
+ to only send the changes that have no origin associated, or "any"
+ to send the changes regardless of their origin. This can be used
+ to avoid loops (infinite replication of the same data) among
+ replication nodes.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
--
2.39.3 (Apple Git-145)
v15-0001-doc-Clarify-pgoutput-options.patchapplication/octet-stream; name=v15-0001-doc-Clarify-pgoutput-options.patchDownload
From 48c1472a066e04465f7061a5abd70ec1ae326110 Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH v15] doc: Clarify pgoutput options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 12
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
doc/src/sgml/logical-replication.sgml | 3 +-
doc/src/sgml/protocol.sgml | 66 +++++++++++++++++++++++++--
2 files changed, 64 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index bc3f5ec78d..4abfb32f77 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1444,21 +1444,22 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <quote>walsender</quote> and <quote>apply</quote>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding output plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 405046f83c..fdd3d05148 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2552,21 +2552,24 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<para>
The WAL location to begin streaming at.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_name</replaceable></term>
<listitem>
<para>
- The name of an option passed to the slot's logical decoding plugin.
+ The name of an option passed to the slot's logical decoding output
+ plugin. See <xref linkend="protocol-logical-replication"/> for
+ options that are accepted by the standard (<literal>pgoutput</literal>)
+ plugin.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_value</replaceable></term>
<listitem>
<para>
Optional value, in the form of a string constant, associated with the
specified option.
@@ -3081,36 +3084,42 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> logical decoding supports output
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal> accepts the following options:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
Protocol version. Currently versions <literal>1</literal>, <literal>2</literal>,
- and <literal>3</literal> are supported.
+ and <literal>3</literal> are supported. A valid version is required.
</para>
<para>
Version <literal>2</literal> is supported only for server version 14
and above, and it allows streaming of large in-progress transactions.
</para>
<para>
Version <literal>3</literal> is supported only for server version 15
and above, and it allows streaming of two-phase commits.
</para>
</listitem>
@@ -3118,20 +3127,69 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<varlistentry>
<term>
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
(receive changes). The individual publication names are treated
as standard objects names and can be quoted the same as needed.
+ At least one publication name is required.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ binary
+ </term>
+ <listitem>
+ <para>
+ Boolean option to use binary transfer mode. Binary mode is faster
+ than the text mode but slightly less robust.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ messages
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable sending the messages that are written
+ by <function>pg_logical_emit_message</function>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ streaming
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable streaming of in-progress transactions.
+ Minimum protocol version 2 is required to turn it on.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ two_phase
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable two-phase transactions. Minimum protocol
+ version 3 is required to turn it on.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
--
2.39.3 (Apple Git-145)
v12-0001-doc-Clarify-pgoutput-options.patchapplication/octet-stream; name=v12-0001-doc-Clarify-pgoutput-options.patchDownload
From 674958d83d25d4e00090befbcf631d5cdc2f7934 Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH v12] doc: Clarify pgoutput options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 12
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
doc/src/sgml/logical-replication.sgml | 3 ++-
doc/src/sgml/protocol.sgml | 11 +++++++++--
2 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index eb4878a1c9..e0e1967c67 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -425,21 +425,22 @@
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <quote>walsender</quote> and <quote>apply</quote>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding output plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 23ee0b84b2..434c9633d7 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2816,26 +2816,32 @@ The commands accepted in replication mode are:
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> logical decoding supports output
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal> accepts the following options:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
Protocol version. Currently only version <literal>1</literal> is
supported.
@@ -2845,20 +2851,21 @@ The commands accepted in replication mode are:
<varlistentry>
<term>
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
(receive changes). The individual publication names are treated
as standard objects names and can be quoted the same as needed.
+ At least one publication name is required.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
--
2.39.3 (Apple Git-145)
v14-0001-doc-Clarify-pgoutput-options.patchapplication/octet-stream; name=v14-0001-doc-Clarify-pgoutput-options.patchDownload
From 2b5dbabe0a75317ee60a908b52f862e1c22ae4fe Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH v14] doc: Clarify pgoutput options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 12
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
doc/src/sgml/logical-replication.sgml | 3 +-
doc/src/sgml/protocol.sgml | 50 +++++++++++++++++++++++++--
2 files changed, 49 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 5ff37a2018..eeddea3e52 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -447,21 +447,22 @@
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <quote>walsender</quote> and <quote>apply</quote>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding output plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 7141f6c277..bb3f3bc37d 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2863,51 +2863,95 @@ The commands accepted in replication mode are:
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> logical decoding supports output
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal> accepts the following options:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
Protocol version. Currently versions <literal>1</literal> and
- <literal>2</literal> are supported. The version <literal>2</literal>
+ <literal>2</literal> are supported. A valid version is required.
+ The version <literal>2</literal>
is supported only for server version 14 and above, and it allows
streaming of large in-progress transactions.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
(receive changes). The individual publication names are treated
as standard objects names and can be quoted the same as needed.
+ At least one publication name is required.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ binary
+ </term>
+ <listitem>
+ <para>
+ Boolean option to use binary transfer mode. Binary mode is faster
+ than the text mode but slightly less robust.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ messages
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable sending the messages that are written
+ by <function>pg_logical_emit_message</function>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ streaming
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable streaming of in-progress transactions.
+ Minimum protocol version 2 is required to turn it on.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
--
2.39.3 (Apple Git-145)
v16-0001-doc-Clarify-pgoutput-options.patchapplication/octet-stream; name=v16-0001-doc-Clarify-pgoutput-options.patchDownload
From e4d2fae2907bd986a90b547cf02f0a3c51e6cbf5 Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH v16] doc: Clarify pgoutput options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 12
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
doc/src/sgml/logical-replication.sgml | 5 +-
doc/src/sgml/protocol.sgml | 85 +++++++++++++++++++++++++--
2 files changed, 84 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index c2a749d882..ca852c5506 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -525,21 +525,21 @@ test_sub=# SELECT * FROM t3;
<sect2 id="logical-replication-subscription-examples-deferred-slot">
<title>Examples: Deferred Replication Slot Creation</title>
<para>
There are some cases (e.g.
<xref linkend="logical-replication-subscription-slot"/>) where, if the
remote replication slot was not created automatically, the user must create
it manually before the subscription can be activated. The steps to create
the slot and activate the subscription are shown in the following examples.
- These examples specify the standard logical decoding plugin
+ These examples specify the standard logical decoding output plugin
(<literal>pgoutput</literal>), which is what the built-in logical
replication uses.
</para>
<para>
First, create a publication for the examples to use.
<programlisting>
test_pub=# CREATE PUBLICATION pub1 FOR ALL TABLES;
CREATE PUBLICATION
</programlisting></para>
<para>
@@ -1636,21 +1636,22 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <literal>walsender</literal> and <literal>apply</literal>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding output plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index b11d9a6ba3..5dff07110d 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2534,21 +2534,24 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<para>
The WAL location to begin streaming at.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_name</replaceable></term>
<listitem>
<para>
- The name of an option passed to the slot's logical decoding plugin.
+ The name of an option passed to the slot's logical decoding output
+ plugin. See <xref linkend="protocol-logical-replication"/> for
+ options that are accepted by the standard (<literal>pgoutput</literal>)
+ plugin.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_value</replaceable></term>
<listitem>
<para>
Optional value, in the form of a string constant, associated with the
specified option.
@@ -3071,36 +3074,43 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> logical decoding supports output
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal> accepts the following options:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
Protocol version. Currently versions <literal>1</literal>, <literal>2</literal>,
- <literal>3</literal>, and <literal>4</literal> are supported.
+ <literal>3</literal>, and <literal>4</literal> are supported. A valid
+ version is required.
</para>
<para>
Version <literal>2</literal> is supported only for server version 14
and above, and it allows streaming of large in-progress transactions.
</para>
<para>
Version <literal>3</literal> is supported only for server version 15
and above, and it allows streaming of two-phase commits.
</para>
<para>
@@ -3113,20 +3123,87 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
<varlistentry>
<term>
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
(receive changes). The individual publication names are treated
as standard objects names and can be quoted the same as needed.
+ At least one publication name is required.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ binary
+ </term>
+ <listitem>
+ <para>
+ Boolean option to use binary transfer mode. Binary mode is faster
+ than the text mode but slightly less robust.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ messages
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable sending the messages that are written
+ by <function>pg_logical_emit_message</function>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ streaming
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable streaming of in-progress transactions.
+ It accepts an additional value "parallel" to enable sending extra
+ information with some messages to be used for parallelisation.
+ Minimum protocol version 2 is required to turn it on. Minimum protocol
+ version 4 is required for the "parallel" option.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ two_phase
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable two-phase transactions. Minimum protocol
+ version 3 is required to turn it on.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ origin
+ </term>
+ <listitem>
+ <para>
+ Option to send changes by their origin. Possible values are "none"
+ to only send the changes that have no origin associated, or "any"
+ to send the changes regardless of their origin. This can be used
+ to avoid loops (infinite replication of the same data) among
+ replication nodes.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
--
2.39.3 (Apple Git-145)
Re: "pgoutput" options missing on documentation
On Thu, Dec 21, 2023 at 2:58 AM Emre Hasegeli <emre@hasegeli.com> wrote:
We don't expect unrecognized option here and for such a thing, we use
elog in the code. See the similar usage in
parseCreateReplSlotOptions()."pgoutput" is useful for a lot of applications other than our logical
replication subscriber. I think we should expect anything and handle
errors nicely.I think we should move to 0002 patch now. In that, I suggest preparing
separate back branch patches.They are attached.
Hi, I checked (just by visual inspection and diffs) the provided
backpatches and I have a couple of questions:
======
1.
The Chapter "Streaming Replication Protocol" START_REPLICATION /
"option_name" part has an xref to the pgoutput options page
e.g. master
- The name of an option passed to the slot's logical decoding plugin.
+ The name of an option passed to the slot's logical decoding output
+ plugin. See <xref linkend="protocol-logical-replication"/> for
+ options that are accepted by the standard
(<literal>pgoutput</literal>)
+ plugin.But the xref seems present only in the master/v16/v15 patches, but not
for the earlier patches v14/v13/v12. Why not?
~~~
2.
The proto_version part now says "A valid version is required.".
e.g. master
- <literal>3</literal>, and <literal>4</literal> are supported.
+ <literal>3</literal>, and <literal>4</literal> are supported. A valid
+ version is required.But the change was only in the patches v14 onwards. Although the new
error message was only added for HEAD, isn't it still correct to say
"A valid version is required." for all the patches including v12 and
v13?
======
Kind Regards,
Peter Smith.
Fujitsu Australia
Re: "pgoutput" options missing on documentation
But the xref seems present only in the master/v16/v15 patches, but not
for the earlier patches v14/v13/v12. Why not?
I missed it.
But the change was only in the patches v14 onwards. Although the new
error message was only added for HEAD, isn't it still correct to say
"A valid version is required." for all the patches including v12 and
v13?
Yes, it's still correct.
Fixed versions are attached.
Attachments:
v12-0001-doc-Clarify-pgoutput-options.patchapplication/octet-stream; name=v12-0001-doc-Clarify-pgoutput-options.patchDownload
From 37c1d8f7619356f19c0d28993c9122133acea010 Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH v12] doc: Clarify pgoutput options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 12
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
doc/src/sgml/logical-replication.sgml | 3 ++-
doc/src/sgml/protocol.sgml | 18 ++++++++++++++----
2 files changed, 16 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index eb4878a1c9..e0e1967c67 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -425,21 +425,22 @@
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <quote>walsender</quote> and <quote>apply</quote>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding output plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 23ee0b84b2..5482f1b792 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2508,21 +2508,24 @@ The commands accepted in replication mode are:
<listitem>
<para>
The WAL location to begin streaming at.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_name</replaceable></term>
<listitem>
<para>
- The name of an option passed to the slot's logical decoding plugin.
+ The name of an option passed to the slot's logical decoding output
+ plugin. See <xref linkend="protocol-logical-replication"/> for
+ options that are accepted by the standard (<literal>pgoutput</literal>)
+ plugin.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_value</replaceable></term>
<listitem>
<para>
Optional value, in the form of a string constant, associated with the
specified option.
</para>
@@ -2816,49 +2819,56 @@ The commands accepted in replication mode are:
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> logical decoding supports output
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal> accepts the following options:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
Protocol version. Currently only version <literal>1</literal> is
- supported.
+ supported. A valid version is required.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
(receive changes). The individual publication names are treated
as standard objects names and can be quoted the same as needed.
+ At least one publication name is required.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
--
2.39.3 (Apple Git-145)
v14-0001-doc-Clarify-pgoutput-options.patchapplication/octet-stream; name=v14-0001-doc-Clarify-pgoutput-options.patchDownload
From b98cc614403e57079baa79467b5daef2c5915fe0 Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH v14] doc: Clarify pgoutput options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 12
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
doc/src/sgml/logical-replication.sgml | 3 +-
doc/src/sgml/protocol.sgml | 55 +++++++++++++++++++++++++--
2 files changed, 53 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 5ff37a2018..eeddea3e52 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -447,21 +447,22 @@
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <quote>walsender</quote> and <quote>apply</quote>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding output plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 7141f6c277..a23a103363 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2518,21 +2518,24 @@ The commands accepted in replication mode are:
<listitem>
<para>
The WAL location to begin streaming at.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_name</replaceable></term>
<listitem>
<para>
- The name of an option passed to the slot's logical decoding plugin.
+ The name of an option passed to the slot's logical decoding output
+ plugin. See <xref linkend="protocol-logical-replication"/> for
+ options that are accepted by the standard (<literal>pgoutput</literal>)
+ plugin.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_value</replaceable></term>
<listitem>
<para>
Optional value, in the form of a string constant, associated with the
specified option.
</para>
@@ -2863,51 +2866,95 @@ The commands accepted in replication mode are:
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> logical decoding supports output
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal> accepts the following options:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
Protocol version. Currently versions <literal>1</literal> and
- <literal>2</literal> are supported. The version <literal>2</literal>
+ <literal>2</literal> are supported. A valid version is required.
+ The version <literal>2</literal>
is supported only for server version 14 and above, and it allows
streaming of large in-progress transactions.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
(receive changes). The individual publication names are treated
as standard objects names and can be quoted the same as needed.
+ At least one publication name is required.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ binary
+ </term>
+ <listitem>
+ <para>
+ Boolean option to use binary transfer mode. Binary mode is faster
+ than the text mode but slightly less robust.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ messages
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable sending the messages that are written
+ by <function>pg_logical_emit_message</function>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ streaming
+ </term>
+ <listitem>
+ <para>
+ Boolean option to enable streaming of in-progress transactions.
+ Minimum protocol version 2 is required to turn it on.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
--
2.39.3 (Apple Git-145)
v13-0001-doc-Clarify-pgoutput-options.patchapplication/octet-stream; name=v13-0001-doc-Clarify-pgoutput-options.patchDownload
From 8f981fb2c16dc52b51c195eaad65bb076dfdd2b5 Mon Sep 17 00:00:00 2001
From: Emre Hasegeli <emre@hasegeli.com>
Date: Mon, 20 Mar 2023 17:02:45 +0100
Subject: [PATCH v13] doc: Clarify pgoutput options
Author: Emre Hasegeli <emre@hasegeli.com>
Reviewed-by: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 12
Discussion: https://www.postgresql.org/message-id/flat/CAE2gYzwdwtUbs-tPSV-QBwgTubiyGD2ZGsSnAVsDfAGGLDrGOA%40mail.gmail.com
---
doc/src/sgml/logical-replication.sgml | 3 ++-
doc/src/sgml/protocol.sgml | 18 ++++++++++++++----
2 files changed, 16 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 1ac374ce40..77c7404568 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -435,21 +435,22 @@
transactional consistency is guaranteed for the publications within any
single subscription.
</para>
<para>
Logical replication is built with an architecture similar to physical
streaming replication (see <xref linkend="streaming-replication"/>). It is
implemented by <quote>walsender</quote> and <quote>apply</quote>
processes. The walsender process starts logical decoding (described
in <xref linkend="logicaldecoding"/>) of the WAL and loads the standard
- logical decoding plugin (pgoutput). The plugin transforms the changes read
+ logical decoding output plugin (<literal>pgoutput</literal>). The plugin
+ transforms the changes read
from WAL to the logical replication protocol
(see <xref linkend="protocol-logical-replication"/>) and filters the data
according to the publication specification. The data is then continuously
transferred using the streaming replication protocol to the apply worker,
which maps the data to local tables and applies the individual changes as
they are received, in correct transactional order.
</para>
<para>
The apply process on the subscriber database always runs with
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 8670230d48..a22c8520c0 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -2511,21 +2511,24 @@ The commands accepted in replication mode are:
<listitem>
<para>
The WAL location to begin streaming at.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_name</replaceable></term>
<listitem>
<para>
- The name of an option passed to the slot's logical decoding plugin.
+ The name of an option passed to the slot's logical decoding output
+ plugin. See <xref linkend="protocol-logical-replication"/> for
+ options that are accepted by the standard (<literal>pgoutput</literal>)
+ plugin.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><replaceable class="parameter">option_value</replaceable></term>
<listitem>
<para>
Optional value, in the form of a string constant, associated with the
specified option.
</para>
@@ -2856,49 +2859,56 @@ The commands accepted in replication mode are:
flow started by the <literal>START_REPLICATION</literal>
<literal>SLOT</literal> <replaceable class="parameter">slot_name</replaceable>
<literal>LOGICAL</literal> replication command.
</para>
<para>
The logical streaming replication protocol builds on the primitives of
the physical streaming replication protocol.
</para>
+ <para>
+ <productname>PostgreSQL</productname> logical decoding supports output
+ plugins. <literal>pgoutput</literal> is the standard one used for
+ the built-in logical replication.
+ </para>
+
<sect2 id="protocol-logical-replication-params">
<title>Logical Streaming Replication Parameters</title>
<para>
- The logical replication <literal>START_REPLICATION</literal> command
- accepts following parameters:
+ Using the <literal>START_REPLICATION</literal> command,
+ <literal>pgoutput</literal> accepts the following options:
<variablelist>
<varlistentry>
<term>
proto_version
</term>
<listitem>
<para>
Protocol version. Currently only version <literal>1</literal> is
- supported.
+ supported. A valid version is required.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
publication_names
</term>
<listitem>
<para>
Comma separated list of publication names for which to subscribe
(receive changes). The individual publication names are treated
as standard objects names and can be quoted the same as needed.
+ At least one publication name is required.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="protocol-logical-messages">
<title>Logical Replication Protocol Messages</title>
--
2.39.3 (Apple Git-145)