logical replication empty transactions

Started by Jeff Janesabout 6 years ago108 messages
#1Jeff Janes
jeff.janes@gmail.com

After setting up logical replication of a slowly changing table using the
built in pub/sub facility, I noticed way more network traffic than made
sense. Looking into I see that every transaction in that database on the
master gets sent to the replica. 99.999+% of them are empty transactions
('B' message and 'C' message with nothing in between) because the
transactions don't touch any tables in the publication, only non-replicated
tables. Is doing it this way necessary for some reason? Couldn't we hold
the transmission of 'B' until something else comes along, and then if that
next thing is 'C' drop both of them?

There is a comment for WalSndPrepareWrite which seems to foreshadow such a
thing, but I don't really see how to use it in this case. I want to drop
two messages, not one.

* Don't do anything lasting in here, it's quite possible that nothing will
be done
* with the data.

This applies to all version which have support for pub/sub, including the
recent commits to 13dev.

I've searched through the voluminous mailing list threads for when this
feature was being presented to see if it was already discussed, but since
every word I can think to search on occurs in virtually every message in
the threads in some context or another, I didn't have much luck.

Cheers,

Jeff

#2Euler Taveira
euler@timbira.com.br
In reply to: Jeff Janes (#1)
1 attachment(s)
Re: logical replication empty transactions

Em seg., 21 de out. de 2019 às 21:20, Jeff Janes
<jeff.janes@gmail.com> escreveu:

After setting up logical replication of a slowly changing table using the built in pub/sub facility, I noticed way more network traffic than made sense. Looking into I see that every transaction in that database on the master gets sent to the replica. 99.999+% of them are empty transactions ('B' message and 'C' message with nothing in between) because the transactions don't touch any tables in the publication, only non-replicated tables. Is doing it this way necessary for some reason? Couldn't we hold the transmission of 'B' until something else comes along, and then if that next thing is 'C' drop both of them?

That is not optimal. Those empty transactions is a waste of bandwidth.
We can suppress them if no changes will be sent. test_decoding
implements "skip empty transaction" as you described above and I did
something similar to it. Patch is attached.

--
Euler Taveira Timbira -
http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

Attachments:

0001-Skip-empty-transactions-for-logical-replication.patchtext/x-patch; charset=US-ASCII; name=0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 433ea40a02ab823f3aa70c18928b9862f0eb004b Mon Sep 17 00:00:00 2001
From: Euler Taveira <euler@timbira.com.br>
Date: Fri, 8 Nov 2019 12:48:03 -0300
Subject: [PATCH] Skip empty transactions for logical replication

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit those empty transactions.
Postpone the BEGIN message until the first change. While processing a
COMMIT message, if there is not a previous wrote change for that
transaction, does not send COMMIT message. It means that pgoutput will
skip BEGIN / COMMIT messages for transactions that do not wrote changes.

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/pgoutput/pgoutput.c | 34 +++++++++++++++++++++++++++++
 src/include/replication/pgoutput.h          |  3 +++
 2 files changed, 37 insertions(+)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 9c08757..eed1093 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -212,6 +212,22 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputData	*data = ctx->output_plugin_private;
+
+	/*
+	 * Don't send BEGIN message here. Instead, postpone it until the first
+	 * change. In logical replication, common scenarios is to replicate a set
+	 * of tables (instead of all tables) and transactions whose changes were to
+	 * table(s) that are not published will produce empty transactions. These
+	 * empty transactions will send BEGIN and COMMIT messages to subscribers,
+	 * using bandwidth on something with little/no use for logical replication.
+	 */
+	data->xact_wrote_changes = false;
+}
+
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
@@ -249,8 +265,14 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
+	PGOutputData	*data = ctx->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/* skip COMMIT message if nothing was sent */
+	if (!data->xact_wrote_changes)
+		return;
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
 	OutputPluginWrite(ctx, true);
@@ -335,6 +357,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/* output BEGIN if we haven't yet */
+	if (!data->xact_wrote_changes)
+		pgoutput_begin(ctx, txn);
+
+	data->xact_wrote_changes = true;
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -415,6 +443,12 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/* output BEGIN if we haven't yet */
+		if (!data->xact_wrote_changes)
+			pgoutput_begin(ctx, txn);
+
+		data->xact_wrote_changes = true;
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  nrelids,
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 8870721..cb57e76 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -20,6 +20,9 @@ typedef struct PGOutputData
 	MemoryContext context;		/* private memory context for transient
 								 * allocations */
 
+	/* control wether messages can already be sent */
+	bool		xact_wrote_changes;
+
 	/* client info */
 	uint32		protocol_version;
 
-- 
2.7.4

#3Jeff Janes
jeff.janes@gmail.com
In reply to: Euler Taveira (#2)
Re: logical replication empty transactions

On Fri, Nov 8, 2019 at 8:59 PM Euler Taveira <euler@timbira.com.br> wrote:

Em seg., 21 de out. de 2019 às 21:20, Jeff Janes
<jeff.janes@gmail.com> escreveu:

After setting up logical replication of a slowly changing table using

the built in pub/sub facility, I noticed way more network traffic than made
sense. Looking into I see that every transaction in that database on the
master gets sent to the replica. 99.999+% of them are empty transactions
('B' message and 'C' message with nothing in between) because the
transactions don't touch any tables in the publication, only non-replicated
tables. Is doing it this way necessary for some reason? Couldn't we hold
the transmission of 'B' until something else comes along, and then if that
next thing is 'C' drop both of them?

That is not optimal. Those empty transactions is a waste of bandwidth.
We can suppress them if no changes will be sent. test_decoding
implements "skip empty transaction" as you described above and I did
something similar to it. Patch is attached.

Thanks. I didn't think it would be that simple, because I thought we would
need some way to fake an acknowledgement for any dropped empty
transactions, to keep the LSN advancing and allow WAL to get recycled on
the master. But it turns out the opposite. While your patch drops the
network traffic by a lot, there is still a lot of traffic. Now it is
keep-alives, rather than 'B' and 'C'. I don't know why I am getting a few
hundred keep alives every second when the timeouts are at their defaults,
but it is better than several thousand 'B' and 'C'.

My setup here was just to create, publish, and subscribe to a inactive
dummy table, while having pgbench running on the master (with unpublished
tables). I have not created an intentionally slow network, but I am
testing it over wifi, which is inherently kind of slow.

Cheers,

Jeff

#4Dilip Kumar
dilipbalaut@gmail.com
In reply to: Euler Taveira (#2)
Re: logical replication empty transactions

On Sat, Nov 9, 2019 at 7:29 AM Euler Taveira <euler@timbira.com.br> wrote:

Em seg., 21 de out. de 2019 às 21:20, Jeff Janes
<jeff.janes@gmail.com> escreveu:

After setting up logical replication of a slowly changing table using the built in pub/sub facility, I noticed way more network traffic than made sense. Looking into I see that every transaction in that database on the master gets sent to the replica. 99.999+% of them are empty transactions ('B' message and 'C' message with nothing in between) because the transactions don't touch any tables in the publication, only non-replicated tables. Is doing it this way necessary for some reason? Couldn't we hold the transmission of 'B' until something else comes along, and then if that next thing is 'C' drop both of them?

That is not optimal. Those empty transactions is a waste of bandwidth.
We can suppress them if no changes will be sent. test_decoding
implements "skip empty transaction" as you described above and I did
something similar to it. Patch is attached.

I think this significantly reduces the network bandwidth for empty
transactions. I have briefly reviewed the patch and it looks good to
me.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

#5Amit Kapila
amit.kapila16@gmail.com
In reply to: Dilip Kumar (#4)
Re: logical replication empty transactions

On Mon, Mar 2, 2020 at 9:01 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Sat, Nov 9, 2019 at 7:29 AM Euler Taveira <euler@timbira.com.br> wrote:

Em seg., 21 de out. de 2019 às 21:20, Jeff Janes
<jeff.janes@gmail.com> escreveu:

After setting up logical replication of a slowly changing table using the built in pub/sub facility, I noticed way more network traffic than made sense. Looking into I see that every transaction in that database on the master gets sent to the replica. 99.999+% of them are empty transactions ('B' message and 'C' message with nothing in between) because the transactions don't touch any tables in the publication, only non-replicated tables. Is doing it this way necessary for some reason? Couldn't we hold the transmission of 'B' until something else comes along, and then if that next thing is 'C' drop both of them?

That is not optimal. Those empty transactions is a waste of bandwidth.
We can suppress them if no changes will be sent. test_decoding
implements "skip empty transaction" as you described above and I did
something similar to it. Patch is attached.

I think this significantly reduces the network bandwidth for empty
transactions. I have briefly reviewed the patch and it looks good to
me.

One thing that is not clear to me is how will we advance restart_lsn
if we don't send any empty xact in a system where there are many such
xacts? IIRC, the restart_lsn is advanced based on confirmed_flush lsn
sent by subscriber. After this change, the subscriber won't be able
to send the confirmed_flush and for a long time, we won't be able to
advance restart_lsn. Is that correct, if so, why do we think that is
acceptable? One might argue that restart_lsn will be advanced as soon
as we send the first non-empty xact, but not sure if that is good
enough. What do you think?

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

#6Dilip Kumar
dilipbalaut@gmail.com
In reply to: Amit Kapila (#5)
Re: logical replication empty transactions

On Mon, Mar 2, 2020 at 4:56 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Mon, Mar 2, 2020 at 9:01 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Sat, Nov 9, 2019 at 7:29 AM Euler Taveira <euler@timbira.com.br> wrote:

Em seg., 21 de out. de 2019 às 21:20, Jeff Janes
<jeff.janes@gmail.com> escreveu:

After setting up logical replication of a slowly changing table using the built in pub/sub facility, I noticed way more network traffic than made sense. Looking into I see that every transaction in that database on the master gets sent to the replica. 99.999+% of them are empty transactions ('B' message and 'C' message with nothing in between) because the transactions don't touch any tables in the publication, only non-replicated tables. Is doing it this way necessary for some reason? Couldn't we hold the transmission of 'B' until something else comes along, and then if that next thing is 'C' drop both of them?

That is not optimal. Those empty transactions is a waste of bandwidth.
We can suppress them if no changes will be sent. test_decoding
implements "skip empty transaction" as you described above and I did
something similar to it. Patch is attached.

I think this significantly reduces the network bandwidth for empty
transactions. I have briefly reviewed the patch and it looks good to
me.

One thing that is not clear to me is how will we advance restart_lsn
if we don't send any empty xact in a system where there are many such
xacts? IIRC, the restart_lsn is advanced based on confirmed_flush lsn
sent by subscriber. After this change, the subscriber won't be able
to send the confirmed_flush and for a long time, we won't be able to
advance restart_lsn. Is that correct, if so, why do we think that is
acceptable? One might argue that restart_lsn will be advanced as soon
as we send the first non-empty xact, but not sure if that is good
enough. What do you think?

It seems like a valid point. One idea could be that we can track the
last commit LSN which we streamed and if the confirmed flush location
is already greater than that then even if we skip the sending the
commit message we can increase the confirm flush location locally.
Logically, it should not cause any problem because once we have got
the confirmation for whatever we have streamed so far. So for other
commits(which we are skipping), we can we advance it locally because
we are sure that we don't have any streamed commit which is not yet
confirmed by the subscriber. This is just my thought, but if we
think from the code and design perspective then it might complicate
the things and sounds hackish.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

#7Amit Kapila
amit.kapila16@gmail.com
In reply to: Dilip Kumar (#6)
Re: logical replication empty transactions

On Tue, Mar 3, 2020 at 9:35 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Mon, Mar 2, 2020 at 4:56 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

One thing that is not clear to me is how will we advance restart_lsn
if we don't send any empty xact in a system where there are many such
xacts? IIRC, the restart_lsn is advanced based on confirmed_flush lsn
sent by subscriber. After this change, the subscriber won't be able
to send the confirmed_flush and for a long time, we won't be able to
advance restart_lsn. Is that correct, if so, why do we think that is
acceptable? One might argue that restart_lsn will be advanced as soon
as we send the first non-empty xact, but not sure if that is good
enough. What do you think?

It seems like a valid point. One idea could be that we can track the
last commit LSN which we streamed and if the confirmed flush location
is already greater than that then even if we skip the sending the
commit message we can increase the confirm flush location locally.
Logically, it should not cause any problem because once we have got
the confirmation for whatever we have streamed so far. So for other
commits(which we are skipping), we can we advance it locally because
we are sure that we don't have any streamed commit which is not yet
confirmed by the subscriber.

Will this work after restart? Do you want to persist the information
of last streamed commit LSN?

This is just my thought, but if we
think from the code and design perspective then it might complicate
the things and sounds hackish.

Another idea could be that we stream the transaction after some
threshold number (say 100 or anything we think is reasonable) of empty
xacts. This will reduce the traffic without tinkering with the core
design too much.

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

#8Dilip Kumar
dilipbalaut@gmail.com
In reply to: Amit Kapila (#7)
Re: logical replication empty transactions

On Tue, Mar 3, 2020 at 1:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Tue, Mar 3, 2020 at 9:35 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Mon, Mar 2, 2020 at 4:56 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

One thing that is not clear to me is how will we advance restart_lsn
if we don't send any empty xact in a system where there are many such
xacts? IIRC, the restart_lsn is advanced based on confirmed_flush lsn
sent by subscriber. After this change, the subscriber won't be able
to send the confirmed_flush and for a long time, we won't be able to
advance restart_lsn. Is that correct, if so, why do we think that is
acceptable? One might argue that restart_lsn will be advanced as soon
as we send the first non-empty xact, but not sure if that is good
enough. What do you think?

It seems like a valid point. One idea could be that we can track the
last commit LSN which we streamed and if the confirmed flush location
is already greater than that then even if we skip the sending the
commit message we can increase the confirm flush location locally.
Logically, it should not cause any problem because once we have got
the confirmation for whatever we have streamed so far. So for other
commits(which we are skipping), we can we advance it locally because
we are sure that we don't have any streamed commit which is not yet
confirmed by the subscriber.

Will this work after restart? Do you want to persist the information
of last streamed commit LSN?

We will not persist the last streamed commit LSN, this variable is in
memory just to track whether we have got confirmation up to that
location or not, once we have confirmation up to that location and if
we are not streaming any transaction (because those are empty
transactions) then we can just advance the confirmed flush location
and based on that we can update the restart point as well and those
will be persisted. Basically, "last streamed commit LSN" is just a
marker that their still something pending to be confirmed from the
subscriber so until that we can not simply advance the confirm flush
location or restart point based on the empty transactions. But, if
there is nothing pending to be confirmed we can advance. So if we are
streaming then we will get confirmation from subscriber otherwise we
can advance it locally. So, in either case, the confirmed flush
location and restart point will keep moving.

This is just my thought, but if we
think from the code and design perspective then it might complicate
the things and sounds hackish.

Another idea could be that we stream the transaction after some
threshold number (say 100 or anything we think is reasonable) of empty
xacts. This will reduce the traffic without tinkering with the core
design too much.

Yeah, this could be also an option.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

#9Amit Kapila
amit.kapila16@gmail.com
In reply to: Dilip Kumar (#8)
Re: logical replication empty transactions

On Tue, Mar 3, 2020 at 2:17 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Tue, Mar 3, 2020 at 1:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Tue, Mar 3, 2020 at 9:35 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Mon, Mar 2, 2020 at 4:56 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

One thing that is not clear to me is how will we advance restart_lsn
if we don't send any empty xact in a system where there are many such
xacts? IIRC, the restart_lsn is advanced based on confirmed_flush lsn
sent by subscriber. After this change, the subscriber won't be able
to send the confirmed_flush and for a long time, we won't be able to
advance restart_lsn. Is that correct, if so, why do we think that is
acceptable? One might argue that restart_lsn will be advanced as soon
as we send the first non-empty xact, but not sure if that is good
enough. What do you think?

It seems like a valid point. One idea could be that we can track the
last commit LSN which we streamed and if the confirmed flush location
is already greater than that then even if we skip the sending the
commit message we can increase the confirm flush location locally.
Logically, it should not cause any problem because once we have got
the confirmation for whatever we have streamed so far. So for other
commits(which we are skipping), we can we advance it locally because
we are sure that we don't have any streamed commit which is not yet
confirmed by the subscriber.

Will this work after restart? Do you want to persist the information
of last streamed commit LSN?

We will not persist the last streamed commit LSN, this variable is in
memory just to track whether we have got confirmation up to that
location or not, once we have confirmation up to that location and if
we are not streaming any transaction (because those are empty
transactions) then we can just advance the confirmed flush location
and based on that we can update the restart point as well and those
will be persisted. Basically, "last streamed commit LSN" is just a
marker that their still something pending to be confirmed from the
subscriber so until that we can not simply advance the confirm flush
location or restart point based on the empty transactions. But, if
there is nothing pending to be confirmed we can advance. So if we are
streaming then we will get confirmation from subscriber otherwise we
can advance it locally. So, in either case, the confirmed flush
location and restart point will keep moving.

Okay, so this might work out, but it might look a bit ad-hoc.

This is just my thought, but if we
think from the code and design perspective then it might complicate
the things and sounds hackish.

Another idea could be that we stream the transaction after some
threshold number (say 100 or anything we think is reasonable) of empty
xacts. This will reduce the traffic without tinkering with the core
design too much.

Yeah, this could be also an option.

Okay.

Peter E, Petr J, others, do you have any opinion on what is the best
way forward for this thread? I think it would be really good if we
can reduce the network traffic due to these empty transactions.

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

#10Euler Taveira
euler.taveira@2ndquadrant.com
In reply to: Amit Kapila (#7)
Re: logical replication empty transactions

On Tue, 3 Mar 2020 at 05:24, Amit Kapila <amit.kapila16@gmail.com> wrote:

Another idea could be that we stream the transaction after some
threshold number (say 100 or anything we think is reasonable) of empty
xacts. This will reduce the traffic without tinkering with the core
design too much.

Amit, I suggest an interval to control this setting. Time is something we

have control; transactions aren't (depending on workload).
pg_stat_replication query interval usually is not milliseconds, however,
you can execute thousands of transactions in a second. If we agree on that
idea I can add it to the patch.

Regards,

--
Euler Taveira http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#11Amit Kapila
amit.kapila16@gmail.com
In reply to: Euler Taveira (#10)
Re: logical replication empty transactions

On Wed, Mar 4, 2020 at 7:17 AM Euler Taveira
<euler.taveira@2ndquadrant.com> wrote:

On Tue, 3 Mar 2020 at 05:24, Amit Kapila <amit.kapila16@gmail.com> wrote:

Another idea could be that we stream the transaction after some
threshold number (say 100 or anything we think is reasonable) of empty
xacts. This will reduce the traffic without tinkering with the core
design too much.

Amit, I suggest an interval to control this setting. Time is something we have control; transactions aren't (depending on workload). pg_stat_replication query interval usually is not milliseconds, however, you can execute thousands of transactions in a second. If we agree on that idea I can add it to the patch.

Do you mean to say that if for some threshold interval we didn't
stream any transaction, then we can send the next empty transaction to
the subscriber? If so, then isn't it possible that the empty xacts
happen irregularly after the specified interval and then we still end
up sending them all. I might be missing something here, so can you
please explain your idea in detail? Basically, how will it work and
how will it solve the problem.

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

#12Dilip Kumar
dilipbalaut@gmail.com
In reply to: Amit Kapila (#11)
Re: logical replication empty transactions

On Wed, Mar 4, 2020 at 9:12 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Wed, Mar 4, 2020 at 7:17 AM Euler Taveira
<euler.taveira@2ndquadrant.com> wrote:

On Tue, 3 Mar 2020 at 05:24, Amit Kapila <amit.kapila16@gmail.com> wrote:

Another idea could be that we stream the transaction after some
threshold number (say 100 or anything we think is reasonable) of empty
xacts. This will reduce the traffic without tinkering with the core
design too much.

Amit, I suggest an interval to control this setting. Time is something we have control; transactions aren't (depending on workload). pg_stat_replication query interval usually is not milliseconds, however, you can execute thousands of transactions in a second. If we agree on that idea I can add it to the patch.

Do you mean to say that if for some threshold interval we didn't
stream any transaction, then we can send the next empty transaction to
the subscriber? If so, then isn't it possible that the empty xacts
happen irregularly after the specified interval and then we still end
up sending them all. I might be missing something here, so can you
please explain your idea in detail? Basically, how will it work and
how will it solve the problem.

IMHO, the threshold should be based on the commit LSN. Our main
reason we want to send empty transactions after a certain
transaction/duration is that we want the restart_lsn to be moving
forward so that if we need to restart the replication slot we don't
need to process a lot of extra WAL. So assume we set the threshold
based on transaction count then there is still a possibility that we
might process a few very big transactions then we will have to process
them again after the restart. OTOH, if we set based on an interval
then even if there is not much work going on, still we end up sending
the empty transaction as pointed by Amit.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

#13Amit Kapila
amit.kapila16@gmail.com
In reply to: Dilip Kumar (#12)
Re: logical replication empty transactions

On Wed, Mar 4, 2020 at 9:52 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Wed, Mar 4, 2020 at 9:12 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Wed, Mar 4, 2020 at 7:17 AM Euler Taveira
<euler.taveira@2ndquadrant.com> wrote:

On Tue, 3 Mar 2020 at 05:24, Amit Kapila <amit.kapila16@gmail.com> wrote:

Another idea could be that we stream the transaction after some
threshold number (say 100 or anything we think is reasonable) of empty
xacts. This will reduce the traffic without tinkering with the core
design too much.

Amit, I suggest an interval to control this setting. Time is something we have control; transactions aren't (depending on workload). pg_stat_replication query interval usually is not milliseconds, however, you can execute thousands of transactions in a second. If we agree on that idea I can add it to the patch.

Do you mean to say that if for some threshold interval we didn't
stream any transaction, then we can send the next empty transaction to
the subscriber? If so, then isn't it possible that the empty xacts
happen irregularly after the specified interval and then we still end
up sending them all. I might be missing something here, so can you
please explain your idea in detail? Basically, how will it work and
how will it solve the problem.

IMHO, the threshold should be based on the commit LSN. Our main
reason we want to send empty transactions after a certain
transaction/duration is that we want the restart_lsn to be moving
forward so that if we need to restart the replication slot we don't
need to process a lot of extra WAL. So assume we set the threshold
based on transaction count then there is still a possibility that we
might process a few very big transactions then we will have to process
them again after the restart.

Won't the subscriber eventually send the flush location for the large
transactions which will move the restart_lsn?

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

#14Dilip Kumar
dilipbalaut@gmail.com
In reply to: Amit Kapila (#13)
Re: logical replication empty transactions

On Wed, Mar 4, 2020 at 10:50 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Wed, Mar 4, 2020 at 9:52 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Wed, Mar 4, 2020 at 9:12 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Wed, Mar 4, 2020 at 7:17 AM Euler Taveira
<euler.taveira@2ndquadrant.com> wrote:

On Tue, 3 Mar 2020 at 05:24, Amit Kapila <amit.kapila16@gmail.com> wrote:

Another idea could be that we stream the transaction after some
threshold number (say 100 or anything we think is reasonable) of empty
xacts. This will reduce the traffic without tinkering with the core
design too much.

Amit, I suggest an interval to control this setting. Time is something we have control; transactions aren't (depending on workload). pg_stat_replication query interval usually is not milliseconds, however, you can execute thousands of transactions in a second. If we agree on that idea I can add it to the patch.

Do you mean to say that if for some threshold interval we didn't
stream any transaction, then we can send the next empty transaction to
the subscriber? If so, then isn't it possible that the empty xacts
happen irregularly after the specified interval and then we still end
up sending them all. I might be missing something here, so can you
please explain your idea in detail? Basically, how will it work and
how will it solve the problem.

IMHO, the threshold should be based on the commit LSN. Our main
reason we want to send empty transactions after a certain
transaction/duration is that we want the restart_lsn to be moving
forward so that if we need to restart the replication slot we don't
need to process a lot of extra WAL. So assume we set the threshold
based on transaction count then there is still a possibility that we
might process a few very big transactions then we will have to process
them again after the restart.

Won't the subscriber eventually send the flush location for the large
transactions which will move the restart_lsn?

I meant large empty transactions (basically we can not send anything
to the subscriber). So my point was if there are only large
transactions in the system which we can not stream because those
tables are not published. Then keeping threshold based on transaction
count will not help much because even if we don't reach the
transaction count threshold, we still might need to process a lot of
data if we don't stream the commit for the empty transactions. So
instead of tracking transaction count can we track LSN, and LSN
different since we last stream some change cross the threshold then we
will stream the next empty transaction.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

#15Amit Kapila
amit.kapila16@gmail.com
In reply to: Dilip Kumar (#14)
Re: logical replication empty transactions

On Wed, Mar 4, 2020 at 11:16 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Wed, Mar 4, 2020 at 10:50 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Wed, Mar 4, 2020 at 9:52 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

IMHO, the threshold should be based on the commit LSN. Our main
reason we want to send empty transactions after a certain
transaction/duration is that we want the restart_lsn to be moving
forward so that if we need to restart the replication slot we don't
need to process a lot of extra WAL. So assume we set the threshold
based on transaction count then there is still a possibility that we
might process a few very big transactions then we will have to process
them again after the restart.

Won't the subscriber eventually send the flush location for the large
transactions which will move the restart_lsn?

I meant large empty transactions (basically we can not send anything
to the subscriber). So my point was if there are only large
transactions in the system which we can not stream because those
tables are not published. Then keeping threshold based on transaction
count will not help much because even if we don't reach the
transaction count threshold, we still might need to process a lot of
data if we don't stream the commit for the empty transactions. So
instead of tracking transaction count can we track LSN, and LSN
different since we last stream some change cross the threshold then we
will stream the next empty transaction.

You have a point and it may be better to keep threshold based on LSN
if we want to keep any threshold, but keeping on transaction count
seems to be a bit straightforward. Let us see if anyone else has any
opinion on this matter?

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

#16Dilip Kumar
dilipbalaut@gmail.com
In reply to: Amit Kapila (#15)
Re: logical replication empty transactions

On Wed, Mar 4, 2020 at 3:47 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Wed, Mar 4, 2020 at 11:16 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Wed, Mar 4, 2020 at 10:50 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Wed, Mar 4, 2020 at 9:52 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

IMHO, the threshold should be based on the commit LSN. Our main
reason we want to send empty transactions after a certain
transaction/duration is that we want the restart_lsn to be moving
forward so that if we need to restart the replication slot we don't
need to process a lot of extra WAL. So assume we set the threshold
based on transaction count then there is still a possibility that we
might process a few very big transactions then we will have to process
them again after the restart.

Won't the subscriber eventually send the flush location for the large
transactions which will move the restart_lsn?

I meant large empty transactions (basically we can not send anything
to the subscriber). So my point was if there are only large
transactions in the system which we can not stream because those
tables are not published. Then keeping threshold based on transaction
count will not help much because even if we don't reach the
transaction count threshold, we still might need to process a lot of
data if we don't stream the commit for the empty transactions. So
instead of tracking transaction count can we track LSN, and LSN
different since we last stream some change cross the threshold then we
will stream the next empty transaction.

You have a point and it may be better to keep threshold based on LSN
if we want to keep any threshold, but keeping on transaction count
seems to be a bit straightforward. Let us see if anyone else has any
opinion on this matter?

Ok, that make sense.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

#17Amit Kapila
amit.kapila16@gmail.com
In reply to: Dilip Kumar (#16)
Re: logical replication empty transactions

On Wed, Mar 4, 2020 at 4:04 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Wed, Mar 4, 2020 at 3:47 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Wed, Mar 4, 2020 at 11:16 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Wed, Mar 4, 2020 at 10:50 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Wed, Mar 4, 2020 at 9:52 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

IMHO, the threshold should be based on the commit LSN. Our main
reason we want to send empty transactions after a certain
transaction/duration is that we want the restart_lsn to be moving
forward so that if we need to restart the replication slot we don't
need to process a lot of extra WAL. So assume we set the threshold
based on transaction count then there is still a possibility that we
might process a few very big transactions then we will have to process
them again after the restart.

Won't the subscriber eventually send the flush location for the large
transactions which will move the restart_lsn?

I meant large empty transactions (basically we can not send anything
to the subscriber). So my point was if there are only large
transactions in the system which we can not stream because those
tables are not published. Then keeping threshold based on transaction
count will not help much because even if we don't reach the
transaction count threshold, we still might need to process a lot of
data if we don't stream the commit for the empty transactions. So
instead of tracking transaction count can we track LSN, and LSN
different since we last stream some change cross the threshold then we
will stream the next empty transaction.

You have a point and it may be better to keep threshold based on LSN
if we want to keep any threshold, but keeping on transaction count
seems to be a bit straightforward. Let us see if anyone else has any
opinion on this matter?

Ok, that make sense.

Euler, can we try to update the patch based on the number of
transactions threshold and see how it works?

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

#18Euler Taveira
euler.taveira@2ndquadrant.com
In reply to: Amit Kapila (#17)
Re: logical replication empty transactions

On Thu, 5 Mar 2020 at 05:45, Amit Kapila <amit.kapila16@gmail.com> wrote:

Euler, can we try to update the patch based on the number of
transactions threshold and see how it works?

I will do.

--
Euler Taveira http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#19Craig Ringer
craig@2ndquadrant.com
In reply to: Amit Kapila (#5)
Re: logical replication empty transactions

On Mon, 2 Mar 2020 at 19:26, Amit Kapila <amit.kapila16@gmail.com> wrote:

One thing that is not clear to me is how will we advance restart_lsn
if we don't send any empty xact in a system where there are many such
xacts?

Same way we already do it for writes that are not replicated over
logical replication, like vacuum work etc. The upstream sends feedback
with reply-requested. The downstream replies. The upstream advances
confirmed_flush_lsn, and that lazily updates restart_lsn.

The bigger issue here is that if you don't send empty txns on logical
replication you don't get an eager, timely response from the
replica(s), which delays synchronous replication. You need to send
empty txns when synchronous replication is enabled, or instead poke
the walsender to force immediate feedback with reply requested.

--
Craig Ringer http://www.2ndQuadrant.com/
2ndQuadrant - PostgreSQL Solutions for the Enterprise

#20Andres Freund
andres@anarazel.de
In reply to: Craig Ringer (#19)
Re: logical replication empty transactions

Hi,

On 2020-03-06 13:53:02 +0800, Craig Ringer wrote:

On Mon, 2 Mar 2020 at 19:26, Amit Kapila <amit.kapila16@gmail.com> wrote:

One thing that is not clear to me is how will we advance restart_lsn
if we don't send any empty xact in a system where there are many such
xacts?

Same way we already do it for writes that are not replicated over
logical replication, like vacuum work etc. The upstream sends feedback
with reply-requested. The downstream replies. The upstream advances
confirmed_flush_lsn, and that lazily updates restart_lsn.

It'll still delay it a bit.

The bigger issue here is that if you don't send empty txns on logical
replication you don't get an eager, timely response from the
replica(s), which delays synchronous replication. You need to send
empty txns when synchronous replication is enabled, or instead poke
the walsender to force immediate feedback with reply requested.

Somewhat independent from the issue at hand: It'd be really good if we
could evolve the syncrep framework to support per-database waiting... It
shouldn't be that hard, and the current situation sucks quite a bit (and
yes, I'm to blame).

I'm not quite sure what you mean by "poke the walsender"? Kinda sounds
like sending a signal, but decoding happens inside after the walsender,
so there's no need for that. Do you just mean somehow requesting that
walsender sends a feedback message?

To address the volume we could:

1a) Introduce a pgoutput message type to indicate that the LSN has
advanced, without needing separate BEGIN/COMMIT. Right now BEGIN is
21 bytes, COMMIT is 26. But we really don't need that much here. A
single message should do the trick.

1b) Add a LogicalOutputPluginWriterUpdateProgress parameter (and
possibly rename) that indicates that we are intentionally "ignoring"
WAL. For walsender that callback then could check if it could just
forward the position of the client (if it was entirely caught up
before), or if it should send a feedback request (if syncrep is
enabled, or distance is big).

2) Reduce the rate of 'empty transaction'/feedback request messages. If
we know that we're not going to be blocked waiting for more WAL, or
blocked sending messages out to the network, we don't immediately need
to send out the messages. Instead we could continue decoding until
there's actual data, or until we're going to get blocked.

We could e.g. have a new LogicalDecodingContext callback that is
called whenever WalSndWaitForWal() would wait. That'd check if there's
a pending "need" to send out a 'empty transaction'/feedback request
message. The "need" flag would get cleared whenever we send out data
bearing an LSN for other reasons.

Greetings,

Andres Freund

#21Craig Ringer
craig@2ndquadrant.com
In reply to: Andres Freund (#20)
Re: logical replication empty transactions

On Tue, 10 Mar 2020 at 02:30, Andres Freund <andres@anarazel.de> wrote:

Hi,

On 2020-03-06 13:53:02 +0800, Craig Ringer wrote:

On Mon, 2 Mar 2020 at 19:26, Amit Kapila <amit.kapila16@gmail.com>

wrote:

One thing that is not clear to me is how will we advance restart_lsn
if we don't send any empty xact in a system where there are many such
xacts?

Same way we already do it for writes that are not replicated over
logical replication, like vacuum work etc. The upstream sends feedback
with reply-requested. The downstream replies. The upstream advances
confirmed_flush_lsn, and that lazily updates restart_lsn.

It'll still delay it a bit.

Right, but we don't generally care because there's no sync rep txn waiting
for confirmation. If we lose progress due to a crash it doesn't matter. It
does delay removal of old WAL a little, but it hardly matters.

Somewhat independent from the issue at hand: It'd be really good if we
could evolve the syncrep framework to support per-database waiting... It
shouldn't be that hard, and the current situation sucks quite a bit (and
yes, I'm to blame).

Hardly, you just didn't get the chance to fix that on top of the umpteen
other things you had to change to make all the logical stuff work. You
didn't break it, just didn't implement every single possible enhancement
all at once. Shocking, I tell you.

I'm not quite sure what you mean by "poke the walsender"? Kinda sounds

like sending a signal, but decoding happens inside after the walsender,
so there's no need for that. Do you just mean somehow requesting that
walsender sends a feedback message?

Right. I had in mind something like sending a ProcSignal via our funky
multiplexed signal mechanism to ask the walsender to immediately generate a
keepalive message with a reply-requested flag, then set the walsender's
latch so we wake it promptly.

To address the volume we could:

1a) Introduce a pgoutput message type to indicate that the LSN has
advanced, without needing separate BEGIN/COMMIT. Right now BEGIN is
21 bytes, COMMIT is 26. But we really don't need that much here. A
single message should do the trick.

It would. Is it worth caring though? Especially since it seems rather
unlikely that the actual network data volume of begin/commit msgs will be
much of a concern. It's not like we're PITRing logical streams, and if we
did, we could just filter out empty commits on the receiver side.

That message pretty much already exists in the form of a walsender
keepalive anyway so we might as well re-use that and not upset the protocol.

1b) Add a LogicalOutputPluginWriterUpdateProgress parameter (and
possibly rename) that indicates that we are intentionally "ignoring"
WAL. For walsender that callback then could check if it could just
forward the position of the client (if it was entirely caught up
before), or if it should send a feedback request (if syncrep is
enabled, or distance is big).

I can see something like that being very useful, because at present only
the output plugin knows if a txn is "empty" as far as that particular slot
and output plugin is concerned. The reorder buffering mechanism cannot do
relation-level filtering before it sends the changes to the output plugin
during ReorderBufferCommit, since it only knows about relfilenodes not
relation oids. And the output plugin might be doing finer grained filtering
using row-filter expressions or who knows what else.

But as described above that will only help for txns done in DBs other than
the one the logical slot is for or txns known to have an empty
ReorderBuffer when the commit is seen.

If there's a txn in the slot's db with a non-empty reorderbuffer, the
output plugin won't know if the txn is empty or not until it finishes
processing all callbacks and sees the commit for the txn. So it will
generally have emitted the Begin message on the wire by the time it knows
it has nothing useful to say. And Pg won't know that this txn is empty as
far as this output plugin with this particular slot, set of output plugin
params, and current user-catalog state is concerned, so it won't have any
way to call the output plugin's "update progress" callback instead of the
usual begin/change/commit callbacks.

But I think we can already skip empty txns unless sync-rep is enabled with
no core changes, and send empty txns as walsender keepalives instead, by
altering only output plugins, like this:

* Stash BEGIN data in plugin's LogicalDecodingContext.output_plugin_private
when plugin's begin callback called, don't write anything to the outstream
* Write out BEGIN message lazily when any other callback generates a
message that does need to be written out
* If no BEGIN written by the time COMMIT callback called, discard the
COMMIT too. Check if sync rep enabled. if it is,
call LogicalDecodingContext.update_progress from within the output plugin
commit handler, otherwise just ignore the commit totally. Probably by
calling OutputPluginUpdateProgress().

We could e.g. have a new LogicalDecodingContext callback that is

called whenever WalSndWaitForWal() would wait. That'd check if there's
a pending "need" to send out a 'empty transaction'/feedback request
message. The "need" flag would get cleared whenever we send out data
bearing an LSN for other reasons.

I can see that being handy, yes. But it won't necessarily help with the
sync rep issue, since other sync rep txns may continue to generate WAL
while others wait for commit-confirmations that won't come from the logical
replica.

While we're speaking of adding output plugin hooks, I keep on trying to
think of a sensible way to do a plugin-defined reply handler, so the
downstream end can send COPY BOTH messages of some new msgkind back to the
walsender, which will pass them to the output plugin if it implements the
appropriate handle_reply_message (or whatever) callback. That much is
trivial to implement, where I keep getting a bit stuck is with whether
there's a sensible snapshot that can be set to call the output plugin reply
handler with. We wouldn't want to switch to a current non-historic snapshot
because of all the cache flushes that'd cause, but there isn't necessarily
a valid and safe historic snapshot to set when we're not within
ReorderBufferCommit is there?

I'd love to get rid of the need to "connect back" to a provider over plain
libpq connections to communicate with it. The ability to run SQL on the
walsender conn helps. But really, so much more would be possible if we
could just have the downstream end *reply* on the same connection using
COPY BOTH, much like it sends replay progress updates right now. It'd let
us manage relation/attribute/type metadata caches better for example.

Thoughts?

--
Craig Ringer http://www.2ndQuadrant.com/
2ndQuadrant - PostgreSQL Solutions for the Enterprise

#22Ajin Cherian
itsajin@gmail.com
In reply to: Craig Ringer (#21)
Re: logical replication empty transactions

The patch no longer applies, because of additions in the test source. Otherwise, I have tested the patch and confirmed that updates and deletes on tables with deferred primary keys work with logical replication.

The new status of this patch is: Waiting on Author

#23Ajin Cherian
itsajin@gmail.com
In reply to: Ajin Cherian (#22)
Re: logical replication empty transactions

Sorry, I replied in the wrong thread. Please ignore above mail.

Show quoted text
#24Rahila Syed
rahila.syed@2ndquadrant.com
In reply to: Craig Ringer (#21)
Re: logical replication empty transactions

Hi,

Please see below review of the
0001-Skip-empty-transactions-for-logical-replication.patch

The make check passes.

 +               /* output BEGIN if we haven't yet */
 +               if (!data->xact_wrote_changes)
 +                       pgoutput_begin(ctx, txn);
 +
 +               data->xact_wrote_changes = true;
 +
IMO, xact_wrote_changes flag is better set inside the if condition as it
does not need to
be set repeatedly in subsequent calls to the same function.

* Stash BEGIN data in plugin's
LogicalDecodingContext.output_plugin_private when plugin's begin
callback called, don't write anything to the outstream
* Write out BEGIN message lazily when any other callback generates a
message that does need to be written out
* If no BEGIN written by the time COMMIT callback called, discard the
COMMIT too. Check if sync rep enabled. if it is,
call LogicalDecodingContext.update_progress
from within the output plugin commit handler, otherwise just ignore
the commit totally. Probably by calling OutputPluginUpdateProgress().

I think the code in the patch is similar to what has been described by
Craig in the above snippet,
except instead of stashing the BEGIN message and sending the message
lazily, it simply maintains a flag
in LogicalDecodingContext.output_plugin_private which defers calling
output plugin's begin callback,
until any other callback actually generates a remote write.

Also, the patch does not contain the last part where he describes
having OutputPluginUpdateProgress()
for synchronous replication enabled transactions.
However, some basic testing suggests that the patch does not have any
notable adverse effect on
either the replication lag or the sync_rep performance.

I performed tests by setting up publisher and subscriber on the same
machine with synchronous_commit = on and
ran pgbench -c 12 -j 6 -T 300 on unpublished pgbench tables.

I see that  confirmed_flush_lsn is catching up just fine without any
notable delay as compared to the test results without
the patch.

Also, the TPS for synchronous replication of empty txns with and without
the patch remains similar.

Having said that, these are initial findings and I understand better
performance tests are required to measure
reduction in consumption of network bandwidth and impact on synchronous
replication and replication lag.

Thank you,
Rahila Syed

#25Michael Paquier
michael@paquier.xyz
In reply to: Rahila Syed (#24)
Re: logical replication empty transactions

On Wed, Jul 29, 2020 at 08:08:06PM +0530, Rahila Syed wrote:

The make check passes.

Since then, the patch is failing to apply, waiting on author and the
thread has died 6 weeks or so ago, so I am marking it as RwF in the
CF.
--
Michael

#26Ajin Cherian
itsajin@gmail.com
In reply to: Michael Paquier (#25)
1 attachment(s)
Re: logical replication empty transactions

On Thu, Sep 17, 2020 at 3:29 PM Michael Paquier <michael@paquier.xyz> wrote:

On Wed, Jul 29, 2020 at 08:08:06PM +0530, Rahila Syed wrote:

The make check passes.

Since then, the patch is failing to apply, waiting on author and the
thread has died 6 weeks or so ago, so I am marking it as RwF in the
CF.

I've rebased the patch and made changes so that the patch supports
"streaming in-progress transactions" and handling of logical decoding
messages (transactional and non-transactional).
I see that this patch not only makes sure that empty transactions are not
sent but also does call OutputPluginUpdateProgress when an empty
transaction is not sent, as a result the confirmed_flush_lsn is kept
moving. I also see no hangs when synchronous_standby is configured.
Do let me know your thoughts on this patch.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v2-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v2-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 3763a3b454f319f561c8c8bac4eedd81488d8160 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Wed, 14 Apr 2021 22:54:52 -0400
Subject: [PATCH v2] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.
Postpone the BEGIN message until the first change. While processing a
COMMIT message, if there is no other change for that
transaction, do not send COMMIT message. It means that pgoutput will
skip BEGIN / COMMIT messages for transactions that are empty.

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/pgoutput/pgoutput.c | 45 +++++++++++++++++++++++++++++
 src/include/replication/pgoutput.h          |  3 ++
 src/test/subscription/t/020_messages.pl     |  5 ++--
 3 files changed, 50 insertions(+), 3 deletions(-)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index f68348d..64c76d1 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -345,10 +345,28 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputData	*data = ctx->output_plugin_private;
+
+	/*
+	 * Don't send BEGIN message here. Instead, postpone it until the first
+	 * change. In logical replication, common scenarios is to replicate a set
+	 * of tables (instead of all tables) and transactions whose changes were to
+	 * table(s) that are not published will produce empty transactions. These
+	 * empty transactions will send BEGIN and COMMIT messages to subscribers,
+	 * using bandwidth on something with little/no use for logical replication.
+	 */
+	data->xact_wrote_changes = false;
+	elog(LOG,"Holding of begin");
+}
+
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	elog(LOG,"Sending begin");
 
 	if (send_replication_origin)
 	{
@@ -384,8 +402,14 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
+	PGOutputData	*data = ctx->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/* skip COMMIT message if nothing was sent */
+	if (!data->xact_wrote_changes)
+		return;
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
 	OutputPluginWrite(ctx, true);
@@ -551,6 +575,13 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/* output BEGIN if we haven't yet */
+	if (!data->xact_wrote_changes && !in_streaming)
+	{
+		pgoutput_begin(ctx, txn);
+		data->xact_wrote_changes = true;
+	}
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -693,6 +724,13 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/* output BEGIN if we haven't yet */
+		if (!data->xact_wrote_changes && !in_streaming)
+		{
+			pgoutput_begin(ctx, txn);
+			data->xact_wrote_changes = true;
+		}
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -725,6 +763,13 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+    /* output BEGIN if we haven't yet, avoid for streaming and non-transactional messages */
+    if (!data->xact_wrote_changes && !in_streaming && transactional)
+	{
+        pgoutput_begin(ctx, txn);
+    	data->xact_wrote_changes = true;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 51e7c03..e820790 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -20,6 +20,9 @@ typedef struct PGOutputData
 	MemoryContext context;		/* private memory context for transient
 								 * allocations */
 
+	/* control wether messages can already be sent */
+	bool        xact_wrote_changes;
+
 	/* client-supplied info: */
 	uint32		protocol_version;
 	List	   *publication_names;
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index c8be26b..2ea790f 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -78,9 +78,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is($result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot');
 
 $node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub ENABLE");
-- 
1.8.3.1

#27Ajin Cherian
itsajin@gmail.com
In reply to: Ajin Cherian (#26)
1 attachment(s)
Re: logical replication empty transactions

On Thu, Apr 15, 2021 at 1:29 PM Ajin Cherian <itsajin@gmail.com> wrote:

I've rebased the patch and made changes so that the patch supports
"streaming in-progress transactions" and handling of logical decoding
messages (transactional and non-transactional).
I see that this patch not only makes sure that empty transactions are not
sent but also does call OutputPluginUpdateProgress when an empty
transaction is not sent, as a result the confirmed_flush_lsn is kept
moving. I also see no hangs when synchronous_standby is configured.
Do let me know your thoughts on this patch.

Removed some debug logs and typos.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v3-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v3-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 07f17491ca2263d152c1651a9da93adbada0aeaf Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Wed, 14 Apr 2021 22:54:52 -0400
Subject: [PATCH v3] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.
Postpone the BEGIN message until the first change. While processing a
COMMIT message, if there is no other change for that
transaction, do not send COMMIT message. It means that pgoutput will
skip BEGIN / COMMIT messages for transactions that are empty.

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/pgoutput/pgoutput.c | 44 +++++++++++++++++++++++++++++
 src/include/replication/pgoutput.h          |  3 ++
 src/test/subscription/t/020_messages.pl     |  5 ++--
 3 files changed, 49 insertions(+), 3 deletions(-)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index f68348d..0aa5729 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -345,6 +345,23 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputData	*data = ctx->output_plugin_private;
+
+	/*
+	 * Don't send BEGIN message here. Instead, postpone it until the first
+	 * change. In logical replication, a common scenario is to replicate a set
+	 * of tables (instead of all tables) and transactions whose changes were on
+	 * table(s) that are not published will produce empty transactions. These
+	 * empty transactions will send BEGIN and COMMIT messages to subscribers,
+	 * using bandwidth on something with little/no use for logical replication.
+	 */
+	data->xact_wrote_changes = false;
+	elog(LOG,"Holding of begin");
+}
+
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
@@ -384,8 +401,14 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
+	PGOutputData	*data = ctx->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/* skip COMMIT message if nothing was sent */
+	if (!data->xact_wrote_changes)
+		return;
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
 	OutputPluginWrite(ctx, true);
@@ -551,6 +574,13 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/* output BEGIN if we haven't yet */
+	if (!data->xact_wrote_changes && !in_streaming)
+	{
+		pgoutput_begin(ctx, txn);
+		data->xact_wrote_changes = true;
+	}
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -693,6 +723,13 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/* output BEGIN if we haven't yet */
+		if (!data->xact_wrote_changes && !in_streaming)
+		{
+			pgoutput_begin(ctx, txn);
+			data->xact_wrote_changes = true;
+		}
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -725,6 +762,13 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+    /* output BEGIN if we haven't yet, avoid for streaming and non-transactional messages */
+    if (!data->xact_wrote_changes && !in_streaming && transactional)
+	{
+		pgoutput_begin(ctx, txn);
+		data->xact_wrote_changes = true;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 51e7c03..acd43b3 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -20,6 +20,9 @@ typedef struct PGOutputData
 	MemoryContext context;		/* private memory context for transient
 								 * allocations */
 
+	/* flag indicating whether messages have previously been sent */
+	bool        xact_wrote_changes;
+
 	/* client-supplied info: */
 	uint32		protocol_version;
 	List	   *publication_names;
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index c8be26b..2ea790f 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -78,9 +78,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is($result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot');
 
 $node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub ENABLE");
-- 
1.8.3.1

#28Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#27)
Re: logical replication empty transactions

On Thu, Apr 15, 2021 at 4:39 PM Ajin Cherian <itsajin@gmail.com> wrote:

On Thu, Apr 15, 2021 at 1:29 PM Ajin Cherian <itsajin@gmail.com> wrote:

I've rebased the patch and made changes so that the patch supports "streaming in-progress transactions" and handling of logical decoding
messages (transactional and non-transactional).
I see that this patch not only makes sure that empty transactions are not sent but also does call OutputPluginUpdateProgress when an empty
transaction is not sent, as a result the confirmed_flush_lsn is kept moving. I also see no hangs when synchronous_standby is configured.
Do let me know your thoughts on this patch.

REVIEW COMMENTS

I applied this patch to today's HEAD and successfully ran "make check"
and also the subscription TAP tests.

Here are a some review comments:

------

1. The patch v3 applied OK but with whitespace warnings

[postgres@CentOS7-x64 oss_postgres_2PC]$ git apply
../patches_misc/v3-0001-Skip-empty-transactions-for-logical-replication.patch
../patches_misc/v3-0001-Skip-empty-transactions-for-logical-replication.patch:98:
indent with spaces.
/* output BEGIN if we haven't yet, avoid for streaming and
non-transactional messages */
../patches_misc/v3-0001-Skip-empty-transactions-for-logical-replication.patch:99:
indent with spaces.
if (!data->xact_wrote_changes && !in_streaming && transactional)
warning: 2 lines add whitespace errors.

------

2. Please create a CF entry in [1]https://commitfest.postgresql.org/33/ for this patch.

------

3. Patch comment

The comment describes the problem and then suddenly just says
"Postpone the BEGIN message until the first change."

I suggest changing it to say more like... "(blank line) This patch
addresses the above problem by postponing the BEGIN message until the
first change."

------

4. pgoutput.h

Maybe for consistency with the context member, the comment for the new
member should be to the right instead of above it?

@@ -20,6 +20,9 @@ typedef struct PGOutputData
MemoryContext context; /* private memory context for transient
* allocations */

+ /* flag indicating whether messages have previously been sent */
+ bool        xact_wrote_changes;
+

------

5. pgoutput.h

+ /* flag indicating whether messages have previously been sent */

"previously been sent" --> "already been sent" ??

------

6. pgoutput.h - misleading member name

Actually, now that I have read all the rest of the code and how this
member is used I feel that this name is very misleading. e.g. For
"streaming" case then you still are writing changes but are not
setting this member at all - therefore it does not always mean what it
says.

I feel a better name for this would be something like
"sent_begin_txn". Then if you have sent BEGIN it is true. If you
haven't sent BEGIN it is false. It eliminates all ambiguity naming it
this way instead.

(This makes my feedback #5 redundant because the comment will be a bit
different if you do this).

------

7. pgoutput.c - function pgoutput_begin_txn

@@ -345,6 +345,23 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,
static void
pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
{

I guess that you still needed to pass the txn because that is how the
API is documented, right?

But I am wondering if you ought to flag it as unused so you wont get
some BF machine giving warnings about it.

e.g. Syntax like this?

pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN * txn) {
(void)txn;
...

------

8. pgoutput.c - function pgoutput_begin_txn

@@ -345,6 +345,23 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+ PGOutputData *data = ctx->output_plugin_private;
+
+ /*
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
+ */
+ data->xact_wrote_changes = false;
+ elog(LOG,"Holding of begin");
+}

Why is this loglevel LOG? Looks like leftover debugging.

------

9. pgoutput.c - function pgoutput_commit_txn

@@ -384,8 +401,14 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  XLogRecPtr commit_lsn)
 {
+ PGOutputData *data = ctx->output_plugin_private;
+
  OutputPluginUpdateProgress(ctx);
+ /* skip COMMIT message if nothing was sent */
+ if (!data->xact_wrote_changes)
+ return;
+

In the case where you decided to do nothing does it make sense that
you still called the function OutputPluginUpdateProgress(ctx); ?
I thought perhaps that your new check should come first so this call
would never happen.

------

10. pgoutput.c - variable declarations without casts

+ PGOutputData *data = ctx->output_plugin_private;

I noticed the new stack variable you declare have no casts.

This differs from the existing code which always looks like:
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;

There are a couple of examples of this so please search new code to
find them all.

------

11. pgoutput.c - function pgoutput_change

@@ -551,6 +574,13 @@ pgoutput_change(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
Assert(false);
}

+ /* output BEGIN if we haven't yet */
+ if (!data->xact_wrote_changes && !in_streaming)
+ {
+ pgoutput_begin(ctx, txn);
+ data->xact_wrote_changes = true;
+ }

If the variable is renamed as previously suggested then the assignment
data->sent_BEGIN_txn = true; can be assigned in just 1 common place
INSIDE the pgoutput_begin function.

------

12. pgoutput.c - pgoutput_truncate function

@@ -693,6 +723,13 @@ pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,

  if (nrelids > 0)
  {
+ /* output BEGIN if we haven't yet */
+ if (!data->xact_wrote_changes && !in_streaming)
+ {
+ pgoutput_begin(ctx, txn);
+ data->xact_wrote_changes = true;
+ }

(same comment as above)

If the variable is renamed as previously suggested then the assignment
data->sent_BEGIN_txn = true; can be assigned in just 1 common place
INSIDE the pgoutput_begin function.

13. pgoutput.c - pgoutput_message

@@ -725,6 +762,13 @@ pgoutput_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
if (in_streaming)
xid = txn->xid;

+    /* output BEGIN if we haven't yet, avoid for streaming and
non-transactional messages */
+    if (!data->xact_wrote_changes && !in_streaming && transactional)
+ {
+ pgoutput_begin(ctx, txn);
+ data->xact_wrote_changes = true;
+ }

(same comment as above)

If the variable is renamed as previously suggested then the assignment
data->sent_BEGIN_txn = true; can be assigned in just 1 common place
INSIDE the pgoutput_begin function.

------

14. Test Code.

I noticed that there is no test code specifically for seeing if empty
transactions get sent or not. Is it possible to write such a test or
is this traffic improvement only observable using the debugger?

------
[1]: https://commitfest.postgresql.org/33/

Kind Regards,
Peter Smith.
Fujitsu Australia

#29Ajin Cherian
itsajin@gmail.com
In reply to: Peter Smith (#28)
1 attachment(s)
Re: logical replication empty transactions

On Mon, Apr 19, 2021 at 6:22 PM Peter Smith <smithpb2250@gmail.com> wrote:

Here are a some review comments:

------

1. The patch v3 applied OK but with whitespace warnings

[postgres@CentOS7-x64 oss_postgres_2PC]$ git apply

../patches_misc/v3-0001-Skip-empty-transactions-for-logical-replication.patch

../patches_misc/v3-0001-Skip-empty-transactions-for-logical-replication.patch:98:
indent with spaces.
/* output BEGIN if we haven't yet, avoid for streaming and
non-transactional messages */

../patches_misc/v3-0001-Skip-empty-transactions-for-logical-replication.patch:99:
indent with spaces.
if (!data->xact_wrote_changes && !in_streaming && transactional)
warning: 2 lines add whitespace errors.

------

Fixed.

2. Please create a CF entry in [1] for this patch.

------

3. Patch comment

The comment describes the problem and then suddenly just says
"Postpone the BEGIN message until the first change."

I suggest changing it to say more like... "(blank line) This patch
addresses the above problem by postponing the BEGIN message until the
first change."

------

Updated.

4. pgoutput.h

Maybe for consistency with the context member, the comment for the new
member should be to the right instead of above it?

@@ -20,6 +20,9 @@ typedef struct PGOutputData
MemoryContext context; /* private memory context for transient
* allocations */

+ /* flag indicating whether messages have previously been sent */
+ bool        xact_wrote_changes;
+

------

5. pgoutput.h

+ /* flag indicating whether messages have previously been sent */

"previously been sent" --> "already been sent" ??

------

6. pgoutput.h - misleading member name

Actually, now that I have read all the rest of the code and how this
member is used I feel that this name is very misleading. e.g. For
"streaming" case then you still are writing changes but are not
setting this member at all - therefore it does not always mean what it
says.

I feel a better name for this would be something like
"sent_begin_txn". Then if you have sent BEGIN it is true. If you
haven't sent BEGIN it is false. It eliminates all ambiguity naming it
this way instead.

(This makes my feedback #5 redundant because the comment will be a bit
different if you do this).

------

Fixed above comments.

7. pgoutput.c - function pgoutput_begin_txn

@@ -345,6 +345,23 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,
static void
pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
{

I guess that you still needed to pass the txn because that is how the
API is documented, right?

But I am wondering if you ought to flag it as unused so you wont get
some BF machine giving warnings about it.

e.g. Syntax like this?

pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN * txn) {
(void)txn;
...

Updated.

------

8. pgoutput.c - function pgoutput_begin_txn

@@ -345,6 +345,23 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,
static void
pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
{
+ PGOutputData *data = ctx->output_plugin_private;
+
+ /*
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were
on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical
replication.
+ */
+ data->xact_wrote_changes = false;
+ elog(LOG,"Holding of begin");
+}

Why is this loglevel LOG? Looks like leftover debugging.

Removed.

------

9. pgoutput.c - function pgoutput_commit_txn

@@ -384,8 +401,14 @@ static void
pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
XLogRecPtr commit_lsn)
{
+ PGOutputData *data = ctx->output_plugin_private;
+
OutputPluginUpdateProgress(ctx);
+ /* skip COMMIT message if nothing was sent */
+ if (!data->xact_wrote_changes)
+ return;
+

In the case where you decided to do nothing does it make sense that
you still called the function OutputPluginUpdateProgress(ctx); ?
I thought perhaps that your new check should come first so this call
would never happen.

Even though the empty transaction is not sent, the LSN is tracked as
decoded, hence the progress needs to be updated.

------

10. pgoutput.c - variable declarations without casts

+ PGOutputData *data = ctx->output_plugin_private;

I noticed the new stack variable you declare have no casts.

This differs from the existing code which always looks like:
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;

There are a couple of examples of this so please search new code to
find them all.

-----

Fixed.

11. pgoutput.c - function pgoutput_change

@@ -551,6 +574,13 @@ pgoutput_change(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
Assert(false);
}

+ /* output BEGIN if we haven't yet */
+ if (!data->xact_wrote_changes && !in_streaming)
+ {
+ pgoutput_begin(ctx, txn);
+ data->xact_wrote_changes = true;
+ }

If the variable is renamed as previously suggested then the assignment
data->sent_BEGIN_txn = true; can be assigned in just 1 common place
INSIDE the pgoutput_begin function.

------

Updated.

12. pgoutput.c - pgoutput_truncate function

@@ -693,6 +723,13 @@ pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,

if (nrelids > 0)
{
+ /* output BEGIN if we haven't yet */
+ if (!data->xact_wrote_changes && !in_streaming)
+ {
+ pgoutput_begin(ctx, txn);
+ data->xact_wrote_changes = true;
+ }

(same comment as above)

If the variable is renamed as previously suggested then the assignment
data->sent_BEGIN_txn = true; can be assigned in just 1 common place
INSIDE the pgoutput_begin function.

13. pgoutput.c - pgoutput_message

@@ -725,6 +762,13 @@ pgoutput_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
if (in_streaming)
xid = txn->xid;

+    /* output BEGIN if we haven't yet, avoid for streaming and
non-transactional messages */
+    if (!data->xact_wrote_changes && !in_streaming && transactional)
+ {
+ pgoutput_begin(ctx, txn);
+ data->xact_wrote_changes = true;
+ }

(same comment as above)

If the variable is renamed as previously suggested then the assignment
data->sent_BEGIN_txn = true; can be assigned in just 1 common place
INSIDE the pgoutput_begin function.

------

Fixed.

14. Test Code.

I noticed that there is no test code specifically for seeing if empty
transactions get sent or not. Is it possible to write such a test or
is this traffic improvement only observable using the debugger?

The 020_messages.pl actually has a test case for tracking empty messages
even though it is part of the messages test.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v4-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v4-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From e2ebbc83c09c11b2751e2dd3b03b57e7bb8aeae0 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Fri, 23 Apr 2021 00:39:07 -0400
Subject: [PATCH v4] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN / COMMIT messages for transactions that are empty.

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/pgoutput/pgoutput.c | 43 +++++++++++++++++++++++++++++
 src/include/replication/pgoutput.h          |  3 ++
 src/test/subscription/t/020_messages.pl     |  5 ++--
 3 files changed, 48 insertions(+), 3 deletions(-)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index f68348d..f4a3576 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -345,10 +345,29 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputData	*data = (PGOutputData *) ctx->output_plugin_private;
+
+	(void)txn; /* keep compiler quiet */
+	/*
+	 * Don't send BEGIN message here. Instead, postpone it until the first
+	 * change. In logical replication, a common scenario is to replicate a set
+	 * of tables (instead of all tables) and transactions whose changes were on
+	 * table(s) that are not published will produce empty transactions. These
+	 * empty transactions will send BEGIN and COMMIT messages to subscribers,
+	 * using bandwidth on something with little/no use for logical replication.
+	 */
+	data->sent_begin_txn = false;
+}
+
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputData    *data = (PGOutputData *) ctx->output_plugin_private;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	data->sent_begin_txn = true;
 
 	if (send_replication_origin)
 	{
@@ -384,8 +403,14 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
+	PGOutputData	*data = (PGOutputData *) ctx->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/* skip COMMIT message if nothing was sent */
+	if (!data->sent_begin_txn)
+		return;
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
 	OutputPluginWrite(ctx, true);
@@ -551,6 +576,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/* output BEGIN if we haven't yet */
+	if (!data->sent_begin_txn && !in_streaming)
+	{
+		pgoutput_begin(ctx, txn);
+	}
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -693,6 +724,12 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/* output BEGIN if we haven't yet */
+		if (!data->sent_begin_txn && !in_streaming)
+		{
+			pgoutput_begin(ctx, txn);
+		}
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -725,6 +762,12 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/* output BEGIN if we haven't yet, avoid for streaming and non-transactional messages */
+	if (!data->sent_begin_txn && !in_streaming && transactional)
+	{
+		pgoutput_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 51e7c03..abd92bd 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -20,6 +20,9 @@ typedef struct PGOutputData
 	MemoryContext context;		/* private memory context for transient
 								 * allocations */
 
+	bool        sent_begin_txn; 	/* flag indicating whether begin
+									 * has already been sent */
+
 	/* client-supplied info: */
 	uint32		protocol_version;
 	List	   *publication_names;
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index c8be26b..2ea790f 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -78,9 +78,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is($result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot');
 
 $node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub ENABLE");
-- 
1.8.3.1

#30Ajin Cherian
itsajin@gmail.com
In reply to: Ajin Cherian (#29)
Re: logical replication empty transactions

An earlier comment from Anders:

We could e.g. have a new LogicalDecodingContext callback that is
called whenever WalSndWaitForWal() would wait. That'd check if there's
a pending "need" to send out a 'empty transaction'/feedback request
message. The "need" flag would get cleared whenever we send out data
bearing an LSN for other reasons.

I think the current Keep Alive messages already achieve this by
sending the current LSN as part of the Keep Alive messages.
/* construct the message... */
resetStringInfo(&output_message);
pq_sendbyte(&output_message, 'k');
pq_sendint64(&output_message, sentPtr); <=== Last sent WAL LSN
pq_sendint64(&output_message, GetCurrentTimestamp());
pq_sendbyte(&output_message, requestReply ? 1 : 0);

I'm not sure if anything more is required to keep empty transactions
updated as part of synchronous replicas. If my understanding on this
is not correct, let me know.

regards,
Ajin Cherian
Fujitsu Australia

#31Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#29)
Re: logical replication empty transactions

On Fri, Apr 23, 2021 at 3:46 PM Ajin Cherian <itsajin@gmail.com> wrote:

On Mon, Apr 19, 2021 at 6:22 PM Peter Smith <smithpb2250@gmail.com> wrote:

Here are a some review comments:

------

1. The patch v3 applied OK but with whitespace warnings

[postgres@CentOS7-x64 oss_postgres_2PC]$ git apply
../patches_misc/v3-0001-Skip-empty-transactions-for-logical-replication.patch
../patches_misc/v3-0001-Skip-empty-transactions-for-logical-replication.patch:98:
indent with spaces.
/* output BEGIN if we haven't yet, avoid for streaming and
non-transactional messages */
../patches_misc/v3-0001-Skip-empty-transactions-for-logical-replication.patch:99:
indent with spaces.
if (!data->xact_wrote_changes && !in_streaming && transactional)
warning: 2 lines add whitespace errors.

------

Fixed.

2. Please create a CF entry in [1] for this patch.

------

3. Patch comment

The comment describes the problem and then suddenly just says
"Postpone the BEGIN message until the first change."

I suggest changing it to say more like... "(blank line) This patch
addresses the above problem by postponing the BEGIN message until the
first change."

------

Updated.

4. pgoutput.h

Maybe for consistency with the context member, the comment for the new
member should be to the right instead of above it?

@@ -20,6 +20,9 @@ typedef struct PGOutputData
MemoryContext context; /* private memory context for transient
* allocations */

+ /* flag indicating whether messages have previously been sent */
+ bool        xact_wrote_changes;
+

------

5. pgoutput.h

+ /* flag indicating whether messages have previously been sent */

"previously been sent" --> "already been sent" ??

------

6. pgoutput.h - misleading member name

Actually, now that I have read all the rest of the code and how this
member is used I feel that this name is very misleading. e.g. For
"streaming" case then you still are writing changes but are not
setting this member at all - therefore it does not always mean what it
says.

I feel a better name for this would be something like
"sent_begin_txn". Then if you have sent BEGIN it is true. If you
haven't sent BEGIN it is false. It eliminates all ambiguity naming it
this way instead.

(This makes my feedback #5 redundant because the comment will be a bit
different if you do this).

------

Fixed above comments.

7. pgoutput.c - function pgoutput_begin_txn

@@ -345,6 +345,23 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,
static void
pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
{

I guess that you still needed to pass the txn because that is how the
API is documented, right?

But I am wondering if you ought to flag it as unused so you wont get
some BF machine giving warnings about it.

e.g. Syntax like this?

pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN * txn) {
(void)txn;
...

Updated.

------

8. pgoutput.c - function pgoutput_begin_txn

@@ -345,6 +345,23 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,
static void
pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
{
+ PGOutputData *data = ctx->output_plugin_private;
+
+ /*
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
+ */
+ data->xact_wrote_changes = false;
+ elog(LOG,"Holding of begin");
+}

Why is this loglevel LOG? Looks like leftover debugging.

Removed.

------

9. pgoutput.c - function pgoutput_commit_txn

@@ -384,8 +401,14 @@ static void
pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
XLogRecPtr commit_lsn)
{
+ PGOutputData *data = ctx->output_plugin_private;
+
OutputPluginUpdateProgress(ctx);
+ /* skip COMMIT message if nothing was sent */
+ if (!data->xact_wrote_changes)
+ return;
+

In the case where you decided to do nothing does it make sense that
you still called the function OutputPluginUpdateProgress(ctx); ?
I thought perhaps that your new check should come first so this call
would never happen.

Even though the empty transaction is not sent, the LSN is tracked as decoded, hence the progress needs to be updated.

------

10. pgoutput.c - variable declarations without casts

+ PGOutputData *data = ctx->output_plugin_private;

I noticed the new stack variable you declare have no casts.

This differs from the existing code which always looks like:
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;

There are a couple of examples of this so please search new code to
find them all.

-----

Fixed.

11. pgoutput.c - function pgoutput_change

@@ -551,6 +574,13 @@ pgoutput_change(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
Assert(false);
}

+ /* output BEGIN if we haven't yet */
+ if (!data->xact_wrote_changes && !in_streaming)
+ {
+ pgoutput_begin(ctx, txn);
+ data->xact_wrote_changes = true;
+ }

If the variable is renamed as previously suggested then the assignment
data->sent_BEGIN_txn = true; can be assigned in just 1 common place
INSIDE the pgoutput_begin function.

------

Updated.

12. pgoutput.c - pgoutput_truncate function

@@ -693,6 +723,13 @@ pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,

if (nrelids > 0)
{
+ /* output BEGIN if we haven't yet */
+ if (!data->xact_wrote_changes && !in_streaming)
+ {
+ pgoutput_begin(ctx, txn);
+ data->xact_wrote_changes = true;
+ }

(same comment as above)

If the variable is renamed as previously suggested then the assignment
data->sent_BEGIN_txn = true; can be assigned in just 1 common place
INSIDE the pgoutput_begin function.

13. pgoutput.c - pgoutput_message

@@ -725,6 +762,13 @@ pgoutput_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
if (in_streaming)
xid = txn->xid;

+    /* output BEGIN if we haven't yet, avoid for streaming and
non-transactional messages */
+    if (!data->xact_wrote_changes && !in_streaming && transactional)
+ {
+ pgoutput_begin(ctx, txn);
+ data->xact_wrote_changes = true;
+ }

(same comment as above)

If the variable is renamed as previously suggested then the assignment
data->sent_BEGIN_txn = true; can be assigned in just 1 common place
INSIDE the pgoutput_begin function.

------

Fixed.

14. Test Code.

I noticed that there is no test code specifically for seeing if empty
transactions get sent or not. Is it possible to write such a test or
is this traffic improvement only observable using the debugger?

The 020_messages.pl actually has a test case for tracking empty messages even though it is part of the messages test.

regards,
Ajin Cherian
Fujitsu Australia

Thanks for addressing my v3 review comments above.

I tested the latest v4.

The v4 patch applied cleanly.

make check-world completed successfully.

So this patch v4 looks LGTM, apart from the following 2 nitpick comments:

======

1. Suggest to add a blank line after the (void)txn; ?

@@ -345,10 +345,29 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+
+ (void)txn; /* keep compiler quiet */
+ /*
+ * Don't send BEGIN message here. Instead, postpone it until the first

======

2. Unnecessary statement blocks?

AFAIK those { } are not the usual PG code-style when there is only one
statement, so suggest to remove them.

Appies to 3 places:

@@ -551,6 +576,12 @@ pgoutput_change(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
Assert(false);
}

+ /* output BEGIN if we haven't yet */
+ if (!data->sent_begin_txn && !in_streaming)
+ {
+ pgoutput_begin(ctx, txn);
+ }

@@ -693,6 +724,12 @@ pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,

  if (nrelids > 0)
  {
+ /* output BEGIN if we haven't yet */
+ if (!data->sent_begin_txn && !in_streaming)
+ {
+ pgoutput_begin(ctx, txn);
+ }

@@ -725,6 +762,12 @@ pgoutput_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
if (in_streaming)
xid = txn->xid;

+ /* output BEGIN if we haven't yet, avoid for streaming and
non-transactional messages */
+ if (!data->sent_begin_txn && !in_streaming && transactional)
+ {
+ pgoutput_begin(ctx, txn);
+ }

------
Kind Regards,
Peter Smith.
Fujitsu Australia

#32Ajin Cherian
itsajin@gmail.com
In reply to: Peter Smith (#31)
1 attachment(s)
Re: logical replication empty transactions

On Mon, Apr 26, 2021 at 4:29 PM Peter Smith <smithpb2250@gmail.com> wrote:

The v4 patch applied cleanly.

make check-world completed successfully.

So this patch v4 looks LGTM, apart from the following 2 nitpick comments:

======

1. Suggest to add a blank line after the (void)txn; ?

@@ -345,10 +345,29 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,
static void
pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
{
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+
+ (void)txn; /* keep compiler quiet */
+ /*
+ * Don't send BEGIN message here. Instead, postpone it until the first

Fixed.

======

2. Unnecessary statement blocks?

AFAIK those { } are not the usual PG code-style when there is only one
statement, so suggest to remove them.

Appies to 3 places:

@@ -551,6 +576,12 @@ pgoutput_change(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
Assert(false);
}

+ /* output BEGIN if we haven't yet */
+ if (!data->sent_begin_txn && !in_streaming)
+ {
+ pgoutput_begin(ctx, txn);
+ }

@@ -693,6 +724,12 @@ pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,

if (nrelids > 0)
{
+ /* output BEGIN if we haven't yet */
+ if (!data->sent_begin_txn && !in_streaming)
+ {
+ pgoutput_begin(ctx, txn);
+ }

@@ -725,6 +762,12 @@ pgoutput_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
if (in_streaming)
xid = txn->xid;

+ /* output BEGIN if we haven't yet, avoid for streaming and
non-transactional messages */
+ if (!data->sent_begin_txn && !in_streaming && transactional)
+ {
+ pgoutput_begin(ctx, txn);
+ }

Fixed.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v5-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v5-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 11bc909ec45dac329c963ad722271788afbf331f Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Mon, 26 Apr 2021 23:39:38 -0400
Subject: [PATCH v5] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN / COMMIT messages for transactions that are empty.

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/pgoutput/pgoutput.c | 38 +++++++++++++++++++++++++++++
 src/include/replication/pgoutput.h          |  3 +++
 src/test/subscription/t/020_messages.pl     |  5 ++--
 3 files changed, 43 insertions(+), 3 deletions(-)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index f68348d..666bd7f 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -345,10 +345,30 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputData	*data = (PGOutputData *) ctx->output_plugin_private;
+
+	(void)txn; /* keep compiler quiet */
+
+	/*
+	 * Don't send BEGIN message here. Instead, postpone it until the first
+	 * change. In logical replication, a common scenario is to replicate a set
+	 * of tables (instead of all tables) and transactions whose changes were on
+	 * table(s) that are not published will produce empty transactions. These
+	 * empty transactions will send BEGIN and COMMIT messages to subscribers,
+	 * using bandwidth on something with little/no use for logical replication.
+	 */
+	data->sent_begin_txn = false;
+}
+
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputData    *data = (PGOutputData *) ctx->output_plugin_private;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	data->sent_begin_txn = true;
 
 	if (send_replication_origin)
 	{
@@ -384,8 +404,14 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
+	PGOutputData	*data = (PGOutputData *) ctx->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/* skip COMMIT message if nothing was sent */
+	if (!data->sent_begin_txn)
+		return;
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
 	OutputPluginWrite(ctx, true);
@@ -551,6 +577,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/* output BEGIN if we haven't yet */
+	if (!data->sent_begin_txn && !in_streaming)
+		pgoutput_begin(ctx, txn);
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -693,6 +723,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/* output BEGIN if we haven't yet */
+		if (!data->sent_begin_txn && !in_streaming)
+			pgoutput_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -725,6 +759,10 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/* output BEGIN if we haven't yet, avoid for streaming and non-transactional messages */
+	if (!data->sent_begin_txn && !in_streaming && transactional)
+		pgoutput_begin(ctx, txn);
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 51e7c03..abd92bd 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -20,6 +20,9 @@ typedef struct PGOutputData
 	MemoryContext context;		/* private memory context for transient
 								 * allocations */
 
+	bool        sent_begin_txn; 	/* flag indicating whether begin
+									 * has already been sent */
+
 	/* client-supplied info: */
 	uint32		protocol_version;
 	List	   *publication_names;
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index c8be26b..2ea790f 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -78,9 +78,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is($result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot');
 
 $node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub ENABLE");
-- 
1.8.3.1

#33Ajin Cherian
itsajin@gmail.com
In reply to: Ajin Cherian (#32)
1 attachment(s)
Re: logical replication empty transactions

On Tue, Apr 27, 2021 at 1:49 PM Ajin Cherian <itsajin@gmail.com> wrote:

Rebased the patch as it was no longer applying.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v6-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v6-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From bc4d6e0d6566051a87c5fb194609bf6ccfabd9df Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Tue, 25 May 2021 08:57:44 -0400
Subject: [PATCH v6] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN / COMMIT messages for transactions that are empty.

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/pgoutput/pgoutput.c | 38 +++++++++++++++++++++++++++++
 src/include/replication/pgoutput.h          |  3 +++
 src/test/subscription/t/020_messages.pl     |  5 ++--
 3 files changed, 43 insertions(+), 3 deletions(-)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index f68348d..666bd7f 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -345,10 +345,30 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputData	*data = (PGOutputData *) ctx->output_plugin_private;
+
+	(void)txn; /* keep compiler quiet */
+
+	/*
+	 * Don't send BEGIN message here. Instead, postpone it until the first
+	 * change. In logical replication, a common scenario is to replicate a set
+	 * of tables (instead of all tables) and transactions whose changes were on
+	 * table(s) that are not published will produce empty transactions. These
+	 * empty transactions will send BEGIN and COMMIT messages to subscribers,
+	 * using bandwidth on something with little/no use for logical replication.
+	 */
+	data->sent_begin_txn = false;
+}
+
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputData    *data = (PGOutputData *) ctx->output_plugin_private;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	data->sent_begin_txn = true;
 
 	if (send_replication_origin)
 	{
@@ -384,8 +404,14 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
+	PGOutputData	*data = (PGOutputData *) ctx->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/* skip COMMIT message if nothing was sent */
+	if (!data->sent_begin_txn)
+		return;
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
 	OutputPluginWrite(ctx, true);
@@ -551,6 +577,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/* output BEGIN if we haven't yet */
+	if (!data->sent_begin_txn && !in_streaming)
+		pgoutput_begin(ctx, txn);
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -693,6 +723,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/* output BEGIN if we haven't yet */
+		if (!data->sent_begin_txn && !in_streaming)
+			pgoutput_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -725,6 +759,10 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/* output BEGIN if we haven't yet, avoid for streaming and non-transactional messages */
+	if (!data->sent_begin_txn && !in_streaming && transactional)
+		pgoutput_begin(ctx, txn);
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 51e7c03..abd92bd 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -20,6 +20,9 @@ typedef struct PGOutputData
 	MemoryContext context;		/* private memory context for transient
 								 * allocations */
 
+	bool        sent_begin_txn; 	/* flag indicating whether begin
+									 * has already been sent */
+
 	/* client-supplied info: */
 	uint32		protocol_version;
 	List	   *publication_names;
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index 52bd92d..2b43ae0 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -86,9 +86,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
-- 
1.8.3.1

#34vignesh C
vignesh21@gmail.com
In reply to: Ajin Cherian (#33)
Re: logical replication empty transactions

On Tue, May 25, 2021 at 6:36 PM Ajin Cherian <itsajin@gmail.com> wrote:

On Tue, Apr 27, 2021 at 1:49 PM Ajin Cherian <itsajin@gmail.com> wrote:

Rebased the patch as it was no longer applying.

Thanks for the updated patch, few comments:
1) I'm not sure if we could add some tests for skip empty
transactions, if possible add a few tests.

2) We could add some debug level log messages for the transaction that
will be skipped.

3) You could keep this variable below the other bool variables in the structure:
+       bool        sent_begin_txn;     /* flag indicating whether begin
+
  * has already been sent */
+
4) You can split the comments to multi-line as it exceeds 80 chars
+       /* output BEGIN if we haven't yet, avoid for streaming and
non-transactional messages */
+       if (!data->sent_begin_txn && !in_streaming && transactional)
+               pgoutput_begin(ctx, txn);

Regards,
Vignesh

#35Ajin Cherian
itsajin@gmail.com
In reply to: vignesh C (#34)
1 attachment(s)
Re: logical replication empty transactions

On Thu, May 27, 2021 at 8:58 PM vignesh C <vignesh21@gmail.com> wrote:

Thanks for the updated patch, few comments:
1) I'm not sure if we could add some tests for skip empty
transactions, if possible add a few tests.

Added a few tests for prepared transactions as well as the existing
test in 020_messages.pl also tests regular transactions.

2) We could add some debug level log messages for the transaction that
will be skipped.

Added.

3) You could keep this variable below the other bool variables in the structure:
+       bool        sent_begin_txn;     /* flag indicating whether begin
+
* has already been sent */
+

I've moved this variable around, so this comment no longer is valid.

4) You can split the comments to multi-line as it exceeds 80 chars
+       /* output BEGIN if we haven't yet, avoid for streaming and
non-transactional messages */
+       if (!data->sent_begin_txn && !in_streaming && transactional)
+               pgoutput_begin(ctx, txn);

Done.

I've had to rebase the patch after a recent commit by Amit Kapila of
supporting two-phase commits in pub-sub [1]/messages/by-id/CAHut+PueG6u3vwG8DU=JhJiWa2TwmZ=bDqPchZkBky7ykzA7MA@mail.gmail.com.
Also I've modified the patch to also skip replicating empty prepared
transactions. Do let me know if you have any comments.

regards,
Ajin Cherian
Fujitsu Australia
[1]: /messages/by-id/CAHut+PueG6u3vwG8DU=JhJiWa2TwmZ=bDqPchZkBky7ykzA7MA@mail.gmail.com

Attachments:

v7-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v7-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From be6e8c62c7484656e7824fd3bd19b9552e023c19 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Wed, 14 Jul 2021 08:19:07 -0400
Subject: [PATCH v7] Skip empty transactions for logical replication.

The current logical replication behaviour is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN / BEGIN PREPARE message
until the first change. While processing a COMMIT message or a PREPARE message,
if there is no other change for that transaction,
do not send COMMIT message or PREPARE message. It means that pgoutput will
skip BEGIN / COMMIT or BEGIN PREPARE / PREPARE  messages for transactions that are empty.

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 contrib/test_decoding/test_decoding.c           |   7 +-
 doc/src/sgml/logicaldecoding.sgml               |  12 +-
 doc/src/sgml/protocol.sgml                      |  15 +++
 src/backend/replication/logical/logical.c       |   9 +-
 src/backend/replication/logical/proto.c         |  16 ++-
 src/backend/replication/logical/reorderbuffer.c |   2 +-
 src/backend/replication/logical/worker.c        |  38 ++++--
 src/backend/replication/pgoutput/pgoutput.c     | 161 +++++++++++++++++++++++-
 src/include/replication/logicalproto.h          |   8 +-
 src/include/replication/output_plugin.h         |   4 +-
 src/include/replication/reorderbuffer.h         |   4 +-
 src/test/subscription/t/020_messages.pl         |   5 +-
 src/test/subscription/t/021_twophase.pl         |  46 ++++++-
 src/tools/pgindent/typedefs.list                |   1 +
 14 files changed, 289 insertions(+), 39 deletions(-)

diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index e5cd84e..408dbfc 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -86,7 +86,9 @@ static void pg_decode_prepare_txn(LogicalDecodingContext *ctx,
 								  XLogRecPtr prepare_lsn);
 static void pg_decode_commit_prepared_txn(LogicalDecodingContext *ctx,
 										  ReorderBufferTXN *txn,
-										  XLogRecPtr commit_lsn);
+										  XLogRecPtr commit_lsn,
+										  XLogRecPtr prepare_end_lsn,
+										  TimestampTz prepare_time);
 static void pg_decode_rollback_prepared_txn(LogicalDecodingContext *ctx,
 											ReorderBufferTXN *txn,
 											XLogRecPtr prepare_end_lsn,
@@ -390,7 +392,8 @@ pg_decode_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 /* COMMIT PREPARED callback */
 static void
 pg_decode_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-							  XLogRecPtr commit_lsn)
+							  XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+							  TimestampTz prepare_time)
 {
 	TestDecodingData *data = ctx->output_plugin_private;
 
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 002efc8..123d2f1 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -884,11 +884,19 @@ typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
       The required <function>commit_prepared_cb</function> callback is called
       whenever a transaction <command>COMMIT PREPARED</command> has been decoded.
       The <parameter>gid</parameter> field, which is part of the
-      <parameter>txn</parameter> parameter, can be used in this callback.
+      <parameter>txn</parameter> parameter, can be used in this callback. The
+      parameters <parameter>prepare_end_lsn</parameter> and
+      <parameter>prepare_time</parameter> can be used to check if the plugin
+      has received this <command>PREPARE TRANSACTION</command> in which case
+      it can commit the transaction, otherwise, it can skip the commit. The
+      <parameter>gid</parameter> alone is not sufficient because the downstream
+      node can have a prepared transaction with the same identifier.
 <programlisting>
 typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
                                                ReorderBufferTXN *txn,
-                                               XLogRecPtr commit_lsn);
+                                               XLogRecPtr commit_lsn,
+                                               XLogRecPtr prepare_end_lsn,
+                                               TimestampTz prepare_time);
 </programlisting>
      </para>
     </sect3>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index e8cb78f..5e68dfb 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7550,6 +7550,13 @@ are available since protocol version 3.
 <varlistentry>
 <term>Int64</term>
 <listitem><para>
+                The end LSN of the prepare.
+</para></listitem>
+</varlistentry>
+
+<varlistentry>
+<term>Int64</term>
+<listitem><para>
                 The LSN of the commit prepared.
 </para></listitem>
 </varlistentry>
@@ -7564,6 +7571,14 @@ are available since protocol version 3.
 <varlistentry>
 <term>Int64</term>
 <listitem><para>
+                Prepare timestamp of the transaction. The value is in number
+                of microseconds since PostgreSQL epoch (2000-01-01).
+</para></listitem>
+</varlistentry>
+
+<varlistentry>
+<term>Int64</term>
+<listitem><para>
                 Commit timestamp of the transaction. The value is in number
                 of microseconds since PostgreSQL epoch (2000-01-01).
 </para></listitem>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index d61ef4c..67c762a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -63,7 +63,8 @@ static void begin_prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn
 static void prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 							   XLogRecPtr prepare_lsn);
 static void commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
-									   XLogRecPtr commit_lsn);
+									   XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+									   TimestampTz prepare_time);
 static void rollback_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 										 XLogRecPtr prepare_end_lsn, TimestampTz prepare_time);
 static void change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
@@ -936,7 +937,8 @@ prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 
 static void
 commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
-						   XLogRecPtr commit_lsn)
+						   XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+						   TimestampTz prepare_time)
 {
 	LogicalDecodingContext *ctx = cache->private_data;
 	LogicalErrorCallbackState state;
@@ -972,7 +974,8 @@ commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 						"commit_prepared_cb")));
 
 	/* do the actual work: call callback */
-	ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn);
+	ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn, prepare_end_lsn,
+									  prepare_time);
 
 	/* Pop the error context stack */
 	error_context_stack = errcallback.previous;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 13c8c3b..8f17007 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -206,7 +206,9 @@ logicalrep_read_prepare(StringInfo in, LogicalRepPreparedTxnData *prepare_data)
  */
 void
 logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
-								 XLogRecPtr commit_lsn)
+								 XLogRecPtr commit_lsn,
+								 XLogRecPtr prepare_end_lsn,
+								 TimestampTz prepare_time)
 {
 	uint8		flags = 0;
 
@@ -222,8 +224,10 @@ logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
 	pq_sendbyte(out, flags);
 
 	/* send fields */
+	pq_sendint64(out, prepare_end_lsn);
 	pq_sendint64(out, commit_lsn);
 	pq_sendint64(out, txn->end_lsn);
+	pq_sendint64(out, prepare_time);
 	pq_sendint64(out, txn->xact_time.commit_time);
 	pq_sendint32(out, txn->xid);
 
@@ -244,12 +248,16 @@ logicalrep_read_commit_prepared(StringInfo in, LogicalRepCommitPreparedTxnData *
 		elog(ERROR, "unrecognized flags %u in commit prepared message", flags);
 
 	/* read fields */
+	prepare_data->prepare_end_lsn = pq_getmsgint64(in);
+	if (prepare_data->prepare_end_lsn == InvalidXLogRecPtr)
+		elog(ERROR,"prepare_end_lsn is not set in commit prepared message");
 	prepare_data->commit_lsn = pq_getmsgint64(in);
 	if (prepare_data->commit_lsn == InvalidXLogRecPtr)
 		elog(ERROR, "commit_lsn is not set in commit prepared message");
-	prepare_data->end_lsn = pq_getmsgint64(in);
-	if (prepare_data->end_lsn == InvalidXLogRecPtr)
-		elog(ERROR, "end_lsn is not set in commit prepared message");
+	prepare_data->commit_end_lsn = pq_getmsgint64(in);
+	if (prepare_data->commit_end_lsn == InvalidXLogRecPtr)
+		elog(ERROR, "commit_end_lsn is not set in commit prepared message");
+	prepare_data->prepare_time = pq_getmsgint64(in);
 	prepare_data->commit_time = pq_getmsgint64(in);
 	prepare_data->xid = pq_getmsgint(in, 4);
 
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 7378beb..5a707e2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2794,7 +2794,7 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid,
 	txn->origin_lsn = origin_lsn;
 
 	if (is_commit)
-		rb->commit_prepared(rb, txn, commit_lsn);
+		rb->commit_prepared(rb, txn, commit_lsn, prepare_end_lsn, prepare_time);
 	else
 		rb->rollback_prepared(rb, txn, prepare_end_lsn, prepare_time);
 
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b9a7a7f..069dc31 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -966,27 +966,39 @@ apply_handle_commit_prepared(StringInfo s)
 	/* Compute GID for two_phase transactions. */
 	TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid,
 						   gid, sizeof(gid));
-
-	/* There is no transaction when COMMIT PREPARED is called */
-	begin_replication_step();
-
 	/*
-	 * Update origin state so we can restart streaming from correct position
-	 * in case of crash.
+	 * It is possible that we haven't received the prepare because
+	 * the transaction did not have any changes relevant to this
+	 * subscription and was essentially an empty prepare. In which case,
+	 * the walsender is optimized to drop the empty transaction and the
+	 * accompanying prepare. Silently ignore if we don't find the prepared
+	 * transaction.
 	 */
-	replorigin_session_origin_lsn = prepare_data.end_lsn;
-	replorigin_session_origin_timestamp = prepare_data.commit_time;
+	if (LookupGXact(gid, prepare_data.prepare_end_lsn,
+					prepare_data.prepare_time))
+	{
 
-	FinishPreparedTransaction(gid, true);
-	end_replication_step();
-	CommitTransactionCommand();
+		/* There is no transaction when COMMIT PREPARED is called */
+		begin_replication_step();
+
+		/*
+		 * Update origin state so we can restart streaming from correct position
+		 * in case of crash.
+		 */
+		replorigin_session_origin_lsn = prepare_data.commit_end_lsn;
+		replorigin_session_origin_timestamp = prepare_data.commit_time;
+
+		FinishPreparedTransaction(gid, true);
+		end_replication_step();
+		CommitTransactionCommand();
+	}
 	pgstat_report_stat(false);
 
-	store_flush_position(prepare_data.end_lsn);
+	store_flush_position(prepare_data.commit_end_lsn);
 	in_remote_transaction = false;
 
 	/* Process any tables that are being synchronized in parallel. */
-	process_syncing_tables(prepare_data.end_lsn);
+	process_syncing_tables(prepare_data.commit_end_lsn);
 
 	pgstat_report_activity(STATE_IDLE, NULL);
 }
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index e4314af..f7d808f 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -56,7 +56,9 @@ static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
 static void pgoutput_prepare_txn(LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn, XLogRecPtr prepare_lsn);
 static void pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx,
-										 ReorderBufferTXN *txn, XLogRecPtr commit_lsn);
+										 ReorderBufferTXN *txn, XLogRecPtr commit_lsn,
+										 XLogRecPtr prepare_end_lsn,
+										 TimestampTz prepare_time);
 static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   ReorderBufferTXN *txn,
 										   XLogRecPtr prepare_end_lsn,
@@ -130,6 +132,11 @@ typedef struct RelationSyncEntry
 	TupleConversionMap *map;
 } RelationSyncEntry;
 
+typedef struct PGOutputTxnData
+{
+	bool sent_begin_txn;    /* flag indicating whether begin has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -410,10 +417,32 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *data = MemoryContextAllocZero(ctx->context,
+														sizeof(PGOutputTxnData));
+
+	/*
+	 * Don't send BEGIN message here. Instead, postpone it until the first
+	 * change. In logical replication, a common scenario is to replicate a set
+	 * of tables (instead of all tables) and transactions whose changes were on
+	 * table(s) that are not published will produce empty transactions. These
+	 * empty transactions will send BEGIN and COMMIT messages to subscribers,
+	 * using bandwidth on something with little/no use for logical replication.
+	 */
+	data->sent_begin_txn = false;
+	txn->output_plugin_private = data;
+}
+
+
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*data = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(data);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	data->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -428,8 +457,22 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData	*data = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(data);
+	skip = !data->sent_begin_txn;
+	pfree(data);
+	txn->output_plugin_private = NULL;
 	OutputPluginUpdateProgress(ctx);
 
+	/* skip COMMIT message if nothing was sent */
+	if (skip)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
 	OutputPluginWrite(ctx, true);
@@ -441,10 +484,28 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	/*
+	 * Don't send BEGIN PREPARE message here. Instead, postpone it until the first
+	 * change. In logical replication, a common scenario is to replicate a set
+	 * of tables (instead of all tables) and transactions whose changes were on
+	 * table(s) that are not published will produce empty transactions. These
+	 * empty transactions will send BEGIN PREPARE and COMMIT PREPARED messages
+	 * to subscribers, using bandwidth on something with little/no use
+	 * for logical replication.
+	 */
+	pgoutput_begin_txn(ctx, txn);
+}
+
+static void
+pgoutput_begin_prepare(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData    *data = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(data);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
+	data->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -459,8 +520,18 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData    *data = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(data);
 	OutputPluginUpdateProgress(ctx);
 
+	/* skip PREPARE message if nothing was sent */
+	if (!data->sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty prepared transaction");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
@@ -471,12 +542,33 @@ pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  */
 static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-							 XLogRecPtr commit_lsn)
+							 XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+							 TimestampTz prepare_time)
 {
+	PGOutputTxnData    *data = (PGOutputTxnData *) txn->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/*
+	 * skip sending COMMIT PREPARED message if prepared transaction
+	 * has not been sent.
+	 */
+	if (data)
+	{
+		bool skip = !data->sent_begin_txn;
+		pfree(data);
+		txn->output_plugin_private = NULL;
+		if (skip)
+		{
+			elog(DEBUG1,
+				 "Skipping replication of COMMIT PREPARED of an empty transaction");
+			return;
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
-	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
+	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn, prepare_end_lsn,
+									 prepare_time);
 	OutputPluginWrite(ctx, true);
 }
 
@@ -489,8 +581,26 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
+	PGOutputTxnData    *data = (PGOutputTxnData *) txn->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/*
+	 * skip sending ROLLBACK PREPARED message if prepared transaction
+	 * has not been sent.
+	 */
+	if (data)
+	{
+		bool skip = !data->sent_begin_txn;
+		pfree(data);
+		txn->output_plugin_private = NULL;
+		if (skip)
+		{
+			elog(DEBUG1,
+				 "Skipping replication of ROLLBACK of an empty transaction");
+			return;
+		}
+	}
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
 									   prepare_time);
@@ -639,11 +749,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
 	Relation	ancestor = NULL;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	if (!in_streaming)
+		Assert(txndata);
+
 	if (!is_publishable_relation(relation))
 		return;
 
@@ -677,6 +792,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/* output BEGIN if we haven't yet */
+	if (!in_streaming && !txndata->sent_begin_txn)
+	{
+		if (rbtxn_prepared(txn))
+			pgoutput_begin_prepare(ctx, txn);
+		else
+			pgoutput_begin(ctx, txn);
+	}
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -779,6 +903,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -786,6 +911,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	Oid		   *relids;
 	TransactionId xid = InvalidTransactionId;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	if (!in_streaming)
+		Assert(txndata);
+
 	/* Remember the xid for the change in streaming mode. See pgoutput_change. */
 	if (in_streaming)
 		xid = change->txn->xid;
@@ -822,6 +951,15 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/* output BEGIN if we haven't yet */
+		if (!in_streaming && !txndata->sent_begin_txn)
+		{
+			if (rbtxn_prepared(txn))
+				pgoutput_begin_prepare(ctx, txn);
+			else
+				pgoutput_begin(ctx, txn);
+		}
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -842,6 +980,7 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				 const char *message)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	TransactionId xid = InvalidTransactionId;
 
 	if (!data->messages)
@@ -854,6 +993,22 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages
+	 */
+	if (!in_streaming && transactional)
+	{
+		txndata = (PGOutputTxnData *) txn->output_plugin_private;
+		if (!txndata->sent_begin_txn)
+		{
+			if (rbtxn_prepared(txn))
+				pgoutput_begin_prepare(ctx, txn);
+			else
+				pgoutput_begin(ctx, txn);
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 63de90d..0be0a07 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -148,8 +148,10 @@ typedef struct LogicalRepPreparedTxnData
  */
 typedef struct LogicalRepCommitPreparedTxnData
 {
+	XLogRecPtr	prepare_end_lsn;
 	XLogRecPtr	commit_lsn;
-	XLogRecPtr	end_lsn;
+	XLogRecPtr	commit_end_lsn;
+	TimestampTz prepare_time;
 	TimestampTz commit_time;
 	TransactionId xid;
 	char		gid[GIDSIZE];
@@ -188,7 +190,9 @@ extern void logicalrep_write_prepare(StringInfo out, ReorderBufferTXN *txn,
 extern void logicalrep_read_prepare(StringInfo in,
 									LogicalRepPreparedTxnData *prepare_data);
 extern void logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
-											 XLogRecPtr commit_lsn);
+											 XLogRecPtr commit_lsn,
+											 XLogRecPtr prepare_end_lsn,
+											 TimestampTz prepare_time);
 extern void logicalrep_read_commit_prepared(StringInfo in,
 											LogicalRepCommitPreparedTxnData *prepare_data);
 extern void logicalrep_write_rollback_prepared(StringInfo out, ReorderBufferTXN *txn,
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 810495e..0d28306 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -128,7 +128,9 @@ typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
  */
 typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
 											   ReorderBufferTXN *txn,
-											   XLogRecPtr commit_lsn);
+											   XLogRecPtr commit_lsn,
+											   XLogRecPtr prepare_end_lsn,
+											   TimestampTz prepare_time);
 
 /*
  * Called for ROLLBACK PREPARED.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b40ff7..11e2e1e 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -442,7 +442,9 @@ typedef void (*ReorderBufferPrepareCB) (ReorderBuffer *rb,
 /* commit prepared callback signature */
 typedef void (*ReorderBufferCommitPreparedCB) (ReorderBuffer *rb,
 											   ReorderBufferTXN *txn,
-											   XLogRecPtr commit_lsn);
+											   XLogRecPtr commit_lsn,
+											   XLogRecPtr prepare_end_lsn,
+											   TimestampTz prepare_time);
 
 /* rollback  prepared callback signature */
 typedef void (*ReorderBufferRollbackPreparedCB) (ReorderBuffer *rb,
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index 0e218e0..3d246be 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/test/subscription/t/021_twophase.pl b/src/test/subscription/t/021_twophase.pl
index c6ada92..677ca50 100644
--- a/src/test/subscription/t/021_twophase.pl
+++ b/src/test/subscription/t/021_twophase.pl
@@ -6,7 +6,7 @@ use strict;
 use warnings;
 use PostgresNode;
 use TestLib;
-use Test::More tests => 24;
+use Test::More tests => 25;
 
 ###############################
 # Setup
@@ -318,10 +318,9 @@ $node_publisher->safe_psql('postgres', "
 
 $node_publisher->wait_for_catchup($appname_copy);
 
-# Check that the transaction has been prepared on the subscriber, there will be 2
-# prepared transactions for the 2 subscriptions.
+# Check that the transaction has been prepared on the subscriber
 $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(2), 'transaction is prepared on subscriber');
+is($result, qq(1), 'transaction is prepared on subscriber');
 
 # Now commit the insert and verify that it IS replicated
 $node_publisher->safe_psql('postgres', "COMMIT PREPARED 'mygid';");
@@ -337,6 +336,45 @@ is($result, qq(2), 'replicated data in subscriber table');
 $node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub_copy;");
 $node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_copy;");
 
+##############################
+# Test empty prepares
+##############################
+
+# create a table that is not part of the publication
+$node_publisher->safe_psql('postgres',
+   "CREATE TABLE tab_nopub (a int PRIMARY KEY)");
+
+# disable the subscription so that we can peek at the slot
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub DISABLE");
+
+# wait for the replication slot to become inactive in the publisher
+$node_publisher->poll_query_until('postgres',
+   "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'tap_sub' AND active='f'", 1);
+
+# create a transaction with no changes relevant to the slot
+$node_publisher->safe_psql('postgres', "
+   BEGIN;
+   INSERT INTO tab_nopub SELECT generate_series(1,10);
+   PREPARE TRANSACTION 'empty_transaction';
+   COMMIT PREPARED 'empty_transaction';");
+
+# peek at the contents of the slot
+$result = $node_publisher->safe_psql(
+   'postgres', qq(
+       SELECT get_byte(data, 0)
+       FROM pg_logical_slot_get_binary_changes('tap_sub', NULL, NULL,
+           'proto_version', '1',
+           'publication_names', 'tap_pub')
+));
+
+# the empty transaction should be skipped
+is($result, qq(),
+   'empty transaction dropped on slot'
+);
+
+# enable the subscription to test cleanup
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub ENABLE");
+
 ###############################
 # check all the cleanup
 ###############################
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 37cf4b2..75639ab 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1606,6 +1606,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#36osumi.takamichi@fujitsu.com
osumi.takamichi@fujitsu.com
In reply to: Ajin Cherian (#35)
RE: logical replication empty transactions

On Wednesday, July 14, 2021 9:30 PM Ajin Cherian <itsajin@gmail.com> wrote:

I've had to rebase the patch after a recent commit by Amit Kapila of supporting
two-phase commits in pub-sub [1].
Also I've modified the patch to also skip replicating empty prepared
transactions. Do let me know if you have any comments.

Hi

I started to test this patch but will give you some really minor quick feedbacks.

(1) pg_logical_slot_get_binary_changes() params.

Technically, looks better to have proto_version 3 & two_phase option for the function
to test empty prepare ? I felt proto_version 1 doesn't support 2PC.
[1]: https://www.postgresql.org/docs/devel/protocol-logicalrep-message-formats.html
are available since protocol version 3." Then, if the test wants to skip empty *prepares*,
I suggest to update the proto_version and set two_phase 'on'.

+##############################
+# Test empty prepares
+##############################
...
+# peek at the contents of the slot
+$result = $node_publisher->safe_psql(
+   'postgres', qq(
+       SELECT get_byte(data, 0)
+       FROM pg_logical_slot_get_binary_changes('tap_sub', NULL, NULL,
+           'proto_version', '1',
+           'publication_names', 'tap_pub')
+));

(2) The following sentences may start with a lowercase letter.
There are other similar codes for this.

+ elog(DEBUG1, "Skipping replication of an empty transaction");

[1]: https://www.postgresql.org/docs/devel/protocol-logicalrep-message-formats.html

Best Regards,
Takamichi Osumi

#37Peter Smith
smithpb2250@gmail.com
In reply to: osumi.takamichi@fujitsu.com (#36)
Re: logical replication empty transactions

Hi Ajin,

I have reviewed the v7 patch and given my feedback comments below.

Apply OK
Build OK
make check OK
TAP (subscriptions) make check OK
Build PG Docs (html) OK

Although I made lots of review comments below, the important point is
that none of them are functional - they are only minore re-wordings
and some code refactoring that I thought would make the code simpler
and/or easier to read. YMMV, so please feel free to disagree with any
of them.

//////////

1a. Commit Comment - wording

BEFORE
This patch addresses the above problem by postponing the BEGIN / BEGIN
PREPARE message until the first change.

AFTER
This patch addresses the above problem by postponing the BEGIN / BEGIN
PREPARE messages until the first change is encountered.

------

1b. Commit Comment - wording

BEFORE
While processing a COMMIT message or a PREPARE message, if there is no
other change for that transaction, do not send COMMIT message or
PREPARE message.

AFTER
If (when processing a COMMIT / PREPARE message) we find there had been
no other change for that transaction, then do not send the COMMIT /
PREPARE message.

------

2. doc/src/sgml/logicaldecoding.sgml - wording

@@ -884,11 +884,19 @@ typedef void (*LogicalDecodePrepareCB) (struct
LogicalDecodingContext *ctx,
       The required <function>commit_prepared_cb</function> callback is called
       whenever a transaction <command>COMMIT PREPARED</command> has
been decoded.
       The <parameter>gid</parameter> field, which is part of the
-      <parameter>txn</parameter> parameter, can be used in this callback.
+      <parameter>txn</parameter> parameter, can be used in this callback. The
+      parameters <parameter>prepare_end_lsn</parameter> and
+      <parameter>prepare_time</parameter> can be used to check if the plugin
+      has received this <command>PREPARE TRANSACTION</command> in which case
+      it can commit the transaction, otherwise, it can skip the commit. The
+      <parameter>gid</parameter> alone is not sufficient because the downstream
+      node can have a prepared transaction with the same identifier.

=>

(some minor rewording of the last part)

AFTER:

The parameters <parameter>prepare_end_lsn</parameter> and
<parameter>prepare_time</parameter> can be used to check if the plugin
has received this <command>PREPARE TRANSACTION</command> or not. If
yes, it can commit the transaction, otherwise, it can skip the commit.
The <parameter>gid</parameter> alone is not sufficient to determine
this because the downstream node may already have a prepared
transaction with the same identifier.

------

3. src/backend/replication/logical/proto.c - whitespace

@@ -244,12 +248,16 @@ logicalrep_read_commit_prepared(StringInfo in,
LogicalRepCommitPreparedTxnData *
elog(ERROR, "unrecognized flags %u in commit prepared message", flags);

  /* read fields */
+ prepare_data->prepare_end_lsn = pq_getmsgint64(in);
+ if (prepare_data->prepare_end_lsn == InvalidXLogRecPtr)
+ elog(ERROR,"prepare_end_lsn is not set in commit prepared message");

=>

There is missing space before the 2nd elog param.

------

4. src/backend/replication/logical/worker.c - comment typos

  /*
- * Update origin state so we can restart streaming from correct position
- * in case of crash.
+ * It is possible that we haven't received the prepare because
+ * the transaction did not have any changes relevant to this
+ * subscription and was essentially an empty prepare. In which case,
+ * the walsender is optimized to drop the empty transaction and the
+ * accompanying prepare. Silently ignore if we don't find the prepared
+ * transaction.
  */

4a. =>

"and was essentially an empty prepare" --> "so was essentially an empty prepare"

4b. =>

"In which case" --> "In this case"

------

5. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin_txn

@@ -410,10 +417,32 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+ PGOutputTxnData    *data = MemoryContextAllocZero(ctx->context,
+ sizeof(PGOutputTxnData));
+
+ /*
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
+ */
+ data->sent_begin_txn = false;
+ txn->output_plugin_private = data;
+}

=>

I felt that since this message postponement is now the new behaviour
of this function then probably this should all be a function level
comment instead of the comment being in the body of the function

------

6. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin

+
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)

=>

Even though it is kind of obvious, it is probably better to provide a
function comment here too

------

7. src/backend/replication/pgoutput/pgoutput.c - pgoutput_commit_txn

@@ -428,8 +457,22 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  XLogRecPtr commit_lsn)
 {
+ PGOutputTxnData *data = (PGOutputTxnData *) txn->output_plugin_private;
+ bool            skip;
+
+ Assert(data);
+ skip = !data->sent_begin_txn;
+ pfree(data);
+ txn->output_plugin_private = NULL;
  OutputPluginUpdateProgress(ctx);
+ /* skip COMMIT message if nothing was sent */
+ if (skip)
+ {
+ elog(DEBUG1, "Skipping replication of an empty transaction");
+ return;
+ }
+

7a. =>

I felt that the comment "skip COMMIT message if nothing was sent"
should be done at the point where you *decide* to skip or not. So you
could either move that comment to where the skip variable is assigned.
Or (my preference) leave the comment where it is but change the
variable name to be sent_begin = !data->sent_begin_txn;

------

Regardless I think the comment should be elaborated a bit to describe
the reason more.

7b. =>

BEFORE
/* skip COMMIT message if nothing was sent */

AFTER
/* If a BEGIN message was not yet sent, then it means there were no
relevant changes encountered, so we can skip the COMMIT message too.
*/

------

8. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin_prepare_txn

@@ -441,10 +484,28 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
 static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+ /*
+ * Don't send BEGIN PREPARE message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN PREPARE and COMMIT PREPARED messages
+ * to subscribers, using bandwidth on something with little/no use
+ * for logical replication.
+ */
+ pgoutput_begin_txn(ctx, txn);
+}

8a. =>

Like previously, I felt that this big comment should be at the
function level of pgoutput_begin_prepare_txn instead of in the body of
the function.

------

8b. =>

And then the body comment would be something simple like:

/* Delegate to assign the begin sent flag as false same as for the
BEGIN message. */
pgoutput_begin_txn(ctx, txn);

------

9. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin_prepare

+
+static void
+pgoutput_begin_prepare(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)

=>

Probably this needs a function comment.

------

10. src/backend/replication/pgoutput/pgoutput.c - pgoutput_prepare_txn

@@ -459,8 +520,18 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  XLogRecPtr prepare_lsn)
 {
+ PGOutputTxnData    *data = (PGOutputTxnData *) txn->output_plugin_private;
+
+ Assert(data);
  OutputPluginUpdateProgress(ctx);
+ /* skip PREPARE message if nothing was sent */
+ if (!data->sent_begin_txn)

=>

Maybe elaborate on that "skip PREPARE message if nothing was sent"
comment in a way similar to my review comment 7b. For example,

AFTER
/* If the BEGIN was not yet sent, then it means there were no relevant
changes encountered, so we can skip the PREPARE message too. */

------

11. src/backend/replication/pgoutput/pgoutput.c - pgoutput_commit_prepared_txn

@@ -471,12 +542,33 @@ pgoutput_prepare_txn(LogicalDecodingContext
*ctx, ReorderBufferTXN *txn,
  */
 static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
- XLogRecPtr commit_lsn)
+ XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+ TimestampTz prepare_time)
 {
+ PGOutputTxnData    *data = (PGOutputTxnData *) txn->output_plugin_private;
+
  OutputPluginUpdateProgress(ctx);
+ /*
+ * skip sending COMMIT PREPARED message if prepared transaction
+ * has not been sent.
+ */
+ if (data)

=>

Similar to previous review comment 10, I think the reason for the skip
should be elaborated a little bit. For example,

AFTER
/* If the BEGIN PREPARE was not yet sent, then it means there were no
relevant changes encountered, so we can skip the COMMIT PREPARED
message too. */

------

12. src/backend/replication/pgoutput/pgoutput.c - pgoutput_rollback_prepared_txn

=> Similar as for pgoutput_comment_prepared_txn (see review comment 11)

------

13. src/backend/replication/pgoutput/pgoutput.c - pgoutput_change

@@ -639,11 +749,16 @@ pgoutput_change(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
  Relation relation, ReorderBufferChange *change)
 {
  PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
  MemoryContext old;
  RelationSyncEntry *relentry;
  TransactionId xid = InvalidTransactionId;
  Relation ancestor = NULL;
+ /* If not streaming, should have setup txndata as part of
BEGIN/BEGIN PREPARE */
+ if (!in_streaming)
+ Assert(txndata);
+
  if (!is_publishable_relation(relation))
  return;

13a. =>

I felt the streaming logic with the txndata is a bit confusing. I
think it would be easier to have another local variable (sent_begin)
and use it like this...

bool sent_begin;
if (in_streaming)
{
sent_begin = true;
else
{
PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
Assert(txndata)
sent_begin = txn->sent_begin_txn;
}

...

------

+ /* output BEGIN if we haven't yet */

13b. =>

I thought the comment is not quite right

AFTER
/* Output BEGIN / BEGIN PREPARE if we haven't yet */

------

+ if (!in_streaming && !txndata->sent_begin_txn)
+ {
+ if (rbtxn_prepared(txn))
+ pgoutput_begin_prepare(ctx, txn);
+ else
+ pgoutput_begin(ctx, txn);
+ }
+

13.c =>

If you introduce the variable (as suggested in 13a) this code becomes
much simpler:

AFTER

if (!sent_begin)
{
if (rbtxn_prepared(txn))
pgoutput_begin_prepare(ctx, txn)
else
pgoutput_begin(ctx, txn);
}

------

14. src/backend/replication/pgoutput/pgoutput.c - pgoutput_truncate

=>

All the similar review comments made for pg_change (13a, 13b, 13c)
apply to pgoutput_truncate here also.

------

15. src/backend/replication/pgoutput/pgoutput.c - pgoutput_message

@@ -842,6 +980,7 @@ pgoutput_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
const char *message)
{
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ PGOutputTxnData *txndata;
TransactionId xid = InvalidTransactionId;

=>

This variable should be declared in the block where it is used,
similar to the suggestion 13a.

Also is it just an accidental omission that you did Assert(txndata)
for all the other places but not here?

------

Kind Regards,
Peter Smith.
Fujitsu Australia

#38Ajin Cherian
itsajin@gmail.com
In reply to: Peter Smith (#37)
1 attachment(s)
Re: logical replication empty transactions

On Mon, Jul 19, 2021 at 3:24 PM Peter Smith <smithpb2250@gmail.com> wrote:

1a. Commit Comment - wording

updated.

1b. Commit Comment - wording

updated.

2. doc/src/sgml/logicaldecoding.sgml - wording

@@ -884,11 +884,19 @@ typedef void (*LogicalDecodePrepareCB) (struct
LogicalDecodingContext *ctx,
The required <function>commit_prepared_cb</function> callback is called
whenever a transaction <command>COMMIT PREPARED</command> has
been decoded.
The <parameter>gid</parameter> field, which is part of the
-      <parameter>txn</parameter> parameter, can be used in this callback.
+      <parameter>txn</parameter> parameter, can be used in this callback. The
+      parameters <parameter>prepare_end_lsn</parameter> and
+      <parameter>prepare_time</parameter> can be used to check if the plugin
+      has received this <command>PREPARE TRANSACTION</command> in which case
+      it can commit the transaction, otherwise, it can skip the commit. The
+      <parameter>gid</parameter> alone is not sufficient because the downstream
+      node can have a prepared transaction with the same identifier.

=>

(some minor rewording of the last part)

updated.

3. src/backend/replication/logical/proto.c - whitespace

@@ -244,12 +248,16 @@ logicalrep_read_commit_prepared(StringInfo in,
LogicalRepCommitPreparedTxnData *
elog(ERROR, "unrecognized flags %u in commit prepared message", flags);

/* read fields */
+ prepare_data->prepare_end_lsn = pq_getmsgint64(in);
+ if (prepare_data->prepare_end_lsn == InvalidXLogRecPtr)
+ elog(ERROR,"prepare_end_lsn is not set in commit prepared message");

=>

There is missing space before the 2nd elog param.

fixed.

4a. =>

"and was essentially an empty prepare" --> "so was essentially an empty prepare"

4b. =>

"In which case" --> "In this case"

------

fixed.

I felt that since this message postponement is now the new behaviour
of this function then probably this should all be a function level
comment instead of the comment being in the body of the function

------

6. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin

+
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)

=>

Even though it is kind of obvious, it is probably better to provide a
function comment here too

------

Changed accordingly.

I felt that the comment "skip COMMIT message if nothing was sent"
should be done at the point where you *decide* to skip or not. So you
could either move that comment to where the skip variable is assigned.
Or (my preference) leave the comment where it is but change the
variable name to be sent_begin = !data->sent_begin_txn;

Updated the comment to where the skip variable is assigned.

------

Regardless I think the comment should be elaborated a bit to describe
the reason more.

7b. =>

BEFORE
/* skip COMMIT message if nothing was sent */

AFTER
/* If a BEGIN message was not yet sent, then it means there were no
relevant changes encountered, so we can skip the COMMIT message too.
*/

Updated accordingly.

------

Like previously, I felt that this big comment should be at the
function level of pgoutput_begin_prepare_txn instead of in the body of
the function.

------

8b. =>

And then the body comment would be something simple like:

/* Delegate to assign the begin sent flag as false same as for the
BEGIN message. */
pgoutput_begin_txn(ctx, txn);

Updated accordingly.

------

9. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin_prepare

+
+static void
+pgoutput_begin_prepare(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)

=>

Probably this needs a function comment.

Updated.

------

10. src/backend/replication/pgoutput/pgoutput.c - pgoutput_prepare_txn

@@ -459,8 +520,18 @@ static void
pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
XLogRecPtr prepare_lsn)
{
+ PGOutputTxnData    *data = (PGOutputTxnData *) txn->output_plugin_private;
+
+ Assert(data);
OutputPluginUpdateProgress(ctx);
+ /* skip PREPARE message if nothing was sent */
+ if (!data->sent_begin_txn)

=>

Maybe elaborate on that "skip PREPARE message if nothing was sent"
comment in a way similar to my review comment 7b. For example,

AFTER
/* If the BEGIN was not yet sent, then it means there were no relevant
changes encountered, so we can skip the PREPARE message too. */

Updated.

------

11. src/backend/replication/pgoutput/pgoutput.c - pgoutput_commit_prepared_txn

@@ -471,12 +542,33 @@ pgoutput_prepare_txn(LogicalDecodingContext
*ctx, ReorderBufferTXN *txn,
*/
static void
pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
- XLogRecPtr commit_lsn)
+ XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+ TimestampTz prepare_time)
{
+ PGOutputTxnData    *data = (PGOutputTxnData *) txn->output_plugin_private;
+
OutputPluginUpdateProgress(ctx);
+ /*
+ * skip sending COMMIT PREPARED message if prepared transaction
+ * has not been sent.
+ */
+ if (data)

=>

Similar to previous review comment 10, I think the reason for the skip
should be elaborated a little bit. For example,

AFTER
/* If the BEGIN PREPARE was not yet sent, then it means there were no
relevant changes encountered, so we can skip the COMMIT PREPARED
message too. */

------

Updated accordingly.

12. src/backend/replication/pgoutput/pgoutput.c - pgoutput_rollback_prepared_txn

=> Similar as for pgoutput_comment_prepared_txn (see review comment 11)

------

Updated,

13. src/backend/replication/pgoutput/pgoutput.c - pgoutput_change

@@ -639,11 +749,16 @@ pgoutput_change(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
Relation relation, ReorderBufferChange *change)
{
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
TransactionId xid = InvalidTransactionId;
Relation ancestor = NULL;
+ /* If not streaming, should have setup txndata as part of
BEGIN/BEGIN PREPARE */
+ if (!in_streaming)
+ Assert(txndata);
+
if (!is_publishable_relation(relation))
return;

13a. =>

I felt the streaming logic with the txndata is a bit confusing. I
think it would be easier to have another local variable (sent_begin)
and use it like this...

bool sent_begin;
if (in_streaming)
{
sent_begin = true;
else
{
PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
Assert(txndata)
sent_begin = txn->sent_begin_txn;
}

I did not make the change, because in case of streaming "Sent_begin"
is not true, so it seemed incorrect coding it
that way. Instead , I have modified the comment to mention that
streaming transaction do not send BEG / BEGIN PREPARE.

...

------

+ /* output BEGIN if we haven't yet */

13b. =>

I thought the comment is not quite right

AFTER
/* Output BEGIN / BEGIN PREPARE if we haven't yet */

------

Updated.

+ if (!in_streaming && !txndata->sent_begin_txn)
+ {
+ if (rbtxn_prepared(txn))
+ pgoutput_begin_prepare(ctx, txn);
+ else
+ pgoutput_begin(ctx, txn);
+ }
+

13.c =>

If you introduce the variable (as suggested in 13a) this code becomes
much simpler:

Skipped this. (reason mentioned above)

------

14. src/backend/replication/pgoutput/pgoutput.c - pgoutput_truncate

=>

All the similar review comments made for pg_change (13a, 13b, 13c)
apply to pgoutput_truncate here also.

------

Updated.

15. src/backend/replication/pgoutput/pgoutput.c - pgoutput_message

@@ -842,6 +980,7 @@ pgoutput_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
const char *message)
{
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ PGOutputTxnData *txndata;
TransactionId xid = InvalidTransactionId;

=>

This variable should be declared in the block where it is used,
similar to the suggestion 13a.

Also is it just an accidental omission that you did Assert(txndata)
for all the other places but not here?

Moved location of the variable and added an assert.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v8-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v8-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 7c0c403625bef87ef67b3930be7fd3171628cc3e Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Wed, 21 Jul 2021 06:29:57 -0400
Subject: [PATCH v8] Skip empty transactions for logical replication.

The current logical replication behaviour is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN / BEGIN
PREPARE messages until the first change is encountered.
If (when processing a COMMIT / PREPARE message) we find there had been
no other change for that transaction, then do not send the COMMIT /
PREPARE message. This means that pgoutput will skip BEGIN / COMMIT
or BEGIN PREPARE / PREPARE  messages for transactions that are empty.

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 contrib/test_decoding/test_decoding.c           |   7 +-
 doc/src/sgml/logicaldecoding.sgml               |  13 +-
 doc/src/sgml/protocol.sgml                      |  15 ++
 src/backend/replication/logical/logical.c       |   9 +-
 src/backend/replication/logical/proto.c         |  16 +-
 src/backend/replication/logical/reorderbuffer.c |   2 +-
 src/backend/replication/logical/worker.c        |  38 +++--
 src/backend/replication/pgoutput/pgoutput.c     | 188 +++++++++++++++++++++++-
 src/include/replication/logicalproto.h          |   8 +-
 src/include/replication/output_plugin.h         |   4 +-
 src/include/replication/reorderbuffer.h         |   4 +-
 src/test/subscription/t/020_messages.pl         |   5 +-
 src/test/subscription/t/021_twophase.pl         |  46 +++++-
 src/tools/pgindent/typedefs.list                |   1 +
 14 files changed, 316 insertions(+), 40 deletions(-)

diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index e5cd84e..408dbfc 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -86,7 +86,9 @@ static void pg_decode_prepare_txn(LogicalDecodingContext *ctx,
 								  XLogRecPtr prepare_lsn);
 static void pg_decode_commit_prepared_txn(LogicalDecodingContext *ctx,
 										  ReorderBufferTXN *txn,
-										  XLogRecPtr commit_lsn);
+										  XLogRecPtr commit_lsn,
+										  XLogRecPtr prepare_end_lsn,
+										  TimestampTz prepare_time);
 static void pg_decode_rollback_prepared_txn(LogicalDecodingContext *ctx,
 											ReorderBufferTXN *txn,
 											XLogRecPtr prepare_end_lsn,
@@ -390,7 +392,8 @@ pg_decode_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 /* COMMIT PREPARED callback */
 static void
 pg_decode_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-							  XLogRecPtr commit_lsn)
+							  XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+							  TimestampTz prepare_time)
 {
 	TestDecodingData *data = ctx->output_plugin_private;
 
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 89b8090..27811e5 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -884,11 +884,20 @@ typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
       The required <function>commit_prepared_cb</function> callback is called
       whenever a transaction <command>COMMIT PREPARED</command> has been decoded.
       The <parameter>gid</parameter> field, which is part of the
-      <parameter>txn</parameter> parameter, can be used in this callback.
+      <parameter>txn</parameter> parameter, can be used in this callback. The
+      parameters <parameter>prepare_end_lsn</parameter> and
+      <parameter>prepare_time</parameter> can be used to check if the plugin
+      has received this <command>PREPARE TRANSACTION</command> command or not.
+      If yes, it can commit the transaction, otherwise, it can skip the commit.
+      The <parameter>gid</parameter> alone is not sufficient to determine this
+      because the downstream may already have a prepared transaction with the
+      same identifier.
 <programlisting>
 typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
                                                ReorderBufferTXN *txn,
-                                               XLogRecPtr commit_lsn);
+                                               XLogRecPtr commit_lsn,
+                                               XLogRecPtr prepare_end_lsn,
+                                               TimestampTz prepare_time);
 </programlisting>
      </para>
     </sect3>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index e8cb78f..5e68dfb 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7550,6 +7550,13 @@ are available since protocol version 3.
 <varlistentry>
 <term>Int64</term>
 <listitem><para>
+                The end LSN of the prepare.
+</para></listitem>
+</varlistentry>
+
+<varlistentry>
+<term>Int64</term>
+<listitem><para>
                 The LSN of the commit prepared.
 </para></listitem>
 </varlistentry>
@@ -7564,6 +7571,14 @@ are available since protocol version 3.
 <varlistentry>
 <term>Int64</term>
 <listitem><para>
+                Prepare timestamp of the transaction. The value is in number
+                of microseconds since PostgreSQL epoch (2000-01-01).
+</para></listitem>
+</varlistentry>
+
+<varlistentry>
+<term>Int64</term>
+<listitem><para>
                 Commit timestamp of the transaction. The value is in number
                 of microseconds since PostgreSQL epoch (2000-01-01).
 </para></listitem>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index d61ef4c..67c762a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -63,7 +63,8 @@ static void begin_prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn
 static void prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 							   XLogRecPtr prepare_lsn);
 static void commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
-									   XLogRecPtr commit_lsn);
+									   XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+									   TimestampTz prepare_time);
 static void rollback_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 										 XLogRecPtr prepare_end_lsn, TimestampTz prepare_time);
 static void change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
@@ -936,7 +937,8 @@ prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 
 static void
 commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
-						   XLogRecPtr commit_lsn)
+						   XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+						   TimestampTz prepare_time)
 {
 	LogicalDecodingContext *ctx = cache->private_data;
 	LogicalErrorCallbackState state;
@@ -972,7 +974,8 @@ commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 						"commit_prepared_cb")));
 
 	/* do the actual work: call callback */
-	ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn);
+	ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn, prepare_end_lsn,
+									  prepare_time);
 
 	/* Pop the error context stack */
 	error_context_stack = errcallback.previous;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index a245252..47a7489 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -206,7 +206,9 @@ logicalrep_read_prepare(StringInfo in, LogicalRepPreparedTxnData *prepare_data)
  */
 void
 logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
-								 XLogRecPtr commit_lsn)
+								 XLogRecPtr commit_lsn,
+								 XLogRecPtr prepare_end_lsn,
+								 TimestampTz prepare_time)
 {
 	uint8		flags = 0;
 
@@ -222,8 +224,10 @@ logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
 	pq_sendbyte(out, flags);
 
 	/* send fields */
+	pq_sendint64(out, prepare_end_lsn);
 	pq_sendint64(out, commit_lsn);
 	pq_sendint64(out, txn->end_lsn);
+	pq_sendint64(out, prepare_time);
 	pq_sendint64(out, txn->xact_time.commit_time);
 	pq_sendint32(out, txn->xid);
 
@@ -244,12 +248,16 @@ logicalrep_read_commit_prepared(StringInfo in, LogicalRepCommitPreparedTxnData *
 		elog(ERROR, "unrecognized flags %u in commit prepared message", flags);
 
 	/* read fields */
+	prepare_data->prepare_end_lsn = pq_getmsgint64(in);
+	if (prepare_data->prepare_end_lsn == InvalidXLogRecPtr)
+		elog(ERROR, "prepare_end_lsn is not set in commit prepared message");
 	prepare_data->commit_lsn = pq_getmsgint64(in);
 	if (prepare_data->commit_lsn == InvalidXLogRecPtr)
 		elog(ERROR, "commit_lsn is not set in commit prepared message");
-	prepare_data->end_lsn = pq_getmsgint64(in);
-	if (prepare_data->end_lsn == InvalidXLogRecPtr)
-		elog(ERROR, "end_lsn is not set in commit prepared message");
+	prepare_data->commit_end_lsn = pq_getmsgint64(in);
+	if (prepare_data->commit_end_lsn == InvalidXLogRecPtr)
+		elog(ERROR, "commit_end_lsn is not set in commit prepared message");
+	prepare_data->prepare_time = pq_getmsgint64(in);
 	prepare_data->commit_time = pq_getmsgint64(in);
 	prepare_data->xid = pq_getmsgint(in, 4);
 
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 7378beb..5a707e2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2794,7 +2794,7 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid,
 	txn->origin_lsn = origin_lsn;
 
 	if (is_commit)
-		rb->commit_prepared(rb, txn, commit_lsn);
+		rb->commit_prepared(rb, txn, commit_lsn, prepare_end_lsn, prepare_time);
 	else
 		rb->rollback_prepared(rb, txn, prepare_end_lsn, prepare_time);
 
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b9a7a7f..63e19bc 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -966,27 +966,39 @@ apply_handle_commit_prepared(StringInfo s)
 	/* Compute GID for two_phase transactions. */
 	TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid,
 						   gid, sizeof(gid));
-
-	/* There is no transaction when COMMIT PREPARED is called */
-	begin_replication_step();
-
 	/*
-	 * Update origin state so we can restart streaming from correct position
-	 * in case of crash.
+	 * It is possible that we haven't received the prepare because
+	 * the transaction did not have any changes relevant to this
+	 * subscription and so was essentially an empty prepare. In this case,
+	 * the walsender is optimized to drop the empty transaction and the
+	 * accompanying prepare. Silently ignore if we don't find the prepared
+	 * transaction.
 	 */
-	replorigin_session_origin_lsn = prepare_data.end_lsn;
-	replorigin_session_origin_timestamp = prepare_data.commit_time;
+	if (LookupGXact(gid, prepare_data.prepare_end_lsn,
+					prepare_data.prepare_time))
+	{
 
-	FinishPreparedTransaction(gid, true);
-	end_replication_step();
-	CommitTransactionCommand();
+		/* There is no transaction when COMMIT PREPARED is called */
+		begin_replication_step();
+
+		/*
+		 * Update origin state so we can restart streaming from correct position
+		 * in case of crash.
+		 */
+		replorigin_session_origin_lsn = prepare_data.commit_end_lsn;
+		replorigin_session_origin_timestamp = prepare_data.commit_time;
+
+		FinishPreparedTransaction(gid, true);
+		end_replication_step();
+		CommitTransactionCommand();
+	}
 	pgstat_report_stat(false);
 
-	store_flush_position(prepare_data.end_lsn);
+	store_flush_position(prepare_data.commit_end_lsn);
 	in_remote_transaction = false;
 
 	/* Process any tables that are being synchronized in parallel. */
-	process_syncing_tables(prepare_data.end_lsn);
+	process_syncing_tables(prepare_data.commit_end_lsn);
 
 	pgstat_report_activity(STATE_IDLE, NULL);
 }
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index e4314af..d82db45 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -56,7 +56,9 @@ static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
 static void pgoutput_prepare_txn(LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn, XLogRecPtr prepare_lsn);
 static void pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx,
-										 ReorderBufferTXN *txn, XLogRecPtr commit_lsn);
+										 ReorderBufferTXN *txn, XLogRecPtr commit_lsn,
+										 XLogRecPtr prepare_end_lsn,
+										 TimestampTz prepare_time);
 static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   ReorderBufferTXN *txn,
 										   XLogRecPtr prepare_end_lsn,
@@ -130,6 +132,11 @@ typedef struct RelationSyncEntry
 	TupleConversionMap *map;
 } RelationSyncEntry;
 
+typedef struct PGOutputTxnData
+{
+	bool sent_begin_txn;    /* flag indicating whether begin has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -406,14 +413,38 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 
 /*
  * BEGIN callback
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *data = MemoryContextAllocZero(ctx->context,
+														sizeof(PGOutputTxnData));
+
+	data->sent_begin_txn = false;
+	txn->output_plugin_private = data;
+}
+
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*data = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(data);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	data->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -428,23 +459,66 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData	*data = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(data);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !data->sent_begin_txn;
+	pfree(data);
+	txn->output_plugin_private = NULL;
 	OutputPluginUpdateProgress(ctx);
 
+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
 	OutputPluginWrite(ctx, true);
 }
 
 /*
- * BEGIN PREPARE callback
+ * BEGIN PREPARE callback.
+ * Don't send BEGIN PREPARE message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN PREPARE and COMMIT PREPARED messages
+ * to subscribers, using bandwidth on something with little/no use
+ * for logical replication.
  */
 static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	/*
+	 * Delegate to assign the begin sent flag as false same as for the
+	 * BEGIN message.
+	 */
+	pgoutput_begin_txn(ctx, txn);
+}
+
+/*
+ * Send BEGIN PREPARE.
+ * This is where the BEGIN PREPARE is actually sent. This is called while
+ * processing the first change of the prepared transaction.
+ */
+static void
+pgoutput_begin_prepare(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData    *data = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(data);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
+	data->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -459,8 +533,21 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData    *data = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(data);
 	OutputPluginUpdateProgress(ctx);
 
+	/*
+	 * If the BEGIN was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the PREPARE message too.
+	 */
+	if (!data->sent_begin_txn)
+	{
+		elog(DEBUG1, "skipping replication of an empty prepared transaction");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
@@ -471,12 +558,34 @@ pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  */
 static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-							 XLogRecPtr commit_lsn)
+							 XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+							 TimestampTz prepare_time)
 {
+	PGOutputTxnData    *data = (PGOutputTxnData *) txn->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/*
+	 * If the BEGIN PREPARE was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the COMMIT PREPARED
+	 * messsage too.
+	 */
+	if (data)
+	{
+		bool skip = !data->sent_begin_txn;
+		pfree(data);
+		txn->output_plugin_private = NULL;
+		if (skip)
+		{
+			elog(DEBUG1,
+				 "skipping replication of COMMIT PREPARED of an empty transaction");
+			return;
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
-	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
+	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn, prepare_end_lsn,
+									 prepare_time);
 	OutputPluginWrite(ctx, true);
 }
 
@@ -489,8 +598,27 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
+	PGOutputTxnData    *data = (PGOutputTxnData *) txn->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+   /*
+    * If the BEGIN PREPARE was not yet sent, then it means there were no
+    * relevant changes encountered, so we can skip the ROLLBACK PREPARED
+    * messsage too.
+    */
+	if (data)
+	{
+		bool skip = !data->sent_begin_txn;
+		pfree(data);
+		txn->output_plugin_private = NULL;
+		if (skip)
+		{
+			elog(DEBUG1,
+				 "skipping replication of ROLLBACK of an empty transaction");
+			return;
+		}
+	}
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
 									   prepare_time);
@@ -639,11 +767,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
 	Relation	ancestor = NULL;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	if (!in_streaming)
+		Assert(txndata);
+
 	if (!is_publishable_relation(relation))
 		return;
 
@@ -677,6 +810,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/*
+	 * output BEGIN / BEGIN PREPARE if we haven't yet,
+     * while streaming no need to send BEGIN / BEGIN PREPARE.
+	 */
+	if (!in_streaming && !txndata->sent_begin_txn)
+	{
+		if (rbtxn_prepared(txn))
+			pgoutput_begin_prepare(ctx, txn);
+		else
+			pgoutput_begin(ctx, txn);
+	}
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -779,6 +924,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -786,6 +932,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	Oid		   *relids;
 	TransactionId xid = InvalidTransactionId;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	if (!in_streaming)
+		Assert(txndata);
+
 	/* Remember the xid for the change in streaming mode. See pgoutput_change. */
 	if (in_streaming)
 		xid = change->txn->xid;
@@ -822,6 +972,18 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/*
+		 * output BEGIN / BEGIN PREPARE if we haven't yet,
+		 * while streaming no need to send BEGIN / BEGIN PREPARE.
+		 */
+		if (!in_streaming && !txndata->sent_begin_txn)
+		{
+			if (rbtxn_prepared(txn))
+				pgoutput_begin_prepare(ctx, txn);
+			else
+				pgoutput_begin(ctx, txn);
+		}
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -854,6 +1016,24 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		Assert(txndata);
+		if (!txndata->sent_begin_txn)
+		{
+			if (rbtxn_prepared(txn))
+				pgoutput_begin_prepare(ctx, txn);
+			else
+				pgoutput_begin(ctx, txn);
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 63de90d..0be0a07 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -148,8 +148,10 @@ typedef struct LogicalRepPreparedTxnData
  */
 typedef struct LogicalRepCommitPreparedTxnData
 {
+	XLogRecPtr	prepare_end_lsn;
 	XLogRecPtr	commit_lsn;
-	XLogRecPtr	end_lsn;
+	XLogRecPtr	commit_end_lsn;
+	TimestampTz prepare_time;
 	TimestampTz commit_time;
 	TransactionId xid;
 	char		gid[GIDSIZE];
@@ -188,7 +190,9 @@ extern void logicalrep_write_prepare(StringInfo out, ReorderBufferTXN *txn,
 extern void logicalrep_read_prepare(StringInfo in,
 									LogicalRepPreparedTxnData *prepare_data);
 extern void logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
-											 XLogRecPtr commit_lsn);
+											 XLogRecPtr commit_lsn,
+											 XLogRecPtr prepare_end_lsn,
+											 TimestampTz prepare_time);
 extern void logicalrep_read_commit_prepared(StringInfo in,
 											LogicalRepCommitPreparedTxnData *prepare_data);
 extern void logicalrep_write_rollback_prepared(StringInfo out, ReorderBufferTXN *txn,
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 810495e..0d28306 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -128,7 +128,9 @@ typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
  */
 typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
 											   ReorderBufferTXN *txn,
-											   XLogRecPtr commit_lsn);
+											   XLogRecPtr commit_lsn,
+											   XLogRecPtr prepare_end_lsn,
+											   TimestampTz prepare_time);
 
 /*
  * Called for ROLLBACK PREPARED.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b40ff7..11e2e1e 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -442,7 +442,9 @@ typedef void (*ReorderBufferPrepareCB) (ReorderBuffer *rb,
 /* commit prepared callback signature */
 typedef void (*ReorderBufferCommitPreparedCB) (ReorderBuffer *rb,
 											   ReorderBufferTXN *txn,
-											   XLogRecPtr commit_lsn);
+											   XLogRecPtr commit_lsn,
+											   XLogRecPtr prepare_end_lsn,
+											   TimestampTz prepare_time);
 
 /* rollback  prepared callback signature */
 typedef void (*ReorderBufferRollbackPreparedCB) (ReorderBuffer *rb,
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index 0e218e0..3d246be 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/test/subscription/t/021_twophase.pl b/src/test/subscription/t/021_twophase.pl
index c6ada92..b954630 100644
--- a/src/test/subscription/t/021_twophase.pl
+++ b/src/test/subscription/t/021_twophase.pl
@@ -6,7 +6,7 @@ use strict;
 use warnings;
 use PostgresNode;
 use TestLib;
-use Test::More tests => 24;
+use Test::More tests => 25;
 
 ###############################
 # Setup
@@ -318,10 +318,9 @@ $node_publisher->safe_psql('postgres', "
 
 $node_publisher->wait_for_catchup($appname_copy);
 
-# Check that the transaction has been prepared on the subscriber, there will be 2
-# prepared transactions for the 2 subscriptions.
+# Check that the transaction has been prepared on the subscriber
 $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(2), 'transaction is prepared on subscriber');
+is($result, qq(1), 'transaction is prepared on subscriber');
 
 # Now commit the insert and verify that it IS replicated
 $node_publisher->safe_psql('postgres', "COMMIT PREPARED 'mygid';");
@@ -337,6 +336,45 @@ is($result, qq(2), 'replicated data in subscriber table');
 $node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub_copy;");
 $node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_copy;");
 
+##############################
+# Test empty prepares
+##############################
+
+# create a table that is not part of the publication
+$node_publisher->safe_psql('postgres',
+   "CREATE TABLE tab_nopub (a int PRIMARY KEY)");
+
+# disable the subscription so that we can peek at the slot
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub DISABLE");
+
+# wait for the replication slot to become inactive in the publisher
+$node_publisher->poll_query_until('postgres',
+   "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'tap_sub' AND active='f'", 1);
+
+# create a transaction with no changes relevant to the slot
+$node_publisher->safe_psql('postgres', "
+   BEGIN;
+   INSERT INTO tab_nopub SELECT generate_series(1,10);
+   PREPARE TRANSACTION 'empty_transaction';
+   COMMIT PREPARED 'empty_transaction';");
+
+# peek at the contents of the slot
+$result = $node_publisher->safe_psql(
+   'postgres', qq(
+       SELECT get_byte(data, 0)
+       FROM pg_logical_slot_get_binary_changes('tap_sub', NULL, NULL,
+           'proto_version', '3',
+           'publication_names', 'tap_pub')
+));
+
+# the empty transaction should be skipped
+is($result, qq(),
+   'empty transaction dropped on slot'
+);
+
+# enable the subscription to test cleanup
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub ENABLE");
+
 ###############################
 # check all the cleanup
 ###############################
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 37cf4b2..75639ab 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1606,6 +1606,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#39Ajin Cherian
itsajin@gmail.com
In reply to: osumi.takamichi@fujitsu.com (#36)
Re: logical replication empty transactions

On Thu, Jul 15, 2021 at 3:50 PM osumi.takamichi@fujitsu.com
<osumi.takamichi@fujitsu.com> wrote:

I started to test this patch but will give you some really minor quick feedbacks.

(1) pg_logical_slot_get_binary_changes() params.

Technically, looks better to have proto_version 3 & two_phase option for the function
to test empty prepare ? I felt proto_version 1 doesn't support 2PC.
[1] says "The following messages (Begin Prepare, Prepare, Commit Prepared, Rollback Prepared)
are available since protocol version 3." Then, if the test wants to skip empty *prepares*,
I suggest to update the proto_version and set two_phase 'on'.

Updated accordingly.

(2) The following sentences may start with a lowercase letter.
There are other similar codes for this.

+ elog(DEBUG1, "Skipping replication of an empty transaction");

Fixed this.

I've addressed these comments in version 8 of the patch.

regards,
Ajin Cherian
Fujitsu Australia

#40Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#39)
Re: logical replication empty transactions

Hi Ajin.

I have reviewed the v8 patch and my feedback comments are below:

//////////

1. Apply v8 gave multiple whitespace warnings.

------

2. Commit comment - wording

If (when processing a COMMIT / PREPARE message) we find there had been
no other change for that transaction, then do not send the COMMIT /
PREPARE message. This means that pgoutput will skip BEGIN / COMMIT
or BEGIN PREPARE / PREPARE messages for transactions that are empty.

=>

Shouldn't this also mention some other messages that may be skipped?
- COMMIT PREPARED
- ROLLBACK PREPARED

------

3. doc/src/sgml/logicaldecoding.sgml - wording

@@ -884,11 +884,20 @@ typedef void (*LogicalDecodePrepareCB) (struct
LogicalDecodingContext *ctx,
       The required <function>commit_prepared_cb</function> callback is called
       whenever a transaction <command>COMMIT PREPARED</command> has
been decoded.
       The <parameter>gid</parameter> field, which is part of the
-      <parameter>txn</parameter> parameter, can be used in this callback.
+      <parameter>txn</parameter> parameter, can be used in this callback. The
+      parameters <parameter>prepare_end_lsn</parameter> and
+      <parameter>prepare_time</parameter> can be used to check if the plugin
+      has received this <command>PREPARE TRANSACTION</command> command or not.
+      If yes, it can commit the transaction, otherwise, it can skip the commit.
+      The <parameter>gid</parameter> alone is not sufficient to determine this
+      because the downstream may already have a prepared transaction with the
+      same identifier.

=>

Typo: Should that say "downstream node" instead of just "downstream" ?

------

4. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin_txn
callback comment

@@ -406,14 +413,38 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,

 /*
  * BEGIN callback
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on

=>

Typo: "BEGIN callback" --> "BEGIN callback." (with the period).

And, I think maybe it will be better if it has a separating blank line too.

e.g.

/*
* BEGIN callback.
*
* Don't send BEGIN ....

(NOTE: this review comment applies to other callback function comments
too, so please hunt them all down)

------

5. src/backend/replication/pgoutput/pgoutput.c - data / txndata

 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+ PGOutputTxnData    *data = MemoryContextAllocZero(ctx->context,
+ sizeof(PGOutputTxnData));

=>

There is some inconsistent naming of the local variable in the patch.
Sometimes it is called "data"; Sometimes it is called "txdata" etc. It
would be better to just stick with the same variable name everywhere.

(NOTE: this comment applies to several places in this patch)

------

6. src/backend/replication/pgoutput/pgoutput.c - Strange way to use Assert

+ /* If not streaming, should have setup txndata as part of
BEGIN/BEGIN PREPARE */
+ if (!in_streaming)
+ Assert(txndata);
+

=>

This style of Assert code seemed strange to me. In production mode
isn't that going to evaluate to some condition with a ((void) true)
body? IMO it might be better to just include the streaming check as
part of the Assert. For example:

BEFORE
if (!in_streaming)
Assert(txndata);

AFTER
Assert(in_streaming || txndata);

(NOTE: This same review comment applies in at least 3 places in this
patch, so please hunt them all down)

------

7. src/backend/replication/pgoutput/pgoutput.c - comment wording

@@ -677,6 +810,18 @@ pgoutput_change(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
Assert(false);
}

+ /*
+ * output BEGIN / BEGIN PREPARE if we haven't yet,
+     * while streaming no need to send BEGIN / BEGIN PREPARE.
+ */
+ if (!in_streaming && !txndata->sent_begin_txn)

=>

English not really that comment is. The comment should also start with
uppercase.

(NOTE: This same comment was in couple of places in the patch)

------
Kind Regards,
Peter Smith.
Fujitsu Australia

#41Ajin Cherian
itsajin@gmail.com
In reply to: Peter Smith (#40)
1 attachment(s)
Re: logical replication empty transactions

On Thu, Jul 22, 2021 at 6:11 PM Peter Smith <smithpb2250@gmail.com> wrote:

Hi Ajin.

I have reviewed the v8 patch and my feedback comments are below:

//////////

1. Apply v8 gave multiple whitespace warnings.

------

2. Commit comment - wording

If (when processing a COMMIT / PREPARE message) we find there had been
no other change for that transaction, then do not send the COMMIT /
PREPARE message. This means that pgoutput will skip BEGIN / COMMIT
or BEGIN PREPARE / PREPARE messages for transactions that are empty.

=>

Shouldn't this also mention some other messages that may be skipped?
- COMMIT PREPARED
- ROLLBACK PREPARED

Updated.

------

3. doc/src/sgml/logicaldecoding.sgml - wording

@@ -884,11 +884,20 @@ typedef void (*LogicalDecodePrepareCB) (struct
LogicalDecodingContext *ctx,
The required <function>commit_prepared_cb</function> callback is called
whenever a transaction <command>COMMIT PREPARED</command> has
been decoded.
The <parameter>gid</parameter> field, which is part of the
-      <parameter>txn</parameter> parameter, can be used in this callback.
+      <parameter>txn</parameter> parameter, can be used in this callback. The
+      parameters <parameter>prepare_end_lsn</parameter> and
+      <parameter>prepare_time</parameter> can be used to check if the plugin
+      has received this <command>PREPARE TRANSACTION</command> command or not.
+      If yes, it can commit the transaction, otherwise, it can skip the commit.
+      The <parameter>gid</parameter> alone is not sufficient to determine this
+      because the downstream may already have a prepared transaction with the
+      same identifier.

=>

Typo: Should that say "downstream node" instead of just "downstream" ?

------

Updated.

4. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin_txn
callback comment

@@ -406,14 +413,38 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,

/*
* BEGIN callback
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on

=>

Typo: "BEGIN callback" --> "BEGIN callback." (with the period).

And, I think maybe it will be better if it has a separating blank line too.

e.g.

/*
* BEGIN callback.
*
* Don't send BEGIN ....

(NOTE: this review comment applies to other callback function comments
too, so please hunt them all down)

------

Updated.

5. src/backend/replication/pgoutput/pgoutput.c - data / txndata

static void
pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
{
+ PGOutputTxnData    *data = MemoryContextAllocZero(ctx->context,
+ sizeof(PGOutputTxnData));

=>

There is some inconsistent naming of the local variable in the patch.
Sometimes it is called "data"; Sometimes it is called "txdata" etc. It
would be better to just stick with the same variable name everywhere.

(NOTE: this comment applies to several places in this patch)

------

I've changed all occurance of PGOutputTxnData to txndata. Note that
there is another structure PGOutputData which still uses the name
data.

6. src/backend/replication/pgoutput/pgoutput.c - Strange way to use Assert

+ /* If not streaming, should have setup txndata as part of
BEGIN/BEGIN PREPARE */
+ if (!in_streaming)
+ Assert(txndata);
+

=>

This style of Assert code seemed strange to me. In production mode
isn't that going to evaluate to some condition with a ((void) true)
body? IMO it might be better to just include the streaming check as
part of the Assert. For example:

BEFORE
if (!in_streaming)
Assert(txndata);

AFTER
Assert(in_streaming || txndata);

(NOTE: This same review comment applies in at least 3 places in this
patch, so please hunt them all down)

Updated.

------

7. src/backend/replication/pgoutput/pgoutput.c - comment wording

@@ -677,6 +810,18 @@ pgoutput_change(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
Assert(false);
}

+ /*
+ * output BEGIN / BEGIN PREPARE if we haven't yet,
+     * while streaming no need to send BEGIN / BEGIN PREPARE.
+ */
+ if (!in_streaming && !txndata->sent_begin_txn)

=>

English not really that comment is. The comment should also start with
uppercase.

(NOTE: This same comment was in couple of places in the patch)

Updated.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v9-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v9-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From a9ae97394096b1de31cebd6de0b504619e5a7b34 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Wed, 21 Jul 2021 06:29:57 -0400
Subject: [PATCH v9] Skip empty transactions for logical replication.

The current logical replication behaviour is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN / BEGIN
PREPARE messages until the first change is encountered.
If (when processing a COMMIT / PREPARE message) we find there had been
no other change for that transaction, then do not send the COMMIT /
PREPARE message. This means that pgoutput will skip BEGIN / COMMIT
or BEGIN PREPARE / PREPARE  messages for transactions that are empty.
pgoutput will also skip COMMIT PREPARED and ROLLBACK PREPARED messages
for transactions which were skipped.
Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 contrib/test_decoding/test_decoding.c           |   7 +-
 doc/src/sgml/logicaldecoding.sgml               |  13 +-
 doc/src/sgml/protocol.sgml                      |  15 ++
 src/backend/replication/logical/logical.c       |   9 +-
 src/backend/replication/logical/proto.c         |  16 +-
 src/backend/replication/logical/reorderbuffer.c |   2 +-
 src/backend/replication/logical/worker.c        |  38 +++--
 src/backend/replication/pgoutput/pgoutput.c     | 189 +++++++++++++++++++++++-
 src/include/replication/logicalproto.h          |   8 +-
 src/include/replication/output_plugin.h         |   4 +-
 src/include/replication/reorderbuffer.h         |   4 +-
 src/test/subscription/t/020_messages.pl         |   5 +-
 src/test/subscription/t/021_twophase.pl         |  46 +++++-
 src/tools/pgindent/typedefs.list                |   1 +
 14 files changed, 316 insertions(+), 41 deletions(-)

diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index e5cd84e..408dbfc 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -86,7 +86,9 @@ static void pg_decode_prepare_txn(LogicalDecodingContext *ctx,
 								  XLogRecPtr prepare_lsn);
 static void pg_decode_commit_prepared_txn(LogicalDecodingContext *ctx,
 										  ReorderBufferTXN *txn,
-										  XLogRecPtr commit_lsn);
+										  XLogRecPtr commit_lsn,
+										  XLogRecPtr prepare_end_lsn,
+										  TimestampTz prepare_time);
 static void pg_decode_rollback_prepared_txn(LogicalDecodingContext *ctx,
 											ReorderBufferTXN *txn,
 											XLogRecPtr prepare_end_lsn,
@@ -390,7 +392,8 @@ pg_decode_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 /* COMMIT PREPARED callback */
 static void
 pg_decode_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-							  XLogRecPtr commit_lsn)
+							  XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+							  TimestampTz prepare_time)
 {
 	TestDecodingData *data = ctx->output_plugin_private;
 
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 89b8090..beb09ce 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -884,11 +884,20 @@ typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
       The required <function>commit_prepared_cb</function> callback is called
       whenever a transaction <command>COMMIT PREPARED</command> has been decoded.
       The <parameter>gid</parameter> field, which is part of the
-      <parameter>txn</parameter> parameter, can be used in this callback.
+      <parameter>txn</parameter> parameter, can be used in this callback. The
+      parameters <parameter>prepare_end_lsn</parameter> and
+      <parameter>prepare_time</parameter> can be used to check if the plugin
+      has received this <command>PREPARE TRANSACTION</command> command or not.
+      If yes, it can commit the transaction, otherwise, it can skip the commit.
+      The <parameter>gid</parameter> alone is not sufficient to determine this
+      because the downstream node may already have a prepared transaction with the
+      same identifier.
 <programlisting>
 typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
                                                ReorderBufferTXN *txn,
-                                               XLogRecPtr commit_lsn);
+                                               XLogRecPtr commit_lsn,
+                                               XLogRecPtr prepare_end_lsn,
+                                               TimestampTz prepare_time);
 </programlisting>
      </para>
     </sect3>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index e8cb78f..5e68dfb 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7550,6 +7550,13 @@ are available since protocol version 3.
 <varlistentry>
 <term>Int64</term>
 <listitem><para>
+                The end LSN of the prepare.
+</para></listitem>
+</varlistentry>
+
+<varlistentry>
+<term>Int64</term>
+<listitem><para>
                 The LSN of the commit prepared.
 </para></listitem>
 </varlistentry>
@@ -7564,6 +7571,14 @@ are available since protocol version 3.
 <varlistentry>
 <term>Int64</term>
 <listitem><para>
+                Prepare timestamp of the transaction. The value is in number
+                of microseconds since PostgreSQL epoch (2000-01-01).
+</para></listitem>
+</varlistentry>
+
+<varlistentry>
+<term>Int64</term>
+<listitem><para>
                 Commit timestamp of the transaction. The value is in number
                 of microseconds since PostgreSQL epoch (2000-01-01).
 </para></listitem>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index d61ef4c..67c762a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -63,7 +63,8 @@ static void begin_prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn
 static void prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 							   XLogRecPtr prepare_lsn);
 static void commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
-									   XLogRecPtr commit_lsn);
+									   XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+									   TimestampTz prepare_time);
 static void rollback_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 										 XLogRecPtr prepare_end_lsn, TimestampTz prepare_time);
 static void change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
@@ -936,7 +937,8 @@ prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 
 static void
 commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
-						   XLogRecPtr commit_lsn)
+						   XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+						   TimestampTz prepare_time)
 {
 	LogicalDecodingContext *ctx = cache->private_data;
 	LogicalErrorCallbackState state;
@@ -972,7 +974,8 @@ commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 						"commit_prepared_cb")));
 
 	/* do the actual work: call callback */
-	ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn);
+	ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn, prepare_end_lsn,
+									  prepare_time);
 
 	/* Pop the error context stack */
 	error_context_stack = errcallback.previous;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index a245252..47a7489 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -206,7 +206,9 @@ logicalrep_read_prepare(StringInfo in, LogicalRepPreparedTxnData *prepare_data)
  */
 void
 logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
-								 XLogRecPtr commit_lsn)
+								 XLogRecPtr commit_lsn,
+								 XLogRecPtr prepare_end_lsn,
+								 TimestampTz prepare_time)
 {
 	uint8		flags = 0;
 
@@ -222,8 +224,10 @@ logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
 	pq_sendbyte(out, flags);
 
 	/* send fields */
+	pq_sendint64(out, prepare_end_lsn);
 	pq_sendint64(out, commit_lsn);
 	pq_sendint64(out, txn->end_lsn);
+	pq_sendint64(out, prepare_time);
 	pq_sendint64(out, txn->xact_time.commit_time);
 	pq_sendint32(out, txn->xid);
 
@@ -244,12 +248,16 @@ logicalrep_read_commit_prepared(StringInfo in, LogicalRepCommitPreparedTxnData *
 		elog(ERROR, "unrecognized flags %u in commit prepared message", flags);
 
 	/* read fields */
+	prepare_data->prepare_end_lsn = pq_getmsgint64(in);
+	if (prepare_data->prepare_end_lsn == InvalidXLogRecPtr)
+		elog(ERROR, "prepare_end_lsn is not set in commit prepared message");
 	prepare_data->commit_lsn = pq_getmsgint64(in);
 	if (prepare_data->commit_lsn == InvalidXLogRecPtr)
 		elog(ERROR, "commit_lsn is not set in commit prepared message");
-	prepare_data->end_lsn = pq_getmsgint64(in);
-	if (prepare_data->end_lsn == InvalidXLogRecPtr)
-		elog(ERROR, "end_lsn is not set in commit prepared message");
+	prepare_data->commit_end_lsn = pq_getmsgint64(in);
+	if (prepare_data->commit_end_lsn == InvalidXLogRecPtr)
+		elog(ERROR, "commit_end_lsn is not set in commit prepared message");
+	prepare_data->prepare_time = pq_getmsgint64(in);
 	prepare_data->commit_time = pq_getmsgint64(in);
 	prepare_data->xid = pq_getmsgint(in, 4);
 
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 7378beb..5a707e2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2794,7 +2794,7 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid,
 	txn->origin_lsn = origin_lsn;
 
 	if (is_commit)
-		rb->commit_prepared(rb, txn, commit_lsn);
+		rb->commit_prepared(rb, txn, commit_lsn, prepare_end_lsn, prepare_time);
 	else
 		rb->rollback_prepared(rb, txn, prepare_end_lsn, prepare_time);
 
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b9a7a7f..63e19bc 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -966,27 +966,39 @@ apply_handle_commit_prepared(StringInfo s)
 	/* Compute GID for two_phase transactions. */
 	TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid,
 						   gid, sizeof(gid));
-
-	/* There is no transaction when COMMIT PREPARED is called */
-	begin_replication_step();
-
 	/*
-	 * Update origin state so we can restart streaming from correct position
-	 * in case of crash.
+	 * It is possible that we haven't received the prepare because
+	 * the transaction did not have any changes relevant to this
+	 * subscription and so was essentially an empty prepare. In this case,
+	 * the walsender is optimized to drop the empty transaction and the
+	 * accompanying prepare. Silently ignore if we don't find the prepared
+	 * transaction.
 	 */
-	replorigin_session_origin_lsn = prepare_data.end_lsn;
-	replorigin_session_origin_timestamp = prepare_data.commit_time;
+	if (LookupGXact(gid, prepare_data.prepare_end_lsn,
+					prepare_data.prepare_time))
+	{
 
-	FinishPreparedTransaction(gid, true);
-	end_replication_step();
-	CommitTransactionCommand();
+		/* There is no transaction when COMMIT PREPARED is called */
+		begin_replication_step();
+
+		/*
+		 * Update origin state so we can restart streaming from correct position
+		 * in case of crash.
+		 */
+		replorigin_session_origin_lsn = prepare_data.commit_end_lsn;
+		replorigin_session_origin_timestamp = prepare_data.commit_time;
+
+		FinishPreparedTransaction(gid, true);
+		end_replication_step();
+		CommitTransactionCommand();
+	}
 	pgstat_report_stat(false);
 
-	store_flush_position(prepare_data.end_lsn);
+	store_flush_position(prepare_data.commit_end_lsn);
 	in_remote_transaction = false;
 
 	/* Process any tables that are being synchronized in parallel. */
-	process_syncing_tables(prepare_data.end_lsn);
+	process_syncing_tables(prepare_data.commit_end_lsn);
 
 	pgstat_report_activity(STATE_IDLE, NULL);
 }
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index e4314af..66496b0 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -56,7 +56,9 @@ static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
 static void pgoutput_prepare_txn(LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn, XLogRecPtr prepare_lsn);
 static void pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx,
-										 ReorderBufferTXN *txn, XLogRecPtr commit_lsn);
+										 ReorderBufferTXN *txn, XLogRecPtr commit_lsn,
+										 XLogRecPtr prepare_end_lsn,
+										 TimestampTz prepare_time);
 static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   ReorderBufferTXN *txn,
 										   XLogRecPtr prepare_end_lsn,
@@ -130,6 +132,11 @@ typedef struct RelationSyncEntry
 	TupleConversionMap *map;
 } RelationSyncEntry;
 
+typedef struct PGOutputTxnData
+{
+	bool sent_begin_txn;    /* flag indicating whether begin has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -405,15 +412,40 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(txndata);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -428,23 +460,67 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !txndata->sent_begin_txn;
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
 	OutputPluginUpdateProgress(ctx);
 
+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
 	OutputPluginWrite(ctx, true);
 }
 
 /*
- * BEGIN PREPARE callback
+ * BEGIN PREPARE callback.
+ *
+ * Don't send BEGIN PREPARE message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN PREPARE and COMMIT PREPARED messages
+ * to subscribers, using bandwidth on something with little/no use
+ * for logical replication.
  */
 static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	/*
+	 * Delegate to assign the begin sent flag as false same as for the
+	 * BEGIN message.
+	 */
+	pgoutput_begin_txn(ctx, txn);
+}
+
+/*
+ * Send BEGIN PREPARE.
+ * This is where the BEGIN PREPARE is actually sent. This is called while
+ * processing the first change of the prepared transaction.
+ */
+static void
+pgoutput_begin_prepare(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(txndata);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -459,8 +535,21 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
 	OutputPluginUpdateProgress(ctx);
 
+	/*
+	 * If the BEGIN was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the PREPARE message too.
+	 */
+	if (!txndata->sent_begin_txn)
+	{
+		elog(DEBUG1, "skipping replication of an empty prepared transaction");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
@@ -471,12 +560,34 @@ pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  */
 static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-							 XLogRecPtr commit_lsn)
+							 XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+							 TimestampTz prepare_time)
 {
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/*
+	 * If the BEGIN PREPARE was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the COMMIT PREPARED
+	 * messsage too.
+	 */
+	if (txndata)
+	{
+		bool skip = !txndata->sent_begin_txn;
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+		if (skip)
+		{
+			elog(DEBUG1,
+				 "skipping replication of COMMIT PREPARED of an empty transaction");
+			return;
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
-	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
+	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn, prepare_end_lsn,
+									 prepare_time);
 	OutputPluginWrite(ctx, true);
 }
 
@@ -489,8 +600,27 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+   /*
+    * If the BEGIN PREPARE was not yet sent, then it means there were no
+    * relevant changes encountered, so we can skip the ROLLBACK PREPARED
+    * messsage too.
+    */
+	if (txndata)
+	{
+		bool skip = !txndata->sent_begin_txn;
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+		if (skip)
+		{
+			elog(DEBUG1,
+				 "skipping replication of ROLLBACK of an empty transaction");
+			return;
+		}
+	}
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
 									   prepare_time);
@@ -639,11 +769,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
 	Relation	ancestor = NULL;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	if (!is_publishable_relation(relation))
 		return;
 
@@ -677,6 +811,17 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/*
+	 * Output BEGIN / BEGIN PREPARE if we haven't yet, unless streaming.
+	 */
+	if (!in_streaming && !txndata->sent_begin_txn)
+	{
+		if (rbtxn_prepared(txn))
+			pgoutput_begin_prepare(ctx, txn);
+		else
+			pgoutput_begin(ctx, txn);
+	}
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -779,6 +924,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -786,6 +932,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	Oid		   *relids;
 	TransactionId xid = InvalidTransactionId;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	/* Remember the xid for the change in streaming mode. See pgoutput_change. */
 	if (in_streaming)
 		xid = change->txn->xid;
@@ -822,6 +971,18 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/*
+		 * output BEGIN / BEGIN PREPARE if we haven't yet,
+		 * while streaming no need to send BEGIN / BEGIN PREPARE.
+		 */
+		if (!in_streaming && !txndata->sent_begin_txn)
+		{
+			if (rbtxn_prepared(txn))
+				pgoutput_begin_prepare(ctx, txn);
+			else
+				pgoutput_begin(ctx, txn);
+		}
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -854,6 +1015,24 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		Assert(txndata);
+		if (!txndata->sent_begin_txn)
+		{
+			if (rbtxn_prepared(txn))
+				pgoutput_begin_prepare(ctx, txn);
+			else
+				pgoutput_begin(ctx, txn);
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 63de90d..0be0a07 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -148,8 +148,10 @@ typedef struct LogicalRepPreparedTxnData
  */
 typedef struct LogicalRepCommitPreparedTxnData
 {
+	XLogRecPtr	prepare_end_lsn;
 	XLogRecPtr	commit_lsn;
-	XLogRecPtr	end_lsn;
+	XLogRecPtr	commit_end_lsn;
+	TimestampTz prepare_time;
 	TimestampTz commit_time;
 	TransactionId xid;
 	char		gid[GIDSIZE];
@@ -188,7 +190,9 @@ extern void logicalrep_write_prepare(StringInfo out, ReorderBufferTXN *txn,
 extern void logicalrep_read_prepare(StringInfo in,
 									LogicalRepPreparedTxnData *prepare_data);
 extern void logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
-											 XLogRecPtr commit_lsn);
+											 XLogRecPtr commit_lsn,
+											 XLogRecPtr prepare_end_lsn,
+											 TimestampTz prepare_time);
 extern void logicalrep_read_commit_prepared(StringInfo in,
 											LogicalRepCommitPreparedTxnData *prepare_data);
 extern void logicalrep_write_rollback_prepared(StringInfo out, ReorderBufferTXN *txn,
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 810495e..0d28306 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -128,7 +128,9 @@ typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
  */
 typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
 											   ReorderBufferTXN *txn,
-											   XLogRecPtr commit_lsn);
+											   XLogRecPtr commit_lsn,
+											   XLogRecPtr prepare_end_lsn,
+											   TimestampTz prepare_time);
 
 /*
  * Called for ROLLBACK PREPARED.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b40ff7..11e2e1e 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -442,7 +442,9 @@ typedef void (*ReorderBufferPrepareCB) (ReorderBuffer *rb,
 /* commit prepared callback signature */
 typedef void (*ReorderBufferCommitPreparedCB) (ReorderBuffer *rb,
 											   ReorderBufferTXN *txn,
-											   XLogRecPtr commit_lsn);
+											   XLogRecPtr commit_lsn,
+											   XLogRecPtr prepare_end_lsn,
+											   TimestampTz prepare_time);
 
 /* rollback  prepared callback signature */
 typedef void (*ReorderBufferRollbackPreparedCB) (ReorderBuffer *rb,
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index 0e218e0..3d246be 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/test/subscription/t/021_twophase.pl b/src/test/subscription/t/021_twophase.pl
index c6ada92..b954630 100644
--- a/src/test/subscription/t/021_twophase.pl
+++ b/src/test/subscription/t/021_twophase.pl
@@ -6,7 +6,7 @@ use strict;
 use warnings;
 use PostgresNode;
 use TestLib;
-use Test::More tests => 24;
+use Test::More tests => 25;
 
 ###############################
 # Setup
@@ -318,10 +318,9 @@ $node_publisher->safe_psql('postgres', "
 
 $node_publisher->wait_for_catchup($appname_copy);
 
-# Check that the transaction has been prepared on the subscriber, there will be 2
-# prepared transactions for the 2 subscriptions.
+# Check that the transaction has been prepared on the subscriber
 $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(2), 'transaction is prepared on subscriber');
+is($result, qq(1), 'transaction is prepared on subscriber');
 
 # Now commit the insert and verify that it IS replicated
 $node_publisher->safe_psql('postgres', "COMMIT PREPARED 'mygid';");
@@ -337,6 +336,45 @@ is($result, qq(2), 'replicated data in subscriber table');
 $node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub_copy;");
 $node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_copy;");
 
+##############################
+# Test empty prepares
+##############################
+
+# create a table that is not part of the publication
+$node_publisher->safe_psql('postgres',
+   "CREATE TABLE tab_nopub (a int PRIMARY KEY)");
+
+# disable the subscription so that we can peek at the slot
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub DISABLE");
+
+# wait for the replication slot to become inactive in the publisher
+$node_publisher->poll_query_until('postgres',
+   "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'tap_sub' AND active='f'", 1);
+
+# create a transaction with no changes relevant to the slot
+$node_publisher->safe_psql('postgres', "
+   BEGIN;
+   INSERT INTO tab_nopub SELECT generate_series(1,10);
+   PREPARE TRANSACTION 'empty_transaction';
+   COMMIT PREPARED 'empty_transaction';");
+
+# peek at the contents of the slot
+$result = $node_publisher->safe_psql(
+   'postgres', qq(
+       SELECT get_byte(data, 0)
+       FROM pg_logical_slot_get_binary_changes('tap_sub', NULL, NULL,
+           'proto_version', '3',
+           'publication_names', 'tap_pub')
+));
+
+# the empty transaction should be skipped
+is($result, qq(),
+   'empty transaction dropped on slot'
+);
+
+# enable the subscription to test cleanup
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub ENABLE");
+
 ###############################
 # check all the cleanup
 ###############################
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 37cf4b2..75639ab 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1606,6 +1606,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#42Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#41)
Re: logical replication empty transactions

I have reviewed the v9 patch and my feedback comments are below:

//////////

1. Apply v9 gave multiple whitespace warnings

$ git apply v9-0001-Skip-empty-transactions-for-logical-replication.patch
v9-0001-Skip-empty-transactions-for-logical-replication.patch:479:
indent with spaces.
* If the BEGIN PREPARE was not yet sent, then it means there were no
v9-0001-Skip-empty-transactions-for-logical-replication.patch:480:
indent with spaces.
* relevant changes encountered, so we can skip the ROLLBACK PREPARED
v9-0001-Skip-empty-transactions-for-logical-replication.patch:481:
indent with spaces.
* messsage too.
v9-0001-Skip-empty-transactions-for-logical-replication.patch:482:
indent with spaces.
*/
warning: 4 lines add whitespace errors.

------

2. Commit comment - wording

pgoutput will also skip COMMIT PREPARED and ROLLBACK PREPARED messages
for transactions which were skipped.

=>

Is that correct? Or did you mean to say:

AFTER
pgoutput will also skip COMMIT PREPARED and ROLLBACK PREPARED messages
for transactions that are empty.

------

3. src/backend/replication/pgoutput/pgoutput.c - typo

+ /*
+ * If the BEGIN PREPARE was not yet sent, then it means there were no
+ * relevant changes encountered, so we can skip the COMMIT PREPARED
+ * messsage too.
+ */

Typo: "messsage" --> "message"

(NOTE this same typo is in 2 places)

------

Kind Regards,
Peter Smith.
Fujitsu Australia

#43Greg Nancarrow
gregn4422@gmail.com
In reply to: Ajin Cherian (#41)
Re: logical replication empty transactions

On Thu, Jul 22, 2021 at 11:37 PM Ajin Cherian <itsajin@gmail.com> wrote:

I have some minor comments on the v9 patch:

(1) Several whitespace warnings on patch application

(2) Suggested patch comment change:

BEFORE:
The current logical replication behaviour is to send every transaction to
subscriber even though the transaction is empty (because it does not
AFTER:
The current logical replication behaviour is to send every transaction to
subscriber even though the transaction might be empty (because it does not

(3) Comment needed for added struct defn:

typedef struct PGOutputTxnData

(4) Improve comment.

Can you add a comma (or add words) in the below sentence, so we know
how to read it?

+ /*
+ * Delegate to assign the begin sent flag as false same as for the
+ * BEGIN message.
+ */

Regards,
Greg Nancarrow
Fujitsu Australia

#44Ajin Cherian
itsajin@gmail.com
In reply to: Greg Nancarrow (#43)
1 attachment(s)
Re: logical replication empty transactions

On Fri, Jul 23, 2021 at 10:26 AM Greg Nancarrow <gregn4422@gmail.com> wrote:

On Thu, Jul 22, 2021 at 11:37 PM Ajin Cherian <itsajin@gmail.com> wrote:

I have some minor comments on the v9 patch:

(1) Several whitespace warnings on patch application

Fixed.

(2) Suggested patch comment change:

BEFORE:
The current logical replication behaviour is to send every transaction to
subscriber even though the transaction is empty (because it does not
AFTER:
The current logical replication behaviour is to send every transaction to
subscriber even though the transaction might be empty (because it does not

Changed accordingly.

(3) Comment needed for added struct defn:

typedef struct PGOutputTxnData

Added.

(4) Improve comment.

Can you add a comma (or add words) in the below sentence, so we know
how to read it?

Updated.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v10-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v10-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From f60263204b86df83b8a62294ba587085f75504b3 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Wed, 21 Jul 2021 06:29:57 -0400
Subject: [PATCH v10] Skip empty transactions for logical replication.

The current logical replication behaviour is to send every transaction to
subscriber even though the transaction might be empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN / BEGIN
PREPARE messages until the first change is encountered.
If (when processing a COMMIT / PREPARE message) we find there had been
no other change for that transaction, then do not send the COMMIT /
PREPARE message. This means that pgoutput will skip BEGIN / COMMIT
or BEGIN PREPARE / PREPARE  messages for transactions that are empty.
pgoutput will also skip COMMIT PREPARED and ROLLBACK PREPARED messages
for transactions that are empty.
Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 contrib/test_decoding/test_decoding.c           |   7 +-
 doc/src/sgml/logicaldecoding.sgml               |  13 +-
 doc/src/sgml/protocol.sgml                      |  15 ++
 src/backend/replication/logical/logical.c       |   9 +-
 src/backend/replication/logical/proto.c         |  16 +-
 src/backend/replication/logical/reorderbuffer.c |   2 +-
 src/backend/replication/logical/worker.c        |  38 +++--
 src/backend/replication/pgoutput/pgoutput.c     | 195 +++++++++++++++++++++++-
 src/include/replication/logicalproto.h          |   8 +-
 src/include/replication/output_plugin.h         |   4 +-
 src/include/replication/reorderbuffer.h         |   4 +-
 src/test/subscription/t/020_messages.pl         |   5 +-
 src/test/subscription/t/021_twophase.pl         |  46 +++++-
 src/tools/pgindent/typedefs.list                |   1 +
 14 files changed, 322 insertions(+), 41 deletions(-)

diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index e5cd84e..408dbfc 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -86,7 +86,9 @@ static void pg_decode_prepare_txn(LogicalDecodingContext *ctx,
 								  XLogRecPtr prepare_lsn);
 static void pg_decode_commit_prepared_txn(LogicalDecodingContext *ctx,
 										  ReorderBufferTXN *txn,
-										  XLogRecPtr commit_lsn);
+										  XLogRecPtr commit_lsn,
+										  XLogRecPtr prepare_end_lsn,
+										  TimestampTz prepare_time);
 static void pg_decode_rollback_prepared_txn(LogicalDecodingContext *ctx,
 											ReorderBufferTXN *txn,
 											XLogRecPtr prepare_end_lsn,
@@ -390,7 +392,8 @@ pg_decode_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 /* COMMIT PREPARED callback */
 static void
 pg_decode_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-							  XLogRecPtr commit_lsn)
+							  XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+							  TimestampTz prepare_time)
 {
 	TestDecodingData *data = ctx->output_plugin_private;

diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 89b8090..beb09ce 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -884,11 +884,20 @@ typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
       The required <function>commit_prepared_cb</function> callback is called
       whenever a transaction <command>COMMIT PREPARED</command> has been decoded.
       The <parameter>gid</parameter> field, which is part of the
-      <parameter>txn</parameter> parameter, can be used in this callback.
+      <parameter>txn</parameter> parameter, can be used in this callback. The
+      parameters <parameter>prepare_end_lsn</parameter> and
+      <parameter>prepare_time</parameter> can be used to check if the plugin
+      has received this <command>PREPARE TRANSACTION</command> command or not.
+      If yes, it can commit the transaction, otherwise, it can skip the commit.
+      The <parameter>gid</parameter> alone is not sufficient to determine this
+      because the downstream node may already have a prepared transaction with the
+      same identifier.
 <programlisting>
 typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
                                                ReorderBufferTXN *txn,
-                                               XLogRecPtr commit_lsn);
+                                               XLogRecPtr commit_lsn,
+                                               XLogRecPtr prepare_end_lsn,
+                                               TimestampTz prepare_time);
 </programlisting>
      </para>
     </sect3>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index e8cb78f..5e68dfb 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7550,6 +7550,13 @@ are available since protocol version 3.
 <varlistentry>
 <term>Int64</term>
 <listitem><para>
+                The end LSN of the prepare.
+</para></listitem>
+</varlistentry>
+
+<varlistentry>
+<term>Int64</term>
+<listitem><para>
                 The LSN of the commit prepared.
 </para></listitem>
 </varlistentry>
@@ -7564,6 +7571,14 @@ are available since protocol version 3.
 <varlistentry>
 <term>Int64</term>
 <listitem><para>
+                Prepare timestamp of the transaction. The value is in number
+                of microseconds since PostgreSQL epoch (2000-01-01).
+</para></listitem>
+</varlistentry>
+
+<varlistentry>
+<term>Int64</term>
+<listitem><para>
                 Commit timestamp of the transaction. The value is in number
                 of microseconds since PostgreSQL epoch (2000-01-01).
 </para></listitem>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index d61ef4c..67c762a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -63,7 +63,8 @@ static void begin_prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn
 static void prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 							   XLogRecPtr prepare_lsn);
 static void commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
-									   XLogRecPtr commit_lsn);
+									   XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+									   TimestampTz prepare_time);
 static void rollback_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 										 XLogRecPtr prepare_end_lsn, TimestampTz prepare_time);
 static void change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
@@ -936,7 +937,8 @@ prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,

 static void
 commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
-						   XLogRecPtr commit_lsn)
+						   XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+						   TimestampTz prepare_time)
 {
 	LogicalDecodingContext *ctx = cache->private_data;
 	LogicalErrorCallbackState state;
@@ -972,7 +974,8 @@ commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 						"commit_prepared_cb")));

 	/* do the actual work: call callback */
-	ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn);
+	ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn, prepare_end_lsn,
+									  prepare_time);

 	/* Pop the error context stack */
 	error_context_stack = errcallback.previous;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index a245252..47a7489 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -206,7 +206,9 @@ logicalrep_read_prepare(StringInfo in, LogicalRepPreparedTxnData *prepare_data)
  */
 void
 logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
-								 XLogRecPtr commit_lsn)
+								 XLogRecPtr commit_lsn,
+								 XLogRecPtr prepare_end_lsn,
+								 TimestampTz prepare_time)
 {
 	uint8		flags = 0;

@@ -222,8 +224,10 @@ logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
 	pq_sendbyte(out, flags);

 	/* send fields */
+	pq_sendint64(out, prepare_end_lsn);
 	pq_sendint64(out, commit_lsn);
 	pq_sendint64(out, txn->end_lsn);
+	pq_sendint64(out, prepare_time);
 	pq_sendint64(out, txn->xact_time.commit_time);
 	pq_sendint32(out, txn->xid);

@@ -244,12 +248,16 @@ logicalrep_read_commit_prepared(StringInfo in, LogicalRepCommitPreparedTxnData *
 		elog(ERROR, "unrecognized flags %u in commit prepared message", flags);

 	/* read fields */
+	prepare_data->prepare_end_lsn = pq_getmsgint64(in);
+	if (prepare_data->prepare_end_lsn == InvalidXLogRecPtr)
+		elog(ERROR, "prepare_end_lsn is not set in commit prepared message");
 	prepare_data->commit_lsn = pq_getmsgint64(in);
 	if (prepare_data->commit_lsn == InvalidXLogRecPtr)
 		elog(ERROR, "commit_lsn is not set in commit prepared message");
-	prepare_data->end_lsn = pq_getmsgint64(in);
-	if (prepare_data->end_lsn == InvalidXLogRecPtr)
-		elog(ERROR, "end_lsn is not set in commit prepared message");
+	prepare_data->commit_end_lsn = pq_getmsgint64(in);
+	if (prepare_data->commit_end_lsn == InvalidXLogRecPtr)
+		elog(ERROR, "commit_end_lsn is not set in commit prepared message");
+	prepare_data->prepare_time = pq_getmsgint64(in);
 	prepare_data->commit_time = pq_getmsgint64(in);
 	prepare_data->xid = pq_getmsgint(in, 4);

diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 7378beb..5a707e2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2794,7 +2794,7 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid,
 	txn->origin_lsn = origin_lsn;

 	if (is_commit)
-		rb->commit_prepared(rb, txn, commit_lsn);
+		rb->commit_prepared(rb, txn, commit_lsn, prepare_end_lsn, prepare_time);
 	else
 		rb->rollback_prepared(rb, txn, prepare_end_lsn, prepare_time);

diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b9a7a7f..63e19bc 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -966,27 +966,39 @@ apply_handle_commit_prepared(StringInfo s)
 	/* Compute GID for two_phase transactions. */
 	TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid,
 						   gid, sizeof(gid));
-
-	/* There is no transaction when COMMIT PREPARED is called */
-	begin_replication_step();
-
 	/*
-	 * Update origin state so we can restart streaming from correct position
-	 * in case of crash.
+	 * It is possible that we haven't received the prepare because
+	 * the transaction did not have any changes relevant to this
+	 * subscription and so was essentially an empty prepare. In this case,
+	 * the walsender is optimized to drop the empty transaction and the
+	 * accompanying prepare. Silently ignore if we don't find the prepared
+	 * transaction.
 	 */
-	replorigin_session_origin_lsn = prepare_data.end_lsn;
-	replorigin_session_origin_timestamp = prepare_data.commit_time;
+	if (LookupGXact(gid, prepare_data.prepare_end_lsn,
+					prepare_data.prepare_time))
+	{

-	FinishPreparedTransaction(gid, true);
-	end_replication_step();
-	CommitTransactionCommand();
+		/* There is no transaction when COMMIT PREPARED is called */
+		begin_replication_step();
+
+		/*
+		 * Update origin state so we can restart streaming from correct position
+		 * in case of crash.
+		 */
+		replorigin_session_origin_lsn = prepare_data.commit_end_lsn;
+		replorigin_session_origin_timestamp = prepare_data.commit_time;
+
+		FinishPreparedTransaction(gid, true);
+		end_replication_step();
+		CommitTransactionCommand();
+	}
 	pgstat_report_stat(false);

-	store_flush_position(prepare_data.end_lsn);
+	store_flush_position(prepare_data.commit_end_lsn);
 	in_remote_transaction = false;

 	/* Process any tables that are being synchronized in parallel. */
-	process_syncing_tables(prepare_data.end_lsn);
+	process_syncing_tables(prepare_data.commit_end_lsn);

 	pgstat_report_activity(STATE_IDLE, NULL);
 }
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index e4314af..2cdd9aa 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -56,7 +56,9 @@ static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
 static void pgoutput_prepare_txn(LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn, XLogRecPtr prepare_lsn);
 static void pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx,
-										 ReorderBufferTXN *txn, XLogRecPtr commit_lsn);
+										 ReorderBufferTXN *txn, XLogRecPtr commit_lsn,
+										 XLogRecPtr prepare_end_lsn,
+										 TimestampTz prepare_time);
 static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   ReorderBufferTXN *txn,
 										   XLogRecPtr prepare_end_lsn,
@@ -130,6 +132,17 @@ typedef struct RelationSyncEntry
 	TupleConversionMap *map;
 } RelationSyncEntry;

+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN or BEGIN PREPARE. BEGIN or BEGIN PREPARE
+ * is only sent when the first change in a transaction is processed.
+ * This make it possible to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+	bool sent_begin_txn;    /* flag indicating whether begin has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;

@@ -405,15 +418,40 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }

 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;

+	Assert(txndata);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;

 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -428,23 +466,67 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !txndata->sent_begin_txn;
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
 	OutputPluginUpdateProgress(ctx);

+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
 	OutputPluginWrite(ctx, true);
 }

 /*
- * BEGIN PREPARE callback
+ * BEGIN PREPARE callback.
+ *
+ * Don't send BEGIN PREPARE message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN PREPARE and COMMIT PREPARED messages
+ * to subscribers, using bandwidth on something with little/no use
+ * for logical replication.
  */
 static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	/*
+	 * Delegate to assign the begin sent flag as false, same as for the
+	 * BEGIN message.
+	 */
+	pgoutput_begin_txn(ctx, txn);
+}
+
+/*
+ * Send BEGIN PREPARE.
+ * This is where the BEGIN PREPARE is actually sent. This is called while
+ * processing the first change of the prepared transaction.
+ */
+static void
+pgoutput_begin_prepare(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;

+	Assert(txndata);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
+	txndata->sent_begin_txn = true;

 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -459,8 +541,21 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
 	OutputPluginUpdateProgress(ctx);

+	/*
+	 * If the BEGIN was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the PREPARE message too.
+	 */
+	if (!txndata->sent_begin_txn)
+	{
+		elog(DEBUG1, "skipping replication of an empty prepared transaction");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
@@ -471,12 +566,34 @@ pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  */
 static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-							 XLogRecPtr commit_lsn)
+							 XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+							 TimestampTz prepare_time)
 {
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);

+	/*
+	 * If the BEGIN PREPARE was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the COMMIT PREPARED
+	 * message too.
+	 */
+	if (txndata)
+	{
+		bool skip = !txndata->sent_begin_txn;
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+		if (skip)
+		{
+			elog(DEBUG1,
+				 "skipping replication of COMMIT PREPARED of an empty transaction");
+			return;
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
-	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
+	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn, prepare_end_lsn,
+									 prepare_time);
 	OutputPluginWrite(ctx, true);
 }

@@ -489,8 +606,27 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);

+	/*
+	 * If the BEGIN PREPARE was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the ROLLBACK PREPARED
+	 * message too.
+	 */
+	if (txndata)
+	{
+		bool skip = !txndata->sent_begin_txn;
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+		if (skip)
+		{
+			elog(DEBUG1,
+				 "skipping replication of ROLLBACK of an empty transaction");
+			return;
+		}
+	}
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
 									   prepare_time);
@@ -639,11 +775,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
 	Relation	ancestor = NULL;

+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	if (!is_publishable_relation(relation))
 		return;

@@ -677,6 +817,17 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}

+	/*
+	 * Output BEGIN / BEGIN PREPARE if we haven't yet, unless streaming.
+	 */
+	if (!in_streaming && !txndata->sent_begin_txn)
+	{
+		if (rbtxn_prepared(txn))
+			pgoutput_begin_prepare(ctx, txn);
+		else
+			pgoutput_begin(ctx, txn);
+	}
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);

@@ -779,6 +930,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -786,6 +938,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	Oid		   *relids;
 	TransactionId xid = InvalidTransactionId;

+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	/* Remember the xid for the change in streaming mode. See pgoutput_change. */
 	if (in_streaming)
 		xid = change->txn->xid;
@@ -822,6 +977,18 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,

 	if (nrelids > 0)
 	{
+		/*
+		 * output BEGIN / BEGIN PREPARE if we haven't yet,
+		 * while streaming no need to send BEGIN / BEGIN PREPARE.
+		 */
+		if (!in_streaming && !txndata->sent_begin_txn)
+		{
+			if (rbtxn_prepared(txn))
+				pgoutput_begin_prepare(ctx, txn);
+			else
+				pgoutput_begin(ctx, txn);
+		}
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -854,6 +1021,24 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;

+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		Assert(txndata);
+		if (!txndata->sent_begin_txn)
+		{
+			if (rbtxn_prepared(txn))
+				pgoutput_begin_prepare(ctx, txn);
+			else
+				pgoutput_begin(ctx, txn);
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 63de90d..0be0a07 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -148,8 +148,10 @@ typedef struct LogicalRepPreparedTxnData
  */
 typedef struct LogicalRepCommitPreparedTxnData
 {
+	XLogRecPtr	prepare_end_lsn;
 	XLogRecPtr	commit_lsn;
-	XLogRecPtr	end_lsn;
+	XLogRecPtr	commit_end_lsn;
+	TimestampTz prepare_time;
 	TimestampTz commit_time;
 	TransactionId xid;
 	char		gid[GIDSIZE];
@@ -188,7 +190,9 @@ extern void logicalrep_write_prepare(StringInfo out, ReorderBufferTXN *txn,
 extern void logicalrep_read_prepare(StringInfo in,
 									LogicalRepPreparedTxnData *prepare_data);
 extern void logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
-											 XLogRecPtr commit_lsn);
+											 XLogRecPtr commit_lsn,
+											 XLogRecPtr prepare_end_lsn,
+											 TimestampTz prepare_time);
 extern void logicalrep_read_commit_prepared(StringInfo in,
 											LogicalRepCommitPreparedTxnData *prepare_data);
 extern void logicalrep_write_rollback_prepared(StringInfo out, ReorderBufferTXN *txn,
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 810495e..0d28306 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -128,7 +128,9 @@ typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
  */
 typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
 											   ReorderBufferTXN *txn,
-											   XLogRecPtr commit_lsn);
+											   XLogRecPtr commit_lsn,
+											   XLogRecPtr prepare_end_lsn,
+											   TimestampTz prepare_time);

 /*
  * Called for ROLLBACK PREPARED.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b40ff7..11e2e1e 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -442,7 +442,9 @@ typedef void (*ReorderBufferPrepareCB) (ReorderBuffer *rb,
 /* commit prepared callback signature */
 typedef void (*ReorderBufferCommitPreparedCB) (ReorderBuffer *rb,
 											   ReorderBufferTXN *txn,
-											   XLogRecPtr commit_lsn);
+											   XLogRecPtr commit_lsn,
+											   XLogRecPtr prepare_end_lsn,
+											   TimestampTz prepare_time);

 /* rollback  prepared callback signature */
 typedef void (*ReorderBufferRollbackPreparedCB) (ReorderBuffer *rb,
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index 0e218e0..3d246be 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));

-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );

diff --git a/src/test/subscription/t/021_twophase.pl b/src/test/subscription/t/021_twophase.pl
index c6ada92..b954630 100644
--- a/src/test/subscription/t/021_twophase.pl
+++ b/src/test/subscription/t/021_twophase.pl
@@ -6,7 +6,7 @@ use strict;
 use warnings;
 use PostgresNode;
 use TestLib;
-use Test::More tests => 24;
+use Test::More tests => 25;

 ###############################
 # Setup
@@ -318,10 +318,9 @@ $node_publisher->safe_psql('postgres', "

 $node_publisher->wait_for_catchup($appname_copy);

-# Check that the transaction has been prepared on the subscriber, there will be 2
-# prepared transactions for the 2 subscriptions.
+# Check that the transaction has been prepared on the subscriber
 $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(2), 'transaction is prepared on subscriber');
+is($result, qq(1), 'transaction is prepared on subscriber');

 # Now commit the insert and verify that it IS replicated
 $node_publisher->safe_psql('postgres', "COMMIT PREPARED 'mygid';");
@@ -337,6 +336,45 @@ is($result, qq(2), 'replicated data in subscriber table');
 $node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub_copy;");
 $node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_copy;");

+##############################
+# Test empty prepares
+##############################
+
+# create a table that is not part of the publication
+$node_publisher->safe_psql('postgres',
+   "CREATE TABLE tab_nopub (a int PRIMARY KEY)");
+
+# disable the subscription so that we can peek at the slot
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub DISABLE");
+
+# wait for the replication slot to become inactive in the publisher
+$node_publisher->poll_query_until('postgres',
+   "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'tap_sub' AND active='f'", 1);
+
+# create a transaction with no changes relevant to the slot
+$node_publisher->safe_psql('postgres', "
+   BEGIN;
+   INSERT INTO tab_nopub SELECT generate_series(1,10);
+   PREPARE TRANSACTION 'empty_transaction';
+   COMMIT PREPARED 'empty_transaction';");
+
+# peek at the contents of the slot
+$result = $node_publisher->safe_psql(
+   'postgres', qq(
+       SELECT get_byte(data, 0)
+       FROM pg_logical_slot_get_binary_changes('tap_sub', NULL, NULL,
+           'proto_version', '3',
+           'publication_names', 'tap_pub')
+));
+
+# the empty transaction should be skipped
+is($result, qq(),
+   'empty transaction dropped on slot'
+);
+
+# enable the subscription to test cleanup
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub ENABLE");
+
 ###############################
 # check all the cleanup
 ###############################
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 37cf4b2..75639ab 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1606,6 +1606,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
--
1.8.3.1

#45Ajin Cherian
itsajin@gmail.com
In reply to: Peter Smith (#42)
Re: logical replication empty transactions

On Fri, Jul 23, 2021 at 10:13 AM Peter Smith <smithpb2250@gmail.com> wrote:

I have reviewed the v9 patch and my feedback comments are below:

//////////

1. Apply v9 gave multiple whitespace warnings

Fixed.

------

2. Commit comment - wording

pgoutput will also skip COMMIT PREPARED and ROLLBACK PREPARED messages
for transactions which were skipped.

=>

Is that correct? Or did you mean to say:

AFTER
pgoutput will also skip COMMIT PREPARED and ROLLBACK PREPARED messages
for transactions that are empty.

------

Updated.

3. src/backend/replication/pgoutput/pgoutput.c - typo

+ /*
+ * If the BEGIN PREPARE was not yet sent, then it means there were no
+ * relevant changes encountered, so we can skip the COMMIT PREPARED
+ * messsage too.
+ */

Typo: "messsage" --> "message"

(NOTE this same typo is in 2 places)

Fixed.

I have made these changes in v10 of the patch.

regards,
Ajin Cherian
Fujitsu Australia

#46Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#45)
Re: logical replication empty transactions

I have reviewed the v10 patch.

Apply / build / test was all OK.

Just one review comment:

//////////

1. Typo

@@ -130,6 +132,17 @@ typedef struct RelationSyncEntry
TupleConversionMap *map;
} RelationSyncEntry;

+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN or BEGIN PREPARE. BEGIN or BEGIN PREPARE
+ * is only sent when the first change in a transaction is processed.
+ * This make it possible to skip transactions that are empty.
+ */

=>

typo: "make it possible" --> "makes it possible"

------

Kind Regards,
Peter Smith.
Fujitsu Australia

#47Ajin Cherian
itsajin@gmail.com
In reply to: Peter Smith (#46)
1 attachment(s)
Re: logical replication empty transactions

On Fri, Jul 23, 2021 at 7:38 PM Peter Smith <smithpb2250@gmail.com> wrote:

I have reviewed the v10 patch.

Apply / build / test was all OK.

Just one review comment:

//////////

1. Typo

@@ -130,6 +132,17 @@ typedef struct RelationSyncEntry
TupleConversionMap *map;
} RelationSyncEntry;

+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN or BEGIN PREPARE. BEGIN or BEGIN PREPARE
+ * is only sent when the first change in a transaction is processed.
+ * This make it possible to skip transactions that are empty.
+ */

=>

typo: "make it possible" --> "makes it possible"

fixed.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v11-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v11-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From f60263204b86df83b8a62294ba587085f75504b3 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Wed, 21 Jul 2021 06:29:57 -0400
Subject: [PATCH v10] Skip empty transactions for logical replication.

The current logical replication behaviour is to send every transaction to
subscriber even though the transaction might be empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN / BEGIN
PREPARE messages until the first change is encountered.
If (when processing a COMMIT / PREPARE message) we find there had been
no other change for that transaction, then do not send the COMMIT /
PREPARE message. This means that pgoutput will skip BEGIN / COMMIT
or BEGIN PREPARE / PREPARE  messages for transactions that are empty.
pgoutput will also skip COMMIT PREPARED and ROLLBACK PREPARED messages
for transactions that are empty.
Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 contrib/test_decoding/test_decoding.c           |   7 +-
 doc/src/sgml/logicaldecoding.sgml               |  13 +-
 doc/src/sgml/protocol.sgml                      |  15 ++
 src/backend/replication/logical/logical.c       |   9 +-
 src/backend/replication/logical/proto.c         |  16 +-
 src/backend/replication/logical/reorderbuffer.c |   2 +-
 src/backend/replication/logical/worker.c        |  38 +++--
 src/backend/replication/pgoutput/pgoutput.c     | 195 +++++++++++++++++++++++-
 src/include/replication/logicalproto.h          |   8 +-
 src/include/replication/output_plugin.h         |   4 +-
 src/include/replication/reorderbuffer.h         |   4 +-
 src/test/subscription/t/020_messages.pl         |   5 +-
 src/test/subscription/t/021_twophase.pl         |  46 +++++-
 src/tools/pgindent/typedefs.list                |   1 +
 14 files changed, 322 insertions(+), 41 deletions(-)

diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index e5cd84e..408dbfc 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -86,7 +86,9 @@ static void pg_decode_prepare_txn(LogicalDecodingContext *ctx,
 								  XLogRecPtr prepare_lsn);
 static void pg_decode_commit_prepared_txn(LogicalDecodingContext *ctx,
 										  ReorderBufferTXN *txn,
-										  XLogRecPtr commit_lsn);
+										  XLogRecPtr commit_lsn,
+										  XLogRecPtr prepare_end_lsn,
+										  TimestampTz prepare_time);
 static void pg_decode_rollback_prepared_txn(LogicalDecodingContext *ctx,
 											ReorderBufferTXN *txn,
 											XLogRecPtr prepare_end_lsn,
@@ -390,7 +392,8 @@ pg_decode_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 /* COMMIT PREPARED callback */
 static void
 pg_decode_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-							  XLogRecPtr commit_lsn)
+							  XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+							  TimestampTz prepare_time)
 {
 	TestDecodingData *data = ctx->output_plugin_private;
 
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 89b8090..beb09ce 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -884,11 +884,20 @@ typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
       The required <function>commit_prepared_cb</function> callback is called
       whenever a transaction <command>COMMIT PREPARED</command> has been decoded.
       The <parameter>gid</parameter> field, which is part of the
-      <parameter>txn</parameter> parameter, can be used in this callback.
+      <parameter>txn</parameter> parameter, can be used in this callback. The
+      parameters <parameter>prepare_end_lsn</parameter> and
+      <parameter>prepare_time</parameter> can be used to check if the plugin
+      has received this <command>PREPARE TRANSACTION</command> command or not.
+      If yes, it can commit the transaction, otherwise, it can skip the commit.
+      The <parameter>gid</parameter> alone is not sufficient to determine this
+      because the downstream node may already have a prepared transaction with the
+      same identifier.
 <programlisting>
 typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
                                                ReorderBufferTXN *txn,
-                                               XLogRecPtr commit_lsn);
+                                               XLogRecPtr commit_lsn,
+                                               XLogRecPtr prepare_end_lsn,
+                                               TimestampTz prepare_time);
 </programlisting>
      </para>
     </sect3>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index e8cb78f..5e68dfb 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7550,6 +7550,13 @@ are available since protocol version 3.
 <varlistentry>
 <term>Int64</term>
 <listitem><para>
+                The end LSN of the prepare.
+</para></listitem>
+</varlistentry>
+
+<varlistentry>
+<term>Int64</term>
+<listitem><para>
                 The LSN of the commit prepared.
 </para></listitem>
 </varlistentry>
@@ -7564,6 +7571,14 @@ are available since protocol version 3.
 <varlistentry>
 <term>Int64</term>
 <listitem><para>
+                Prepare timestamp of the transaction. The value is in number
+                of microseconds since PostgreSQL epoch (2000-01-01).
+</para></listitem>
+</varlistentry>
+
+<varlistentry>
+<term>Int64</term>
+<listitem><para>
                 Commit timestamp of the transaction. The value is in number
                 of microseconds since PostgreSQL epoch (2000-01-01).
 </para></listitem>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index d61ef4c..67c762a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -63,7 +63,8 @@ static void begin_prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn
 static void prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 							   XLogRecPtr prepare_lsn);
 static void commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
-									   XLogRecPtr commit_lsn);
+									   XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+									   TimestampTz prepare_time);
 static void rollback_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 										 XLogRecPtr prepare_end_lsn, TimestampTz prepare_time);
 static void change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
@@ -936,7 +937,8 @@ prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 
 static void
 commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
-						   XLogRecPtr commit_lsn)
+						   XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+						   TimestampTz prepare_time)
 {
 	LogicalDecodingContext *ctx = cache->private_data;
 	LogicalErrorCallbackState state;
@@ -972,7 +974,8 @@ commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 						"commit_prepared_cb")));
 
 	/* do the actual work: call callback */
-	ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn);
+	ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn, prepare_end_lsn,
+									  prepare_time);
 
 	/* Pop the error context stack */
 	error_context_stack = errcallback.previous;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index a245252..47a7489 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -206,7 +206,9 @@ logicalrep_read_prepare(StringInfo in, LogicalRepPreparedTxnData *prepare_data)
  */
 void
 logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
-								 XLogRecPtr commit_lsn)
+								 XLogRecPtr commit_lsn,
+								 XLogRecPtr prepare_end_lsn,
+								 TimestampTz prepare_time)
 {
 	uint8		flags = 0;
 
@@ -222,8 +224,10 @@ logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
 	pq_sendbyte(out, flags);
 
 	/* send fields */
+	pq_sendint64(out, prepare_end_lsn);
 	pq_sendint64(out, commit_lsn);
 	pq_sendint64(out, txn->end_lsn);
+	pq_sendint64(out, prepare_time);
 	pq_sendint64(out, txn->xact_time.commit_time);
 	pq_sendint32(out, txn->xid);
 
@@ -244,12 +248,16 @@ logicalrep_read_commit_prepared(StringInfo in, LogicalRepCommitPreparedTxnData *
 		elog(ERROR, "unrecognized flags %u in commit prepared message", flags);
 
 	/* read fields */
+	prepare_data->prepare_end_lsn = pq_getmsgint64(in);
+	if (prepare_data->prepare_end_lsn == InvalidXLogRecPtr)
+		elog(ERROR, "prepare_end_lsn is not set in commit prepared message");
 	prepare_data->commit_lsn = pq_getmsgint64(in);
 	if (prepare_data->commit_lsn == InvalidXLogRecPtr)
 		elog(ERROR, "commit_lsn is not set in commit prepared message");
-	prepare_data->end_lsn = pq_getmsgint64(in);
-	if (prepare_data->end_lsn == InvalidXLogRecPtr)
-		elog(ERROR, "end_lsn is not set in commit prepared message");
+	prepare_data->commit_end_lsn = pq_getmsgint64(in);
+	if (prepare_data->commit_end_lsn == InvalidXLogRecPtr)
+		elog(ERROR, "commit_end_lsn is not set in commit prepared message");
+	prepare_data->prepare_time = pq_getmsgint64(in);
 	prepare_data->commit_time = pq_getmsgint64(in);
 	prepare_data->xid = pq_getmsgint(in, 4);
 
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 7378beb..5a707e2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2794,7 +2794,7 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid,
 	txn->origin_lsn = origin_lsn;
 
 	if (is_commit)
-		rb->commit_prepared(rb, txn, commit_lsn);
+		rb->commit_prepared(rb, txn, commit_lsn, prepare_end_lsn, prepare_time);
 	else
 		rb->rollback_prepared(rb, txn, prepare_end_lsn, prepare_time);
 
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b9a7a7f..63e19bc 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -966,27 +966,39 @@ apply_handle_commit_prepared(StringInfo s)
 	/* Compute GID for two_phase transactions. */
 	TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid,
 						   gid, sizeof(gid));
-
-	/* There is no transaction when COMMIT PREPARED is called */
-	begin_replication_step();
-
 	/*
-	 * Update origin state so we can restart streaming from correct position
-	 * in case of crash.
+	 * It is possible that we haven't received the prepare because
+	 * the transaction did not have any changes relevant to this
+	 * subscription and so was essentially an empty prepare. In this case,
+	 * the walsender is optimized to drop the empty transaction and the
+	 * accompanying prepare. Silently ignore if we don't find the prepared
+	 * transaction.
 	 */
-	replorigin_session_origin_lsn = prepare_data.end_lsn;
-	replorigin_session_origin_timestamp = prepare_data.commit_time;
+	if (LookupGXact(gid, prepare_data.prepare_end_lsn,
+					prepare_data.prepare_time))
+	{
 
-	FinishPreparedTransaction(gid, true);
-	end_replication_step();
-	CommitTransactionCommand();
+		/* There is no transaction when COMMIT PREPARED is called */
+		begin_replication_step();
+
+		/*
+		 * Update origin state so we can restart streaming from correct position
+		 * in case of crash.
+		 */
+		replorigin_session_origin_lsn = prepare_data.commit_end_lsn;
+		replorigin_session_origin_timestamp = prepare_data.commit_time;
+
+		FinishPreparedTransaction(gid, true);
+		end_replication_step();
+		CommitTransactionCommand();
+	}
 	pgstat_report_stat(false);
 
-	store_flush_position(prepare_data.end_lsn);
+	store_flush_position(prepare_data.commit_end_lsn);
 	in_remote_transaction = false;
 
 	/* Process any tables that are being synchronized in parallel. */
-	process_syncing_tables(prepare_data.end_lsn);
+	process_syncing_tables(prepare_data.commit_end_lsn);
 
 	pgstat_report_activity(STATE_IDLE, NULL);
 }
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index e4314af..2cdd9aa 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -56,7 +56,9 @@ static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
 static void pgoutput_prepare_txn(LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn, XLogRecPtr prepare_lsn);
 static void pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx,
-										 ReorderBufferTXN *txn, XLogRecPtr commit_lsn);
+										 ReorderBufferTXN *txn, XLogRecPtr commit_lsn,
+										 XLogRecPtr prepare_end_lsn,
+										 TimestampTz prepare_time);
 static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   ReorderBufferTXN *txn,
 										   XLogRecPtr prepare_end_lsn,
@@ -130,6 +132,17 @@ typedef struct RelationSyncEntry
 	TupleConversionMap *map;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN or BEGIN PREPARE. BEGIN or BEGIN PREPARE
+ * is only sent when the first change in a transaction is processed.
+ * This makes it possible to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+	bool sent_begin_txn;    /* flag indicating whether begin has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -405,15 +418,40 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(txndata);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -428,23 +466,67 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !txndata->sent_begin_txn;
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
 	OutputPluginUpdateProgress(ctx);
 
+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
 	OutputPluginWrite(ctx, true);
 }
 
 /*
- * BEGIN PREPARE callback
+ * BEGIN PREPARE callback.
+ *
+ * Don't send BEGIN PREPARE message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN PREPARE and COMMIT PREPARED messages
+ * to subscribers, using bandwidth on something with little/no use
+ * for logical replication.
  */
 static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	/*
+	 * Delegate to assign the begin sent flag as false, same as for the
+	 * BEGIN message.
+	 */
+	pgoutput_begin_txn(ctx, txn);
+}
+
+/*
+ * Send BEGIN PREPARE.
+ * This is where the BEGIN PREPARE is actually sent. This is called while
+ * processing the first change of the prepared transaction.
+ */
+static void
+pgoutput_begin_prepare(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(txndata);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -459,8 +541,21 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
 	OutputPluginUpdateProgress(ctx);
 
+	/*
+	 * If the BEGIN was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the PREPARE message too.
+	 */
+	if (!txndata->sent_begin_txn)
+	{
+		elog(DEBUG1, "skipping replication of an empty prepared transaction");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
@@ -471,12 +566,34 @@ pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  */
 static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-							 XLogRecPtr commit_lsn)
+							 XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+							 TimestampTz prepare_time)
 {
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/*
+	 * If the BEGIN PREPARE was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the COMMIT PREPARED
+	 * message too.
+	 */
+	if (txndata)
+	{
+		bool skip = !txndata->sent_begin_txn;
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+		if (skip)
+		{
+			elog(DEBUG1,
+				 "skipping replication of COMMIT PREPARED of an empty transaction");
+			return;
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
-	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
+	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn, prepare_end_lsn,
+									 prepare_time);
 	OutputPluginWrite(ctx, true);
 }
 
@@ -489,8 +606,27 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/*
+	 * If the BEGIN PREPARE was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the ROLLBACK PREPARED
+	 * message too.
+	 */
+	if (txndata)
+	{
+		bool skip = !txndata->sent_begin_txn;
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+		if (skip)
+		{
+			elog(DEBUG1,
+				 "skipping replication of ROLLBACK of an empty transaction");
+			return;
+		}
+	}
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
 									   prepare_time);
@@ -639,11 +775,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
 	Relation	ancestor = NULL;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	if (!is_publishable_relation(relation))
 		return;
 
@@ -677,6 +817,17 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/*
+	 * Output BEGIN / BEGIN PREPARE if we haven't yet, unless streaming.
+	 */
+	if (!in_streaming && !txndata->sent_begin_txn)
+	{
+		if (rbtxn_prepared(txn))
+			pgoutput_begin_prepare(ctx, txn);
+		else
+			pgoutput_begin(ctx, txn);
+	}
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -779,6 +930,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -786,6 +938,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	Oid		   *relids;
 	TransactionId xid = InvalidTransactionId;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	/* Remember the xid for the change in streaming mode. See pgoutput_change. */
 	if (in_streaming)
 		xid = change->txn->xid;
@@ -822,6 +977,18 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/*
+		 * output BEGIN / BEGIN PREPARE if we haven't yet,
+		 * while streaming no need to send BEGIN / BEGIN PREPARE.
+		 */
+		if (!in_streaming && !txndata->sent_begin_txn)
+		{
+			if (rbtxn_prepared(txn))
+				pgoutput_begin_prepare(ctx, txn);
+			else
+				pgoutput_begin(ctx, txn);
+		}
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -854,6 +1021,24 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		Assert(txndata);
+		if (!txndata->sent_begin_txn)
+		{
+			if (rbtxn_prepared(txn))
+				pgoutput_begin_prepare(ctx, txn);
+			else
+				pgoutput_begin(ctx, txn);
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 63de90d..0be0a07 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -148,8 +148,10 @@ typedef struct LogicalRepPreparedTxnData
  */
 typedef struct LogicalRepCommitPreparedTxnData
 {
+	XLogRecPtr	prepare_end_lsn;
 	XLogRecPtr	commit_lsn;
-	XLogRecPtr	end_lsn;
+	XLogRecPtr	commit_end_lsn;
+	TimestampTz prepare_time;
 	TimestampTz commit_time;
 	TransactionId xid;
 	char		gid[GIDSIZE];
@@ -188,7 +190,9 @@ extern void logicalrep_write_prepare(StringInfo out, ReorderBufferTXN *txn,
 extern void logicalrep_read_prepare(StringInfo in,
 									LogicalRepPreparedTxnData *prepare_data);
 extern void logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
-											 XLogRecPtr commit_lsn);
+											 XLogRecPtr commit_lsn,
+											 XLogRecPtr prepare_end_lsn,
+											 TimestampTz prepare_time);
 extern void logicalrep_read_commit_prepared(StringInfo in,
 											LogicalRepCommitPreparedTxnData *prepare_data);
 extern void logicalrep_write_rollback_prepared(StringInfo out, ReorderBufferTXN *txn,
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 810495e..0d28306 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -128,7 +128,9 @@ typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
  */
 typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
 											   ReorderBufferTXN *txn,
-											   XLogRecPtr commit_lsn);
+											   XLogRecPtr commit_lsn,
+											   XLogRecPtr prepare_end_lsn,
+											   TimestampTz prepare_time);
 
 /*
  * Called for ROLLBACK PREPARED.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b40ff7..11e2e1e 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -442,7 +442,9 @@ typedef void (*ReorderBufferPrepareCB) (ReorderBuffer *rb,
 /* commit prepared callback signature */
 typedef void (*ReorderBufferCommitPreparedCB) (ReorderBuffer *rb,
 											   ReorderBufferTXN *txn,
-											   XLogRecPtr commit_lsn);
+											   XLogRecPtr commit_lsn,
+											   XLogRecPtr prepare_end_lsn,
+											   TimestampTz prepare_time);
 
 /* rollback  prepared callback signature */
 typedef void (*ReorderBufferRollbackPreparedCB) (ReorderBuffer *rb,
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index 0e218e0..3d246be 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/test/subscription/t/021_twophase.pl b/src/test/subscription/t/021_twophase.pl
index c6ada92..b954630 100644
--- a/src/test/subscription/t/021_twophase.pl
+++ b/src/test/subscription/t/021_twophase.pl
@@ -6,7 +6,7 @@ use strict;
 use warnings;
 use PostgresNode;
 use TestLib;
-use Test::More tests => 24;
+use Test::More tests => 25;
 
 ###############################
 # Setup
@@ -318,10 +318,9 @@ $node_publisher->safe_psql('postgres', "
 
 $node_publisher->wait_for_catchup($appname_copy);
 
-# Check that the transaction has been prepared on the subscriber, there will be 2
-# prepared transactions for the 2 subscriptions.
+# Check that the transaction has been prepared on the subscriber
 $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(2), 'transaction is prepared on subscriber');
+is($result, qq(1), 'transaction is prepared on subscriber');
 
 # Now commit the insert and verify that it IS replicated
 $node_publisher->safe_psql('postgres', "COMMIT PREPARED 'mygid';");
@@ -337,6 +336,45 @@ is($result, qq(2), 'replicated data in subscriber table');
 $node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub_copy;");
 $node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_copy;");
 
+##############################
+# Test empty prepares
+##############################
+
+# create a table that is not part of the publication
+$node_publisher->safe_psql('postgres',
+   "CREATE TABLE tab_nopub (a int PRIMARY KEY)");
+
+# disable the subscription so that we can peek at the slot
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub DISABLE");
+
+# wait for the replication slot to become inactive in the publisher
+$node_publisher->poll_query_until('postgres',
+   "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'tap_sub' AND active='f'", 1);
+
+# create a transaction with no changes relevant to the slot
+$node_publisher->safe_psql('postgres', "
+   BEGIN;
+   INSERT INTO tab_nopub SELECT generate_series(1,10);
+   PREPARE TRANSACTION 'empty_transaction';
+   COMMIT PREPARED 'empty_transaction';");
+
+# peek at the contents of the slot
+$result = $node_publisher->safe_psql(
+   'postgres', qq(
+       SELECT get_byte(data, 0)
+       FROM pg_logical_slot_get_binary_changes('tap_sub', NULL, NULL,
+           'proto_version', '3',
+           'publication_names', 'tap_pub')
+));
+
+# the empty transaction should be skipped
+is($result, qq(),
+   'empty transaction dropped on slot'
+);
+
+# enable the subscription to test cleanup
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub ENABLE");
+
 ###############################
 # check all the cleanup
 ###############################
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 37cf4b2..75639ab 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1606,6 +1606,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#48Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#47)
Re: logical replication empty transactions

FYI - I have checked the v11 patch. Everything applies, builds, and
tests OK for me, and I have no more review comments. So v11 LGTM.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

#49Greg Nancarrow
gregn4422@gmail.com
In reply to: Ajin Cherian (#47)
Re: logical replication empty transactions

On Fri, Jul 23, 2021 at 8:09 PM Ajin Cherian <itsajin@gmail.com> wrote:

fixed.

The v11 patch LGTM.

Regards,
Greg Nancarrow
Fujitsu Australia

#50osumi.takamichi@fujitsu.com
osumi.takamichi@fujitsu.com
In reply to: Ajin Cherian (#47)
RE: logical replication empty transactions

On Friday, July 23, 2021 7:10 PM Ajin Cherian <itsajin@gmail.com> wrote:

On Fri, Jul 23, 2021 at 7:38 PM Peter Smith <smithpb2250@gmail.com> wrote:

I have reviewed the v10 patch.

The patch v11 looks good to me as well.
Thanks for addressing my past comments.

Best Regards,
Takamichi Osumi

#51Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#47)
7 attachment(s)
Re: logical replication empty transactions

Hi Ajin.

I have spent some time studying how your "empty transaction" (v11)
patch will affect network traffic and transaction throughput.

BLUF
====

For my test environment the general observations with the patch applied are:
- There is a potentially large reduction of network traffic (depends
on the number of empty transactions sent)
- Transaction throughput improved up to 7% (average ~2% across
mixtures) for Synchronous mode
- Transaction throughput improved up to 7% (average ~3% across
mixtures) for NOT Synchronous mode

So this patch LGTM.

TEST INFORMATION
================

Overview
-------------

1. There are 2 similar tables. One table is published; the other is not.

2. Equivalent simple SQL operations are performed on these tables. E.g.
- INSERT/UPDATE/DELETE using normal COMMIT
- INSERT/UPDATE/DELETE using 2PC COMMIT PREPARED

3. pg_bench is used to measure the throughput for different mixes of
empty and not-empty transactions sent. E.g.
- 0% are empty
- 25% are empty
- 50% are empty
- 75% are empty
- 100% are empty

4. The apply_dispatch code has been temporarily modified to log the
number of protocol messages/bytes being processed.
- At the conclusion of the test run the logs are processed to extract
the numbers.

5. Each test run is 15 minutes elapsed time.

6. The tests are repeated without/with your patch applied
- So, there are 2 (without/with patch) x 5 (different mixes) = 10 test results
- Transaction throughput results are from pg_bench
- Protocol message bytes are extracted from the logs (from modified
apply_dispatch)

7. Also, the entire set of 10 test cases was repeated with
synchronous_standby_names setting enable/disabled.
- Enabled, so the results are for total round-trip processing of the pub/sub.
- Disabled. no waiting at the publisher side.

Configuration
-------------------

My environment is a single test machine with 2 PG instances (for pub and sub).

Using default configs except:

PUB-node
- wal_level = logical
- max_wal_senders = 10
- logical_decoding_work_mem = 64kB
- checkpoint_timeout = 30min
- min_wal_size = 10GB
- max_wal_size = 20GB
- shared_buffers = 2GB
- synchronous_standby_names = 'sync_sub' (for synchronous testing only)

SUB-node
- max_worker_processes = 11
- max_logical_replication_workers = 10
- checkpoint_timeout = 30min
- min_wal_size = 10GB
- max_wal_size = 20GB
- shared_buffers = 2GB

SQL files
-------------

Contents of test_empty_not_published.sql:

-- Operations for table not published
BEGIN;
INSERT INTO test_tab_nopub VALUES(1, 'foo');
UPDATE test_tab_nopub SET b = 'bar' WHERE a = 1;
DELETE FROM test_tab_nopub WHERE a = 1;
COMMIT;

-- 2PC operations for table not published
BEGIN;
INSERT INTO test_tab_nopub VALUES(2, 'fizz');
UPDATE test_tab_nopub SET b = 'bang' WHERE a = 2;
DELETE FROM test_tab_nopub WHERE a = 2;
PREPARE TRANSACTION 'gid_nopub';
COMMIT PREPARED 'gid_nopub';

~~

Contents of test_empty_published.sql:

(same as above but the table is called test_tab)

SQL Tables
----------------

(tables are the same apart from the name)

CREATE TABLE test_tab (a int primary key, b text, c timestamptz
DEFAULT now(), d bigint DEFAULT 999);

CREATE TABLE test_tab_nopub (a int primary key, b text, c timestamptz
DEFAULT now(), d bigint DEFAULT 999);

Example pg_bench command
------------------------

(this example is showing a test for a 25% mix of empty transactions)

pgbench -s 100 -T 900 -c 1 -f test_empty_not_published.sql@5 -f
test_empty_published.sql@15 test_pub

RESULTS / OBSERVATIONS
======================

Synchronous Mode
----------------

- As the percentage mix of empty transactions increases, so does the
transaction throughput. I assume this is because we are using
synchronous mode; so when there is less waiting time, then there is
more time available for transaction processing

- The performance was generally similar before/after the patch, but
there was an observed throughput improvement of ~2% (averaged across
all mixes)

- The number of protocol bytes is associated with the number of
transactions that are processed during the test time of 15 minutes.
This adds up to a significant number of bytes even when the
transactions are empty.

- For the unpatched code as the transaction rate increases, then so
does the number of traffic bytes.

- The patch improves this significantly by eliminating all the empty
transaction traffic.

- Before the patch, even "empty transactions" are processing some
bytes, so it can never reach zero. After the patch, empty transaction
traffic is eliminated entirely.

NOT Synchronous Mode
--------------------

- Since there is no synchronous waiting for round trips, the
transaction throughput is generally consistent regardless of the empty
transaction mix.

- There is a hint of a small overall improvement in throughput as the
empty transaction mix approaches near 100%. For my test environment
both the pub/sub nodes are using the same machine/CPU, so I guess is
that when there is less CPU spent processing messages in the Apply
Worker then there is more CPU available to pump transactions at the
publisher side.

- The patch transaction throughput seems ~3% better than for
non-patched. This might also be attributable to the same reason
mentioned above - less CPU spent processing empty messages at the
subscriber side leaves more CPU available to pump transactions from
the publisher side.

- The number of protocol bytes is associated with the number of
transactions that are processed during the test time of 15 minutes.

- Because the transaction throughput is consistent, the traffic of
protocol bytes here is determined mainly by the proportion of "empty
transactions" in the mixture.

- Before the patch, even “empty transactions” are processing some
bytes, so it can never reach zero. After the patch, the empty
transaction traffic is eliminated entirely.

- Before the patch, even “empty transactions” are processing some
bytes, so it can never reach zero. After the patch, the empty
transaction traffic is eliminated entirely.

ATTACHMENTS
===========

PSA

A1. A PDF version of my test report (also includes raw result data)
A2. Sync: Graph of Transaction throughput
A3. Sync: Graph of Protocol bytes (total)
A4. Sync: Graph of Protocol bytes (per transaction)
A5. Not-Sync: Graph of Transaction throughput
A6. Not-Sync: Graph of Protocol bytes (total)
A7. Not-Sync: Graph of Protocol bytes (per transaction)

------
Kind Regards,
Peter Smith.
Fujitsu Australia.

Attachments:

PS-empty-tx-testing-15min.pdfapplication/pdf; name=PS-empty-tx-testing-15min.pdfDownload
%PDF-1.7
%����
1 0 obj
<</Type/Catalog/Pages 2 0 R/Lang(en-AU) /StructTreeRoot 55 0 R/MarkInfo<</Marked true>>/Metadata 1660 0 R/ViewerPreferences 1661 0 R>>
endobj
2 0 obj
<</Type/Pages/Count 12/Kids[ 3 0 R 26 0 R 30 0 R 34 0 R 38 0 R 40 0 R 42 0 R 44 0 R 46 0 R 48 0 R 50 0 R 52 0 R] >>
endobj
3 0 obj
<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 9 0 R/F3 11 0 R/F4 16 0 R/F5 18 0 R>>/ExtGState<</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/Annots[ 23 0 R 24 0 R 25 0 R] /MediaBox[ 0 0 595.2 841.92] /Contents 4 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 0>>
endobj
4 0 obj
<</Filter/FlateDecode/Length 1983>>
stream
x��Z�O�F�����RU)����w]�H��IU�#�CU?���#��P���wfl�A�\HL�D�gw~��]�{<������u�g�tx�]�?�'��lr�gw��!�^�7w�tv7w/�f8t>���i��NN�������Bp	�&�+���b�����o����dpx�=�LJ.�$&�W\(���.�{������Vf7t�����\D���7Q�:��d�7�g�������Q5`�U�`C�Z���Nv)$���^���3\5��D�3j]�RZ�u=O�����\����s��v?��x�m0�i.�a��f=��$	D>���mE�2Ft�2�&	������u/0��������Z3�u���\9���`

B��=��p�LO�#!]R\�iq����������*��	����8~FtB�{�#��I
���<�����4�y��^����i�'T9�|K>8.\%��������<m�<%�A�U��|���=^�^���:K�0��F��d9;��U<k.#wrS�%\�-]�����R;��<q��,=��2.k��k������5<�b{�L-#�������n~�S�3���B��"l(�q�2�T(��a����0�ut�+�1��2��Y��Z�*��\�y�)p{�pI	��$"B��mdA��:�
���8�q��U��s�X������m���s��1w����6�-%�M]/�?�%0��=���/�$����dM����@#�����p���5t ��.qP�$��j���F��n�!=I#�F��������N"i;�c��\������\����K��m[�Y��]�� J:������<��%[�C�.4�.r�}j���H�8(���(���J�I�:��+�Gsz_�qe�!���81�P�U%O
},��qE�8��/����,���l��A�KdB���b}�*T!�,�b�/�W����#Z�q[�������I���zM��2�V���Z-��16��"��R�y������wQ:?2����m�[Y�
w��[?(0�p�Fv�2��� ��fSv��Z�|���u:�@�8@@8���7�	L�>;Ho��]�B
v\���n�������4��E��L����u�j�3�#�h���AH�Mz���
��t|��p�N�:YX�8�7z:���d:�c���5[.:��`��m�K{�}aJ�B��<RCjh;>E�Fn�&k�@h��3>��F��d�O��d�iv��t�7�UY��x+�D
7W���VSA���@���$m��@9�uhb�z.P�4����|EK;7c<����������G�����`A��6}��/R�!t�lq����O���������Y=������S���]�K�����RH�S�8�_���z����uG�8	�(���b�Z�����cI�]����
��s����s�O��/g�|���|~�Z�[/�Z�-%^��I��2}��z/�!w{���������.R��P���[v}��8�*��G���w4x�1��.$�i�eS����1���}H�����.X�
.��������ZVo4`A�8���|	wm��{[6=�W�`x���u,�"��5�>�����?T[D�OA?�����W�~z�~�E��/��<��(��r��Le����Y��0�2;��48���w��j~���&�C�,��X��F#j��V���@=���H�������68��O��?��H.�^`�����|���vP�x�h��s�)����	?��
��y��,##UT>���\5C�����������~m`�/#�H�/�~�?"_�GIR��-���V_]'Q�l#2
�x��Gs�B|f ����l'Wj������HbS3��;��5�v]�2W�5�#�F
���K��(���N��R!���im�AI6zE�W�E��������j���R�Xz]1��0����'/�L�8��\�b ��62Z������!��l�.�_����
endstream
endobj
5 0 obj
<</Type/Font/Subtype/TrueType/Name/F1/BaseFont/BCDEEE+Calibri/Encoding/WinAnsiEncoding/FontDescriptor 6 0 R/FirstChar 32/LastChar 126/Widths 1649 0 R>>
endobj
6 0 obj
<</Type/FontDescriptor/FontName/BCDEEE+Calibri/Flags 32/ItalicAngle 0/Ascent 750/Descent -250/CapHeight 750/AvgWidth 521/MaxWidth 1743/FontWeight 400/XHeight 250/StemV 52/FontBBox[ -503 -250 1240 750] /FontFile2 1647 0 R>>
endobj
7 0 obj
<</Type/ExtGState/BM/Normal/ca 1>>
endobj
8 0 obj
<</Type/ExtGState/BM/Normal/CA 1>>
endobj
9 0 obj
<</Type/Font/Subtype/TrueType/Name/F2/BaseFont/BCDFEE+Calibri-Bold/Encoding/WinAnsiEncoding/FontDescriptor 10 0 R/FirstChar 32/LastChar 126/Widths 1650 0 R>>
endobj
10 0 obj
<</Type/FontDescriptor/FontName/BCDFEE+Calibri-Bold/Flags 32/ItalicAngle 0/Ascent 750/Descent -250/CapHeight 750/AvgWidth 536/MaxWidth 1781/FontWeight 700/XHeight 250/StemV 53/FontBBox[ -519 -250 1263 750] /FontFile2 1651 0 R>>
endobj
11 0 obj
<</Type/Font/Subtype/Type0/BaseFont/BCDGEE+Calibri-Light/Encoding/Identity-H/DescendantFonts 12 0 R/ToUnicode 1652 0 R>>
endobj
12 0 obj
[ 13 0 R] 
endobj
13 0 obj
<</BaseFont/BCDGEE+Calibri-Light/Subtype/CIDFontType2/Type/Font/CIDToGIDMap/Identity/DW 1000/CIDSystemInfo 14 0 R/FontDescriptor 15 0 R/W 1654 0 R>>
endobj
14 0 obj
<</Ordering(Identity) /Registry(Adobe) /Supplement 0>>
endobj
15 0 obj
<</Type/FontDescriptor/FontName/BCDGEE+Calibri-Light/Flags 32/ItalicAngle 0/Ascent 750/Descent -250/CapHeight 750/AvgWidth 520/MaxWidth 1820/FontWeight 300/XHeight 250/StemV 52/FontBBox[ -511 -250 1309 750] /FontFile2 1653 0 R>>
endobj
16 0 obj
<</Type/Font/Subtype/TrueType/Name/F4/BaseFont/BCDHEE+Calibri-Light/Encoding/WinAnsiEncoding/FontDescriptor 17 0 R/FirstChar 32/LastChar 121/Widths 1655 0 R>>
endobj
17 0 obj
<</Type/FontDescriptor/FontName/BCDHEE+Calibri-Light/Flags 32/ItalicAngle 0/Ascent 750/Descent -250/CapHeight 750/AvgWidth 520/MaxWidth 1820/FontWeight 300/XHeight 250/StemV 52/FontBBox[ -511 -250 1309 750] /FontFile2 1653 0 R>>
endobj
18 0 obj
<</Type/Font/Subtype/Type0/BaseFont/BCDIEE+Calibri/Encoding/Identity-H/DescendantFonts 19 0 R/ToUnicode 1646 0 R>>
endobj
19 0 obj
[ 20 0 R] 
endobj
20 0 obj
<</BaseFont/BCDIEE+Calibri/Subtype/CIDFontType2/Type/Font/CIDToGIDMap/Identity/DW 1000/CIDSystemInfo 21 0 R/FontDescriptor 22 0 R/W 1648 0 R>>
endobj
21 0 obj
<</Ordering(Identity) /Registry(Adobe) /Supplement 0>>
endobj
22 0 obj
<</Type/FontDescriptor/FontName/BCDIEE+Calibri/Flags 32/ItalicAngle 0/Ascent 750/Descent -250/CapHeight 750/AvgWidth 521/MaxWidth 1743/FontWeight 400/XHeight 250/StemV 52/FontBBox[ -503 -250 1240 750] /FontFile2 1647 0 R>>
endobj
23 0 obj
<</Subtype/Link/Rect[ 69.75 527.46 525.55 541.95] /BS<</W 0>>/F 4/A<</Type/Action/S/URI/URI(https://www.postgresql.org/message-id/flat/CAFPTHDaQFuASQPjxrYTcRPjF6exewjxXVyfuz1hCWJeCJpOSsQ%40mail.gmail.com#d9dbe3d195f1acccddbd81e46ded2315) >>/StructParent 1>>
endobj
24 0 obj
<</Subtype/Link/Rect[ 69.75 512.97 525.55 527.46] /BS<</W 0>>/F 4/A<</Type/Action/S/URI/URI(https://www.postgresql.org/message-id/flat/CAFPTHDaQFuASQPjxrYTcRPjF6exewjxXVyfuz1hCWJeCJpOSsQ%40mail.gmail.com#d9dbe3d195f1acccddbd81e46ded2315) >>/StructParent 2>>
endobj
25 0 obj
<</Subtype/Link/Rect[ 69.75 490.48 203.78 512.97] /BS<</W 0>>/F 4/A<</Type/Action/S/URI/URI(https://www.postgresql.org/message-id/flat/CAFPTHDaQFuASQPjxrYTcRPjF6exewjxXVyfuz1hCWJeCJpOSsQ%40mail.gmail.com#d9dbe3d195f1acccddbd81e46ded2315) >>/StructParent 3>>
endobj
26 0 obj
<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 9 0 R/F4 16 0 R/F6 28 0 R>>/ExtGState<</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 595.2 841.92] /Contents 27 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 4>>
endobj
27 0 obj
<</Filter/FlateDecode/Length 3492>>
stream
x��ko�8�{��}9�Z���H
��@��]l���w�C�(�DN�M�R����������IcY���yqN�l���8���~:yR���ey�?yzS�7��,�m��7��j]��������.�������?GO�?�>%$�J�<J�,���$g��<>�����������%�(%	���#�oN"IF�����[\��~y'��J�9��O���������x�g�E<�2������o������adrA�RCd�4;�m�4�DQ��Vg�	s������;E)��L�0��~���.�����2I�����E&$S<)��G<���QB��s��/��zYI�#�"Ty[��5z��Y���������Qj�#*H�l(KM�����L�PO���v�-.j�����(Ng�?��|Vnc�f�O9[�1��/���
e�����h���aD\��fJ-��U*��R��|��J���ge���]�s5��SM�|V��K�Em��l��(��&�3����/�xNi� ������e������|�\��r�_��������l��m7�-��zUn�
q(���^0�i�L�T*���r��&:�~��Hj�,���!�6p)��\�N�$s�{���&�5�0�c�@h�(n$b�K��}!'#
��~�W���,w�CRoZ��m��+�9k�\���Ry�U�C����H+S�Z����i�\0���si%0��cA�l�\���5H�_@l`��o?{��o��jb�K���q�"&� j:s�%bp.��l�	�9�(k-�_n�1�����,���4F�����#����vVi>����8;8�����V�*�\����w�!xN����q�����0w�w�[k�������x�������6�tfw�a��c�V`��<h�m^���a�P 2:�����
@V�k|�BGQ�E>��$U.p�� |v�C2���!�
�^�����Q�2��.|�p
RzSQ��Rzp��4~��?���T����o�#s$�jb$Vs���y��DN���S^#�TJqB�6���ARmC���|<Z/�#$BH�n��m�v����Y1���0��KS�Q9�i��};���P6��au���:�-��������O	�3���?��V]���D+E��8�;w�yA��,�u$��@3����>�&��c�u
���*�W+�(�t_���f�����9��w����Xv�wy^_��K����{���/�������vzr\[(�M,�Y�m���X������C�5�E���_�m�l�<�O����}F/}�R(t�w�2-n�tt�rF��@�V���7'���)��<����J��&��iB��T�Q@��+m�����.�5�$(��B��.�p����\�`�0�IW�0;l���F����&����}e|l������X�
���
o�(��/�C���h�:��y������f����'�6�-�>��1S�'�&Z�L@��������������J�5��u%X���3�����^)�+X
\K|
2n�H���l�r���R���m��fd�\��r+�`Rn�J���7G�w�m�B�G� B�����Gy����s�X#Zo��K�y�^�(>��x,jj9�������z��Zh�BEp�m^�^��K91�7�L�2�����Hd�vp�Nph���4���X�!)���qJG��a=j�����]o�#f�
����w��I"�ky�&�[�M�_&��C=X�u�y�ZB
�M?`��4f���[���;)��b8#!��u'C��CwG�`�D�p����v���>���B���Ia�D��
N�ta��l0�
N�`2�,��abK�H��=�?�)���^7�$Q���Pn�\(�.��o�\zy
��;��<#����<���cx�y:�[�����L����o�c��k2<g���6X��m���Z�[n?��1��mq�)'���a�\0����������1���r��<�
�D|w��w8�*��
/`3��*#�#D�{#F*xb��\�������������m�_P�����9��l
61��"�Y��6	�����.D�F�s�Y�����'������k)VB������,UD�&{N���k������b���-H����hEM
"]���!r����^=��H��vq�@�������F�4	��u��[��8@��:��
�,�#��1��[8�c��>�OQ]�f�����~h���C��Q���:���J��P��:�'0m��F��W�$����Of�������i�`���t���q����~vpXF��||��������#��X!�9�i�t.w^��`����<moc��s�����X�4m�P�������z�TS~Em�����?
af����\7�Z����9����s\;���3�G���pa:g�.��o�T�3����R��A
�v/��l!�]�f���7R2'��&�����:�������_��^��W��rc
Z�^4����J�*���1�4���\AO�w["��Q��V��,�\02��-�;�
�^�����R(�I����t��M��<R�f�&\�|���������Z!
��v�v�}jD_����9u�dk�0���X1:H��Z�{�R�Bs?1�������g��L��;�������ha��kn��[�i�Z��m����G���p���4��t0������?��?�`��z�_m�dlL�~���e��q6��q&�������~��0��@	���������+����$�W��{7
���\B0`���-������va����@4K�bn��A ��Qx;��h+���i�>�.t������\�I���yoa<�A@�	�\0!����s�o5��.?�MulZd�
&���Z�y]�]�WI������#c���%O���S���~\�pBM��Se�����xC�)b�@��ojgT�X���S<�;�!�D&��d��w����a��
�Y|6P���%TE�]�]i�M���|��:����Pts,���S�n{�+p�@YXj
m�f�S��;e�dp�?w��A>�� ���� ������A-$�>Vh�}W+YJ�t���������~L���n���	������k����f��M�������qm��\c�:��9R�{�d�C���\B){tc]���<���/������y�����-Uxf�������NQ��O�
;���`6����\w���94�SlTQh�����N����(������e��b�~v��p:Cwd�;8���L)IYa���D����27�|������P���} Z�Q�nO�3-$����-��^e,2Lhy���Y;��P��Z�B��;0	���t�t��S�Lr�3�5���d�I�����:����|�e��Uk�g���;!����$�XQ�uz��)��b���	
��?�h��
endstream
endobj
28 0 obj
<</Type/Font/Subtype/TrueType/Name/F6/BaseFont/ArialMT/Encoding/WinAnsiEncoding/FontDescriptor 29 0 R/FirstChar 32/LastChar 32/Widths 1656 0 R>>
endobj
29 0 obj
<</Type/FontDescriptor/FontName/ArialMT/Flags 32/ItalicAngle 0/Ascent 905/Descent -210/CapHeight 728/AvgWidth 441/MaxWidth 2665/FontWeight 400/XHeight 250/Leading 33/StemV 44/FontBBox[ -665 -210 2000 728] >>
endobj
30 0 obj
<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 9 0 R/F4 16 0 R/F7 32 0 R>>/ExtGState<</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 595.2 841.92] /Contents 31 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 5>>
endobj
31 0 obj
<</Filter/FlateDecode/Length 3769>>
stream
x��ks�8�{f������_�x��sM_��}t����f'��N�i����������(9�^g[H� �<^�./f������V�����������������/���W�w������������z~s��/~�!:|�$������eYj�$�V3e�3+��|��o����������<��%*:�����D<2�%BE�������k�[B��;�����{o&��������t2��?�����=�����1���2�eMb�����k�<�,�>t�F��N1�Ew��Ep�r����8�)tn�����	��+KBs�S��I��T�J������(a�X���7�o��sDf�g�F�����D��+�?=y�4J���
X�l�E���)��L�'L
��lz�u��u<��r�\�S=n"tgA1�l{��"ms$m��fq6���M%��i�=�kls��C�p��C�IdJh����Y�2��H�Uf8�������l��n�H�ocI���Nh�Y���>�O��y��@0�v��e<5y�;�\�V1��KlY���2vD�Kf|���	�};DHC�@5���=PU�Y��	Px��y��S;9t�63������3L{��P�����\�F�e����T���FJ��'$�`8�C�_3�9`XN?���I>o,��G�%hw��b�_����\V
Z���LR&S�C�3�g2f�C���O��&V;]�S�ZN�X"���>Ixv�
����!ve���j�PBz������9�Zi�1���O�B�����X���U��$��b(|�� a��Q���{���I���(���&�'�)�/���|
`����{����7����+w�*��n�o��]�����,��A�������;�A�����'��FW��'�������[0��4������I�h�����\B{��!tPB"�����p��]I���kM-E����g���A�����&��)qC��|w>4|������;��]��Cf-�(��|������l72J!&��;�+u��o���>H��3��p������w)Y~)�lr"Q/7qZ���.O�+�5���{jvz=�B3_��s���x<�G@�����3��h�������g�)���y���K| �:�U�pw��+A�n�|�jzu�DEt�7hM���;`b���G���7���L�#�&SBX�����,o�������e���z��M]>k��$���h|�3�e+��P�����,���K�q�O���*�Y���@I���-^������<Rh��:���%h
r��H��m8�����!����M,LG�
����P�[�h<\������E�Y�u/�e��r�r��P�����9��X��������J�]~N���l��2<%�&�F�3
�����o�A���	:��o��x�e,K C7ZD�>��f��.��3t�D���x��L{s���I��f�]�'�d�F�@�7�����%3f�$]�3��z������������IX�Lq����o��X<S</�gZ"3����TP^���Ri��$�.��������B4 �N��A��]BG�2�G�P����Y*Hi�u���2�EC"�EB����S�4�z��I'��kR����Gra<U_c�)� �\�)���,��T�iE����[4Fge�����{�S*/w[~�E�
���L����Ip��
P���(e"�I�-�h=��k+R�?��Sou��X%������L�L�Pv`��[*���n��&1�r�{	�<N�;��<).�6e~-�\Pow������7������!c5�U��g��x�E=?c��?��U�X�<8��
T� e4�8&�I�K<����4��8�����pE:��F����f���)7�����GxF���H?_�@�M�%�#�/�����I^JJ����
�-,-�Tb2�A��Jj�c��8�I��M�.	����Z�����4�Nx�(a���P���Q�(�w�~�7*n������*W���Lk��03\s�bF+��nZZ�j��gc�U�����_p���{����7�wu��n��h~��@D<�9H6�2���,0)����b�n%�hV���e��a�&"���=�5�;�L�����������e�z�S��4�z�*I�5��t5�-&�I�Rm���]��pO��@&���W���1��2L��M�H�s��`��$�����U�6�%`�eamm2���V���?L���%|T)a�&c���&�*�.m����5w�\�������R��'��E~.�����we���)��*��H3��$j��^���$?�y����I&K�s~h<7��b^l����^��&��C�n�\
����eG���=r���xZ���J�X�V��c����cX��
~J|��	��fSc�D��6P���s�Ro%����_7�BS���Pj5va�~�=%��U�J-��^M�����"g~�1T��Y@��A�Bm��M!��3E�����W������[4(3-K0�'�->�v��6���-����J���Y����2:��AT`,-H�R����\�c}��6��
��
�.:��������*��7����C���/�~/\�`�?)!�y����.�����������b:0�&��f
��uWF����d��i����x��m��B�p�?&�=*�$V�����T�KC��4�u�X�*�����~z�y���\W����?/�4���`��y�Xb��g	�p�3�R�N��6X����C�����:z�5����<����~�V�U�y�B��L�
B1Uy�/�;�����m�;��t���=�P�,��J`Zx�����t7]��+��-s"v-`Z���T��nK/:�i�4�&��G%�rQ�`��+�KE[PJ-?����Q~~����:�V��a������7��e����a���
c�^8�\�e�?��_�N���u�%�I���$?��m���8+_*RG�Mi��o�������E������a[��\7��=���o�T�La��*;L
k�p*
�F�b�\���T���NE�Z��~���5����e����9~�Q
����
|���"J|��vq�������a������|8���#����(wC��a�?�][rI��B7�������,�&4�Y.�������\��<��c*+����/��G:�~�"����Ij��K45@a�o�������&k0^�S���h=��s+��/��c�/�:�_�c����A�9������|�T
�=]T�K=�|�������6.eR#4�� �e��k��0��B��+�����|L�����>�I���y4OS��CvIn[G��2�"u�`B�$M���������:�.@��_	�%k��pI\���h�iQ��,�.PF���%_���c�'s����������v�]�U^#PY|?������C�	����v{�V3�@�������v����Y]�,m����.-�e"8f��"d�`���1^�'F������
��w�:_��	,�JAg��|��e
������C�IF+��H��mW�S�&/]��IB�p�=�1*��p8
�t���CIA�����������!��?�K(����U���w�uS��2���D�����BH�� �� �#�����i����i B���jI���fP��6�����\fxj�>3����Xx��h�x�4d�%iQ�0���C���
endstream
endobj
32 0 obj
<</Type/Font/Subtype/TrueType/Name/F7/BaseFont/BCDJEE+Calibri-Italic/Encoding/WinAnsiEncoding/FontDescriptor 33 0 R/FirstChar 32/LastChar 121/Widths 1657 0 R>>
endobj
33 0 obj
<</Type/FontDescriptor/FontName/BCDJEE+Calibri-Italic/Flags 32/ItalicAngle -11/Ascent 750/Descent -250/CapHeight 750/AvgWidth 521/MaxWidth 1984/FontWeight 400/XHeight 250/StemV 52/FontBBox[ -725 -250 1260 750] /FontFile2 1658 0 R>>
endobj
34 0 obj
<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 9 0 R/F4 16 0 R/F8 36 0 R>>/ExtGState<</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 841.92 595.2] /Contents 35 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 6>>
endobj
35 0 obj
<</Filter/FlateDecode/Length 12774>>
stream
x���k���q���8_
�o�/�E�&i�H�&�"(���q$^��F�_J��(r.����k�!g���(����?����?��������?}����?}���w�~�������w_����~�������������O������������?�������������=&o��}�s���?|������/����_~��_���g�_���/L��1�gg��7O7�_��~������������&��_~���^~���y���W�����?���/�����|��9c,a�<<�p0f�T�u����<��S�tu�:�4������oQc��H�||�G������{����9�=��������9�c�=���=����\����k��\	������J����������������/����3<��2�������C��2P�4=~�����Y/�7K�����_M���ejP��0��p���(�
�����_,��~�����n����
V1����?�O?q�����x^���^�^��y{�	��-g�6����C4��pW�@H�i�����0u~�:����+���7�q��
��)����/�_�2��>4N��rLw�u��1�����O;W�:��;�nY���s�B�?��L��wsk�n�N��i�������X16��t.����0<���d��\�����t�����>���_��g_�[�__��T�o?��L9#�p��_�f���%���u��7K#V/����>Mb��r��.J��>���1,�Ml��=���������B-?R����W���:��e�����i����o>/����!�|*��b�����M�3���~M<~r!��O��1NTDK/��	?�8c��a��x�$k�*���E���-����?�~5�;���?����!���������{�������O�?�l��{��mu��Fb�	�����]�>?�C��#\)2�@f)���}���+%5;�r�:9��][������e��\��v�'V�;wQ
F
\���1 `���Q�M�s��f|�v��(��.�n�A�[.��9�i������h'��`���}2,
}�>�FXB����x��Q�X����-<dL^#{��kmY[��^���_�[���Wc^~O����u~��g����9*���6��5|M[��>�{��#np��z��v�c�m�z>�p\�H����l�3NK�t��L�0Zy84��E���&P��a����4,t���b�5�z������5��9�,a�yJ%\�O���1���[V0����UU��Q���/5����,un���GB�X/����p���c�';>�Gs��
���n�E��>.[n��}�����,	����i|NQ��C�9K-�eqyog�4�������&��*��0-�6��������}��&�n��m�Px���e�8,��9<�,�t.��y�T��6Y������K�[��,XU�%K�������W[Y���d�O��M�m�Y�nj��[�vxp�,JxWK�e�b��W��d�Q����m��J�����Y�fc�Mv��0���m�nX6,a�t�Y�6Y��.6|��<D��!�u�%�����Q��1���j�!<�l�W9�{����$A�p.5|�W6
,�C�RC�����-��ZL���V�p;��A���%K�~r]���N%g�
HI1?d����������&q��>�+�&���oQ;����D%|�K�>,!����]6�%��J1?F~�����K2���2���^J�4��0�Ch�]�n	�~�����d{��J�$��+�~�����{�K5���B
/�B�[<���[-��N�]��a����sb�
�$�\*A���)����K�
5��!Z�n�!RnEm�}���T7������;� F��C?';p���%Y�Z���u����`�l�G�:��a�8V��(Y����`����I�t��|�4�Q����0����i���}[^��E�. �4/�}�Tq?��~�#"������C���q -l���������<��E�mc�%G+l��J}l'� ��u�[���,�O�
n�Q�~^�2�C�}�0����<
L��O;�9�#�����3+����C���i����"�@��2�V�c���>3cy�������Y���������R�d���`����{w��i��]*/��I�1�O�]�A6����
~�������EC�>���Y�(f���;7���>S�m�!��;��y�+8;��}�a�"{�)A���������8�@��d7�"~^���d0v%K08�~0��~0��~ v�~06�~p�'�O�8��~p���6ss��R?8�b?�}+��o��S�Q�x��u�!��>�#���[{�nyW�'�Z����)���C�	�>;�z�f�kaf���B����r����g�`Q�C����eqc��)EI��5P��H�LQ�
�<Oi����C��{����;���h04o�S@X�8�%��Z��)>�/�U,�
|�����K�G���������rj�Y����S��l[W�2�L
.�}�r��#$�4�6�P���������X���@���~��u��'rUc���mr9��c0�C�c�n�v(�'�0Qb�����V
�3�����|���5j���L���H�5��e1�W��H�PN�E	�����N�H~M�i���� jo�,���{3_"��1���$�I\��g���1yw�F-;x.���������|��t���;T�Y@���O|�xPD	����3}��U�f~Q�
��j��C���@�&�/�S������/<������M�1���?}���;��z�a��r*�_�#^��2zg����L�\�mQ�
�������:��\�]��aK�Fj�_�r��\`�����S=�H �"F������y��T���<��<�����Q;0
�����z��D�$9G^ ;D�EJ�"��Gh��}���i�u����`�Kf!�F+dA�~��m����SsP�<�2�S	NF�����!�72\��9���m��� F80Gj��= 5�	��U��� wWmH�Y[X,es�.�l����T��(I�x�	������p.�B�}GL������`u�SWgg�}=#Q�=�6�^@m�X�:�Q�@m6��.Q�A@m���x�&v��6����C��
��vj���6���@'<j#����`����j��6�!����
�Mj��@'<j#��M��j�}�
���6����v$����6[�$J1���\v���&I�x�B(��� ��2 ���d	������������N���|����N]���M*SS����Sc����Z&�*H#�@9f���&�@d��T��X�3�^U��ys��9i�9��g����������\��K�FP��fZ�r�:1���=��\2	��7r$�!���X:R�#��4�#7�F�#A��:�f^�-~)?6��]~L�/-��E��c�W�nL�����m�\~]+,��F�gZ��KBEh��)��~�Y<��b�S����c��L�������K�������`2%���T����n����Hy���������4
)Hx�f,��I@Qd_��D��J��5��A(7�s(��|
�
_!|K��>�*�2���e�u
|�c�O��[���z�BfA]����8��Q�(���������|��q��G(�K�p� 
��-�c{�'���`��s�E?�����)���"�6��g�6��,WfVT�>�l��;g7��Xy�q�Li�[j����\�wXt��^7N.�~�Y�n�LImMfc}U[%c�5�e6.�b�z�#w���gV��1�6�5���b�]�j�:������|:�d��1�AB�m��H������Hr�}gl�������u�`�
������x#�`��2,�QN2 �{Cv�X��B�8=V�C�=��v�e���x#]��� G�q� ��Nu�\"f}�{C��X�oy@��/B��{����Lk$/��x������]=u�J��s	�\l���k����.�9�[l=�?H[lC���
�Z�Y��7I����m@��7��#��@gp�o2�H
�EbCXl]��o�8�9�z����l������
,Y��A�����#��%8���KJ�����s,�Q&oF�F�F���N|���I�^	�[���>/SR�;��W�,Qr��p�JH�r�b�.C'\}������{�%W,B���E�
W]����j3���x%1h+X
Q�JS�v��$��Rr�**������*%�$�V�a��L������4�l��^m�
�������������
���G�����bW4#hAi�!-\m5�_�[��E�h��h5����x����.BKRc���U�	7�����ES�p�x���f�ot@���aD	���{?�DEc�����U�yo�d�1���H3���&�bMc�����~@=�1�������k������8�X�*�����8��5��3��)���o\�d�4��d\e�
�~Q��1��A�����P����H�����s_��Fx�x�}���~���k3-j����U36n��	��,�a�T�Rc��P��6���mM��8��G��V�0�"�Z����w��l��4���!= ���Z{��$�9~yW_|������B��s���q�$�:�h���B|�V+o���KmFw���&�fd�f��#�7�Qo�H`\]�
B�r��X��3��0�g$U�*�q~��+�o\����r' 
q)JRN2�#��^~����������e�!��}�
�+�7�R�k��W�ox�@r ^��!�$�W�op�,�W�o�=
�����j�<�W�oC�D�W�ov��B!�H�x���`r!�!���V#:�CCC�
�j�����Z�
��C�<��C D�VC����0_������$�]���!��7E��w�oH�`pJ�`�I�`p��@�H�`lH���O���oH���E����N�����ar����>(KRK�)cI��{�\���\�$�j�&1��r��&�d�$gM:���i��Q�������W�� E����R�"%���Vvh3����-�`]��(����omF��oc�����JD�*W��Z��J�Y������,��0��,��hIr5L��+?����Ju
[��Kb���p��\�K�3��l��r�KN
����F�:Qj��0����=��1����ZEL�2�0�<br�.`L���������|��R��2������;����]�L���S6�����Uc��5�+I�����	�?���?N��q�`M��5�`M�v��&�V�������f4���h6eN���6`"���k\I���U����G��'��F���4��JS���(H�����=*��a''\�}��o�cOn��p�I�����$O��z�.t�0�r��������o@��u8��K�Dn��'��������N�~�O2r1U����"J
��(�-bE9]�%�"�V����W�f2(o�h�XE|P��
#E��k��+� ����������[�P���Z��(o�M@*�8v�^���h3%nkC����dQx�*�r~�|��'�8��a@���O���PY����x��]�g�H�Q��h��g�I�iQ��/�RlZ�� ��|��cy�������Q�{(Rr�f�TlZ�� ����E!�b�K�2pR-���=�-9��i��jDdB�b��K2x�-
+6-ar��Sq��V����D��\4�{(gr��u��WD��^6t�P��cz� �X����Q���h�8�c9��y�h�,�B&�f�N
�8�#�rn@���d�=@�� �mI�:��&H@r�%��G*�G:��P�0g��D���a���`(�=a��=a�=A��=a��=ad�=[F8���pN��^,%�	+���q�	K��K�J���<<*�����k�@��a]��~,9A����uV�0wc.W,B���E�(���h
#I9t�ph�����kT��_���]z8����)�����
�Gp�9�BK~��(G��ZU5%�Hy�f~�Bj\>���V�h�5��m���	#��MI?Z^����p?\�����t,������P�k�a-~�����eQj�q���Y?��b�2���%L#���)�Uw)��o6Z`�e3�F����.�$��>Cn��55]��$n$�����G��a�a@���_�j��0��g�����Uu��u��M��`���y{BF��^�%��yMc����I�$���n�����A�9��4����X�+����O�#�rj9�~��J������l��%7�V`6e�R��(I��x>�m�PW�`$�I�@��������^�`$#mf��Vn�u�������S�hN]������l2c����K!M�hF��W3M���K�o~��\�M�:uKJ8��4#A��y���z�p��f�PX�z`����������%�a�	�Sy��$�*�s>@}E	�K&!��F9i�������%	�N9��^U�f"�I������
����r$�+`�V���\�s?q����D`?�Z��9?l����l����r'0,q)JRN����%h&���Q�q�f��g���1��
<��g��m���~�����H��>�?s��G���B(�uVc���@<i�����{��������{��N���A"��3=�c����A �?��G���1}Z&������@���{����`��g�*�}IvF�5I���X� ��xC"b��������!H�,c���9HDB���;���^&���v�yNL��ox��gr��{���@�x;��a����GZ� �i���y*f^��.����d�'5�"~^�;
%h$K08�~0��~0��~ v�~06�~p���p%h�~p���T�F����OU	���5f��L���XO�X
Ir�Z(Cs��^�FZ
�d��Vs��|7g+���<�?�5����}���#q�X0�k���|��_)�e�
:l�V�XA���y����>n���TF�is��6H��d�8I�p���5��X;F��\�5�^�n�>�7�j�d�in��F��9��J�8A��[��3��^�{Nj��_������]���TV����z�u���3(��
���(��X�Js�v�����6�:C�>����XU��($Aj|38����fu��Hj�Z4���K6��l8��O)='JR���t�^mFr^	���5��r���Xx��,�'M����p����k����$H9�a��'Sv��g������� �W/y�C�&K6��V�m�H��[�,]g����������6���n\K;^�{M���,��y�lN�X
�uF��Hm�:6�}�$/MoxQ��E����$�b�b��f�J2����Q+�UZ]���$o0e]����F��/�JVEj���[���%��rnr���^���Sk�j*����d�%]��A*-�h�=��#�(H
q��`B�Ej��P*�^~|�i��=�:L�d	p)w�[����'f�I$<g2#9��<gJE���������q�b�t�d�+��-�3
�����R��2�d����3��J��U��'��4����ef.���&�zl�&*�M���&{�u��J�����9�i����������l��%v��~%�D$�D��E�L�~���P���6)	8�.
��G�N�~6�������?��DE"�Pp�E�icQv�8�b)���(6����kFmt��<PQ�FG\�TF��
v%�*�Q�����#+�'�Z��$pE�icWD��!Hl����%�7�w�[��X �HH�������)��rC�=V;)b^��Y��T��O��J�F�!�%���.��SE�j2��z������d���3�����RL���V9��5�*&c
���UfeL�%i`V����J�	g��QN����c+��WT�)�Q��M��A�?��d	�+w�x0��sv��$�gsx�WS�Z�^�4k�=:�=�+VM��������W���`����������~r�����+Z#OB)��G�[z!�~����J���eb�uBrj�h��+�0���"��PQa����Q�b$�s����P5���I�����
'J�o�H�_U�~��J�����[y ��$��A��5�#�R��b�T�Rc�cEL��7������6��Y�{��i;��H5M��/��X+�����9�v]	�\[~"AR7�kv���A��5������[���Vk�W`�z�?��A�*)�6�$i��+"Q1�%9j�m����^��.�I.���&��`6��J14�E['i��f�j��~����T-�y���j J�>A ���W>���Q�Z�L�>\�s�%@�6��4�$%b^�*�0��0V�����M��Z�Y�|R�c������Jn|�@L">�kNI�uS��@@�T8��WOk4�E���v�W,GrMyxX�w��e�;��q�L���9���W����?��tK����XoJ��U��-uC��i�KV�r�\�t�/JR���C_/�b�O��h sb���6�aS�|Jn��f 67��b3��[��E��k�d���n�$�fRj��}M3��[�A�f�j��=A3)�[������-�P 53��[$;�����-�H�LJ��ddC�A�J�LJ���b&�v�nO�LJ��da&�v���L�LJ���_X;vA%_��Iyb�� ��l�Q�x�C(��� ��_�E��S��O��K�bG�cC��~�S�E��.��[�~p����P����"fO����$��
�����H����w�kT�<L���n�a.��@L�
g�0G�>�"J�n��p��*����`bd�*S3pG8�/�r��b��!p`Q�)��������.���u�d�=.l3��&&��fE,Z����J6Fj,�1�Lz�v��
�&���P�k�p�^�D.S�1rs��l���4*B�����XK�)B���f��,�"f����U�UJ�t�pn(��F[��1�#���C(*��}����+��������L����A�F�h�2������������e��-C��������^pyh4��t�$F��U���}��s��~*���E�X����~��D�!��H����D��%�"�V9��`����\�@�� X�������HE�e����O��Z�X��O�D+�=�t� ��_{�� i�#�0#�8��-Q����T���Hk�
0���
�Q ��'�x������8�>b]���@4�O������g9Qj�}F�@���Z���H��[ej���'�g���fB57>o ���'vWI��7dX
�v,�"�o,�Q�
�\��4j��N�����8����6Zs�L`b�$9jM\���t�<������xM����%�c����w�k�dY��`��+61tL��Z�[�>�p��<csU���s�=|�J�m6is�L:�6�^�f���� r3����!��l����b6��!�3����g�(f3�e>�j���C�X����m8'�_�sd��*���6!�#��b�:�y�#���PS�&5�Gw��b6�Q��H�I�x��b6�M�H6!T��$!=[n��k1�	��v	���
AZ���S�.L&���(K�$B�Dk0���0p��0&�� ���0���0D���-9��bO8�Qh/u�
��{������n����g	l_S�����6C|��uo�/�W(Vo�I��\1��:��=`ME�F������Z����g�kT�lO���%S�,��N��HeN��������U�I?�^�Y�zN{�,W�d	P=M�K������8�P��VezR��-c?���C�fDz(�5H����M����*����&��J�n�yN����ohIb��7�����PS�&��{�b��9��a"��&S��1�#���T�aD)�����U�I��^�[�x��m�����m� ����~�L�W��|	�H�5��N{E�Jo ���2m������C�X�*�����sji:
%�smE�W��rc�U{��8��3���5{�j�@$o`��p����4V�!z������q���C�Lp�!�V�"�$9j����N�9��.��
�9;�{v����N�l���+��]��H�5j�j�\~$�I5@;g�(�z������~w��������
��J��Q=;
y����
8�$5�6�u 7TU�I�1\�Y�u�B�&���u.�����O�|��w�-I��g�tU��/II�
�\]I�)N�Ny8��*�s�$�t��C��=ZB��=C��}��
8�X2:�Z�9��`J�����9U�h�o���;'
`&�G������A1�T�����h�]N<m�74�N��'�m�x�rp�I@�8�dP���Bm""$��d�"�A{?]�d���A��H	��d�#�AR?@A?"�� �I&��!B2�H"�d������ �$$��+��mB2H6
Y�( ���2H6
� q�J��� ���� �($�b�d�$�A��H)6$�Q�h�Dm��8���@���b?�����P�y���b?�%�R?�R?�b?bR?<R?i?\����(#�������sC�{�aS��$�r���.��lV��_�����Ve��=E~.X�P�g ��~X"$� �0�@6MZ��+�I�����vW3`���Y���%����:ZlV��t�$H��g�����H��K�G���W�m��N{N(�s���� 4�X�u�Pr?|S��p�Z��o�Z?�r7�A5r?��ZV6��S��$5��yd������n�8���TrF�b~���t�9�����8��46��S�K��5g��=��e�:����,�+}UYM�tt�����y��w�$�<c2�����$���8�4��Q����z'G��A������������p%n$������hFgw|�u:�7"�0���%k�8j�� ��/�eIj�}��Y	����.-@���/��A�������j����$H9c����x�'!������x�F�T;^G\�\94Y�����U������8��NuZ]��ot�c,ot�����<6��SzM$���������r\*Q+�K?�\�W ���E_\���^��]��������%)gEtgY��Gl�WDA$I/?`���'7�q�k�����]����3�Q� ���.����%�"	zU��|V�(�+V du����[%��)����lV
I�QI�+V����5����S|A��~�d Vw� @wlZ���>�5w�*H��a��3Q�S��M��~�����02z�K���\^�(�z������������5��J��A��XT��3�@5-l�a���6��a�&�T�A��^�j�^"�t6`���PM�RMG ���j���j:���piH5�=�j:
� �T��T�Q���
���F5AO�j:!� RM0D3�}�d��j���� ���lRM�RMG`)d��j���B1
�����T�`�lRM0��j:
��EK�����R�QH5�H!&� �S��
�}�G��(��bPM�M���6�t��,h�������yk���	I9���Z0���Cn����;E�V`4�`�
�`�I�@\	�`��` ��d;8��^p���^����B/8��^��Q�*7�n�<&}�@�����Hm��$��i����&�H�aX���@��x��� 
��I�@3�P�-;����dH�������1����:�}IjH�yc�a]Y�0���w�R����d����N�1��A���q���+���5p�T8�u�i�B�M�I��Z=�6�K�;��}o�M5K��h����L��Zj����p+Y����r�Z����bk�Vp�~�zM5��p+��g���<P�<�����'���������.��v�W��1�Gy��Y�g�qTL�J�'�~����v�u��xsgc�ec}y�$�fop�u�Z8��3]I~�8F����	�	Sq����-������5�������>��&�����^�cawq-[N�f���5wkk�=nC�go���
|�A}����L|��_�E���/�9����gP���� ��l'���>T��))������%���}�EP��lF�b��fND?�%��+*_�����_J$^o��+Uo�*���O.��f�����Nu�h����w~j�NU��i�����a�F�z�U��w�s��y��}����l	+Fy�����
���au�}*��Y�/n�U�3�KE'�qsD8����k}���9�E���W���"����1�,(��zIW���{z	O���
��Wy�Q�����������UR�Qp�a�0��c�eW�1�C@��BA����p�a}�mw�y^
]������{�71��`�9M�C�������e��������7x`��r-�/�e����7�0,
endstream
endobj
36 0 obj
<</Type/Font/Subtype/TrueType/Name/F8/BaseFont/TimesNewRomanPSMT/Encoding/WinAnsiEncoding/FontDescriptor 37 0 R/FirstChar 32/LastChar 32/Widths 1659 0 R>>
endobj
37 0 obj
<</Type/FontDescriptor/FontName/TimesNewRomanPSMT/Flags 32/ItalicAngle 0/Ascent 891/Descent -216/CapHeight 693/AvgWidth 401/MaxWidth 2614/FontWeight 400/XHeight 250/Leading 42/StemV 40/FontBBox[ -568 -216 2046 693] >>
endobj
38 0 obj
<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 9 0 R/F4 16 0 R/F8 36 0 R>>/ExtGState<</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 841.92 595.2] /Contents 39 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 7>>
endobj
39 0 obj
<</Filter/FlateDecode/Length 12444>>
stream
x���m��������_�w�VK"��df�] ����`��8&n�����XEQd�P�n���>��:�*�"�S��/?}��O_��������|���7���/����O�?��~���?~���_��_�����?|������������_~���|��/�����o';��/��m|���_~���/��_~���_|���}x���/_���/'��/����}�������8�_�ny��g�����o+��W_~�����}��_��a~����?_���_~�oN�����{���3�������0=��������r���m�vk�������Gt�n�M�|p����|���_���o;������7hY�nrZ��������m{�;������������7�1K�����}��_�����o�	�������WU�uC�m3]�������<�F���8M����w��?�w���������_������������������?������a�^�����y���_�����-iD}�a[��T�$:��OO��]0MZ�~8�x��v��������v��L�����%�zo������Kg�p��l���Wx�<w��"9lN������<����*�=��K��������7��~�����\�	���Y;�c���}��_���+�a�hpJ�-�pt8F��sXxY4x?,��:��(����{�_;EA+�����#h?�-�_�Y�������_�������asS��y�|�q/��Bw�9&��QF�������"��q�XX�6r����K(q\�8�e���c�G�������n���{��]-.�?�������}�(�o�����Y<�|��
cf}++��������U�������-�|��s�V��E;���&���a����}��H��m���/3Adtk�u�J,���YTR ���v|t�y�����}�2�u��a�*��H2��t��������1�s��^n�����f���v���aiv��ar�?��wK�<�[������
�wp0�+�m�
\��u�������6�c���6�`��uf�=r������`��!Z�\�{��1WtqC�bo�:K��[�x�n<x@-`(9bP�d�2-�����\��G����
c�z������7j9���@���[���[����i?q>�it������ Y�.�G�[����Y�=CTb��q��M��6��=��9������H�Dxp?:�z�G��9
�n���D�����O�m��_�W�G���������y��D�I�$���q[�����3V�T��)������nag�_aX��^�C�d�?��������6]�}5�9m�C�qsC�k�]�Uq�$9"Z�2�n�����v��k�
�Q~��q�bU��Ou?��.�������+������;�5%K�d�|0k�M�Jj��%f��Y'!NQIe1i��	�G��j^�M���5���������s^����J��������5�Lw<t�Wo�f)�����d���W(�&����)w���������3�����-f����}	[w/l2�f��5��3	i2^�}IR����lKB!g&J��|WR�����o�h�������&s����)8�{|=������`��d2������
1&Z����g������I������7�����2�����Y&���53������g�L�$A�(%�{�[S��(Z���
��3�~t�����1j0=������-a��Z��#Kq�E����3.���c�UA�]~���aQ��N��yh}8������`���"b���y�]�����$Zh&�AA�E�������7~[�>.��E��R���D|�����0����O]
��#���z����[p�nZ����'�K�M����9�	�/6|n�]nd�����
�����[s,&��j���� �#s0*���Q/r��_>�m\������K������4���i������t�[�;e��1�l���!����]d
1q����?�/VV��L�a������XR�?����c�&�����V~��0������1����gQ��&����[����8i������.��D`�MOF�1�'�~��<��~2����1�7�����"�f�ME�(S)G��<�N_~���N���9�-����� r=���`���C]&�j%�JAkN���D|�	�%lV��� ��,�@�
"���J�(�h���n�����(�+��{���g����$C����������������sE2�rPh�M<�F�I����O�Z�L��^�����?n�v�7�X:������\�R�c��>���_�0!UD	|2��@L�����3%�y�r4������p�l|\� �����y������
�,J���b9��������q��2?�`�)�COE��/���+��x��!+Jq�w��{bH�k�gPb��������?|=�9��cq�P���A����l�SF����)�Y'��T��<�>��������%�"J4�� s��R���Yd�%g}\Zg5&�m��N�"0�\8�#������p���DNE��N�o��l����G�	W�n��i��������T8eI"��6D�($�� h��~OUb�>��i��nT�k��a��?���6�<T�^A���J5Y"��8��^�F��*3���7�D�6{2vv|v��Ky��9�S��;4���`p�0���P�m����?�O�?��4�IW))E��%ay����<�Y���(�VO�����z�RhT��-�o���_�-��|I�f��[��&`4�&���9##_����:b��O�j7�����L���x�5c(��@���a���L02�Yv���`��s8�C���h*�����F�����N���M�-��d#��G�?q/hve;���
�	��l09"�)��J�H�9�Y~�V�����dS��|I��;�Q$VJJT�%"�=�J�`z*��?Q�������pA��k[��X7���?�h�D'����g������'�I���=�*=��`^�X��
XO,�s_
����{.B�1?���g�G:�����h:��X(�U���mD�#�@R���S� ��|�HI@�O�hHZQ\�<.�T|#�A��0h:a���	HH�	U	�e"�B��`P(j�"�BRx�
x�e�9����BB�S�!8������_��	��N	�7B4%���E�-�M	�(DC8�BP���,E)�r2��J$��$A��B���$�(�9�l���(j���fw�&���UX���N�HX�-I�[U9"���y�#�p)�������@��-��rI	"�II��"�.�s�q�t;��f�8��P�S@�W@��r�3e�HA"�)9
��O�
��v�t���P�lq
�����/��]�Bt,������j�kb����l��;���,� 96�i�2�@�t����C5�Co��j30����a����v�$xI@�$�#Mk��fU
�i�p��6�M������}J���=�$�-�o ��I�������
�4_�#�yr�j�N���:d�p%�'���<r���S�D;��o+��p�K�#{i������_�$�z�b-�r��TKU���R��O�<�����>���&��{4��K��0sn��'X��o�8���.C�~Q��������T%]��@������G�����J��.U�}�:�y ^��b����}`?���f]3lE�T�V��Z�}3m��M�Yi���zCR��-�U��Y��>�U��2�ko�����j�(�wZ�#r���&0�d�mH��-���1T$�*���L���V�Y���=��RD�L��������K�l�T���`�e���n!��N�D�B/HY����(�
r�q��:�(�FG�g��e�IHd��<p(�	-s=(p��L���6�%|3IP'^�o&�}�M����H���k&#|dL@��LFP�t	�LFR\	�G8��� Q��N���~��6�����|3A�#�Ch&#)r���d���4g��HJ�%|3�82L3��p,�;����>�S=�}��an9?g��"�s��
>�9��`�V��D��$�)I�WL����H"���f�$�*�`*�_�i&#h�C���(<�A
��%�?������!�f$�U��]��x��]�� }��E@I"�[���i!�C�g�hY�@nG4����.2�^�^Q��O���-�Gu�����2bT��,�,��Qm�
`������
�� ��*2h?��]��i-��L�!}���,�eL��2[/%���{���	�5`4r:033�,I
r�6�>��"����$�*�4cfZ(���1 C��V��C��l�GCQ*������g�bl�78��U<�y���!O����F	|T����UuN[��#B����EP
��L���=��0�"cQPp�(���M�Bn��0�DJ]���'�����<Yn�s�[�����U,������]�&}8A���:W�Ki(�-
�@��F�$h��:��R�}h��:ht��)��P}������)��<������z���d���y�is~��Q/Vu����8.�Ri �����>��S*�$5�K�����>K�9�"�UI����tT�s	P�wz�@�"<��S�"JEtKh�>m� E5~i�m�Wyb�o�d�L�[�
N�\$y1"��-������� �9���U����f>���YU�_���S8�((NE�
�I��{�ist��$������������f�7
��G���}T-H������G����>*�& Z���GE�j���?rI�-A��GE�^���&����
I"����x����]P����StH/)�E�)����P}T�%����	9�$�QQT
��>*�&dbB�H}TU@��*��J=����TD.N�4�_����/S�)�=BrPF����p"�`���`��2P����(�(��"��HN��2P�&�(�,�`��_+]
a�Iznp3f+��B�QK�����uE���7\��a3f{��)55s��[�:��Ma�`.Wh�Y���/��13���v�hq�#3�R~�o9��3� ��
�`H������5^�-�d/��<.�\DO���z�f�h�R���zG�u��
�����`��=e�_�^r�����e����n��I�{�E�2�}�{
-H
��r28Qq2���k�L�D1X�L���4��)�����@���)�$����`X��*�7������lw� d��=�����Eb0K
�H�
�J�o0�	v��/mT���(4�R����Lh@`��,I
�z�Un����*u���k�R���<��y�~����k5T��;��$��2�_�baP�:O<��R��((Q������+��M��5L��h���T���z`�h=Tk2HE�VcT�E����=�Zl{D����chR��2�o�-�VT���Nd�X@����SZ�`-���D��Fx��)�1Z���N��jpJEMNz�:��wm�r>jkmTI*���L�h�~�5#SD����t���V�[w��0�2Qd��nW*��z.��gT
kU�R*�WXV����&��B	�EAQ��DH��	�W���)u��?�L������o�G��f*��L��1�^m���-3@��f*�@Oz��
�E:��f*��M�����-4��f*�DLz�������pR�7S���z3�8�����Z3AB$��LER�H�5St 8��m�")d������a������z��
L��A�>��~n�>���H"�c��
>���k�T�(��R��E|M�"G��l3Q���$�,��\�D /_(Y�@�IZ�@!���� |7F�Z���W�`Hj�z���������7�
�r	���.��C���3�T���:%�D�$D���"" �BP��y8�Y��/�D�+�<�w��{�p�Tp����VD����"�m~��f�����Uj�^���F���S����rbD�XeF�o�0#v��+u�������by�,�:K(��8����v��}u���<���l\@F��\��u��K�+�^y��
�����
�,J�sL������5�om��K�m�.�F�qy������0�;���T|m��@|��wh��s.�u�ti�'\����w,`E`�]����"^,���G��1�;R�-*6�/����Zw<��pC���$4Yj��+�r�"��LC��DA4{��w���#a��r��T8q�g�E����=��s!�������'��I�����F��y��z���VQ"�K
����%'u*�?Q���xZ��{R/�`��D���x������-��F��G����u�:�0z�f�,�v��Up�`����A����p�G<mR��2�(5�)���3����U��~���e�z�>��N��R���p/�d�������G���<O�
�]�*�����+NA�
e
�����H�����������U�����y�x"�RR��,�5�z���J`=�u�<Q������6�h�~���������"�����fP��jF��y��Z�H_
��Gl5#������o5�|/ �P��jFQ��jFS�i ��N�[�H>E��jF�)�z�tIj5�hJ5��f4U�BUR�ECg��f4U�BUR��K��V3�S�������:�)N�������*y�Y�z�D���D���B4�S�,�-
a��BP���/� ���mg!�Q�_���0g�/�-�0EM���-��2��B�F����4��K1�����3+�VU�h��
v	�c������:��-���of������OE��D���)G�*e�|�����99$�����+���Ri�/z0d��xor^3��(9*d)�CV�<f%x ���;,���v5����1��p��u�2^��X�����c������G�rt�*��O��[�@��@
��*j�PD����;�l�#\+4���Q�>d����&�@��o�;��������6�v��S��f#���J����05x��s3�%����3� ���q+k<�0�`X%q��k��I����G�3f��_�?<�W�<y�Yq!Gx1*p��2����g�d �-�k�[���:�>d���
��<.@W�Z��%]nn����*�����q|O��:����}�������)���'������:�B`��<�����|q<O���/H�>Q�M���9%unT����NU�����k-����������T����=�����LN��{&��4�8w�Z�A�����,�p�
�h��������_��6@8�3�$���=����6�o`OA����)�7�9���;$�8����m�[)!*�;j�R�
YJ�����{������i,���o�����yx������qBT�R�w�b6
cSm�S4D3,�����Q3wAd��X!���(H�lH��;�6tLMn/�\��_RhM�������fY�Z�9$D���0��!�0A���xE"	#��9�K!�}�)A@0�E``�o���#0�"`��H������p�.)B���'��j}���x�#�/��{� ���������Hd^D=��`IH�����H�]������P�$�p�;���d�/�c����l�^�r����Y�������-H@��X���� ��&9��</��,H@�����WIV.�Y�������z��R�Z%���M:���$�]������_�q�2�,6���2,�$��b�i1l9�"�V_�T2h�kt�<�����a�F-<�BKR�K�����G -�e��H��G�����Z��a������F�V�c�a��	����6�1��	e�0�iv����]�B&�p
�:.���*�grk�h��t`f�Y��!�Ml�#F�����XVy��S�`�!)������!
��.��NQ*�&�Nl�Cz1�6OD�*zR6n��@u��:�>i�����I�>S��H�
�_�n�����<��a,��h��s!
3���R>_a�{���4He�y�w����9Od���LFk�#�P��k�eT��t��/��v����/���s������dIj����U�OKL��FY�-��k� �Jw�h. l���
G�����UV�l���+�&��k�]��`�$�x��!^�2�x�����+���J��m4L�$��On���(����Gg}��9�!�U���� �������|��#5?�+�T����i�1zs�p���c�������n�C�2�����7$��	P�'3	H	���,���Z�$lY�$��%�Y))*��s������ U�`���(�Q*|[����������-�g>�Z�o�/l�����Joy0�.|�[M��I`dt���}�{=�-p=�*�H�*'T���I�D�a_��~�Y��UN� Y��UN��@�t��r�� �C���IT�
�Gt\�)Ar��#:OgU�"�����UN��*'I�*�����HP���I���JHs�UN� T@	�*�Kd�W9	�D��N���$gSx�]Nl1��)���X�0���@uKV�xE�NI�O���e|����$'VD�}Is[����d05���.�0�$=��S�t�#a���H����+c�Zq�����G������J��;��)l�B�Q�2	�b�����+�u�*�R0X������R��*��S�u���b6%��)��2�o������#!kp����E��)i�2�%GS�+U0�(?�s�[~DYo��n	�b���������������i�yS�&���(�E	���'�5�[�����'c�p��
��-���%^<��	(�M�HR���'� ���o�?�������.0��'E�����h�Oj�A��q�^���B�L���W��	9;\c�����3����q��9
�	5�EAj���4���S�T��.I/m�sg�-� c����f�&��&i�xIA�/p��?s�����%�������5�K���J�5�J�2M���
����%��P�)����T����!�-�kN��������9������g�p���h�d�~\��rI�m�G�'�"La�J��!AGLa��y���������������������������e^���D0���H�x��H�iO��/7z�.��L������t�����B��KRf�����/k�����u���b���Z�� �%�Bn���S!5��[��T��T�vK����0��]C��"����!���#�A���I~K_�4�M�����i�����'�a.�H{�m����_9�t5�������9���t��O�YH��K�����W��C8�fU �����y�~�>��u��"#?��WI~��j@��&]J��o^P���$?g5 ��	�"�
 >��'I~��6$�x��p��:�S����p��>f��������6�1�.�+S����%��	(A�W
�}H	Hl^W����%0�����!�u�&9?1a�
�	-I������W�@Gj�ru�n��*�1�{�7SU> �Q����9�k��H8Q"t����w@�N�C����q�(8��E�+z<C`���	-IQ*�&�= ���Chs�C��B�CH��is���F��"��b1���t������7`~���)�YW�q�@(�%�l�!��� ��*����~ca�Z����39=��YB�zI��%"�k���U6��,�f��8%h����z$�����B�N����`"�����c�V�2�*+R6vgs�Y�RR��"��>�0u��|L�!cF��Ul��D�O��k�	����U&��T������q��B�d�A���:09CR�v���2X���F%i-1n���q���C��N�k�������F_�*i(@%J*A�F}J����0gxIm�����0���"&��e�\@l@���#�����x�
��,j��H�6���}t���?�[3�D�Z�N�M��2�����>�'
���A��!4/aD������"~�k�a�C)+��M�_R��(��oO��d�����E$�g�f�m���������T��lJ��	V[�������)O��8���$�QydL%�w���A�����
�Ap��V�py�c�D�\��R1�l��\0��Je�#�R;��~���������Z��ay0�.����8�S��\�\D�gN-&ior:O��},�R��k����y�-i��\Z�\D��n��!�b��'W��D���K���#�����4��EG��*`����U�<��#����|�N�%E��>��|#�����F��D���	A�4H�
�L��
�sm�r���A�3&�Qr:!I4��2�:��0��kJ"�EY�8Q�S8
���:R�)N������6!DQ�zzD���D���B4�S�,�-
a��BP���/� :��Td!�Q�_���0g�/���0EM�1�=5�7��%G�������T���������{�_xP�H��yp��g��*� �-!o bE�UL����vc�h��X���U�<����C��Lc����y�Lx�;7��+&~�{�a~���d���	=V����)p��}�����%���*g{Os6�Y�i��4R6��R�V�����4%���|�������r�G�,P5����%�:���Gz���V9�Q����T���	�����~��������������=O�p���-I�oM�_Q��o9�#�V���A3oi��r���[�00Gk]1L��w�R�����r���+�o�S��Gy�3H���/KR�=\1�[��������<��M� ������?�D����0;�.zn[w����PM�[�&��|�p���p�*Oz2t��kO�B��C�j/�����W�N����f�����t��r��$H�/t���-�M��1��h��9~�S�Bo������x�t���7� ����������m�{�H�8i���U1������=j�F��Go�C
��>=�3�G�h�#XU	���"GHUy|��1C�&����zhI*ji��ZQ3�z$������A���:�����G��m��m��-��L$@}���$+E%P��'�QK��h��Y���=/O��#�}��A=*$i���%����{��<���*�����o4�tS�����os��@
��5r9��t8����w7f"��� Z��>!d�l�r�
T��>
dM@y���'	0����V?!D=@=[>��k�P��ly�AT��FD?�"�D�G!�BJ�)��"�d���@M��U�P��H�7`�&�@��*�"�p��
(PZ�2�5!
�������U���e�A�$�^n(�����N�4����t'8�s������B����-��s�9L�	,zI�Y�2�D�%_XA��.$�@%"����� �9!����%b�zW�������w�q��E���Jb��pqcp*��|��o`2�F�����<�s����@�.��w���m�%��z�������K�aIvv��
�b��e�WVw��9g���yk2�A��up�[��c���������kyW=Jp�U����[h�N���|�vx=n��-����}�i3
��i�LS~S������*��wb
G�wb�d���{����:^N�����Y�3�;!�s�1]�����������C�;A�������~RP7�\���?��S<�a�m��)~Gx���������4�ql��!=V�/fop�u�Y,��}�-;�������3�;���7���r�&;M��f����[����3������+^�;�7��8{�w�v�vm+��m��M���8��}~��~�X��_�s{�����?w�y�70��n�ik�5b�v3/�sr�2����
7���`
��)���������x��w�>��Q�n���{��p l��[���
f�����b����!7��8����J�s�F�}UD�os���;����E���os�����'2�����'uSW������X���l�vo�Al�:��?Gq�v�d����L��-�>��O��x
d���sn�������"����qGz����.��u�]A^�.������v������\�^V{`k��j�X���k���������ay��i�o���^
endstream
endobj
40 0 obj
<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 9 0 R/F4 16 0 R/F6 28 0 R/F7 32 0 R>>/ExtGState<</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 595.2 841.92] /Contents 41 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 8>>
endobj
41 0 obj
<</Filter/FlateDecode/Length 2688>>
stream
x��[IoG���K�� ,��`���Do1e�9�E�P�CR��2�}��jv7�����>�b�����mU�>>�/o���2��������GW����l����y|�����]ys;-�������r�U/g��h��/���g�_G=J(��V��f�)�3+q<���z��M�z��G���,c�P��_�t��'��L1������:0��f�n|�V�_�z���/�����|�f��<����?�}�4����x�h�q��3��e��=iuqq����f����3��"F�ef}��@���"g<�r�!45�U�J����I�I���i������~�$+�lDX�l��@����gYv�=��gg�3�(�����d�(1��N��0p,��j��5��D3_��3��E��%Z��`:��}�O&��St&�)M�{4�'�O�!���):���y����H��l�,���~Q�y��������"�*�����A����G)m�Y��VH"l��m���q@(�&�s4��Yz����P?/��(�o��(l(
��/oA�Wm�]�RD�]�7�N�Z'u���
����7���[$�d���P�i����Y&��`���6��5�����I�1IH2�[�N��,2��%(������3>��P5��{�Mng�;����o��`��P��v��8�tdX��y9�^�r�Sa��#�����5X�}��r1JM���h�E�6�!�Y��\&��;,�!�
^ ��W�����n���>��M�~�6������kp����'8�`:������]��Yz�L!��^��b�����\Sh�d���'v�%�v�+Ac���g�Q���|�a�2�r>�dgQ[���'�7��h�x��*���p"0CW��0����Kn��c�w�;��Iw!��}z?��W��2�%�#�j�"2��+�0M������h�����	>���>�P�v�-[pM�a����_���QN1G�n�_���x0���Cf��sur�.PH����[jRmJdS{Xmc4�����Me����}*��	ZU^�������_c��k�
��i��}'���4�iW�fj�5
N��w��*��B0<���Hi���bK����-I�Mjy���m���r�������X�8�C���9�MD����t4/'��V����+�m����3�E�z8����1^;����7�GO��o�t��;+��,�'7�i]�'<�$��Tg"��%1�s�
Ix�1L��i��$ *�s����5����Kg�c��~� 	�1I��u�?��p�Ah��B�r�b����W����f������F��5���
 �G�7x0�����P��=����j�15tl�_OhrS������!���$��kL���..��"����%69�QI�������hCx��Y+��,\&-6/!���iBu�*J'?����`�|��%�uQ[p����C��v�{��Un��&��
�c��MAp��p�����~9����{������B��;��.]I#����{(`T�M��h 3X���SbT���V1�����Y�IJ�	N/x�j����U+�D�[����[�
�f_�[%3x�Q7W���e�=��v�iM_��vEQ�i���>8��8M��NUw�Z7�l�M!��Xj����@��V�6�e�>�=
?���Z���d|�m&(d�~s�����U����T�a1\�����s)�-�'u�����>��1A%�G����bRW�f	C���#��� ,'J�*&uE��0dm	O�����%���.��_[�m�3�&	v2@�-�7(oP�k��W�����x��`cB�����-Z6�o��� �b��hM�I��!�z�q\���U�E�&��c
��	��?:�w���0|"��r�U��
g9Y��7���5�\N� ��_���,M�E��`�����h�b��'k��J��q@��F���|������� �@xT����]��oo�9��8PR�T�T����}��}���������@�>@��2K�H)���et�&R���@�jD}�C�T"|_�R{0Nxt������J�]dx����~���IPq!������KKPA���T7[<
,����e��I`I����NXz�E�����'	!��WgsJ��e}�������v&�E���m����WYahUx�w]'o�Nh�/�N��5\h;�������|t��:�Z��+�>_��b�+E\����ja~���z�s!:����S�7�;�{��������3���_��V���[��2����TmgxU�����L�0T�0	������M����������,;x�w��i�H���)�PW>�%�����������/$e�	'�8��_�T�GG��2\�HU_�E4�����5*�a�3�1�wt��2�������K�#bd
�s��]h�$s���*�������E��������g�Q��
����Q�-���n���c����]���s�z�C$u��%#������O�Z72_���*��w��.�_�|�_C���|��N�o�����HX}���zqg�_]x���c� &��q�I�����w�~G���GH���#��e{s����}�����o�
endstream
endobj
42 0 obj
<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 9 0 R/F6 28 0 R>>/ExtGState<</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 595.2 841.92] /Contents 43 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 9>>
endobj
43 0 obj
<</Filter/FlateDecode/Length 2717>>
stream
x��[Yo9~�����+�y��`;�l���l�y�8��k���L��UE�!�����#7�����u����buq:=Y���?X��'��������ju}������������|�����O>�����������_���'�����
���
�a&.��������b������h��`B�F�������	�$o�fFp������cgK��Q����{���z���Y=��������F�`���F�#3`����!��X�����.����������\����6�iq�
a�Sy�l�;x{|���������v?�Q\����*.�f��fk�v!x"?���-"I�����$R������`x���S�|�[��=Av&p�	�X4��v`u��+�5�	 5/�������}u�Y�����5~��?j @��z������-�[�w�������h���kI���3w"������E�p*�������8v�,!����H�� �2=6�Iq��8=B��j��t&x�	��Usx��_�����&������,���`�O�X�����n�pY�}y�Ha��(����dV�-���]��/P�eyY���:].Q�I��}��G������4q^�4:����+���qZ�;�X�o�B ��t�����4$AG����u��N�z,��9y�x0��qjg�������c����1���V�N)��9f��$&��D
i<���\�8'�1Qnr2)���$t+�Vxq;�.�CR�#���MNmjh���5�MiP����A�P����Qy����'����w1�a����?K�5�����)�%y.����6����D�������|�J1�B�����oZB�cn:�����KX�AN��Z����e���Js�a)'v�sp�� �A�@(��MN�������3�{A���$��:��I69=�kopz~
�f�Z�M�\zi��t��q-?��������9>�R�xAp�~����q�VX��9]f+�>��sU/#��b������b6]R�;[����51�/q��.�.����Vd����X�*�~�M�����QrL��QYZ���$�G��39N��J�3P�������G�d:p���?(�J����0���L���Vvep7�l��(<���q�S�������E��0B�����`q��w{��b�'j��9qH��,i���:D�L���z��}�i�$W�6���r�R��F$�^��@I.)�Da/��M������Yi8
�+�)(HM��xY��NJ�5���{����f�Mf����Ef���f�65mOIhZ�������5�c+��|�f��\��<�(Y�m��$]�s���D�|1xC�!���^Bi-H��{����\lg�y���a�xG��?Zjn
���
w�}���2c���F��M�D��3X������B7
Wr@�: �P{��K��� v���!������2��*u(�p�
TJHT@O�:�^�����Xg=�HG1����/���-�6����� &�I�&-n�����z���B���dA�^e��8����C4a�3��VE2 �^E

Fc�i�!���.�#a$�U�v$u��
�<������;;��%��O/y �����K�m]�-��#��;���U��#)�PM����N���W];�_v���q��{�`"��6���������U��{q�{��=��
X�/n��'��a������j��t��~}������6)��MZ��L���DFN
Lr�o�$���)�������G�q�8��kv����1:;�8�(�?�|���U��B��^|o�����i��<����0�i=�%H�����L:w�!������}`�����N�q&��((��l�Dk��5O���`[��~@�j�U����/������U������K�T+x*ht���Y������@�mp&�p�Z�
�����J�&[������[��o�.����&��}(�!���T�G*<��B�b���
�~,�Zy��NPw��y4����6����j����\���Zu��j(e�������U"��;���qJ�~-w���t��8P�@,��jv��l��t�nw�n��Yi�������B8�i����8m4
t6l�'(��!�B�oopg��1��{]��t�����;��z�?�+|zR{�T����?�2�<�s������1�gFa���BP�w��7�������)�!n�C	2�|W����.�������}� �e��9"f�{LG�oq�E��	�t��v���Z���Q���)��s|zM3������-��"����m}�
�-�E���,o�l����^�a�����}�H�O��=M�7��!}$�j\��RI(+p���jA��U�|�X0)���,Z���c����	��	���m ��A(FB.�%�0��+C�������=��3I��U������NRp
`Dc�0��W������4����\�o��������'e��TZ�	.���;��AQ�����������v����U�h�
�H������� �H�Ip����=���*
���L�����������M����ooV�@!�������uM�&@�I��(�&���Ao�.�,�W�m������8������L���1n_�ms����������d�u~�� y��',J��a��;���Y�����M�
endstream
endobj
44 0 obj
<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 9 0 R/F6 28 0 R/F7 32 0 R/F5 18 0 R>>/ExtGState<</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 595.2 841.92] /Contents 45 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 10>>
endobj
45 0 obj
<</Filter/FlateDecode/Length 2856>>
stream
x��[�o7�n���~9`��h�@ �q�^����������w����\����!�!iU�5� ����������������E�����b1=��/��GO����������Go��o��������$=��[����x�������3������	���k��,f���?�+n��=�������@3/D�$�RF0_g7����� ��H-�Z?�/�V#]N?V#[�E�[q����)H����a`��`����0$��1���.�`^lR���
��LnTwW��2�E�0��a���;xzv��2�p�c<�uORg���*&�.����3�B��~���2�#�3���X��W'Eq�3����
���zh�dgS��!�/g���p^��)����	�E�_�����QW�\|ic���Z2=�F����z^�T$!�8k:LQ�-~�A������l�z�������MX���S��� �yf��!�����LdK��{�M��e�'�C`� �l���G2�a����C��|H��n4���3p�+pmX�\9�/>�/�g�n$d���~]�<%�[�Q4�G�� $�Ml���F;�^^�8i���,gJ|�1J!����1�M!���?����+�8��������r��������-?��G��{��E)������U���l��A�Xt^�J*+�DmCs���gm��:��X���{ ��(�a����=F�-S����(e����[�������kF)�RI��}��u!�W%yy��ktopnzN���J�T��
,T�
@|�1��&�����{�����T�P���>����D���E%d��������i�H�D|o02.����g��y�=�X�<�I���AQ���o�*a5�rI����'���p��4���47��� {�k+i]�t���FR�����y�{���r���%u�����T�W���v��/�L���������L�s�����\vGRD���~I�
C� �Y�5
�!���}�{�aa�A��rXK[�o��X���2b�UMO8��r!=��X?����s5����_�x�����pI<��S�#�2f<�6���8���M���G�����4dY�����G����8K���x$�h�<�p@0Ld�*�P�9R�(�g��!�'Ir�(�@O���
���B��^6+N}�p�h�������x���gBR�}�Ei��������t��lbbE�������������!���`\�]��M
�}���F���a2xp����/���C�>��^=K3��"�W�������������p_c�#C�?���*s�����`��v@S����0>���":���k�a�a��hT��ltdV�/q3h�u�Fr�~�oT�;��X8���EQ���������{���C�Y:W�����cF
`�^�i���}k��2���Y�XF�lt�L�Bq���C ��V�^���y��?��vEkc�i�"�k k�OK�4�?�i�2p�L�����E���@H����e��Z�/�����Cs����8���J�e$B�*�$#zA����	������D�q��1P�O$����xR[Ax�LO�b�cP���Z������,�2L��h���b������)Ms��w�0]V��Q�����0�����w��z���*��\_s���m@g����e�g
�o�e���&��&�A�H��p�v�NS��o��h�'�����'��DS��6�	�c�'G�1������?5���$���:�"[���nd�I%���7!v��uie4B[�)�\k����u$$��7=�j9���Fh#5iE�H�"�!!�uG��D��Ve�e��c����Z���}����~L+H6J��v)��k��K?d����	a����~�iE��N
��c��=$<����O:/L5N�vG�����7������@f�3��]K�sr�fNl?@`��1��q4�������������J*���0rz��r&m�(��n�E����^�)�{��{	PI����Oo�*$���~,��v�����/*H4�X��p�H�*�@gK�x���hT:���
t��~Py���%l��������5l����J�A�]��}�P�]p��p����=���^�v�������T�T�
(���jv�i����
�.@�N@�����\�4�n'�n�S�y��;���m�>?P	��s*�.H��������]l*��2)X��$�����rKnj�Cl
��.P��N:������@�HK�t�<l����-^�x�o*��W��_'����Ee���-�I#�)]����b 1�H��Y�}�����	����^T������{�LA��[������/_!�4�_��,-�.��tA��|�G��`,|�s���P�����)X�G!H��r�Nf�����";�mvmc2���0o��~F�yZ	_�|���H8���t[�I8~��^�Rc�m<����������}dd$��J��gdF�#p�nd����.LHP�l���|&8z�vt�R��OEc!\G.`��v�:��B;|�4r��:��YLi���;�z���UK}���_��$4�;�aJ)qKk��	<��o[&��+
�(��09N�2��8���xx���3nzN�{��Hj:O��x��������iz)Y�-��?H��S���Rx~UW���n
������z�V",�Z�	rh<2&�o�)��4t��M�q4'f��������mp�d��LGw[&�m%�����zX2��*��z�d��uA��1o|=EZ���C�&i���?4�m
endstream
endobj
46 0 obj
<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 9 0 R/F4 16 0 R/F6 28 0 R>>/ExtGState<</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 595.2 841.92] /Contents 47 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 11>>
endobj
47 0 obj
<</Filter/FlateDecode/Length 3300>>
stream
x��\[o7~7��0/f�=������Y����v��a��-��R*���e��s��HCY�M�iD���;��N*'����Uu�J~���t��.'�������j5������/�����fV�n������
���W����g���?��r��ku��IQL$VqV�dQ��}2;>zv~|t��'��\%�W�G�	O�`�PI����oa�Oc�\/���5��o�t|�)}��TZ]g#��I�{r������}Sj&���!~��1��h�9g�o[�Q���SLl]n��Ut�r^0#�k&#x�����O)����������YaU�%�J%���$g�����W��{�P�uDZ�m4J`h���Y���G�����$�Z��k��d�1�A&�9�^�6]��~ ��]��/�E�M�5��B�_e\�7�l$�Y�e�L~K����� ��8"�R�%��,��}F����H�������i�[���.E�
�p�76t�K�A���f�@���%X":0Zp�����cEB0�`+��EF�Q����2��u	����4�����m�
��r�
�9���)�Q�� &����d�,�yZ�[��.%����k�U4%�W6��2m�F��ZI2��V:�Wul�4�q���1Z1`��tO�4��e#6���F�^�������������Qw����%�)�����d�F!��Bl��L�>���	���Ac����^�S[vD�;��L������*�G'H&2$w���g�����
7����GQ^�4�=jJM����h%�/	S|D�`�!;���F�����~U���Ut5/KfB[_
s�xh�E}M���s��zX/3^:���5�2i}���Z���(�1�jS����j�c�#�K�9�KM��,:�&;��-�&���kZx����������U2{�J����2H�'�$d�t0�V:���W��o.O�[�t���l.��(n�
4Dh�Z{V���9�!�)��C+-o�9VS4����2R+��>�K
+��}EK[/�H���[�_�X���,��&v��)����l��������l�X�p�	�������������GRaR2�$�N�<D��="�6yhc��P�h7���f�E������'b��;��I�2��Z :�@��O�~?b�,xmiJ v�/W���C8��A���' �d�����t���/���X�#��9�t�zV����N1��$�������M�]d.nr����W�����S�8y�}��k��"�e��FZ�['�-��$=�Fe���G��e�KW��.G��W�p�RV:M����J$w$'��P#U+�NjH�`���6s@�s���s�g_ VmT�+W�Z�l�
��P�1�rZ�
�L��i�q�Y���F����~��)S�_bp�5�2N.�]/�Cg��}�������{���[W�/R�J[�yU�)M#�U�6���3$h ��Fk��D��P�
��^�o�	�#br���Re����$����E�.�+/��O
-0��&vi��H@>Ii`���*�����k�Ty�-�������;@����r�������~=:
�HQ�Pa~w���O�`�MP�)dDh�	�p%����j��"���6�;g�c��p�T�E#��U������G�s�G=T�bhe�*O3oN�]�D�)�}�l}Q���#��J��2�R����d(zH�!�
r�pI�*8z��9.�����n0�i�FM��
}�I��!%+���[.���`/z�,�a1����~�����������(��LUZ�+	�/0)C�w�����QV�7z��K��|�G��Y�B�e:�hW
�����v��H����e�~x ]����/�&��n���&���De�%��SHLLl?eH��_�9��������h�^i6�	���g�V0�mw(o��g�F�������
��j�}������
������X�.�u��==�m�06]0^��A����2e#��JSm�LYA��^��B��7A��4�����Q}[��G����`"��h\3�{�!��K�	W_�V�x�a{�sw�����D��,�=�kK�����p<�Wr�U�
*Tk�>e5����q�k��w��RQ��w���&�����K��������~p�\��]��/
������~�iL����~�=]�kB����O�����6���T	xlH7���R�z��u)����n}��
�j�u�c�PY�ot�w��,�R9l���@�
	�kO�v�&?�{0J�"f��;��bR4���/h��K0%�0��-�x�����A�-n��k��U�9so/_h���������(D�'RblM�V��kd��V������G�������A����t��9+{A�|k�Z=�lrz�c�a���S�`�>�1e���'��f��^�`�>��f������N�
�oO�v���<w\���
?8�_C������8��U��n���tm�`h�By�6�R�:'O^W��$�g�����v���A��9�:�-Jd��/��tUb�1	bt�|!��]�:8up���u�RnG�}�tA��h��<UrH%����w'I��S���t�C�>@���Z�j�P�P{����yw�x?P����<R���7�;���M"?����}���O�)��r��\G��B	q8��I��L)��Q��0�H�S[��"�G�(�E\�w�`�8��Tx,�',Q��U��%s���q���#��rQ�/.��_&.%K�$�W���0�u
��I<�����9����'kO 2
��>n������]�����Ifr�������X�cy��K���q�eR�o��x���%�s�����JC��xj��3jG��+���y���)Z	m�7���Ap�����;�Q���%BO���N?�& D<��2���ng����c7U[�|���a�[/8vx>H�|�����.0"���My���5�Y�m��U�����j?�GQ�o��3���5v
�\����oV�%�U@XS
P�:�.�>2H�H������c�����q��4/L�t���S:�A�p�����dZ�����,Q�[>g
�yW5�(?d��p����Y'���!P���+��*��bI���M!
��*�L� n���g���P��a��"�-���O���W���,��r^����e�!�����4���x�Ku�b�k�8��q�f�}�]��c���	���d���.�|�s�����pd�-�?�{
endstream
endobj
48 0 obj
<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 9 0 R/F6 28 0 R/F5 18 0 R>>/ExtGState<</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 595.2 841.92] /Contents 49 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 12>>
endobj
49 0 obj
<</Filter/FlateDecode/Length 2892>>
stream
x��[Yo9~7���/t"��|e&�$��l�y�����+��A��oU�}HjG�	l�Hb���X���&�{�����lQ<�w�XL����������������������������no����^��/���������������P���d��`A����?~*�vwOww�^�B�uqz��#@��p�q�#����[��y���9�\\R�����;���H���jd�iQ�Q����s�����40�L�Lz�C�t1���S����R��1���S������F�o.0��������+�`����>\%����O�������	�m4���L�<�_�D�fI�����F�������{�����������������:dh��Y��u��4���7 /�P�^V�|�i���������
P����+~,��������(VyQ��1�=��2s"���b��"[j�3n
�,s>!��Af����������L������}���:�[�,�����.��~|���H�?Q:+��)���D3���C�g���F���"��0��-��n�}P�H�e��&�9�L���d�?���1(�����}���r�g�Pr7�d~�R3H�����	�M���lr���&49������l��s�M'�i5�b{q�`�cKT��B��|��:���1���)g������}��>�����$��L�J3	
&�OTM�u����n�y$`��{���Y��_�x�"�r/"�y��XD�-m��k��g����.�SL�k�l�~��]J��sMw�=t�Y����]����G���(y<KT���
E���H��Yv�;x2�?bb��}�VWn���fm��r�� �f��c.��h�s��{����@��{��8�W�u�E8�w��?�������9�y�v��F��<�i��2���*��}�����6U_PO+R0�_����$!L����@��
hLh��*b�-$��$���s��	�pT8�N�	O?J��#�<p,��$b<b��	[(�%u��~�����u�#����i
�����{\����q)�����?	����*xu?d5]���L�j		��0��A.��aHI? �1`�@��$*+l�rM{4t�c<Rl�&���,v��A�&��s��6 Nr���X��s�x�����v���>��G���i�
-��g�z�x�:>�mmW'9�K+�l������f
�]��c�}Xea��rp�o�Y����|�p=���?g�*7.)�E��z������K�}-������k��8�\���l&��`kH���y%C,���*���*;���'�q
Aq�P�A'�1������|`�oo���G�s�U��f�0R4���U�i���k���~�<��>��A��t��'A��� �fn��(RX&6ZkHc����_ZL�
�79�9S|�oo�]��_���J���.���gVj��n�a��hDAG�f}W#
Z
�9S���
`}�]�(�h�[�������.�$�hx�|g��+<hm5���!5�v4�4��;;R�D�ZI����P�R�`U��
`�Y���Vc�t�������f�b���P6i���)v����/	�*����C[���|��%��&@De�w���6L�!`�i}���8j}�U#���:����%�1���U0���Ms-������i���m]N�7�~��[j�����5��1	*[,��16�����z���-�GJx$�V"�H���]�J��R��\j�;���@)@s�sJ�$��bl��Vp�
b���I���Z��R�R�DdS$���6��V���M:}��>�
ac�^#��H�G!z����)}�Qk
���$"��t�����B2�l9hSVD��h��.I����Jv�� ��u�"��{���i����������'��NN��Y��,*.��4%Nv���f�$S�����i����>�M��J
SF�x�Qc����
!IX)b<&VdfM�!tZ�1���h$P��7{�����}������]��n�~\���������t}����GM�6(lA�J{.(z7�~���n�������G���2��~�yd�6�������A��TfT9��T bC���P���X���z T�-�w�`B�[�
O6��jBu[�
+� ���H���z�x]��B
[��+0[|+��0���<�x�	����7V�!X��Zi��e��zP���3�R%%$~�����R��8���A��L)�0�	@� ��_��yY�'�p�4��r�(!rQ�D�$	�Y�#6z�xl�zJ���������r�Uv�;�h���Q���,��l�ReT9�7g����u������W�����/8��JH�-Ty����+��8�%�������;����%y�C�,����EEW�XqJ^���A��Q�R��Jj��:��<��	��q�Zo����/�,�As��
nm��x,
���
�-��=^uV���^��'T>N/��4����	6�K�'�F�.Z��$��U��
�}l<0�'|d���fGA��� w6���m~+Rf/�dz���Py��fPn�"�G�4�=���c�:�<>uxq9� �@���Y7�F�an�V{���@�]����-Y����{>C���x�k (��36-�l���'l�4�9F�73x�����T����K�Z�i[wV����d��0X���<�)s�/�%5���I�� Byv5��)�7����~��X�������[���!:��{)����N�j
�^Xw5�s��}��q>3Y�����X��������{[	WN�m\B��*��?�X�������Z�*����E��nV����z��
endstream
endobj
50 0 obj
<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 9 0 R/F6 28 0 R/F5 18 0 R/F4 16 0 R>>/ExtGState<</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 595.2 841.92] /Contents 51 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 13>>
endobj
51 0 obj
<</Filter/FlateDecode/Length 2751>>
stream
x��[mo�F�n��a�@�z�����H�4rq=$�������YE/��ofv��l��4�����<;3;;2G�����|��V������}8xq�Z���vp��a~�nvu���n��?>�����~5_N&���c�yOp���&�
�+V��b����?�c���g�{�$����.��$0&�W\(����������gW� �]��N���>������r��������N@���{_F��	�����!��U �{���?R~��>;H�����?���~u�����K[\�n]�$Wv�u���������9I����0��4�k��Lp�C����;�;�J�~�5��h+Vv�����w�
��~��_��}~O��
\?�Z�bYz�:��+���9 
o���P|,��8�������|����dzQV�����c	R�x�|��,O���~<��Y���)]��;����Mx��S�f���k�4���!�&���L��������k�}���}A�!�|SC�(�� T����Is#�4� �F�>�tz
�}SVu�� .���3����A0�-+_�a�c�\�M�8#��s���� �}xV�k��{��	�N��~Y��Ray�`?��79�����h�4�5�4����D��2�8@����)���]7;���
"���V�����%��h��f2�i��]��uFw����}����������	�wl�5�F��c4<`����bg�=n[�T�iGU����Je����������#l\�X�v��7TQ`�_�}��O%��}B3>��/`��$�(X�/�1;�Y�uR[I,*�)WP��D���*������s�,�EV�A�X��]
���C-�����.��D�qlNr�"m��=�!o��(s`�w�?!��t�m��}2���KfG������X�=_��jK�]�9�4�WA0c|��
����{�i�#2����C!���'B�ZH�rb����$���0G�_�h���Q�")�����Y[;�p�3B�,���r��wt	Y�=��L��%)H������DE���;�'����+x9�pA�Ld�:���R�d��Ah���Ir���'a�U�V��kn��,���U��9i�<����IU��L����'��V���0>�
X9v��M��Z�/����5��cu����m '��rtS�|�����|	�&p�[_��~��4��>�t=�����a%=R��nr��p=�[���J�d����8Y-�l�6MT���T��/]
���idge������6�4p��������,�2+] �����TQ��krWsz���D�c7���_Y�� �@V��L�5��VP>p�F;�
$"E�z��q������k�9�.o��W}���LR'����#�zfp�/y�-�[�'~�VR~��_lO*P��w��>��L&��g#��t8��)n�:����������G0\������B^uu�#:V��;4�����=lY����6���D��2�q��k��j��eXS��h�0m���?^�{��m�}�?GP�"�[�������M@h�D.�)����cV�T�\V���d� �;�1�q�:��^4��y�R��u�aI�����������|=�7���<�"@*n(	�:.	�c���m\b3������;�I^;uh�>����� o �K$$��������SYh#�4r���L�h"�$����A7���,4�@Ms��9�5�}
Ttt3�k�����������1�nW+V�h5����6y-\�;��U9zg���Y�Y;���!G`G���*�49<!I������������m���� \��d������Fkg�u@�"g��������'�u��Z��H&-GF4��i�%4>�wy���;��6����iA�v�@���f���Q��k&o/�aH�F4m*���l����lq�����eZ��Y�C�8�B�C%�]Oep�>}g��v����%�N�6�UE��*Z�&��� ��T�X�u�6���1��P��T
�v_��S����iLPGG��@���3���A������A@���Z8�6m�5�nN�3�`�!@� �~�SRCFk�F����3���8�tG�
X�
j�XT��X�p�C=<�bQ��c�8���@���t�s)��*�{��d�dk���A��nS:���|�?~*�$&%�:.k�~�JG�����t1����i)�9	$&I#+��N��N�G��)$^{�mRTs���*=�����`��r�M�"����_q��4
P��>/et��������k��L�",��k�6K�NKi�U�J� �&3�:Mf7��`����k0��PmcGC7_[����C�E)������G�	�_��Mr�������m�q_��}T��-3������d����^��+��G���5UN�n�6��m���o���s���Q�GQ�}	�����h���p7v�z�M�����I���0�*�0�yzm���C�m���\������$!���3�R\��z�:�H��N��v+
�h��xA�
���[�E�'@T�&�:A�{��%�����h1NR���.����7$��U|�)�aV�K2���R��b��"����6�Tsk�m��i�����1��r��F"x|��r�&Zv_3-�r$m�~S#�m��mq��R�;��G��u�|1�����=�m���g������!��gJ�V��4(
endstream
endobj
52 0 obj
<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 9 0 R/F4 16 0 R/F6 28 0 R>>/ExtGState<</GS7 7 0 R/GS8 8 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 595.2 841.92] /Contents 53 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 14>>
endobj
53 0 obj
<</Filter/FlateDecode/Length 1739>>
stream
x��Zmo�6�n���_HEM�o"���+:�[����XN�5v*+����}w'�-�&���c��=<�=�B�W���8������u]���cv<|1��������U9|W�]L��b6���X���f����=c/^���^�3��\�Y��7\2���Ue���#6��^����O�	�3�F�~O���	f%��fF�>F�0��{����fvF�\��u�w��K:)��A��,��F?�{�����{w#;���K�����rk�'B>Dt�Df�R������[��6(n��d]�B
.���]%������[l���,W\h���\Z�qm�w4|���e%��(�����^�=bl�������,���k&r��.�c]��g\Ix!(�t���wZ��,�d�
����|�F��B'�O0�.�BK������-����i���e[8dt��<��1J}r�*dl�T����w
O���M*�E�i*����,*������,u�
K��
�V����dH��������[UL&0���|}�����TU8tSXU����4�.~|�-�[������E!�Y���<�s`��3o=�3��e
&�����2b��s��w������b��7h���������!���H�Qt���>t��K���T-�5d��a6�!�]7��N�7F^����<�<BCK#�\����
x%c%5\|�.]y0�������:(�1�)��=h��v��q�����Z���)J����
4g�����L�`m:Gl���:{0��y��t@����r:��6J��dlr��a�ej��{�]Hp�uz�uX�d�7��j�B��Cl�mAw���g(dv�+9�qG�}]�1(�`�5�6 ��Y�&
����z�B �"v�������� $X��+��CWp�n�J��+��d_��@�����p��9������Z�q�00�X����G�<���
���.��Yt�4�bo1
f����*���^���6���n/,�G����k�'{(�%�&������EH�������UIU�h�-�l�4zXc����p��
]���fV����K@���it0�_���^@�I]�U}1��`�
jsW��D�Y��������'���m}l�N(vmd���R��p������I4����VD�M�����/���b����`����!HY��������d����72�(����}P�2m�3���e~�������{�����K�!������Fsk0�L;f��#��8Yu�%NQ1���X�3�<=�JL���8m�D�C�h6��UNW~����Gl��4��!l_���%���J�n�2:���f�dx.�~����+�j����wt�eF9:��Hb����#����-�r���x��W����W��F;�1��NE����FWm��:v�����q|�!�-���?�/R2r���t����$)ZJ]����y�}N�Z��f�������&8�����b������A��2�6\!�d|B����Lpt�]���S�s�����z����z���z�GzH�������}S�s������,��F�v;�K�5����ly������8B(��
��ym�~w���M������~�M;�����>��5��?���Df�[�-z�
�����u��J�6���������m ������_��~��y��( }��_��|��8��T�v��>��4U�
endstream
endobj
54 0 obj
<</Author(Smith, Peter) /Creator(��Microsoft� Word for Microsoft 365) /CreationDate(D:20210730152554+10'00') /ModDate(D:20210730152554+10'00') /Producer(��Microsoft� Word for Microsoft 365) >>
endobj
63 0 obj
<</Type/ObjStm/N 500/First 4776/Filter/FlateDecode/Length 6009>>
stream
x��]M�%�q���e��f�0�H%`A����l'�c2����K��4��z�ZH���&��������*���*��(Wh�*������x�_����W�t��J5_5\Y�U���]WI�*��W��c��w4��jw�WK�Z�WWG#tn�����.`��x%>�\
�KJW������
1��c�	����J�pi�������&��Q�}�T/@���z�BG��F��W���;�_�q	7�eh$�p�Fn��pM��h�.I�B�$G��/)�_�K)y�Yj`������E
�h}��%=S��p���"�!0O�F`�(�B�7\�0L(B���H��$KF�Q�X	o4L��
���;lN�����Z�i����� �M!C(��])RcP}��F�5��R�]��k�x��Tx��/U�H�*5��&���J=Q��|C��o��3�(Z����c����sb��='P3�09�1���� z.B �+�0�NO�r�Z@���4�4w���dv�]�M�A�i����@
��E�:xZ!���%P2Aq�%S���=������K^������:�E��K���_I"8��w
���;�j�C�K
tj����f0En���.��������b�S/F����	-�b���%7#C:b���nh]�M���F��G2���"�	��9���! |d\(��V�A���[��q'�q����8x����4�oz-��B,�:|B�W���gt���V
B�WA�^�����W�T��*N�KastX�U�����f�1�B��C����B1������<��M�A��qIFD��S�"wl��0�E�%��Ab/�
B����d���B_�k����e��&����n�<7���Ap��e�IpN�/��C
���#�-]�(1�G�_F���g����a����G7�P�L��z\�1t/t������F�e�)J@]g�����o��	��f��D��X��7y$�J)1D��2��`d#F�#�3\3�0�!��hLk2tJG��S�6�b��a���0�����#Y��#Y�8�?�i�D7�?Y5���
��x3��H���;16370y������?""��Rav��	��M�_�dk��%nJL�("�y�9�q-%X4bH LW���B�E�;J���Q�P�}E���*%bt�bb0�E�Z��5�M������
S)�'T��K)Q���Xjdj��<j�y)�qd+j�G��G��[�D���K���Sb����8���/�oa����B�����k"}�Dj�>QjLf�q���\)�0����;(���H?G��F�������1zU���%03��+�qy�Q=��+=6��+-�m#� >�e��%mD�Fx���Y�}�"
�ol��
�t��F�����1xJ��L��>���G�o���>�����<Y����?�~K�$}�er�������m�����Y/�2q�Dd.H�h�R&FO�n�v���{�TB�����:��b�gZ��������<����kb��#���a�2r7F�F:���4*Z�/��;������kgH���
�&F��������D)1g3j%F��<c����v��0�@i����Q"Fa��X�O�����{����n��#1�YYQR��'9�B�N�/'$����C���R����~���5�gBP?2?��!���N�c�� 9���v��Y�gU�YC��Y��3
\V���
�K],�����,6�GJ���3� �>X1 P"3]b�!���,G����$\N�g�H�F�L��k�p����8[a����8"�L��g]�\��,^$��Hv1������9H)X���;�v#-�(���Q�i!��4j�)���N���������N�L,0��pX/�ZoL&8��Dk��Q�pJ���Pb%�1�4*��1W����QK�SJ�@0�DNw����+=��J��g<H,�:�QE���b	���X���SB��D&U�L�Cr����Qcq��P��[b���N��F�ZH)�$DLCF�����!q��k�H�=�.��2j�1I������#�����L^F-�9���#k`fC�F���
��?�tA��� q5��](-�Q�%J�5F�3`p%�)0Kw�!�J)���7q~�#��W�{���#��}d.D�����*��>=&!���Y�~�hcH��q�J
�����"'��R��Yr&I?oc����
23v�1R�t��2}�q^�G�Xx��}�Dp��i(���	��x��B���N���)�B�9}�s�!��o�K�����L��� s�#mD����9�
gLr�Fh�������{����;��L�G�����{�����q�K����r�e���g��j�'�U�=f�c1��Zm?G�	��,{������1�#��/>|������w��������������o��_�����������_����
M~�����?����t����7���\/�t-=:�����p	�[.��c��z����M�v�Y���wZwZM�a��l6�*l��S�����I?��p�h\��_���~���D6���7\�w�|���1#�g���������*�A��C\�UHR��� o����-��2l��i���8�����m����������v�m��j�o[Gx����C�G�QGLb:B�:b��bU_l���_]���]���S1s��*v������k���X::z�J�`*��>�J��c��G��g���%g�a���A2�t_cm�����G����!���r��S)k�Jg�����k��-�;7��j���Uw��;��3�*�>V	.�>������u��>����Scm|T�
R]T%\��_88|*4����$�2�R���fJnZ?\������gN�A�����V��e_3q��k|�X�MbTm-�a"NSF+L::�F��n�"Zn�P
���VK����j�7�}��_�cM���iD��.�i�iD��.�i��R2$QY�fdI3�$���������Q��|�z����"�U+?�(9&;�%�%����0R�I<�z<{(e
S�����k�g������������D�'���=��4��� ����	���#��g-��@
�6�ep?��FQi����W�rp�����}48�4�'��I�z���4�'���U^+�Wv+�Xv+�Yv�#Z�#^��g\�3��������Cu������j���a���6�����gckb���Z��B[���[��(�uZ)�uZ)�uZ)�uZ)�uZ)�uZ��M^���6y�&/��e��l��s�1���.:?���?�su��eaX}���|�����f��l���8HwT�z�����DT��i�8��**���R��i�8��Jq.���\N+���V�s9�4^�������������4n�x����l��<����3�O��q~����g��u~��9����f�>�	d}����������V{_��2�m�0u�H&�����^!o�*�ky�b�l�/�9��?{���,/��������,yYl�#�c��j^�wo���|n#�^%���b�Mic����k��g����iF��������J���D�������R�L�M����='c�f���X��R��s�{�e����g�83L4��Y��+c}������3i�n�^������/zg�~3��5����N�} t�c��~���;����F�6�E�x��m$��'�%'�������0�fS_����
�bA��z�{1���V�Q�I��1��2������}���(s^Y���F�dT�;�d���A�b/
�]��>��S��T����hP�����7�W]P�S���s����]�6�s������Pu�	��~�5XTM������I$��8$R5�TM"�C����I�q"��D�KyH��H� R3�4���Dj&�x��H� R3���H�D�&�b{.���D�&�x*��H����"��"��"��"bn����b�
�+������r��5���[���K)���y\����
��v�F��,=��:�
�X��v�_�F���y
4��vBY(z���T�i��u����.b2����\�x/���x�,���v�^P�E��g��9E'h0XL���2sA
�E��
�����'��,���!o�A�`R���v�(��O�c�A
�����}�0��A!�)��r�[1($&����!o�Hg^�17�f�]�oe��$V�^�t�sVX���e�xqk�{Y`)e��h��(��	���1��a�W�Qk����[)��a|�U�v@������1�s���>�eP�wx{����EL&��`�������P2)�r���U@
%�Bm_������n����A
%�W��n�
���|��!o���h/��}m�#��W������Y���������]>������j�)p?���p��{L��!o��{���)}e���V.Zb��JWt[1g�����_������R��E�x������{�X����e!OC�[q�'�b)czQ���{<���s�j�1�����k5g�V|��4����x�kP��TOv��T�6��=B�4j6��x�!�A�fSI�&�!
*5�J�'T6
}X�J���2;p!
5�F��
u�F|x��7��4�6��|Tew��5�d/���6+��>�A%{�'���C����|�t���J�+C���b�
�+���b�r��9���c���
K-���	����b�����n�dL��[>SW�a}��T���1��������������t����J�p!-�:���s!��G
6���ce[��
��C��8�CX�J�/�CR&�.�A���xh<F���RpW:L0�d����e���+���RR�.�A#qht�0b�H����c��'�G�<��a��J��>�3�!����d/��i	!2�X��J���W�]	qe��6V,_�uE��VLX��<g�yql}Ya�EQ�]4��y��eWC)E�^4�h�y<���U,��)�s>)_�]va�>I���(��>�a�d�A�x��!�>I��So�9�5��l:��e?��a
:%�N����M��n�Cg�N���x��>�����M'�aB���a
:e�N������s:F� v!�A��P���8d����M����C�J���x��!�����,����2�X�wK�G�_yx%���V�X�|��VZqa9���E���e�e����}{���]m;�#�1�dc�
2�;�x42$� ���Y����G�x��.�e[{>�����4�"�Q�0���t�8���g��AJI��t��3��r�WI|X�N��>�An��ua
:�GD�X�!
:�����3X�HC��4����CT�&�D�X�!
u�F�������M#�xG�X�au*�A&,�su��lld{�O�e�����$�����+1�L�R���+����B��	�Q��,:/�-�/+,�(*��E5���w�����/I�q�!�gd�mu��`K��c�O7�1!�l������l�o�����}�A�]ZR��d�x��"�.�kP�>�0�R�������S"�
��4�9d��#QyB�kP�>� ���iP�>��G������T�:����C4�F���
E�Fq�D���Qth��4��hP���k�E93��T��$�7K�����t���J�+C���b�
�+���b�r��9���c���
K-����<������h��d��C�=���l����[�8���c��������C�}�A���>�eW{��������>������K����}�A/�?�5�d�����i��>�G�rf��5�d�ec��4�d���D�Xua�C�>!���i��~|�(w}H�F���� 	Q������'�����a
*���|L�(�.��������b�C��U���W�]	qe��6V,_�uE��VLX��<g�yql}Ya�����?��
endstream
endobj
89 0 obj
<</O/List/ListNumbering/None>>
endobj
102 0 obj
<</O/List/ListNumbering/None>>
endobj
109 0 obj
<</O/List/ListNumbering/None>>
endobj
118 0 obj
<</O/List/ListNumbering/None>>
endobj
133 0 obj
<</O/List/ListNumbering/None>>
endobj
142 0 obj
<</O/List/ListNumbering/None>>
endobj
153 0 obj
<</O/List/ListNumbering/None>>
endobj
577 0 obj
<</Type/ObjStm/N 500/First 4894/Filter/FlateDecode/Length 5984>>
stream
x��]��,�q��?�\����a��fgh![od�0����<��3���sK���t1Xd���N��q���~���}���ue��W�J�g�����W��,W���j������n��k���}���u�A�|���t-��w�v�g����WN��n�HF�@�yRW.�B@���N������[�� �
�^�
z;Z�Z�xb$�<��J���������|�x���f#cD
^m�s�&
^n�q�V(t���Uz�0!����������GA�sR@������)���Q�US��!L�����JaB�p���Z:���Z�*��B�j��6�����e���hy@`��,���LP]
F�n����A!_-q����
��8�t����i�=����v��ZO��?u���x�6@�����@���;����X����ms�1F=��1g�3�A���1�=o
�����L�=Gf���+���8�����h�]-O>�i�sRHT�iV2�B!���NRRh� �t8D��9���Ip�c��+xX��d�:mQ�|'�W���_��@����j]bW�v3�83���Q(����	l�����@��2����g�n�3���49�/9'(:��sR@��5�������WA_���<�a%��_W.���4�l;6��U;�r��1��o�I�]�����hM��E�M(�b�T����-��*h�0�k�Fv��51�;
��0@&�Ia\�@�0Z8ZP�]A�	E�-SHOu�}��}���'�2�z{U
�:��i��S�iJg�@"H�����t��iQ��K�=RG	��S�&�-�3N�f��zB�``9R��@wb��j����N�3Z�4����qvbl�rvb�M	h/�)��$��Y ���t1����N)S"��~����;G��DJ
b�3f|��x��E��n��d{����B����B��H
P��h���B�c��!�=N;���i�c��!�����;%Rb��M��M���8�������>4�,UJ�k	�nBoa�;%`�U�xgH�0j%#�6k]G���6t��S?�9A�����+��;i��g%�ge�s�Y�Eo�X��R[��\�ZQn+�Bz�VN{x�U�����=Zt�0�G�!���t�}�{���[���21�����`L`(
J����B��
=A�V����-h:(��%a@pl�	
&�
0`]�	I a�T5w:�}��u�+�i3��{���{UbPuW%G�Z���#T�qzZ��'
����ixs<�oN��(%�,|s�4�'%>�i�������z�E�����A�]����^�}u�u�C��>4���8G�}X]��N�sD��uD�������s�hq)����07x_����,�>LzO���K8���JZ�E��d�"����AMY�s.%`����|��R�Az��h�=�N/��E���$:H�oN�_��:}|���������e��=����6�A����T)
�`�|�L�����T�G�}�
�uz�R/�>��:
I	��Q�7C�E����uzGH����E���-�GH�83��)t����C���%���)pVCH��
��M�GiSB,��z���
���c��1^�0���q:��i�y�=rt3�Nl3@��R.W�o��>3��;�����CI����F�R��y��aj(1No1r��S"��]��0mb�]�A��1�C�c'J��`�fVNO�_H�F9=��~��W,��	�]�����B[���YH��`��������h���1���z36<^cC����� �� 	��wc{�$�y�����;W0�o���i|s���kP�����k����=�����<�(AbO18�\$@�����o>�NJ|�������)Q���%`4��
���9��0�I0�H��I.�hg!������X�����m���
-������o��.��A*|s��b���������c{��;�����'���7_�`|�71�M�Q{��oob��!�r�s����c�qz
��$�c��o�y/�$����p%G�j�~D��"M���DsT���Bl�	E�����>"�NH����x�D���r����A�d���q�K��h	`��fg=���\*9|�ivR����E?�q�<�cg�����4}n���qUZ��`��>��.����m�V�. H�w��+���e@A�*�����L���\���IZ���cT{x���N����n�k�!���:�#e�h'���8�D+��y��m�3��	;�!�Q8�b�x&��v�@�]�N���Yd�����bs����A�c,h	���M�^��-6m!D�a<������FM�<{���k���9b�x(	
�����8Z>_h?�{=��fH�x�h\�C$Wt��#�Y��Mi�~"�h� �b����M���o~����9�������~������������������o��o�4�����y���o����z����ww�����������z�`��v��0��4����>l���!d�MC���]@	���y;����
��
*mM���AnA��iD���CXA����*�T���p2x*mM��B�4����|�0L���F��3�����i�hp?3�L����J8�33���Tb����L�x��U=���s<#e���y(sf�����3d6��4��l�I�Y�a�����!�b��<jZt�0���V
{���~�r�2�Bl�H�!�x1['a��G��ukH���P[�[:m��a���H^sy+�T4��CA������x��"�T4�@���
z[���Sf�������NU��03EV��j:AjyH�*(U5���<�T���R��ByH�&(�4��;���RMP�iJ���!��O�(4y���� �c�1�n��,��#����AFi��M���
�3d2���1����	K� � ku��o����H�k�Z���!dk�.�"�:�B���rR���b
��8�%���}��+��}SAA������.���>���!�����i��q }��O{h}�����i����}Z�)�3�<�>S�g�yh}���������[�ax�^;��S�_;��S�_;������x}o��7^��������{������|}o��gi���O�����n}��M����r��;�c��jM_�C�������h�[����M��Z�������W��|
����!���������=��r��=��*�>3�T���{r���_$~��N2�g��H1�PZ�v���U@J�8������L�c+^�����Q��:m�-�����ov�JK�N8,g��7�	��2�����|BM�l���#�|�:���f:�����:�p~��t
P���b<��A�EIH�{�,X�5��s����HY'������HY���#����HY�5���!� R	~�q��D*�H%�}^�!�� R�1��C"A��9\�!�� R�q?���
"U$��>$RU�RB��/�k~���9��fV����1{`*jZcD6n�t���8�7�����`V����cV?>�:��b�{%�����2��z�{%�A��K,�����M���%�� :F,Q������d*X�$�����K�T��Iu�Z>,j�EK���X���dQ��.X�%�N=�CP���Y���]P�K
���:��?z��������?�^ho����:E�A����`��CP�EQ��r��k���G2'a����8�9fL9M_���-�n�g�dh8x�������C��:�03���~��Xc����%A�C���+BD1�:�0���BgN��CPA!]2�*�����KPH,'�	t����'����,A!��*p�

�]��������z=T��?�.�b��CP�]&�����{������>�X��@Y��t���0�x�>_^���"�f����u3�cf�����Xl�����AqL�,��W����J~P��
Oy��|Bu���
W�k���A*a��J:����E����
���/��KWd���U/�L��,��)$�+���r�_���F�: ���|���EPIWd'GC
*����$��%�a�te@��K�R�HWd'-C
���r���d+����
�y��IM���Jz���;��VPI�������'c���/_o���y(sf�����3d6��4��l�I�Y�aq�LF���39��U����"^l:K�����'���j~������s�CX�����Gu�G)�UWd'�C���+2	qr�1����
�A������]PIW�z�j�!�te@�<W���BXA%]��dl)���2�is��1������Y2D�C�HWd'�C
�
�����p+h��2��kM�T���<+��;�K���\��2X����H}mM|�_s����m�-7�j��L��ST���q�&�f���24������6E��	C�D9��O��L�5��kg���cX��z���C�%�u�5��N��5��t�)��1�!����t��W�Y����NK��I1���JKS��:I�v:��N<��1�!����t�q�N�6�t��N�!�����Tr��1����T�)~N�6��)��N�J�������{���z���[��o��/\�������.�����c4Oe����X�xf��.������c6�66,����q����G���5;�a���N���L�*�T1��d'5�������9��^bX1���!���$���"[�1�s��A�a�t%C�8����e���!��V�IW3�sR�}��
:�z��$VcHA']�P�	��=�t���I�����aWI��$VcHA�������m����5�����a�t����2�T�i�Q�Lg+�QH�r����1��2�a���Y<3AfLQMs���1�t�g�d�������M��lQ��ua�!)N���3S~��uiC98�������Jv�G)J�.m(N�6��]�PN��}!����n��!������J���'
!�t�/7(���"XQ��u�Cq�1����q(<��I����J���8��R�hh9��R�hh�s��3X����/Xp�1���N�������J�E�"��t���wG^����9D�P�6���q5kg&�l�)�i���8f�n�`���y����3�;�$�t8i�O��3%jvpIoq��1�X�%8��1�����'KC��Gpr/Rq��1��RpIojq��!�(��E	N�n)h\���e���VP)�(�I����J�%	�;�I���>���O�Y3��N)����~�����J��K���d:7bI`�����������Rs2�_V�������k,RaX�*�y�bs������]7Ck�����S�����f�{����o�t�{��<���K�'�j5��Y�;����3�7k~0y�n�����^A���/Q��p���|UI���$��N~�_����$��te��#+f�Z	^�W���W��t�DqJ~���X��&~yw�w#���s��D~�E�l�\e(Q?������Y�t�8E�_AU���'��T�*}R���N���b��\)��yG�S�d�� m��P���� +f��������Eo����ow��0o����o���Ro�����Z���&����y��7�:��N�pNM����J}d]`�K�������L�(�8�
��x���.1�Z��089����9�a���[���8�u���z. FV�
neH�[�~��dQyq.�U�N��+��Y���������k`
��9�
�bWp[C�E������K`���?��(�8K����#+�W8����a"�?j��������?'k�
endstream
endobj
1078 0 obj
<</Type/ObjStm/N 500/First 5332/Filter/FlateDecode/Length 6534>>
stream
x��][���}_`�C��}�$Q7 ��3�c�/�A&���!�	H��������E��<���"%J�!EU����4On.����?��yE��&/ $OaN�$N�92��Ad�� ���k%���_���A�����U����^�T�B��U��_j�.BB�?��[��g����o�R*ia�H�G��s��V����MR���G�z��3��>8N�V�o��T���,����g�������Hu�J~+��H��JU~�tJ#�Iy�~�T5�u�1{���h�2�HU�t���Y���E
����@��)x���"��
����Q)��NC�W8�A�Er�#9~����.(o��'���D��E�.T�o����
�,3%~�Y�T��&��ym"��T����FR&�����B���#���8+U0W:�R�GTy�c,�O�s,S�cZL+�@)I)��Kn�>�]��{�Jq��Z�$��N��?c�*�����T^���R`�.����N�U��2��tQ���N-�q��S��1:5���nj�)p^�LI9�R�u����?S��:�)q�j�)�����k�K�*�@-~��X������);j�V�=�E�0���C���,	���#=P�>'���f��I5��9�:�\8C���r�ts������� ��RS//�l�.�~�P<,��o��Y��B�zx���0W^��p^��������������d*Zz�h?sJ%~�
w!UL�RE�l�HS
�ZC&����j�Ud�v���lk���(����V���*��k�
�%Cs��Z1kf�������`��EJ�=���g_(�f9'�N��H����rS�@o��,q`�A������������&���.d�Stg��������� B$@�_��W�e��-��^F6:HF�>��`1U4H�m�G:s�`��q�iYH���I�8�ei���.�'r�SB��rSa��W���?������>���f���Q\�\�*) +���2(�h rz�V�E��P-��=
��D�%(�R:.��0g������^�(/�T�A�#��Z�C�C�d�7I�!zDIH1[��P��
a\~I��A2�)���]�D�L	$�p��>s�`
�N�&��xcp�Z$	���\�L�13"6+	�>q1&�&yu�3da[�-0��X� 2�s���Mk��RIA������������X1���7E�i��bS�])����'��m!�Y�Z�!���l�@.&�]f�n������]"A��jn�	U�
"������m�'9q���3
���J�2���u�t� L�h������
�U!��T�e�na�"sX*�*��J�JZ$0���T������J��*�t�{U������*�Z������W.N��k���*P @b�3�&��I|M4e,S��x�O������1`5�!�.*g��J������5b
�\c������j���+��������J5�D�:�\g�U�t�

�X^$�
�T�Bn&�T�aYc���[����� ���"I���D �|�5D'�/n)����P�8]t���Bn$R���a
2C.n%!7c�ED�:D���F��&�hQ�����4�z�U����`P����(H�=�$v�A���Bn�)���������sC���9��� �k��V�*���-s8Q��lAlV���%9%���R��K(:h�+�m�\�[��h���H�K[�V��
�x����(�16�3\
�5�X����s~��4� ��H��h����SB�~i�+ 9�`�U?Q�m�f!��-8x���5����&�������>�*�Q$���y �����j��V�uK�K�B+��M�[h�	���OU�X#I�b�������k�Ba��J"��*�yV'Y
��Q|PqH4i�4��U������E�D����E�T�e$@EZ��g���� $ ���?/QY��M-g��K��(����	�?G �@t��=�L	��� ������@���5R���=%����&5@
W1AI8C��:C����2������)Pn	��p
i�?h�(S��
s�UP2������GHuK��]?�+��27����/$<+@*��P���g��/D7U�C ��@[�-�
�P���K�4i6��@7���"�B0{�E(�f���&fl��^��QC�#�)�t0'�f�����2����m
	3�
���%�:�j�����nF�)�(Q��T����(�P5���)A��1P��*����X	�:��
>���V�L t��
�.l��V~�Qp����Og@I�����~���$�t��qD5M��$r��P���T7�t�E��b��$�L��r���C��2�>s`[A1������u��G��j��1��)	 ��i�H�QPP/f����$6'H�5����ff�
+Jr-
�2�kn�d]�d��"��

�^
�P�r��
����*F���ir����0t5x�^�z�����K�r��������^����_}������������~��c�������3�nJ��W��+�j�~����J����Nb�g�l�g�|y�g�������}f!J�F����H�5N[��5N[��5N[��5�[��5�[�ei��jM���X[|a��oS�^��k�����2�R�^���V��:���g{���Vo���5X�����^-k�k�O�jYS�XS�����3���d���v����^
k������M�^
k�����<�M�'�^mk�g�	;�v���&y�PQ4z��I��&�B�^mk�g�	��F�kz��B?���-[([�P���n�E�R��5�[��6^�h��gw7�-{7�}��%�g��2K"e�����}O������"Xw��YG�S<������8vZ����8�t����6����/?���?��:����q:�6��~��q����[w�m�3���=����	wyxZ�p�i���87kz��}����������k�\������rW��w�=�S�s�h�����5�I��<���|����w�W�Q��d����>����������5���h�����|�5I�GW����\
���&���������fTi[�w�D6MZX!f��36�5��&eP��i����l���2�AM������4)�&-4�f��g��j���j���J^c��b���*b��GU�}�>��	/[�8Ui�E>M6�����@�I_��YU���)�2�b���$��%>#tZ��R/CJ\U��3J>�g�b��*���0�U��@�:�>���B�<���*�H�;�>������<��7]Z�>wb}>����y4���{y�fz������z�����f���;�?������2�_�}����~�)�JJ'(�S�b�|M~����__����w�Oo��]Ul��]|�
j�aA�����>|������/�|��A+J�N���^�l�F���b�2
�~���iU�:����Z�kX���o�S��a��x�����~X����}.�����_N|&>K~z��������B���}������������i!�}������D���3!�����W��K��������d�X���/?��c����Y���^x�����w�5�{�}����{��'�{��':{���=������3Vb���V�s;�����e�l ��qwy���lW}�s��FU�_�;�����_��
)�/����������t���j������8���]������^cg��{v��<���77T;�M�����w[9>�o�9�Ss�.o!�:�����K�����A��W��[�ui���L��O��#��u���2�Y��*/V��a�N��p�ie�(������+��`�+}�ldP�l���r��X��;~{�ldL�����2,��/_s�5�dP�v��������*�A�]��b���*������|��AD�8U�`=>^�����%W[�:�T�::��D���c'���a?Z�+X��������������^]j�����i@� >=[�l���
���@�`>}��e+A��������%�V�L�f�����y���t�B����e��<��}qS�}Z�Y���p�9���
~+����c0��p��>}:�,[��9���<���<�?�r���,[`�;R�'�3�����@�o�-�e#����W��Y�e����xO.��lOl��`�{���o����b�����5�5�x�Ek��
���ly�<�����j�y�@y����@c]�W�(^���b(o�]���^������(P^�����	����,$�����N��u�^[������C��m�}a��z|��S��[�����6�E3���oK�Q�n=\���*�Re4��C� ����1�J�x���*r��D�����T�yK��/��Qvb�a��DK��'�ok�Q��<8�:FUiGF��4�J>TN��]y��u��
<�������${o�a��p�pu�P9=��k���1���!�F�c���J'����>X�>��c�� ���a��X��~����du�}8��������N�����0wSC=��I�|"�D`���~��� w��M��iA��c+Vv^�
�n����R��@-VN#�@}Yrl��aa�O�>-�.9�B�tB�VF	��ZB<�2�"�M����Q@?�[�;��{���3���G-�������K!�O:����*9��������M��e�7\�veO��Ty�4�<m�r�a�{��pw��W����7�������w5x�����'x�<�./<��J~�N�3\��r���o|�����^�CjW�V�W�S=2���z���[���zU}=(�/%>�qy��>�4�=��vP0�(��%��H7�C��Xx��Is�f���(b��ySo:H�lq�^�yR������1M���t������kb!]���j�ARg3PM������C����r���AR��YM���
b��zL�O����W^��X�1�Ji��y������C./��T��2�!Udn�f^>�����d���2[�1����2�G7���3A^xn�bX��*��-g������W���8�:Uq��{�Ac��b_#���Okb�������������H/�t�g����oFh�zqv�:��3�\��~����^�������
8����.���h/�����1����2�~Q�f�F����F�O�V�~���1�V
�y�H�� X)��G��h��]lo������5������d+��������lf���������Z�sX����5,���P:G`"kQS����gp�!��V8.����������B���v"�Q]����L��a?��q0;�6I<�r�������@�?���^���v#K��'�� a�� l��{�}'��c�]��D�)�������;��vQ;N�^:���Vb���o���N�@u�q��L[z�.]w��?o4�uc����������{���ox���W�s�z��v�^�����k��ri��-��g�T/W�����v�~��'xrKWRw����h�F��1���;����%r���|�����J���:Q5��fX���������|K��;���$��3�L1D_�\�p���F��)m8����)#��2�����X62��X���G����:�b���2�QG+�i>^�pP�S#����)S�a���7�L�P��_�/
�V�B���(�'V���� �S(���*c����n�B��f�[�K���1�=����A���� �N�|:���qM�$��?����
l���?���g;���t�������F�x8�9��ct�ct�G7����� �������������|8�9��t�y��b�z��������Dg�����70>��<��h��Do�����7`>��\��:A#C����c��5�Zq��)L��l���8��Yq���h�@�~qs�� �h�R:'mq{:ne��;H�n����8|���h������b8��a(Y����!K�Q�y���v�c!r>���d�X�d��G�zd�>D��Cd�Y�[�e�~�!���}g��j�
��O���������F�1w�U;`����V�Jl\��*q;
���CcP?��b}|�gw��?�>18�s�F�;^�����zw\O��K��R���w+����T����y���]���W�=�q|�'�t�AA��`�8j���
m��:����j�������?P��>�8��l�S�f�Ui�~�@y����(����|��YU�!"?P��>�8��D�lY��*�
�����S�����>[�1��X�<P��>�8��d����AU�!�~������Jg3_,�S��#�~�@Q��|�(���/�u�b���+N�W�%u6��2�AM,8~��p��QT��_-�T����w�W�Eu�}=����/9�<�$����Z�
endstream
endobj
1126 0 obj
<</O/List/ListNumbering/None>>
endobj
1131 0 obj
<</O/List/ListNumbering/None>>
endobj
1207 0 obj
<</O/List/ListNumbering/None>>
endobj
1212 0 obj
<</O/List/ListNumbering/None>>
endobj
1217 0 obj
<</O/List/ListNumbering/None>>
endobj
1287 0 obj
<</O/List/ListNumbering/None>>
endobj
1292 0 obj
<</O/List/ListNumbering/None>>
endobj
1367 0 obj
<</O/List/ListNumbering/None>>
endobj
1374 0 obj
<</O/List/ListNumbering/None>>
endobj
1457 0 obj
<</O/List/ListNumbering/None>>
endobj
1464 0 obj
<</O/List/ListNumbering/None>>
endobj
1549 0 obj
<</O/List/ListNumbering/None>>
endobj
1554 0 obj
<</O/List/ListNumbering/None>>
endobj
1597 0 obj
<</Type/ObjStm/N 66/First 633/Filter/FlateDecode/Length 3173>>
stream
x���_o�6���;�e������I�(��	4�
��mQ�a��&AO�N���/)�C�wfl�x_��#�Ryy(�++��0�%;x5(?�:�
i��-F�we������5�j�i����]a�s;q�1n�z�e�4�]J
��b���M�2�R�Q��_�r|���/�r;��b�VM�y����0�|��K{�����:�f��#�i[>8�|�sy[���Z��;h��b�A�����-W�4���c�O�mi��q�N�S���l��^+��s/h�m��`�m����m��8�lr�IM�t�B�f���S{V
F���Vg����< z�5k�5{�2Hq�fLTCn��<��g&=�0�P����<���l���f��i_�������
c��_�|�j

�-�R��i�|1�{��XW������6j�q���7|���b�q�\]�.V�_?oVW��/ov�?n>�~����e�o���]��b�����9���|��k����#M�;�Hl����f������������8��CiQ�cm�;��L�G�K:�D���������g�����\?�.�_��O���<�~y~��������K�����;�{�]�~��p�bygW��������y�)����c����W/�~|��p�����Wh�k��YS�8���F|���1��l?�N�����_��
�E��9��
e�~�2���C���y��7������Hw�R�i<~��)������"��'������������s���X��{�&<�'�>��F-.[?4b.�^8&���������?�`Lu�N���ow�#?����ai@w\�1���4��i8Fb��-�HE��-���-�����1������:S}~T��8C��p"u��>����cO�q'���>�3����������i��I�^�Kf���y���O>o������������������r�4����)g��a������&�)�E�2]�)�E�2a���U����/�67���g���A����a�]s�0���8���>�s��
=N��zOuvU�)����z���y~wh@:��������
���(��+��#
M���4���e���W��p
�Q�04
�F��Q<�W�5����X
��w�#������cR=���n8R%���p4�nK����
�H]g0V�:M��p4���<��a�la}��'	���'	��@�IO����Z����	%'��V���
��(� |���e���]<��[,E���i~}��\n����������0���b}���������u�Y~��i�5�*�~����6���������O�W�7��������l�/o>~��\�_�K,���������}��:��oo��}���iX�����fW�r�z�~s�]��C�����}��p��������<��w��O��}���^��|�3?�
������1���1�8g%�c�TN��I
�����sjl��976����'���'����x��-��l�i�����_LI�CL>O��`��,�`���r�^�!!i>:�8��;��:	V��������<�����e�R@�j.I�N��nv��t�l�~�������NVXO�z(�TE�"5��(���:E�������R�p]z�>a�4Ic������������`��7����q5��zpQ����sE���1����jH�6�.��k�Fy������B#bCO����>jSOO�t�A0�D�4A'SOQ�����Ap5jAz��A������P� k��^0���4A'��?
����-��G\6�qi���'���z�����~��.���'D�`�i�?��~51k��^0v��F#�8����5A�� �]#� v����'���z��+d����=��(�h� ��(�g�F+�cO�f�t�A�
�l4�N0��d#���d����::�e����'��Q����'t�A�]l�gg�����'t��:������F~��F�C����Y���q�h�`���k���!�SO��C�5�q�8�1�����N0����F+�����k��V�:�D6jA+�cO�d������Gz��G�y��G�y��!�<�v|�C|y��G<y����y��H����1����F'�7K�S"H��1��X<�����V�F�	���k����'��/��F/{b�mD;�
��k������L=�C�4��Wa�'��~�F�������S��xm��^0�T�� C1k�����5�"�?�X�x5jA+�SO,��F��W��'D�~�F����!n��SO�]����'�\�pUZA/�|4jA+f��QPZA/�zB7��d�.�d���:A��SO�� ����SO�e����'�1At������b#����������L=���F��lL=����s���O��Y�Y�Y�Za�H�`^wa�=��Q��L��/�g������vX�[t7tgya�=��~�H�`�ya�=�����H�`�za�V�F�3�4��^f�p{F���C�t��������xR�+G�)GFo-4+F�b���C��j���V�ju�/�.��*1][����U�kq��/jC�XK�A�X��U�Za�%�:m-Z�jE-O�zD-@(�L-5�ZC-6��B-/����X�J��X���Pd@�*��ZEq��ZQ����k-�#�k������g>��e���v<'�f}Zs6�Z�����1��4���l��#�92�����W���@����*�����-�3[�����`�b8q�c�����K�Z��%;N����`�z��g:<l���p@�
��81'�q�����x���������w
��<f��A/���1ud�m���t6����r�0��,'���`%���rXN����vZ=���h9V��X9U���2X}��k��,>���O=��S���l������l��(`�3�IY�F~�J�b�5���s;�s�Y�Yf��������`;�����l�e
�3�3zY������w�������K��e�]o�B���
���c��4`�2��c�33�����S���b������_z�o92������i�1�����>������D��
endstream
endobj
1633 0 obj
<</O/List/ListNumbering/None>>
endobj
1646 0 obj
<</Filter/FlateDecode/Length 404>>
stream
x�}��n�0��<E���"	�P	!���z���(��P��~�M��IE�������v���{������g�ZW��lJ`{8��1���	���h=�&�C��i���$����]o����=<{�����������r~n�8�����Up��^���8�1m���~�3���9��$� 3eSA�%�B�K�}R�l��z���~@Y�C�]�l4�����@�KG�K�H`�1G\*\��0!8E�1�����K
�����H2&�����"�b�5�)��VD[���m�mQ���v4��F�-J�$��H^Q�d[Dd[�qW!QxkM���a	���P��,�EE���_9���$��+7[��7��5PZ�;	1��g�����9w3��\������g�Mo��z���uY����$
endstream
endobj
1647 0 obj
<</Filter/FlateDecode/Length 68773/Length1 166356>>
stream
x��}\T�����������38�ePl(��HS�2f�Rl��5�XBb�Mo���I��DL5��'��~��Mn4��\�}���
����~����,������^g�}��g��@��,8��� 7�l��sI�+Q����1ys�?����1D���3�����Db#ZUV��Z�rqQ9��+����-��Z�N��{����-�9o�G� �E��"�3�W��Z<�<��I��hVmU��cWz�/���#���2>�]g�[���`=���h����U{<��������U�X������Y����]Zu��[��6����_5�v����I��B�g��K��Zi=�'C�/\\����.������}Er,��|���9uz����)��=���%�]�|�/�7���b(i��vA�n�	����C[B���X�m�c�I7����--�I�b��:ju�S\������H����B�5
!-��i�I�L�Q�����,����vr�b�>o���$Ze���%���LQ�z#^������TA�L�Tu�����mY��}�T��K��4��|�k��N/�������z��Ao��=O^gC��{>i�.��<��M����^y�6�����]���>_u	N�3N�w�>���{�����A3��f��w��Z<�m�7�/���f��TB��������1Jk��y��6�''o��^v�:�=Tw:}W�?�yL�T�8n�S�I��S�v��D7���Sj�pt��E��w�}�gS~���4�t�u��Z}U��.xU}�Gle���BSO��"J���B��4�V���z��N��e������P�#��s�:��c��������rw�~��J������!��7��F���7%dY��]�kO�6���m{���}�{v�x.y]������Tr�6������(�]��)����w�s#���R���|����2�������7Rs���P_N=����	�T~�\�6���N��8�Gi-4R<A]�����%��j��Z�A�^L�z�$�~n��h�6�'p���!�h���C�Z���'�6�rO���[
�5��v/���M�^������-�3VO�+�����f��O�{���S���C�HB��l'�a	�!��k�O'���������:`�����4���F��y������t��C�Yta[p7`����]%�b(�tM�^ ���f%&7u����������|�,fv��T��06���~M��P�����YT����ze�P���b1��3�!�L�
�����������d���l��{/k���}J���3�1M�.�~���|�3��
�)�J�����n�lq��j�4L+�^x���B�)�>Z6u��D���}�q�����s���(����3��i��9��W��������v�:k���� ����b�.�"��Q�L�;��l��l�����
T��c�
�z:Sk�B�
&=z%�
�V<�������,`X���,`���&?c������LC���L#���}S~�4>k�?g����,`X���,`X���L������,`X���,`X��?�����t �t�@����9�h80
�Go��F.�O�������X���,`X���,`X���,`X�����Z��{������������b��(2��I�uj��L�o:E����
�3h4��d�J�i��-t/m�]��6(9#�wr����C��%��C�{{�=�����~����1�1
�����2��Ec�����|m��I����|���:���W�/B>B>�z��3?H^R��U����yk���^�����_mh�u���������l��}�����X��#��Z��k/�q>�������n6�;dW�����N�*}�~�����^����������������g�����D1K��#���d�N�T@�t&����Q
��%��V�*��haI"EtD��*f�z�@4�eb��@\(.����b�xB<-�/�4�n2�OLfS�)�b
5���M�HS�p��b���$������ee��7�4�m���1����1��wF�#����
�����s���Q����w���6r)0M������Y���O*�����yO�����.Y�h��������=kf]m�����N�\Q�q�M,-�0~��1���F�,,�����~���Cg����WF�����.���KtdxXhHp���k�2
��voz����5��,;���j�����*l��Wa���.D���H��Ha��a�2����|��ET�x�7�;����khS�Q�D!5-�����^Qi/�.��TP��|��ay����^����X�,����i�($R����T�x'�x
���������\��<o���>[��.�7g�n���B3*�5���)�^�FMzAS�o����������D\r�7��_�u:�����	���fq��:�8������	J�$)�%&�+M�z��KM�}���E3P�6�x�l�V�2��^�R��V5�nY��j�6�t��[UP��^6+��8��+�o|���v��^9�z����&G~>�[�����pU�����O&��*q��0�x�����8G.�a��`�D���������j+ofA������2�;(s9J<���'�����i��~x�pS��<5u^[���g��cM���1|�Om��K���'8]�qF����h,�<8-����z��[p�qp�C���(�;�;��VRa8�?B�vyP���F�*]6�eM-Oe��.Y�}2�yC����q�O|�Sv��e�z�j��t�]R����l'��&��b���s����0s����p���h����Q�(w�rM��k�cm�������
�q��OIY��gs�K��V-�`���n�Qi��GW]���_MM5�x_�G��,a����;�Y���p:Re?{e4�PDjYe�j!�;Ga�/�������M�.W����YC0/�E5M���aV����5�U���T,��r�J��f��X��'VxvY���<>Mhy����]Q��e���j�+��`����#���E�h���Q�nd�B�OPu��>�(�8������T�	��5rtwtj,��!����J�f��
3�B\��-R��J���*h{���f�,5�-��9�e�ed*�G6"R����s�&���>v�
��B~���\ix
g�����^#������*���A	xV�-��1���c8z�
s��z����#�9���`<�"A�f�E�����3�CV�sM�)�-��e�����S1��o�/7s�h�����{����J����68����R%DH�7B�Qh���
����U9	7���ro�S��3����/�r��sNs�<QfyS�����`���m����D{�(�d�<H��y�U��v~F&b.��"���Z����ZaV%�����#������R���k�9-���;o�6�pn�7=Jo3��T���{�*C��iJZ���K����)�����*���}8<�l�8D.���{�,�<��%���N���6��C����G�]��T�t��;��+#�xo��nj
�<y����l8��j�V��x���U����s,n��DK��FG��I����(ty����2H�	��i#y�e�*	�of�wf�����B	l�z��"�Z<+s��z<�*D�{�������H�J�����?�:9i���x�������InQ�����?�w��]J������x'�+������Oj��l��>�Q%_�z&T[��&��v*�Vo0^LuU��T�A�r���}4��
Y��M^c�"��1��$�{��QU+��ur]k�-Dw�������Z�����a��!�Mr�>�����i�m�n�<oSz��J����n��*+J�"Y*G"M��<do�9������bdE�J=�	*��OR,rz�����/J+<j��eu�����*[��Z��{��E��U�0n������o��b�������>b����4e�M{��R����w�o����[�7�o�_�~�8�1���G����������6�
�Ls�IP8�����|�X
\���nCFAv����b4n�yJ���9J4*q�k�X��j%�Rb�+�X��r%�)���R%�(�H��J,Pb����Wb�s����,%f*Q�D�5JT+1C�*%*����4%�*1E��JT(Q��G�3����[�2%&*Q�D����8%�*1F�b%F+Q��(%F*Q�D��J�)���%\J�(1\�3���P%�(1X�l%)1P�,%(�_�~J�U���J�V��J8���D%�+�M�t%�����C�.J�*aW��D��JtV��D���HT��	J�+�D%b��Q��D�QJD*�D�aJ�*�D�AJ��0)�+�)!� ��JQ���*������?+�/%~R��?*���+���*��_+q@��J|��?��R�/����+�w%>Sb�S�S%>Q�c%>R�C%>P�}%�S�]%�Q�m%�R�M%�P�u%^S�U%^Q�e%�*��/*���+���*��O+��{���O*����x\���xT�G�xX�����D�;�xP�����v%|J4+�U�~%�S�^%�)�U�{��[����S�;��]����U�[��Y�-J���f%nT�%�W�:%�U�%�V�*%�T�
%.W�/J\���J\���JlR�"%.T�I�����%�+�N	��j�#��G�m�P���=Bm{�����j�#��G�m�P���=Bm{�����j�#+��?B������j�#��G���P���?B������j�#��G���P���?B������j�#��G���P���?B������j�#��G���P���=Bm{�����j�#�nG���P��v;B�v�����y��h�����a��K����s|)C@�\:�i�/%��K���bZ����<����Z�������ri	�bv.�%��2-`��!������:��0�f��4����9T���j�LUL�L���q��\��4�������t&�$&7S�D�R��	L���1�e�T�4�g-1��YG�F2�����u(�)�)��Fp;S��t�0��4��f�f�4�)��
`��Y�1�e���2�zs�^LLN��L=��3u���Li��+����Ne�s;S
S2Sg&+S�/i�S�/i<�#S;������)�)��,L���b�d���p�0�P�a
f
�u�2�:��LL:;5.	&2H�21B�a.����!��7�~f��OL}�e�}�A?p�{������o��5���\��?��%�L�`��C�������Kc�������#v~����L�q��\z��m_�3Ao�:N���;_gz��U�W8�e���|��E��������e�3LO3=������$��`���8�=��(;az��!�]L-��K2=���i�/!��%L53y��g���^�mL[���%`�ws������;�ng���V�[�nf��t'��Ynd����g���Z�k���\���J�+��r������R�K�.f��tG^��&��62m`Z���������t�/�t.�9�x7����X���Z������vg1�����Vr�L���150-eZ��s�EL}����l>G�c�g��4�i6���4�{V��k�j8��iSS%�t�i|�S�gS�&�EWp�r>���L��$>����1Md*e*���@|q��}q����;4��4�C��F���/E\�4�������_�P�/�lP�/����-�`r1�0
����.���0_L9h(�_�|43e�bF��b<����
P�
`���������ya}|1rnf2�����LNN���'����)�)�#G�+��sv������YlL)�.��3��)����2���Lu�Y�������:0�r�n`ag4SS$SG�sd;C�B����8���&v�L�`"Wk�����j�����������3|�~?���=��C�[��k�����P�O����|5����Y���}���O���1��!��������k{;���-�����7"�m��A������E�K��9������A?9��L�l����lOE���A��"�����u7����F,�=���p��CKm��`'��n�����/p�J�}��l����m_c���vp7pp'pp{x/�m�[�[��f�������7C��}=r]�\�"�5�]
\\	\\��.C�K���.	o�8l�mS����������l�����D��\w��������k�k��q���k�k����f�����b��V�W����������b�r�C�z�������mmp���6�?6��
"�A�i5X�
z�R�b������x�����������,�h�ki��}�5��Z�8�R�����p����y�9������Y[g���k��[k���3�U�����S���NuO��pO�Z�.����D���2�{k�{bv��tk�{|�8�8��f��l-v���.�:�=2��]���������Ev`\g���"���e�����DV�u�U��N�%i=�;������Ngw�����r��J��Q����w������c����`I�'����������7��V[�#�0:^D�����o��z��]���"�V�?*���I�K��Y�B����	��b�7m�<�J*�A�����i��r�g�q��J���M�(9���<����lI�-/�6J�r�UjBH�s���%N����$��=�q��-:ZDG�Fk�ht>:���Ck����;�0:���Ck�����G^_��	e����p��>>\s�����{�)<�:����3;�N�a���N��r� �N���K��,��29�84}	l�r.��V�����;��o��<#Z���F;88h��k���Y�*`%�X,���`�X����\`0����Z��fU@%0�L���
��g�7PLJ�`0����h��
� �r����g����`0�
Y��?���2��@/ p=�@w���]�H��
H����H:�@G ���@,X�h 
�"�p B�` 0��8����F�'���_�_�C�����?�������o�o����~�+������?��������������oooo����/{���������g�g�����=�_�'�'�����c���#���C�.��	<<��>���������=���]������m���-����&`3p#pp=pp-p
p5pp%pp9��2�R��b`pp!�\l6��uT3�Q`����/0��������_`����/0��������_`���� ��k�� ��k�� ��k�� ��k�� ��k�� ��k�� ��k�� ��k������_`��}��/0�������s_`���?{�/��?���FK����IK�>���7���o�L�9�����6���8�O3�<�ki�Aw�������?�[4�kGV��Q�����Q���G�Z�Qm<����d?�i��~}���#��Z���R��6R{
����Cx���:P��
��F���7������A	U�d�BS���p��7ufcd�R=���Fi>�f�X����',/�>���i)5�2|-�^�/��EF����k��[ng�jZ�?.7<�Q��(�����3����R����|Z����6��Y���j��"�����S�M�J���2���+�J����sq=�p��j�m������+���P��z����~z��j����:cbV�
�k�c��GGk-�]^[��JW�n����(#�C$g�� ��9n$.�5�>vE\����c����[^57�����T�{O���1o�Q��T�@����m����n1���mt;����R��;����0�����
_�t[�|�k�9/5������i'����;����;��E��xB��Xi����<
��~����'��(�(.=M�`�z�^��ez
����Y�^���uz[DB�J_�x�^1FQ4��8�@�h�����x3'Q<mi��uy���(�e�@n�]�A���c��Fa��Q�h�I��~�=��#��~Cf��K�����L�i,�������G(��"x >??�W�c��hd�&���sE����II9��YA�����kGN�&��stxo�����< 2?���O-�������7>�����K��Y��Y���Yz��z=&G�w������M�H���L�������Dg���"&5�@\����[���>��~�����.Q��0p�p��M�S���,��_+��������I��)I�q�Af�sbl�ai������N���tsHp�A�]����������|�}s����Q�����B:%��~MX�f

jII��shj���SxKLBHplLD��)���w�9:��s��c1���C���8�B����]����1����-����W"��$U�E#�c�qtui�:#\���HO�1"<"�K�#,R$�"(����x���CwD8"b�Kc�f7������9ujL��1�1�-�����G8����N�����i?����6O�Jt4�Yp����;�MO��tG���������=��",i6[Z�P�������:8:'�E��3Ev��b��e:K|,�<#�e��#B��#��F���Q��/<*D�C��7>O�6"��s�BN����u%�-b��-�8$F�`�H�������B}�����28Cg���!�3��hj��4���}��H�����i�O�#�b{�d�����;\O��c���]��/�"����(�@�1c����C��
'9��Y�	������Ha�9v�[J�e_=�`��
�9er�vI��0�*�:^N�]��91r�t8&M��=�zQ���:���Q�/���_�sD��)��N�����W:�����sK���KK��Y:�g�����n���Uc{����
�*����1Y��Lu��!�a�#�������j��aN�zc�5�;�P���Q����Z���?�����XeE&�R���u�hzX��,�#z7�N����	���eykF�95�Edn�O���"2v�w��ej=��g������-1p{��q��
j�r��W�����hr���k���!q��g�}����z���9���n
	��7~��I�jeU_:y�����aA�NKblT\�n�������_��o�i�����C��"�K����~g��f���h���X�eK�y��8v�%���	!�jgA��%
��(P�J������6�����^(�-����~}m��y���~PBb����hq-}/����4�j�9�{���sgK�����tKw4ed|���=���}!���
����X��P��)' >1��Q��Q��K��K��Q�	B�@����{��5,�4x���q�a3P�0U0����#Z��5b��*`�3���+��`U��3_�n��������5Y��o�����5_;��������?H�����u�P�������.UL�*�K����r�`PJ������@&����Q�	�o��x9������k��)��$}
B�D�| �N��&'R�i��)0�e�q�� �/�/:$�QF7w?T��3q4
~��e���3�~���iJ�,n��-s�+�G��$����n���{�}r�W���Ut})��]_��/E���5m�b>/$:�(Nf�
�8ap�#y������O�.�sC���:��r������\�
�
���aX #:�s��K
Y9��>��I��]�Jn���
sdM,M�_�P^ ����n�X{Y[��������B��xdU!6L��1<��6����kI��$��G����kI�����f��S�P6�u7e�G�+���N�A4$kJ�}�����eg��s+�I�.px�@g�3xb��]��3)��4X�L�����ZZe3����e��+���n�8�n�}���,���oOv���xJ�)��_����m����������o��;p��b����x.�]���v�{�`�����p���W�R������#��+�p�tR+�Ia�AKL������W��+� t
�zh�#�C���r<�-�C������6x�
�%����c����c�3G�(�uy�Q���\,����)h��%?�>Z;*h��V�p�A=WZ�������
�W:�Vp���
X�����	!�YC�3h��azp���8�k����������'����m|-��
��}<�X�����t�c��1]���pLW8�$����z3J�H9@)��	$�'hI$|mI�-�pV_~���U�����2���q\�_�</�<�����xS@�j8L��8������G��>ZPh�u��+k�)e(m���O��QOkC�gi���s������	�������,��b��h���n�KPb��0=f
i]�M~��4im��6]Z�.�M�������mO�A	O�P�Ne�G�p3?���� ����D	a���(Z���'J�9�U���p�$A�����OiS��.����H��
�����d�`_�I�p
�6�5l�k������Z8�2��SJmj`������Y@�
(q`�GH�#9���d`~2��K��V&��0�����y�e"x�$H�L�����������V����~�U�P�:��@cD5�{i���G�W�nE�����`�,I�R����?V\q��5�c�Z'oP�6�s��e�{���(Zu��WQ5���������H�>cCC�=���81-	�����H�o���A�QN�!�����?��T�������N5UF����n��B}�#��'��ds�Z]�p�V����)~U�[�:��7���x��%�����(~�=`��� y����j���K8��2	x<2:���6o�4�Z����S 7�	,G�za�:C���a_��@+��B�
��g���1�7�F�� }��}:�}:)�����+�w�$F�	(��2�.=6�@����a��B�����L��'�U�
����
�����<��_��;��+�s������?�{��7&��>w��G6����x���5����������o���W���e��:���g�^u�Q68�s`�{��e�#�.*��������7���@9d/T�*�+	&|��b/����5��0�����F�*��A��W3���8���*�3������J-���{��
�����Zn��y�Pb�}�x����l{_����Ww�@�u����f��������)��)1�{Jc���%e�CY�,q�8K��7bO������u�u�u��u��u��=����T��XYC�B�� dN�U������@[S`���
`\�2�Z�LC�)�Po0��u����,VO�i�"`��c����]Mm-���g����]z����+~�b�]���L0���W7������Y��~q�����;�D%L�z��\���'w\��]}?��G�}-X/�#�e�� �r@5�P��Q�P��P��@_��.���P��H�u,�0�u����ex���AI���y<�wttMGzpA��lY�Uu��#
y8t���c}(t~��J)�����;|�^kq(v;��E��l�g��+h������U�����c���5x��X�g}6�\�^g��s�+���}������p
p�M�kr��_�Q
�!�4-���{��6�9�����������N����+@���0Kn8_��5~�}���(�C��:�u5w�Q�[Wk�Q�O���J�'��#i����n����!nh:��D��G���=s���Z��c"$�B�	<���*���@+���|H������Un��0��v���v�t�5+�\=��f%�b*l��}Pq��
p�pAIV�x����/�yi8q�!��I}��k��Q��2���<��[���x��5��B��Y�g`���Hzt��;F3�;�[�\3��0A2<��z��ZV4��+��������?�5m����kaC���uE�uY{S�k���#����N����b�(O��m��iY���t��Y�����:aPf��a,C�OA���]>�k��g��:c,�(���z�
o#�>��NBa�T	���x�R��I%�����P���R&��*y�@P��Ke�o�d��h5t �� ���v��|��z<�Gz��er�D$��=/W����]W�]W�]W�]W�]W��IB���2�V��La��J��
&QR���d�*���'��Yd���_a���s���f�����{�������~���#I�� HM^���t�xl/.i��_����n�mkHGu�x�X;�i,�o<���k���x���9�:w���@�����y|��:�h�T�-Zi��d�-(=6>����'������Ttt���S�E��T�������q���y�AUX�x�[�!O�b��kq|'�W���H���8E���:�:��x�i*��({'�s^�w`�F;�����^~t45�<���	�S��Tx��j�7���x}�������F��.,f���`�D�q����-����0!!KGkb�N��)�����3U|�x�4��`�G$��3�
�#
K�a��k���H<A�^�Q�^_X���Y�Y6���|�tL}����3��NRm��Z�?����i��t�<�����L�O"�'���Gx���1<1�e��\i-�@��<_���1����x�ET����8�mI�O_-�L��/i���=�������n���Q	y�1������D������I�.;�����"E��y�?p&�e�Y�b�V�����.�,���L^���Q7ce�5�W�����������pdZZZ��
Y�%�6��Y�cA_����:�.����"y%��$$���^,;�e�7��k2�Zp]{vO+�1���V��&��@E���k�O���5�x�����|�8C���2Z����+d�������3��:
�%��^�`V)�B��Be�^U���yP����B
�����
k������J��
�	�I	j����#?N|��a��f���t�,ofI��At>P4�''���;��Pf�� H�����(l	��@^��]S[��4�y��^��;u��N��>wS�w�MW�ZA�`3����AG^�AG~��Y�zkw��[#+���N�������_l��;�=�������+{�zeAo��g���C�7H~=�
�n��������Ek�G�mq{�iw}Gq0����nOvA�_u�-�v�[�]	�pD�b3DW�(����9/�
��@��������|�av',����_�Z����c^���WF�y[�^����)����U�?��F������������h!�f��e�X#�Gs���kG�Rc{�v�����&?�
����S��v�/>3 y��p���9������-Md|a#{`~Qp������
���4��7���	�Yz{)��64f�l�<A��������;b����������O������W�Q���m�.�#�w��P�f����?z����o6M~�^�\_���f5���\����L���
.��ZG�R���^@M��
�	�B�
��B��L�<3�:����<�	�7��Yv�����7��z��YMVr[,N�+���\�����7X��$��>��,��@��2����y7�q�ge�A�L8v�oH��e�l����Y���?�C���D�x#���S��S ��r�<gE��m$P��$�bq�k_�i�r���� ����7�7�4��v#^���*���3���qA�'
~�����mV�7��-m���!as(�;����68b�����9Rh�Y�dW�
W(6��1�F,JK�|�f
��4t����77�n��H����K6�7��\�j,q��w-]~��7/�fu��H���%�[�������<�}�������IY�����e-.���'7��?�]n���n[@���3�C�
���M�9j'#�A>BPa2�Y�h����������!���uV��C�� ��Ug����c�Lem���V:&�Z��V���c]B|Hh�":O��I$�-����������k)_�7F;b���e�M��OQ�	��"*���_x�2��8�\�~�9<�B�L���	�u�'a{x`zg���O�#_v-���jLx������rVw���;�@].W@����������$�h��� ��H{��u��5�
�r!�A��0�i����0��	rU��bv�Hl��0{|2���I���N-��>��Q-�����d�8�_
:�F��A���S�NO�a���?��]�h���b����M|/�1��_A�F�<���*.��%��o����Eg6uN�hA-�J���5M�%n>Lm/u~��L�|�H��^i�0���������\7��G��"Q������[�=	|�B�
������$.�
!S�X_j6��d,�`�������w����1H[��p�X�2�����O/ua.�����?���m�$����%�wR���c1���	�p:�6��uA��'(<-8�	�6g��j��?(�������>
���\3�S_���J��-`�vl���>��C5��?P��L��U���P�=S���|
H���U��_$-��
d���[!����T<_!ge�]�Bi��n����A*�1�C��LO(���#C������S��r�*##_��^=����N%���oR�T�F�@����D;p*>�v4�#���.���LxPC�*��`C,;���������7�t����?����}�#�����q0�7
	�m���e�w�����]�{�'�������u���]����A��������L���	�=dp��dr��j��M;�[%�@�@�yc��t�?k:��.[]�%�h�z��%w�_S�7�vy>t������K�K���W�vy�x_�IV�,�X�E���U��A�����-j�,�	qC@��g
a�F�|L9`�P��m�%���-uw���<{���M��-�s���d�?���K�a��"�=��y����C��7��l;KMKh�
�fs���V\
^o
���l
���#������s`�;�;���+�E�mL
1�����U?�b��
=�����(�j0�	�7	�T��������������Z{�#�.\s�W*e`��0��;
�d��Z���[���t?v�"~9s�Ted����������i�n�^B���M�����`��]_�.3��5��L���Sa�3"
f����)Wqy?z�}��a�&���|�����+������3�������y��A�����2m�6���iS��*S���:�]D'��-(�
��nB%g@5hw$R��g�XS!��:[@�8^�Fa*-��ZY�d����?�����}fef����	pg��osG�P�?��n|]m��}KjG5���������uj]��
������u��,n��K6����a�=�Mt�jj3�7w�w$$�S�eUR@~���l5o��!O�n�~���������(��v��@�=��0���'�~��)7�`��>����/gw���i�%�t��\�t)"�@[���	���l����!������/�,"
<Z^�<�����N�������s%W<
?�e����ee������e����|UWP���7f�z��P�~����s;f&���]�s�n��
�z��%vU1� ��K��C� �`��L���R���wx'����a������
/~Qi�QT���2:��P�*P�/0�	��V���`k��
�(��m7��v�	m?�Ur�����q�������U4
)��u46
�����Cp���pX����
�����aX��$+��[����u.U��-Y�N�.w�"0y�"��Q�"�~��^�L��/�Ty�U�%�q ���?,n�|A�%���3�Y�^O������'9j�L0�V$�x���4d<�8��7�wM�����)��[�����`�k(�1'��1���]�3����*���F������'�j:�LeR.a �����Q���,�1��D�XqT��8y�HZ~�n�����xk���[H3^W��@[�ao�f�-
�(��rTd����\��&Q�?�po�`��������Q5���W\.|�(2f�����U�kQ��_��B9b\��t��� �'U�2���A���/��p�?�'5�
�A	9�`��V��|56u�����P�����e���'�k�P��4�/r<CQ���^�?����r�`��X�
~�(|@�1���s�E~�)i�,O��%�1Nx�t�FG5�����;��%�1[�y[W�|P-p�R��U�'�����9+Q�!9Q(	�,.��%���v�[!^���,+ZM	�fs*sM!���_�%s���x������I�����,�p3�6�a�;[�3���f,1\�`k�{�m��6���V����\�*;����f���fk����kgp��_���}�a��%a��R��h����
���B'������4���y�����|!4����p;�l��Qh:_*�y�T�>t�kx\�C�L,�"|�:����PP`��m�e$2'y�.������kG������}��e���K�N0P��gl[��O��~�����e+�wu���0��.�W���{h�@M_��n�9�):���W�[��IG}.������a`�_�{�Z�;��
���-zS�Eo���Z�#��������$�����E�������m$�1o�l��A�4�>p�ICYpx�F���-���h~�}\����6��c���`���VE�����W���L���vD#��z�xri__���mV���.H�-\|p����;����5��]����}����:���?q�O��%���j%G/,%usL,K�n����o��������\���f�.�����O����Z+�U��uf
��.q����o����v���M)3nv���MK����R�?5�}��`�!�L��_�-�'�m`y��|��M�����&�O8�4��g9R0k4u.}���Nf^� @���cpmj�C���<|Yr�w�j`�I�u�n�{K0?���%�_���N����Z�7��O��.�B�YQ-10<�����&#�0�x5�2��o���N��m���+��0�~�h,��z?�q���y������\��6�q(?D�
�
����'����u���
���Po�(|�����e|\��a	,���S�e�����-����m6$
�f���)����/
�o �H��;���$sm���Z6�E����/[y#�I>ui������Fus�6���{��<2X@����������s���p|X�\���G�2�dG���V�j���e�� ���Yf�S��m��L,n	�������>�y�%�+��E��z#kW��Yz����oY��
7�e���"~��adc����B����eY����kV5�,o6�+�����a`lY�2�4o��)"�Mb@��F�����e����M�e�<}�R����\by�hl����D{K����%��1�K��!z���8����i��%����R$j��\��
�b�� �v������L��~����
(9����V3�TE�����", i8`2��5�v�U3S<x��C������_�?���8C�<�X|�%��9�5��Mc`��������h2�$�U*>Y��m.���]�~���)l���$�����N��y>��Y~�Z���3����.%'�������5��[��\�5�u���xj�~x�,���r�jX�b�vnF:��~r��tRFxp���Q��&�����^�p
�
S|G�:[�R����an�l|��d������Y��m
k)�*Pz�W�{e����C���
���r��G��A�!K���9"Gs��=���k��W�P��Y����48}������e�Qr��a�g�-IN�{�=�F�n]���]J$P���h0h�j��L&���v�~ ��dv�cM�����]8�6[����B��:��,d '�b���UX�b?D���k�yw2�,����{)���d!����a)-R����tJg�)=@� ��L���)s-$,����Q�0�#����.��Sc=yci�
GN�3;U@�a��y5�jf	Vz��P����������{���|�]k��-�m\~{���!�1<����9�s��v��b���;<6|��-��8��x�����-g��|�m���,j��}0�$�����[�j���j!���R	����uP��
����qv�7��$oE�����	�� �7�^5�OA}�u�����g(�E
�(O������&�n3a6���Kp�;M5��rR#D�.od�U�\�ch�h�%�;���,�	�4{o<B���q��.��� �w���%�9����v��'���gH����=�]����c�W�^�������v��
�0�`s�X��bRD�w�J�M37]��[�^����m4M��hXs�}����'Q%�.A����[���w����
��X[3s�����Wc�m�wEg���%(�!�I����q��f��[��pfct����BY:��y��*7�*~I���(�VX�/��i���xm���1�[��������=X��6�53��E���T
0�Ms��K�DAqC/PxC��$v��{u���IA��5��
���-�
F�����^O�W�l��y`��7;���6���y(�D��:�A�*�6'3�/fN��f�dp\�m3��U��
�f`����BiN����R�?Jz��d�K��(�tk�}_��}W���.Y����j�%Y��e�d���e[�6$�/	�2I���Bf�9y���2v�m y�w���1'�!$d83/!��Ix�f ���������X�Z�����n������e�Gi���$�����A�h\a���(�(e�B��6�)c|O/�fL����=��MG�r��	��c��[����
C���#�s=��*�wi0��1��.7w�?�%!���*�J��Z���8&�a"�%~��}���g!�_8�c�����������:�1��a�u��@�K��?}�Y��3���'��;k�mcGkE�*)�� ��&���|������A��X"�8�\<F� �����F(?�?�~���Wz����V�M}9x����t���$�a~���*V1����0�z�2XW�pF�����'�>:��i��������0:	J,I�#I�w��h%&7���
�d��0
��%Q����RfTR�*`TZ��w�
����*�k����)����u�T�%<��,������J�Z)�G�+%����`�n[�*�i�w�4��u�g+��l��������4MDo!��o���S|�b�|/���kd����b�"�J�r��Y�s������{�<W?�����8���H\4>�|���Ul*^�Nv���O��V%,6��
H�&���gd20$�A<(�a�����C#p�`������5��UY��olG9��V���	vC-�:���a�aXx��S��%s�,b���Afc<���4���l�W�~-��6����L��`�3��1�F�|���m�k�"�5�x1A���t�^�9����O���{qvvpb+��w�������fp7�ZL��J�����'����.����������&X�����k�;���
/�7L�g��n�I����98���<�rx'Xkxi���uk-\��V�6�W_�O��j��I�<6���x@a���L57x|l�F.A��#���{+3|
H����C����9 Sfb�"==J�L"T���-�~(M��\j���*����v���)����(#o����o�%��f&x���9}I<j���K�����z�@_�V��Za�)}N�����M���<��4]����0��K%�P�Fg�=�G���?��p���q(������DO<{*�*��n���
9��.;���=D�����6�?�x65���[u��*�z}�����F�������z�t�(�����������f5���y�K"��U���^b�[
��J`��Gy�V��98�p/��G\1�����'W`������g����f����V-���DJ8�TY,M�K��J+79���~�R�5�t=u���D[s|�	A�n'�D:�U��v`�oo���<��s����lfw���� ��=5`�%�S������K���z���O�e'w���98�r��`��Y
�Y��B���?��9�MYa�ZJS�J�t$|��G�Q����~t6NR�
��e�+U�t�J\�N.?!��c�w�N�{���v@�;�:x5
��33�Q�������Tx�s��������1����6W�>�3�a���`��
��$z��{M�)���<G=v�G<n��phG[sd#�M���*]�M��%�7��kU�C-T={*�@|����G�zP�8ZDAo8=i��:�[� WK�Zs��5V�IE&�I��7]��c_K�6��c\��##�A�� 6V*J��0�����w��_�C�����0����Sa7��dm�m���oMS%�bJ}A��X!�����Mb�����0�yn�@��H�j��"s���G��G���i�&���� �/������'z��"3)�\���1W��;c�-]����[�3�-'��J����'�7q����c;v��>4�L2�
���?�2����z��'�~��� {#�L��_-}�������r����+dY~��X��EXc=,�O���5�h���~4n,�	C���98���<��ayn�p�T���WH�����z*or�W	���0�
q@�9M��5���,��;c���rr3L�Q��t: ��5��)�Z�Xa� ���[J*�0���fl"�!�����*8�lr���b�-d��f�X	�
+���z�X��E3����t��(���o4$���4�������)[��y�*)ggsY�O�����lv� Z��kG�S.
�y[e{_+�B|���5Q��|u��E	@\��%��(�G���sp���5�d_�e�C��+�?��Cc��~D��7���D�h���W*~'��1<����\���
����{5��--c�LB���c����>&��+c��gXJ��;\,����"d�Il�<�
���#��<�`4<�F���$WV7�`Ms�Fd�US���5I����H�����L!UX`N����O*`������1���2�U�(
�D���)y��:iL����"��B�X5/f�=A�N)����t�+�_-�<��>
:'a������ZN#T����Ak��:�W�SD{2�	�c�i���R�dY��|�K���@�R{F/r��-�Q�x���*`��QXy�m<���=�sx|�C(���+����U�j�\g������z���`VyO�{%���b#^��+R{��p&�BZ(J���A�[A���@j2�
����LrK3C���3-�e���C[:��18}�{}�� �}��1HL;HYh�)]�u����d�R(^��L��
���9���|
�Z��Mdm�%����\�����NI~I�.������m��n��%��p�\[���}NN���T3������������f��-o�����"��}�!��K0ekR�a��C��$�� ����)��Zg�����o88|��7��1�-or�)���]
����@@D�;��H��	x\b��
kV�����"4�#Z/~���XLG��G��{B����'�w�L���w�g2��t��[S�@��=�������%*>�`4./���U������Y������v��w��t�GC������gS����2��
l��?���=R���)8����������7�����{�o��T��n�u�.�)�#��[��=�ATW�����
��ye���M��:Q�U
�������7y%�!��lz������])f�������F��!m��ix�Y.=�'��\�&���I�����#����
�����k�E���p��g�,
��z�)���O�U*1��$Q�$���"����w��&�Vo�h`��0;7���M����=w�����#���J���\~��G��3`J����i�U>+]"�I�l��lkh%
M���6�om^b����1*6Q���0~��
�w�����zBhP�&�bpbG�q�\�p������Y��O?�bE����r������WN�4���x���ii�����84:���~�����
��n�L-n��h�@v+ �9�����>�0T
���B��P������8| �+�\��|3t��&�\Ms��/	��������\_[�|c�����������i�
���MW�J�@l���0������b���1j�������&� ��(9��`��74R�1��m"S�]m
����m��0��m��'��p�s"���(��O�]�.�i[�n��'�������������mr��HU2ebt�Hq�op�<�vW�+�������w�HLt?��c[�����?�;l4��^g�]�h����5�6����Ui������uT��.�7A]������0�#�C��Sy�_*����vD,+��@!�@�%5�p\���:�d��� ����o�}Ut13�6`.��)�P��s�����y�^Ib~IW� 2+{	W(������O����+���-���D�,RZ�����B�5��p�%�V���V��mhPwF���G=�4���	��C�O+��*�~�;f�`�� ��|3M��L���%9�~�����r�v�O
zE���/S1"Hh����&
'�N��x�`�ji��.����2�E0]d
�N.��h� [��v��k��KL��������Y!�]+��������Pc�h��#�N@Q+1]G�?��[��5t���hc���3��{�w%���6�^�g�	n,�v_�"���
\�!�kw����M;r���'x�D���1^:BNG���]�+X�7A��M�k7<�����e���H����0��,�{����IZ��{<\
�t��@�w�N�C��,7>6�F�q+��I@���+��ob�����(�j�F2�GP�\�aR����u��a0���X�g�H�������Y�5|�P�*�e�'����-S��P�}������������z�u�CpWsv��V`��Wm0�!���i�����S��<Uz������QUc��
�Nc�P�5��SW��J����	���U\��D�	=C���K@E�:�*���� �&2a���Q
U�Q9�� +[�����7���T5���W���D� �y=1����@�g�c�$%T���4���:h������g9��=��U��{B������q����=��#���o]���@(p�O��G���9����?y��:^OT����c��c����X�R�������`��i�S2��dM%��J��*�>*�Xq�$j�D���_�e�d�0�n����$��Wk�l!v�7I~a!>2T
7����2����4���[k5I�c#wn�v����[�U�����h�%�C�O�
&����;6z��3���������(~>��gH�������I��I�Fl0E�8	{2������~+��&�'���*����zC�'1d }�c=�'�0���=bK�x�3�jo�c�����]����~��[	5�g��K����/�V�����V!��Y����DoH/�f����7g���zM}�c����N2������ �zA��P|MM��P�	'��HI�������FSr}��x�t��G��YjUw@+{�SF}��)
�'lu�:�B��ad����
�=	M��2^�R�TX2���_k����i�J��A<���<��I�4`�1;^����s��&X������
��]�*�v�N����������[,�et�D���E]HMI-��F�xE�sJ��E��]�&�N�����?�8�c�;� �����yx���m�"��_pF��a4�FZG��?C��cR��T���ao��`�S����m�1*	�VOo�i��b�M[�K�����0����i�C����R�/�l�-�H�(�����}f����.S�+�������:wUS����3�_�8s(��u�����������0���0�V�sb/��|�X$�:N��?��>�Cph��Y�7��L�CLg}��r�����wg6���Y��33����W�����S�x@�;�G`����Vs��.���d��Mx���K�
���W\��8{��s�B3��33���������Zy�;��C�sV�
������0i�>_sSv�n?j5��W�oS��:�����D���Q��{\o���w;2���3�Q�8N�kf��E$����#ibl���m�9���W�^L		#cT��b
�����(������we���5���lt���-��$D��uP�7D���	�J6!-���`+��$��j%�l�4b=Z���7RF�%h8B'����5�s�X[���9+ooN%��K���kR<�p�:�0�[yO��I���"�\��o@���-��(�J��t��%�F�?I?N�!N"ifh(���1����||i�&���Bt��c�a�R�N�jn��e���y4����(d�.	(m������9����02��d���28�i+]�t�N���m�"���{���h(����,��8����t��l"��<��7Z�P���^���9�0��$�\
�B�N����_!�"�_�����?=��5�d�0zOz�I=���39��|'Wt��P�&�K���YTzW%G����$���:��9I�x�$��gT���[��S���������FJ��~���%H�%?S����]�$'�P��v��
����fb�b���l���9]��dMPQ#�����y��S��q�t<�XJM��s:HH@F�@F���E��P[��5}��(���5���*�8_�.5���zo�5����P��I���J=��'`���71�h1j����A�"r�����F�����!N�ztrj��L�m<����,�F��No��2L�s�����p���C�T���{���p�����H �3s������z�+�_g���E��{SJ_��
�����C�[��}�\�rnb��.�q�8�_����s���\�B5�������H�2���}�z��)�3��&n���X�PW�
�uX
j������^�.72�(�/XD�,��g�E�R�E�
���������!�����??�_G�&a��v��y&�����m�[d��d�����z�a%�XWQ0E<�C��d	��v��5�"z=��D�����"�U���U�6�^�s����L�mf�����m����P���6�M����u�t����-zs���I��@��b�+v����b0*��wbh1!f@�������'xr>�������M�e�"���������jL-uT�Hi�!��M����R8AF
��UH�u��U���Ff�~Fn�Xm�l�I�4(��lY67x|Tn��d���bx�������
��h�oj�pdnu���D;��S�V@
��A��}h������_�$#P�*�Z����T�#���K�p�9#5�1���8�`C���� ���>��a?��w��'P�l����I	������ �p|
81���������#��he1����q0w��AI�D�4H����B�10.`��`����L�VzDD/E')�Z���"�q�N���D
���yBf�dYB��R�.��(��J�Ti���R�������rO���
�+�wH)��/���(��e�l�|�\�Ni B,EJ����5�!�2�'�>2�%�2�&�2A�}�d��&R��o%'B|-����`�^����t��c�P�^4L>e�J&�5��j��Sd
�-W6f��=uW���j������b��9�Lc�*�e��XR�U��^J���)�mJ�4��	\$��JQU	���� PNr��?��*���:��~��b����%���i�-���94��)�9Z�Lx}1�n����������)�	�����~L�oPJs�i�@N�-����*�ip]o�+���r-`�R	������v��P	H��%��%�P�h�e�@���!�@�AtE;i�#~��J�)m0w� ����&�~��|V��s�*�2�j��@�c���bPq��$�SM���������M���
s��
Y����X������T�����9���,*��HTF��1Q��?�tf����{���}���<O~]g�Jh�J�|����������k�'�(�-�-����6�E&\���dT���bJ�	��������A������L�B�=������|���������(�������q�EN>.k�;�FJvBi���Na{�A�RJ��2c��e�/�R�EG�JL*z�����
�����Y-�s"����1}�Hbl�	xR�E�}f�1�%��/��7�>��2�u���K�b���8�!/p�i�a��a�KQ?�)t��������]������$j�F?[>F~M��2u�:�q���l���_g�9
>)7����K��."M��A>���w���,+�/��J}�����>����L)Ce2*���R����a�mLoDX���w�N�X���!������q�).���~�Ccy����Mp]����Z3�'Ury��|_j����7k}�hd �UhM��=F�{������|*US������H�I�2�i&�jskMVc�m�x������5�����1@+���23%����V*_�er+L	����������.9��NK��	��^�L=������8RL7����Y��`y�YP+N6Q_��`�|&mw
�N*38-�A��V(�������G��x%������MI�h��4�vK!z9���j:���$2��Ot};��uC�]��v���3��Z����*�S���K����3��xo{�����*1�S�G�"�_J�Os)y�N~t�Zh/�:G��e����);%�o������AK2��e44OrU������CQ�\"��I���������Lm4�t&��bRI���v��	)����Ou���X'�>�tt�L��P&M>R2}J:�<�����I�{1���Mx�c�4�/q�f�[���������,�k`�-�N���b�L\�G�=��� ��x��p5C�+��P�=;������d4Jf�
	
�D��l&������d'��V�3�!2�	T��Q�Cg]F�+z��Z�.�I�N�y�����/�����G�]���-)��+H��|��H2@�%����_��Kv�)�~��M�������_�u�40��CN�������h1P��`����Tl��HBIFGBa3%!����6F����n�N��~�Tyi�-^^�2��m�/��a���K27�E����M&B�\ �J��hP�d�P�)�����#��h����#��/��KN�$�;�|�kc�7>-D4���k�L�>��F/�%����
F{�4:��.��e�fO��}�����m�V���������W�������������=n{W�L�S�Th-M���S{��5��W8�3M�q�D��)m��=��_����91����L�T�G$'{��O������}\���]���h������[�������*���N����������c�� __�9D��P�o#��R{��$.`�>�V��F�>�������Uz���"�'�t���(������W���u,�+��w�)����%�2�����U��t���:}=&�6�"S�p�����������r2�����;k=�V����pi�:�I�6��}(dI�����9s��a�N�Fe3�����F{����������3�ocz�Y���I���6zb�\��^���RW�	��v?����>�I0#	v�t�9�&�@������N'��mG�G������d�I�i�~�X���"!�s�jl����������bR<���Y(ss�o:�P�Z����M/��7�?m���l/�T���R�?�J��^��)x8<"q�u�s����,�v��%�'�8���\���{wS���K�N�x�V�^�
�2�`&h�3e�)d~�o�zm>/A]�	�>���t\�����Ih8��@��<�,Z���	�'����F���3��%����4'Er-�n#y��s�]�o�$Li���0���
�Ro5F�r��F�f�L6@�������r�����R:�Ku�y,M��4u�����v�/��f��x����"/�=�`%����Q�+A��.g������Z��kC���F����0�.���0n�R���e��Der�r����W+��+��O-+��L%��+,����5�������}�Y�� z*��� ~��R�o���q�k�J���e�����l�d�PP�D��������t��@W"��8bk��P��}����
�Z����V�����
Z��=�(\F��{{ISG�8.�;�V�^��n���#����[��B6k��l������Sv
��������ux�L� ���R�Yac�������R��_	�+�UC���t�sMG4�^�����8d�<jWO�����������
���GB,�(������.aoV���x�J��I�R��8Zh��U�x8��������z]�WSw�S��`�vlr�P5�Y}dC�mC�_'���/��c����������j���uwTRm���[����2�[���W�s*H~+�r,��8�\�'��
�������G9�C�cxt��.���bm�S\��L��m�Ju�!�,�h�:l:^��0��,M%�f�e�C)��J�(�'`Q{.;u�:��o�30�Qx����GQ���GY�������y$��b�����/��J�����

��#go{��/�:���cqlP\YX(6��*zc����64@�:K�z������t*���c�����N
�^��g& �"�a|ce|�4��!Z(�>������@�T]�-�V���H�^����]S]���*5�nxkl�� ��*�`����}	|�����$�H�E�����l�����,o������q�8�"bg���M�Rh�@����
-�$N��G���BI��]~P�Wh�G�@���X~��H�B�G��b�vgt��s���s�=��h�:���E�����j�6�KA6j(�b��+����������6�\?)�'�"|�����XX~�[�-��������������e��muT�.��s���`=L���ql���c������j�����i�^��[q��R�����9V����?XuUe�������LVyJ=z��4$��T�45w%
�u����������c�r�zt'��}�T��r�^�Lz�9����H�*����^���7���Z�d���:F=���:�I,�	�^�M��oc�7�G��<�������E��x�M���6S����6�k�dU|��&�MQ�j?�4$���s��X�>n��%,���Y��vx���-��h��p������# ����)-E���zE7z���#YQ�����)3�hj�f]@�.$M���(��&�O�U�|z5�U������>g���s&���������(������2+
�����5u�E�9Q�W���ziu���:�b% �;�����C�����y�D��N�|ZDCIV����	�lB����Hk��-�_Wd6_9�>
�7������3c�;�C��(�4[�"M��[q-���g�9C��E����]/��������hi�G1~��1|�V�je���Wf���%�e���,6����&sY"��z"�d&�	�����i]�)���e�f(��������u�K�/��sv���$5�(�5���E���&BgFGf�x�$4��,9
~
�_4��b�i�F�
���L,g��L�����g����F����*h��Z6{�Fo���
c�]��$&cA(�u�2�`��wLZ��L�3P7����K�&�<�R��'s�$��+�~n�9����a�:7QYc
���&e!�F��Q���e�I�_}=�������'�����m.AK��Ch9����p�!�/:����p~�E���l��h
�$�����l\��4���f��\0�>��2�������Y�S�����:',:nB���*���S�<3����z�&���!fg^�dl[BD�$���i��m���B������K���p���Zs�rf=�k�l�J|Y��!��xL�\/��)#�}����n3�`�)��$��	������n(������%�>��@���a�����9	��a0�3P#�WX��(Kv�K$MFs���QA����p�|���rf��j���Y�kw��(�LNsf�2����3�*j�c�\^�]���@��AU�&uDR��l���Y�������"��1�BvIe�8z6�=������<{��go���{��j��u���)��&ZXS���f��5jB�B�@���
�yv�u��4U�T��a����pD(��t>D�B-�bq�h-�#`�u���S(�tr�������U	�3zQO��]~R��:X���Jl,k+A-{q�"8F�*5�UG���y1^w���`}�a���s9M�4:�t��vt~�^y�JUG����@�MA��o��t����t:)�t�:P�������E���S�+,��v�"6��E Me~�`��K�U�����Q���l�Cnm����hT�[c�We�t�m��d3	�C����{������s4|�$/0���"M��P|�
�e��P����t<h>�/����;�x�q��b`5�vB�1O���5�1;:	mK�t��z�A�����
�p+em�R�z)h�a�	���h��_��%���W8�e4�����+�w8;�a��X����<�_w�ql�G����
u3:����O_�����3�^x���F|�	H�v��^{�~X���wY�P�#=5s��,�#�C��u�.I�*�S�������R�wEg����w���G�RWC�������[�;�D����h��
�%���()9�s����QV
�������o����x+��t��$;���"M�^��v����O���\@�%��/�{6,U��N���w���"D��9/Iu�K����QFU$2R�&�l��9�K���NT��(_����Z=���^�x
^��.����x��+8M�`k�q�x�:�����?���$��y
S\����8cJ��g�bk�C�.��'�4��4���W$��8-�1��uN��d)��T�'�W����#X�|�+��+�N�[A=�R��H�Y�$��|=�L��x1���J�^$14��8�$�^L����9+�n���RQ����xDk�:�Ga�'r���S;��{����q�l�H��~�K!5����>pb�?v�d�����~��M�O&Q4?�q:
����Np�����O��^,
���=$6#c�X�d����$
7��V����%�I�K�q�s����V��O�Q��B���������C�����n�*oJZ�6C�eZ��AX<�!,�~�$�	t,'�\��E)���PF��������k(`t�D�IOT/�uz��U�hX%'�S��|�����KAMm~�7�����o���i�f,u-���a
}X��D��N��:A�S��"J)��z0;��8Z��2:��2�9��j����\������&�&�hw5<22B��S�8-��J����������8�
���������x��&/d����0swP�0��kN�^���#��]TP)g{!����U��C!9����sV8��3�`z����U0��������|�s�9:��Dh^���iNR�f���s���`���g�a�_H�h��4����"�);�3�'8���~�����G����$�����$���sz��5����pp�?%'!u���|�1;���'N@���3��^}�UeB�� G[�K���z�{�H�[m&�(N�X��TU5D�$��#��;����������{�N��|���)�)������v�*ey��',v1��gg�����j��	9�!$7���S�8��mdw��^��\�6�xQ�To�<�=XHJ���������GtF�"HX��>=	k����M���J�5�����Z���gU��m�p�z6���g��P�l���<��
���md�����������>x�5u;]^J�w�F+�IF���J��-�*�������I��������s��0#���%��TEP=�YW ������Y��=p�l#���f�����9�o@���V#��#��C3$�V�	��~E�]������p������F�xh��:�I�rM�'���+�$_���=f;�zgy������)����������%�+/7���6RT��iTD�e��s�r��E�F��1)�	�.)T����K:�����������
~���Q����XguQ�,r��/>��q;ih����K#�o�,��k��:�����!�E����fPG]	��C���HE�T���!�Y}'7�NE�Q#1�$�,:@��_d�%��NED�7�����W�P'����NS��r���<&Z�{��X-���DQ�h�1�
�u s��s�����L��s<�bQa3�j���`U�����S��dY�^.����W�+��
{��D<e�����������
R�~F��M��yox�eu��(���w��h���)+�~����PW�&�C���}3
�*�7�7�<8
�=��^z
��s��O�F��g�� �S���%��j��*��S��K�%RX	ZA��$�u����B�yh�����M�gQ�/�����[;����P]nHg������T����v��������^d�T�����VU3h:@��������#�XO��
O���G��F���U������\^�}�)��r���
S�� R�_�qQQt��]�5�7�[m4�P#�T�(�X�%]����:+�����������I�p�`9=c6W?���T���C_�eS���~q.�V��le���O���e�"2���T��c�_(t���cd6��4t�-�Z�n�@������Dc��,�96�	�B�_����N����\����������5�c�<x�?�K{�n-��:����b�D��m1���D"f��s������ `�����i�����t����a:���� �~B�D�g�+!'���5���pW{�M^�
�I�*�c��}y�����&@�UZ��&��������5V�b��Gw�_9X��Jm�o������ru�;��c�p|�p�����LdnN����}�a� z�Q���~�|�c�o^]��_}��u��;����}c����e� 
��d\��_�����O��t����wV?u�*�V0?N34I�M~���u��k�n�������r��c; �
�|L��2c���_��[W���M]oU�C��5o����[�WH+��S���p�n�s�s����r�{����sn�*��h�9� T�8���}��<�2�d�)7s���!�������t���^�Dk
�CE��UrCYQ�y��8�$��B��^{s�5����Z4�[��Y�a�����2R ��3M�Kq� s�����l�@n��K����.tUz����h��pt�s�=�2���
������_�>8m��k!��)����Pi��,����L�@��d���m�R��
��%=I�G����4V���������WJ��e�}��m���j�z�.��$�Z�X��|)=����VZy������G���swr:~�/�����NMh|9F��u��"w�����<�P��4pg��x�z���E�~��P���T��Y`s�����I�<����������������>�����{>��	xt�������S����_{pt�"YEV(��E*�7���~3�
XOp��)`=N��s~PY'��0
Xd=���?o�"�������x����i�������Y	��X�wi7]���������i�=��
���H6�8�j�}���4#:�1N��;6*V����'1A5��I�j��4;���I}���L��Kh�0o�T'54���dG(N3�d�Z���*N�fu���P3���N��4d���H_l���!�%��"l����`�6c{Ulm����(��n
?8�z�up��%u����K���)P�^�,N@>�P��=��w�*����=4��D��,V@�����Y�|�Y�|��� ��%]�Q���3X�ZR���Jq����ycw�EZ)Ai��U�S+�������^��j��u�����{�q]_�/�,�{�^���P�0�A����\������,���q
l�!�N�}*����oE�/�����k
�v3��LM2�%r��YQ:��S=�������1��u��DqP���uJ��V��Qn7�W�Kg��(�H�c��0�h6B��b���������g���&�kv?1��gU�^�>ZG�����S�����^��t$������������a}w9��'���W$�n*�6��O�����}k���a
��d�h_��o^YQ�:���v������`uaW�'P�x���
FsA@*]~eG��e�z�*_��z�C�C�]��f�gJ}�NO�I+���3�(�[S�O���YG����P�`H����F�� �J�Ng���|
����o���Uyj�E}T����&��-���4�S�h�Q���)���(s�s��V<S^�����k)J����D���t�D�
����;D��5bFl'�p�U
��>)���Sx�	L�����z�z
4�(��U��~�s��zT�T/F�'��C�u.�|�.{&�6F��T�e�t����?��ns���P���������%���Wzb���){\�[gd)�)��[d[��G�k9�l�x�����m[Wbm���H{�{�Z���d�J�c�"���"�;���c��{6M�m�z�]_1������3��T��	�`j
Xr`�Qrkt�p@��&Tz2m�P�Y��_N�o�bA(~��m��v������e_X�	���k�����-��:&T��b���Q��Z�:������r�P���3a�%G�-����[+�yuW�;��GG�����4������v^k���o��0����M��kM�G����%e}��U����k����[O��$L��'1��@�I��N�b�]p���eG�&��E%9��E�
,WG�J�~����2�(�h���_Ed���g��4�������\@lC��u3&�.�E'��7��%��r�
k�v��R����nK2n.��Q3�g�6(��x[R�T4P�i�`�t�sU�U_���yp��-:��
R��s:�n���Q-yy6�����D�T]m�T�]u�����G�oxhWS��+:�&}e�����������������l���Q��i��h�}K{b]�s��n�+��_��i�����o��*���+��G�/��W��c�����6��+n�~��'���6�����o�#`��)(y�X
���W(�u�������F���q�\��]C���?��E�X���a����fl!��YB��@z|E�)s��uOL�����iu�����;�Jp{K�����+�G�\����c���d�H��V�&��ep~�{�����M&�yC�g��a�e5�o���7�u=���7K��K7U6��(��� %=��+��]����	�KbB)�	�	��������W<����7>-7m���������YLx�O&|a�������Y����
��P���#��'9�-����&1���Lsa��a.�z�3&1	����,���~a!���].*6}�!�1h��5z���@�y�Q�E�+��h�:�
E����$��DJ�,������:��x�n�f���R�z�"$��t,?X�~�$k�����N
�Us0dvH���`*y*
��U�������qch�!��oP�SMW-��`8=������
Mz�
��kbhOb��`��m��E�����11��\<������d�Y�[�'����=�����
V��b��4,-I��,	%	#���!Zlk����g�^�k�����rwW����3i�����b��#X<����-�V�sZ�0tU�m��m���5������Y�n�+����z�E��4���{Z�n]��PC�x_���wm���������������zp(���5���=~9�U�D�1t�EMe���P�G)��"E)��)2�5aG)�$�PV\(*�PtR�TZ-	c�����0���&
���>�B���I�����{���5�-�i?Y����m�
D�F��#�~z['n���>'^;'"
d�)�������+��!��C���W��9xq���i=g��}��;��k�R:��eA�}7O7��W���V�j���m�Z'�K��_X��:���&�����+�?:^wb�����M6� :��m���v7�&<��������ia��B�X�-�~���0s.)q��u� zelN���L�`$����Y�t�����@O���*�U��	xa����b�|�����I��g�q*�����������k�T�h(XT�+Ga
�H�������P��}HJH�,����0�T���K��Y��,6B�
hX�^\������+���5zJ��������H"9��GV<t�|P����3X*{o|<Ko[W?�
"Xz�S�J���`�m�c��T�|Tj��t�I�B����D����R�����3qL��'��������{L&I�~H	�@J`� �<����dcxT��B��F���~�T�LU�Le�Le���X�
���<�u�x�L<�h�%�A|
�_�9��;D��H�}/�����ac��(�7�B(�V)S���X\`q�rc�$�����f^>��v����4g�|v�f87�\R(	�������r�N�*��V:�����LUU!z�<�������gNl\e���������@>�V������>|�?���������>����q"�
�/FA:�{��q�T��!�o��.��oiK�6y���]~`��h����X���m�j��w&*F�wy,?Z��!4�X���?�HiOK�����y4�r��
�oS-m��|�db��$uEI�4Es!��=��.Rn���I��M����}Ip?o���{����q��S1��C-S��/�}u1^u���=��������~8��S�-�
NTqC��M�������l��!���"�}���=��-w,[���:�Z���; ����n�l\�Z�Z�H�o�c����FA�5
�7��6Co<Nm�,�7B��!|#���>?��n��WIW�O<1��OE:���mj��;�7�-.����z[�x�M?��7y���{��.R���a�-l�~
�?�1x�c>�>o�5��)V����M�&��)b{�O����e2�\A<������V����W�<�[�EO$���5IQ�Yq�G�a�MnpI�$���kt�X�a���d9����p������9�o��
+]��{���[Tso�����I]Z�{� b��5��]��[����VXI�A;}��b�)F4��mu�Y*i-�X���P0!��
�K��V^�����w�c�5Q���������I�����z}��b���k��`��e)��XW�q�m�	�=��
�~ ��k?���Ik�A���l�H������,d���?*p}3e�(G���:�������jF��8(��,[�Z�T?����m���;�1h���~SQ*V���vF
���]����]�[��.E��ZV�Kv���Af��"E��6�h3��y{���[S(��v�h3q�����@�����%A����|J"����V����3�a��bS��$g
;��%>�g|����"��X���<���!R���ST����=7-����W��!����������b�RH���fO��/u�	��y�|
l
>��9��7)�%�NY2����Q�����6�o�����G�	������y�^�b��)�'���oC�(�(�����u�>�m~;��B���h��_��>k�EcR�F����<�2����@�
�`�u}�uI�7�#/��v�_($8���2���%���
�\uh8�
f��&^2j���]Z�n�����thC�I��/@]:ib����V����\L�v^(�9;�9i���*-5QO2��is�Y��M����+�V�)�-n
~�6m����K��=�������o�s�k�@}����u��Gy-N3Z�W�*Z2��-5����7,?1����Piq��WV.)������5�������_���@\�r�MFK�=S?��Mo����}�[t����P�����Ql�c�8Q�duB���'����K jt
"jt�R��r>�6���+I�5`��
|{�CAy�.�&K�Nt�$��O���3��B� ��A����`_"D��]�Zx�8�Xx�9����
�n��6�T��T����W���[���k���E_I�6S����v�b���zU�?"f�w[����_x}��m�p�V2|����UQ�r�!	���6�w�%s|i��U^[�P*�����M��U��l�����N����nUW/U�]U������t��������%�L�}O&!4	�	K�,�
.����@v���	�(
�O}��P��'*
F�t����gz&�=�{	tU���Ru�����9������({l-������D�{e11���t����?v��By�;��`K��
�b6cw5��No1jq6�K�!%gCJ��~).�a\���@t�Q���L��R��������j��Q��5����*��w*���~��:����9cLW��!D)A�l]���/l������JR�bhM�e$3|�P�4�C��-�d������RH|��2����m�7��Ky������XZ��ex��d�)���y��YZ8R�w�\���SZ���M�o���I�cx3��(��\H#��Q�����K���qR�}<,s��dE�C?2T�����sk����a1�����|ny�1{�������Q���9/%���_�c.\B����k����[^���{6�:*w-��d{�.8�o����1����?�E������x`SL�@�[�7`=�`(P�2S�u�����M`R��xN�o�a����Y�����z[4q
4�5
��z�[�|N�v�m�J���z��"�v���h�Y�^JA��/��.����ib��<�%�#
��]�T
�)%���s+��*g�sAMKw�������I������/��*~����X=U��Pt,�\���<�v��T'�� ���*{�^�y�����9r=+J���E^r����[�OT�^��A����H�%���e~��m��Nm�~�q2?�*�|�����~p�2�l6],�e�\���������Y�N������uUw��=6���H�[I��'{����FW��&.�aEZg4j8�n��Z�����b�?cS((��C����k�$�������cI}�pJ}���A�O�y�+�&���@��NM(�!�
��o���
���gJ\��lzl����\}���K�}���r��P_s�S�2�J�-�j��m�����x������CM}��M�
�s�f�%,t��q�`���_��Q��,�5p;�d�l��}�xz����`����q~0TAR~5%]���e���![sG�g��vI���+����]�c�����yDP��N�{V!��/iEfR�?�r*Y�3�^�1�����%�j�>Nz"{�qj=N=���=MRpJ.�TYT�J��xPIE^��u�A}�����D^�X���7*�I�<��W���;������6��X��4Zp$���r������t�����|����p���^Gqq[av��u��[���/��7�d��jwX[�f7�m��{W��q�z���u�������%��j
��2vK:���7*����K����#:�q�.���_U|�a�������Z������`n��J����NM�_��V1�x�!��.�s}l�-,hH%����.c�s(�f��/�����.]���=���*�����b�v�����v.�it���B�8A��n_��������`���(�C�9��`�3P���}C���4d�>B0�q����0�X�����[xt�X�_�<?�#X�\���n}�{��������{��{-<���9���!_l���%W���w��������/�<�f��'���xy��g��!k~}�����8��bYcY�u���#/����R�q�-�B���_���i�w�����;\_�����)�A\]�������{��-�z'��;�z'zX��^�3����)yR��gV�D�T`�z%�����p�C��b�����5�N�jj];�f�7��R#�.�����U)���� ��@;�w4*�m6�--`V~�����H����E�IguB(h/������;�xC��3�Q�!�����3�=����,���im�����j�e%+�_��g����������n�����K���j���������C����v��C���F���1�l�Z��)�>�u�����n�Nkt��L>^�//mF�|����Z���t����~����s%�
����������?����Z�u��E���G%+-[f+u�����j���Oa���6bk���;�^}/�F_��zt/��ox��z�o!9�@�|�%��q:D��V#9���G�qjX������]5���$�M�~��T�_��g��Z����
��_1��l�����A��{�un������pVa�=�f���O�xCV�������-���r�Q�	j�����:�������~����c��I���O�����G*��	9����-�QP� �J��c��{����6��������CN@���Qcp�adB��.���w�6A`��?P�pfA>[���l����$����VM�a$����~�_���Gs��:[��?��"�Z��=�n.��I�����h=�Rp"�[�c�{��|�7%����������HS�6#�q���8��G���4	��l�F�)����F+:{�L�����+�[�r�Y��.�bGQda2Y�0^�!e�B����"��eHg2�q�ql"}[�	�`�
	���0�����(����U����.W��RBR`g)�0��z)}�7�_��X"S�������:G�4�Y������4P\��j������uK:y��'���uw�Z�l�����^���Y}�
D��b��p/�nt�� hA@i, �{G�2Od-��C2�@�'br.��L����'&�<&����C$�s[Q#+�~YQ�=�#�E������y���F[�#�B:����q����
���HL$�����
�SKV�T%��+�J��)�r6e���U�z�.Fr�@S(���������)�Q���O��&�p����)��\V���F����^�9�LN�9�GZ�"��DBe�aQ>��y�4/#��)�~���f���s�E-zga4��n0�!MI��m�.n�@���O���}Q�Z�6�fAg
��]}���0��z�w��#�fIre9$�z9��!)�h��bjg��P���X
`��r�a�0J�3��B-�0n�@-�p�02y�vAD������^'�$�n�<][^��qJ�6���<<$��OFqB����I��M�	���)��m,�I��T��/>��Rmh]��\�S.[v�X����a�B�7���\��rZ]�E�O���m�5@NI-�BF���(���@�	������9%A�# !';@O�!+Y@��&6# �iT����T/��I����k�����ov�R�����.XC@�^@<%�ti�
�"<M�D�PH3�
04�8a��Pd����r�������>��S#��-��~�\#�R:�S��-�b���w���u�M������p���tiB���l���8�!������N�J���R_�����H�S#� ��pF��s�����,����������)H�(��(,��1���m���Q9���.4�]O���"������l�gO W)��"�����~��)g�U�K�
ytN�zM�R7�~���l(�����)~����8�����M�'���l���[E�U��v��4]vo��I�)�/;hB�,{hB��2���H;���_�9Y�s,#��lQ�0���I�
QN0���'��k�AJ��[	r+|�E9����T���|5���:��7�#�IF%&9e�I~&<��8l�M���!,�=�����������]�Lek:�c#��4�k������+�(\�4��D�n���!Y����W&��#�K�8
b8]<��E�����DmM������`��,��(������2���Lr1<�FB�}�i�sz���6����6�F�@�8�(��!*���U5�G�"��(����iHB�o6W?6��(�^�TBop���%��r��*����5�<�������F��7^������S��-[�XA����v��y����i�u��n�����$��r��9�[R��!2�*,	���+�_��\2L�Jn�uTrIl������
y�~�h��>Xg!�sX��%+`���+��`���7�oM�_�u�rI/,9�Ktr�j�[�&,�������������d%.1��*��?'gc���(�Gq�#(��������5gD�����/V���B*t�Bj����yd���]=��C���/���
������M�z�gV�������5��������M���"[������Z���+����'	8x�z�����J�jy�����p�U���=������P�(8��.��A6H�Sq���U��P��FE�U��0�e��@A�	����&r3�i��KR���I_�G4���eFcy3����H]%m���v1m�Q��x���X���1i���9�������g�{�Z�|%��g�m���W��()�>�6�\�����ns8�v�[��E�l�/f�������lT�����L:oqN2��hK�&�m�r�2��sj�yI����0�_�����6�Z���x�l�����n���3/t�6[f(,z-g��s���*�[p�f
6���P��M��=�p��m��e'M�L��T�g5�x����s/�O�n�"�+�)��������@fE}D��,-�;�>��$)�khw�jw����������E�[�f��(��]i��Bf�����-�d��.�^��,F�3?+1�Q )W�G`�t�<N�����Gj����%��zA�zW2W��oR/T������c�j$�s�]je�1��UJ��=�9�6n���C~�����J-��W��x����'�*%#V�z�W�X���f!�	���~��5�;�j�'t�o�]�?M~������oh���N6����m}��%�O�#F�FA.L.��?�Us
����l��t�+e5&f5��v=o0��!2V����aHY�������=�Nd�h�_��u��)��xtj��X�j�E��#��m��+ 
��#'E�%���H�8��1�je���P�B0D^������x��?/�.=WG�U;fX����XQ�?_w����A��4��x�;ywp�2�p[����s/���^N��<�-�q�6Z�wy[����v�7N��
��P���S�kfE�7|z4�i�����8�I4ii�Z�Z�5���e��B�`5�-o��C��3rp�����rB����������(�O��b���_.k(���|�|�t2�#�R�ZQ�%���:�<R����d��O���Bg���S_Rk�p�����'��������b��a�y��D���e<F�<Vf���.������0���p��N�(" �?@�Q�n���J�n��R�OW�@��s5ll
<Z1��Z�P��Z��&��g���zc�4[;������������������wC�{���Uwt�;x_.��f��GC�082{\|	JT7kCL�����������w�����TC&fx�B���-��`������Bj��B�vcm�t�1�O��_����eP>��M?�����j:�����������+
�U��*�3�\1C�1S������<��������w����X1�</o��p������
nC�~i��B��������e��.�
*Y�p<�7�*/�iD�:����y��Xq��[��m�H���,�*KN�M����py��B�21���D��7�l'.�
�`Ea���XQL8��gD�5�����pr2��V�������e��l.W�Ctbw ����!��|��E����q�C�8�#GM��91��*�vNHq�Q�Z����8�����
�ZM�����B@��>��ZD����'v}x�����t	��Vt��}o���{;IPL���[
�n%~�|-��8�s�Q��H���qbdz�H�����onB�>�J}���4�Yg��T[KKIrV�h�)��4|����{:���H�q�?\��.x��&;��?{�x�=r��)�\�	7#�����G�!�:��p�CF~��d�|�����*�����!7��/�����^<8����V��
�}�~G����F�,���\;:x��|����KZ]���b������R4{�Cp=�y���5�|��`���s/�n.38T�7��h5_�&y�f���i�FkL6�QJe���1���Ql$�j.�I���D,����G�9�`��,h����z�h<D�C��a��gJ���PV���*��$~}�l�2�
�B���
��[���"v���)Z)t ebl�y
p���
�go�����T�-��F�`�s�\o���3���9	yG��:��\
?-kF�����`+�2�^�\�2 �p�'bzx����={=F`|Df/�����;�5����.�)=��+�����h�#5�0cC��<:������ C<-D�:�����h��x)�zx}���[�H���f
�[\��K���P4l�	�7;�Fw�=f~29�wit�`�;���l>��kQno��*���`��A����1�Z[�=���\j�=�=���?$���0��8��o����e�9���@�a/�a�OA�g�r}�RW����[S[�s���ZSU�O��*�(�%`uFl�0����������������`���PI���;-���c���HyQ�|��$�`��`C��s�C��3�����`�Z��������1e��T`�X�`%v-m~xr�:o���4�m���
�������N�R���[��|�<0�j�����Z�����J������z4Z-c�����iqW.����n�.��l�[�#�pD!d&����"�1����@0h ����x�St:
&G4s����9���9h0��:+�����{M�wm��=x�
c"����Z������x�$��7ZnI�<]N��t��������������*���)T*�;��=��l�������7�8����1�?�DF�o�9�Dz�?h��E,�}�����|o��������RM��X��!�
@W_B���q�VE��FQ�T94�+4�����V�5A<�}��}�|V{jY���a$������v���� �U�hY�$�i!"����t����:�6�E�Q���������xN��fN���D���{�����p1�������=]+���R����*�f%c�!����o�zD�3�.<��9��r�����I����5���qy�E x�6�dsT_6��vgux����K@.@�y!�������-1=FpQ@\n�y���~��F��`�g2y=�v��_����+a(���g$����=#[��3p�Rg>����Oc
��������4�/h����v��iw�*xN{�������V���b9��������]��xJb�����Q�&���K+"-m����D�H��'�	�d���z�KLrI�����:f�$	[	��"���d;m�%V�$�h��
�\��%I��.��a+�q�%!X'��8��,��*!�����Am���������
���Y�0��)1
����cz�^���O��N���Rj��39L&G����R���5��T:#�t������ARM���G�J��#M�<���_Q��x�!F�D�&�M�K0���������
����p��K��<��,z��7�����Z�s��Bc�K:'�}
.�^�����6J�#K���������s�����#J�W�m�i)�q'�A��3@��8�
V(-.-����7� 5����3��M=�����|pC���z_���r�D87�p������3{C������%�"��K:u���H:9R��('2
����D���7���c��-�����������������T�*���W�\n���u��[
����A��������$�`!���J���9���"����U%�5���F�|_o�?����
���%pGC~?!c�r�
�BE������Q�h&:�Y�\b	���Ll'>E����i�����J��kJ�D/�����vcp�z`67�(�*z�L���\�s���|�w����TT��+����v���}���=��������v���_l^L�v(;�xJ���g����T������R�����D�D��X��Bx������O��s����a����"�����o�E��A>Z�c��j������*��������=��L>����l�6f������k�f�������O�=u0���� ��g�K�fu%�=�j����/���6V���}^�/����iW��l.]85���d��W�TU����f?�g�)xg>'y�|��/R�>J ��Y�G���_���������8�g�QB�Gkx�\�s�k�oe)G����&����Gc����g�S�@���p�
��d����Rm�o���W�&*���=�u;JU���l1c��Gj)����87�8\,�;G��[��_u����qA�c5v����m�]����q�I�r�B��SY,b��/�Y��J�?���nA%xC[�n�Q�Tg�h�*���D�z)����,�]�k8�H�Vi-���5����jE��u�����s��J�	��	��'*��U1Q��B�ra�E�T��������NZ�p�	C@6�����K������3t���e� '�_������
'�����]+�~_����_?;��]����s���So�l"����m�y���/�����(�Gx��\�r��w��9�����a����
N��x�ny�������,��1��h=���8�+��D������'*�Jv���xfH4���(��`����Z��Vcf��4�32sK�y9�L:��7'��W<�Pj���?����>�V�P�����xrdSs�ft�^���`yyb������[�nK��J�������1
������VH�>b��wJ`��`6�����Me-���FK�������e��K>��>)�NS$U����
��������c���0VL�q��d�`d
���Vk��R�5>�������k(k�XO$�j�����`�{y���
����YXcG��`������u�eY������Q�x�-o����.7�G��j��0fc��f]��=���{M(_��n>��K���@����=LK;��=��P��J�5v8��)T�������z���e���4R��t*��wT����g��j%�����������[.�V��7p��L>g�r:;u�ftj���&��Wi��������	��+����|�'�5����q��(D<=:�����u!���Y�Y�r�Nf��&4���(���>��W+�q?��m&��a����1,�}l����h��j�8��Jc��/D�p��oT���uV#X�6-M�Xu�@���Z�T�����~�O�&t������h�N���	�	VF���]�����8������*����M������{{�fO8��1�b�n�]�(6�3�j��95� �o�
\�A�U��������B�U+`��eQM2~"$��N\��d�O���j)Y����`��HL���e����u[�NA����S"���j����&�[d��c?$��4$�U���Q�5J-j�_�4
�T0����NP59N��Q�2N���o�%Kx�g�J�
r���?��\N����7N�/�rE@�p��bK�) ��\x�B����M��D5:�n����3��s���Z�(h���N�������6�n|`U�Wg��������!�d�7j�������=���@v��v��J~�jp4������@��o�~�����:.�W:W��,�	=�1�������������Q]�m������Y^d�����w%��5{"���X�I��Y����Bh����Rm�M��%�B1����A�,������= @����w��I^�~����{���;�;���;��{G3�l=�`\�!�EQ���LG�$3�l������������(��0������������4vhO��)u[Z�o�������.�5$W�������w�Y�xU�W���4�dA�
e�L$���%ssE��^O�)�����B�z��$�U�U�soC��?��F����jm���*���0A��4U���g�����i��a�
�\��&V��
�B��$� W��7�J=^U�U	��o�9���0<Dn;N�)��2�B(L4,fe�k��Y��#�@���|XZ�����Y�E/T�M\��N�4�^[8���:����Of�+�c��kU����O/���eIVmou���,�}{U�:��ix�T���d95�����������gC�R$����T}�����������,�F��#l��*�Z���z��}��'�)u�3������we'!&�8E����\�����\�F��d��r�0Q�eI�Ike%���*��������~��/R��������D��du:z~o�:=;����=zx ����\�_���/-�@�e�f���lT����H$����i$^�<����i��FVj�#�7���o�\�9��tf�Ii��2$@h�N�0�+�<���V��or�
�|�\����j���H�L�����Kuy�z�D,�IU��T�\+_�u9�������$a����"
�$zK�eXs&�*�11y��J(	8�\��!\��``�I	��xf�b<�Y�I=���InT�H)�T(|�lF����q
b=�Z��g��:VdD��]���1���b��N�C���b����)S�S���\���!f��(�����N�z����d��'��O���Fmr�\�g2R�R'O C�b��*���mz����~A���������D�����Fr�H�'#�_2i�	��(���	�T2�}8�Y���{LZ�9t|}��9Sl
8FQl�B#����;e/rV���}���d�
\���tZ_�[
y�O	��U����9����a��tU�&E����H5U�PP��J46?G_�%Ve�k�Z�����%��!��A+����O��.*o� ��gv|��������Jn����D����Nd,��)1��O=pF�����D?A�O�V}�QJ����?�^����gf�n8�z�g;���S+N����9�T"��W�� ��K"��Y� 5j��2)�����\k9������Zvv����G����2s�H��Dn���hR}7���n3��VX����-�R�np�Z��DB��F ���Xi.7��>��+��1��#QE���������I�D~j�
���
���$TJ����=�P��oXdP���h�+�}h13�F�k��bUL��M��hf��7z<~
;a�+���2�7���N+�U�^���SIn�7'H���i)2��D}ZAj
�6�@(E���|�����2�dk2R
����Y��tc{_f)��^g�1Q�HjIv��cp��4)��l��4��$IRrJ������R�~Q�V~$&	��:�����
*�z1-ep�u�Y���4��7-��ekx>�V�+�>���g�*�XU�F7F�:A@�����Q���������Be��4�~��-��[�~�D���-$t�*%ra�9�D
7��!v���0%2�pR�>C�N����I!P�����N�H��U�l�/�������*S�&Nb�����7�;��������
������y$E����y.���B�"�A3X��,���+,����������XSBdr������K��v-MN�i����a�r*Q���g��sR�n4����6�%�C�4(D��Z�H��'Z���	�����ri����+���$��>C��o�E���}�Z��IK��i�������eS�n-Agj�B���Q����'�
a�)�4��u�����/��i'��os��5#�m__��5�b��n-��*t�&�m�S�������{�k���u5�wL����]:4��}oN��,�V5YE��0}�$���RV+Rf+G�D.y�/jC�j�CW|�U#�.�%*���)�NKL�L���t5�IK��H�d����`��x�N���Zj�`��'�tV2��E�v�����/�5	ms7�p�&Q�d�%�*Y����-_�qS�����X��W��g����)��vyi
�5�h3�����j�O���5j�,,I��*�j;-��&�@�\2�r���Xw�t�+��e����^EVF�-kw�?K��%\�jx>���T[X���L����y������,��xJ9��g6������B�$���h����~�J)�-h-OUfU��6��H�	`m~]G�E�Fk����*�2�lo��%k�MjQ�!�����l2$�,�Y��,���S���D�N#wZ�+s$� ���j:YD�<0�2D�	�Z� ��t�N*��!?�bn�!5�6��L%�42)0n3�~cozZ�i�q�B�@�$�E�����%��tEw��#�Vgf����T2������K�(N�H�j�$���c�����f��S�Bm,|7C�+����w�h5��p��f7w�g�$bqbBjq�m��6��P+�j�������.��s�����6�T�
�*�B�����'Am$�5���?PT��a`h��>��V���,�Xs�E��������G�r����"�>�{g�Pv���_�1)����q]�T&N��t����-5&:g��uw�1���/�����lQ����jk]�
�&
�O��4����6NU$%�'�ufs��
��]������^����BP�BOh��X��>��p����H�?N]�5���tZ���MRJ@:�FIr�Y�J`3ei�N����
~/�`H���i��Y���B�DAc�1�b�Zi�]���������$������<��o�E�g�8r��-��[�2���ZG����������C���z�a�����[(�f||��"�����&#��-�y���Mn�h�Pe(0�$7.���7�V�on"p�q��yT�qCJ-����fO���!�
�J�����8^}���;���7��h�I0T���\���*���1Rg�j�n�5�fH�������R�7�n������$�)9Q�X����d�4n���1rcs�XgQz�����B=���@�&��x>c`>��QP��1��1/���K'5KJ�����u&S�����u�quN���*C���6VeH)Y�Wv���~���r�����{���:\u��{}7!�%��0��K�?D�|<f^�5��
0���d^Cd�����p��j�Xz�H�IS_"��^�Rk |:$U4j��@ ��=�F���x���SD1a����g��^3�W�'PVv��T(,G��Ys���^�4)WHU��,��
��yi�H*�*�����4*7����uo(M�T�BQ��R�m�J��WiDjURJIcNf�cKmk�=JGO�D��F������d��
�Z�aj��VB,/�*��y����g4�X��Q:s����z[:�����:s��-���,���E�je2��M	��]�.st���o'9[��*M�P���=������i�����T�Z�V���Q�J:���ENI�@|'�j����\4���� ?�=B�L��?�?�%��b�T�n@oJ�.�&��%�HK^���4��r&wA��(YP�����r),���K���O�$�Ki�D	a;Vl�	��X����]V_c&;�DV�BWcns�����b�\l���to�wf�r��l�j�������C����	�$Z��g��I��
v�BKB��<By�_�����O����EXo�Y�H[6�+���U�u�il�m��i��7�m�����%��D�$A��^�^��XO�o���.�����u���U9��L�U[����������Z�"�B�X	��������*G����ik���_���N�2�]��0�U�G���H(,������^b��"S�E.!UK�B�w+�t�.Uu�Uy�JN�jt����F_�@�H� 7&$�����h%OM	�h���Jj�&��������3W�e��K�nJ�n�4�4�?���a�:�����ao��h#=;F�7�����b���.k��;H2��
k~=S��o���������������TG'�����v�:����_2a�~?w�il�hW�]%O�$����U�%z*m�j��RmZ��
m;���$���[�)��T������j�I�����^�k.Q��W:��� fkJ�����?�;�����H�:�D���+���	45i+�����E���=�����*�`RT����J�p.?&��c^�����F������_�KM~�MM��V����Q%�����c�T����6j���������OY!�����UT�N�Z-�0/"�#��)��"�c�)I F>][&L�I����QI����k�lVn��aE�*9x.Q&*���WXQ��!�V�G�n,���7�$W�<���h��,EE����]���^dE?#O�b�YS.7F�5	#�����s�G���J�]�d��8��e%3
y,���gs~j�=�)�o������������u��"	'��Bb{��-^J��Rf�r|��U�V���G+�V�b�=*UG�k��+kK��5W���>�J�w�R�*+�&YY_��i��r��~�pz�4�����4�7gD������+�u��e�����~S��;�����q��3��6��Y���.E��n]���7z���z{���]�n.�K\���e�+}��`�S}�E����d���O�)
�#�y`����My��A�����C2�0tx��a�03�7\2\���7���"V�Vm�l��������jG���w�w>��y�/w;w?��g�o�u��#�#t
��4���/����/�xB8��������{����|�[����~�����G}�o�+�?	�<������OO�2C�����u��&��y��=1����o�����=?1�o~~�����o��?X���p�����.~���K�\����/�������<x@t`��,}%|y��w\�v�w�����������l9�up�����<�����'��5F���Ir��K����k���k�]�X�,��@�z�v/�/�a�������k]��k�|���~���%.q���F�>�(�'7���F��Go���O�?���� �����o����_}).q�K\�����Z��K\����%.q���|�����=rs��n��-��\r���t~�9���y��q������|��o�~�.,O`�]\����%.q�K\����%.q��L\���W���QY��U�H)���o���x��re��<��1u��A�G�,��/&f�qe	Q ��+'��r��H}+R_J����ID���,��$��rb#���E�����2I��%\�"��\�&���� ���H2�_�,��/&�
���B��q�Bix�+'����R����\9��3��LL����0C� ��l)�0��Y�l����Y�lYS����E1�Y�l����Y�l����Y�l����erS��Y��QF��D
���"������}�P
���=(��bx�NxA����^o���j���(��P�=nbj�@knhc���%�������i��J�X~�Pg���`":�����#[�����P��~��j�ELru7���E�N�~��=���m��S�1��!�`{�A{���r�v����e�ua{y��pl���Z���'��.�:!:|�s����q
71}"�����4��2x����������82��5l�'b�~��VC�'���|��Q�N������Y(��B`���X� f�����8G�m5�mb��a�\XS�%����^�=(�1�n�����mbY�pT��U'��cn?�����|��>�3�{e�aRQ
P�l;6x���^5(&��EZMA]'��[>�k>�Yfl/�}�]~�v��jk�v>��z��������6�[����Q���>��~�/A
|����Q�"��:�suB�5��+X�D���1�F��2����M���1�.��W�����n��C\���_��A�8w��q��8Q/�DG��<9��u RE.�q�w�����������q;A��GY>�>C����c�� (_�6�Q�9�*z����Ay��8�"��9���Y�|�l�^��`k��9���b4��<�mg)��!�n�}��f�f��#��k�y���n4������"6O0W��V������n�Q<�BV�����sG��\�g,b���2;+�b�an�a�'��5��J�,:�9��xZ��,g��4/S��_���,yP?Y��:����6v|��;���a�9��yr��Yq�^�11�,amaW|�FV�x���<�<��l�9�E���+k[�����O�x�p��m�����e����L�u~�xbV8�y8�(��p�ts6�+�����b�8qy���W+������"/�q���+
�>���!B�P�����{E���Fo4[DW�6��t������N�
&-�{a�'>j�����E��}����s�r�s����Y���f�����fl�w+�9��>���]�s~�������a{��u���G�����+���/"���v�����Qn������;gz�j<�c������r��y���h4�!v<\p{D�����vv���n<��G{�Y�g���^�5Xt�Dg"��V�?;Cga��;&B�����m"f�e������j:���\����y<�G�7�?�����S���Y+cg��1%1�9N��~�g�i|v��q�h0�_Q�Q.{��+f��'��[��xu��8�����V�><G��L��?O��S������g��s��
F��(����Q������~~� Z��=Dl
�l���8`Y������{,P��{��=5����7��8��>x����8���F[��~7���m%��>Z��~\���{;�w+W�{a��qd�����s7'���~&b�r��G^�.����;�w������G���rwD�6NS;f�ZFm6�F�x����P��o�6��vc��}��V���������q� !�:A�V�1��M�_3���Q�����!z��li?���1C�v���U����5�*b��.�i������.}1�-g7����b��s���\�b��������V��}�����Hl������H����e������'F�?��X]��f�3F�V��9O�����1�W��s�c�>�����������,�4�����}����e�<������3��bY�{$��ezn��\��t:���a�����?0DG0���r&���2}No`��p�\~�$�����1��!����'�xc���&����rz�G���N��:�r3H�Yg��L�F�A&<�f�L�������L��f�S#��Q�(�e�2���+�	 �p�����
7;������d��� �������g�sNy�s��'<���G�^7�C��8(U��)8�7
�>w0T�8������CL�
Vx���+deBSN��r������=h�7=�B��;�1��������^�,3p�T��
3F�A38l�A_�1f�3�f;
�/
���Iw1��i	1SN�����z#|>�t�-AOu;����Z�=!�<T���d��L�}��qM8���;X����:�������C�P9�����������)gp��]��q @�]~0��q��;�]y�P>x�i����p8���F��P�d1`���Ag`b���8CU��w����}jE;M^z�����bs�4�P+��@���a������CA���*n��7��S�p���V��� n�A�0�z����`t���p��c��������5��,t������Q��>��<O>;,b�C���E���P8�q��w���o���@/0&P*	��3���y������,*�,0��
��d�Q72��p{��B^��e�#�x�8���x�(?�@�1?-He��q�@W�/�)x'�q����z&=���Y�����
j��rJ>�������Z��W\�NT��y�lBh`,y!�a���$B�,Q�d��9!<x�n@��� �����B�CC�8��+�(��G ��'N�|�]�H!g(�wy�(>`�A����l>�x�Ljq��L?��_����l��a�z8���1�f��
i����@��}����L=�A�,��\�C��H`
M�M�L��B;�(m`x��R�?�a3�9Ue<t��4Vbv�?u�0��@7n`�9����
��c�Qxul�C�q�L�>
6�{�a�F
�Vh�#�e#�chu
C0y�E���|�x�he�{���}�������r���0{?l[���c��gp��}����LOc���lrt�X��-�}���LO����t��>Gws�`����i���{`^w�H�Fz�!����5�������&G�c`��ist�6��Q;�k�p4v���������V����vt��A/�]��0�v�>�u6��{g'��>��a��{z��9�;�����V���
���:[�����N������������h�W���h���?;�op�t#3�{��`�
V�
Dv��Z{��i����N8�7�u��� ��2�@�=�������	m���c+����������u��O���?���{��������+�y���K<��e��e��e��q�	`l�k@_�����Q���?^|g���	��$�P7_h}��Z_�@�����T���W.��J��KZ.�~r2���������*��&L�Y*QA�~8�Jl�	x&�ibq9��8�H\���~a�$��x��E�R��A�O�)$����#-�<YAR��T�����J��Q2��G��������<B�$�Mz��^�����9M�J��o���g��g�u���������6��<IU��Sv�	��|��I>CM�������z����n�����-���.���:E{�'� ��^��������O��>��4�ZM�Bg�G�<�V�������{����;��~�����W�������o�����5>������������o����	j�Ov��@���@�w��X������U`�&��1��H�6k�^��zX��)`}1��
X����?�?�������-`��>K.P`��Y��X��V`���k/���W�C�����;��8�~X?	�O�����?�3��#z?-�Z`m���z�n��@�
���z?��XX��O��������������z	(K�5������%M1��������X����zX�����-�����`�O��%`��>K�"��(i�`]	�;��X�����u��6`}�>	�����������s�K�H?�JNS�����������X���`}=��
X�������������m`�Wz����[�tzp�M�^��9�0��<=H_Fo�S�z?�>���������~	X��?�����6�~_P@ ���"h�?�	F����<��X����.g��Ak#������
Xkt:������������?����{��b��.2X��v`=����`}+��X�?������
�?#�Q	�N�Hz(����z������
`�U`}'��X?���/�����z��-�7�F�����u#�����z�v�Y`})��X/�;����������o������� �~[�B�l���pF��������W������1`�(��9��5�~X�����1�S�u!������v`=	���w�c��	`�<�~��D��a2��J��`�X���~`}�>	������J* (����%��j2�j!{�A`��3�� ��X�������?�O��i	y�����Y�I��|��@>Ao"������S��R`}-��X�
�O����3��9`�*�~X�?�K�~��> ���"��ot�GC�������	`}5��X��g�����-`�1����P��?Z�����'�f�Sa��V�]`��<�9k�D�4`]�7�	`=���G��"���wpM$m��M6�M��"����n1tl��;��8�%"EE;�=��lAQ9�l���(���"|3�!��w���������3���g��?3�@�O��?`~�	���l��G���:��=@�����]��9��`��W#N�u�u�u:�u��f���@�g��o]����>�������p��L���;�:
�z&��|�������]_��t�������3�p��;�p��$|���~�	�����	D"7�z
���K(���]��>t}����+��z�a^[n
�;���}��}�����R�oxy����@�G���]������~$	�)!�HG����k|I$vv��
��)�:;�N�T���d@P�S$NQu�t���:�_�W	s�e�����(nh�wQ8N�&P\�"�lP�
xAE����t�V�
��6o����B��t�P�P/�M���Dc@�l�u.4��K%����zM���c��JP(|}��$J�Q�ti�����|0�RF
�M����tT�R+"y8�����J�+>��N8�����G�,
����z�I��Z^'�"����P&��=������c5$�6Q\)��<�a�T�_���pRx�\&�I�.M� �^�|��`F`�� E�\�$�l-`d<��%j>��	��4����L�0��*�`�@�(�a�g�Ade@��t��r�O
.�^�1���p.��8���3H�8Nr�`p���p��x��"&�Q�D"�u3�Q�$F�
���lr@#P�\����R�T5�M�&������������\ �rA�N��cVX��ld63k09��!;��K��Fv����
�h��5�i�\���(M1�����,��F�.G��&��������k%�h"%j��Q\9=h\�^���P%���;����?����z��oh��=�a4����P�%L	P�F&Dh\_S����WhB���
�N���B����J������� ��>����PZ.@Ahv�p�{`h��/�@��  p �
h����"�7l@$����������{�_�A����?���Zi��J7�d�����M��	�g	!�B	!����2A�W��A���F��1$t�@�fF�	
#@B���1z��P�e�N�V��"F�y\LH�@i��
DJ0��`�`�%�L[�#�|�cX�@�	f"�R/l2F�(��z�G�`}P�Hy�0%�O�~�2h��]��a�u���T���oj�_�3t�D���YT/��
�By!��y�aIa��������w�)�}�u�W(���4m���3�bJ�B�w�]BS$�c~cs��}G�BJ�5{�A�A�Q�@� ai#c c�zT�������w�W�$������*���	I\H�J�����"v���G"� ��(+�O�|�_j`�T!�	yZ��AI>?�
$}U'"R�7���$��G����MJ��l����)b��/.�*���:����S�l����#p`R�w0
>��5b�5�i(�vh�#�Q}�:0`�&�����|�����-���ap�R�Wx
��N�Q'�9���	�,��A���"9�����8�����8\@/d`vv6pp�m@b"���J8�S��!��>��\\���aI�r��Hs�:����H�\�K��|L�C�3�/�"hnF�{i�4S�#��/A���!#���kk��1l
�PJ�(i�F�"��X��7��T6PT�kLQ�Z�XP��
H@����2U,�D�*����:�U�D-bBQ��*0ByK����!��o��F���kpA��|\(@��� �@������'L
i@
���S
����bPat���4����&t�7^��O�E���6E��@�
�I�r�r��ER�T)��/�(��S3��� �z
���L$�n�>��*�,=]���*��%�cBJ�3�vL,/�V�4
��7L�6o�tP5D5T��j
�T��"A�s������Z0n-��@6��[�����G��x����m"����78�$�����p�Q3��k���,�������xT�l����H�n�j(K�7(DYz|�c��zA���&�d��'B�����h�\���	2�K~�f��X(f�����u��o���/r�S8M<���jA���QI�
�d�3�:i�eJ�}�I=�[c�Z����&�k�Is���\LLhz�b:xS]�5\��{|
bO$�DB1�E+��<tH��E|\��{D>���������ia}�>EF=r�>
�ti����Hd��2r�����	e,1$�Me���~�O�{%:.�Vt\��"#�������.;0;�M���B����AT2� *@Lc�f��@/�6�4j@{RD�_���fY�@:HJ��)8�">&�������������y_��8Gb�`D~6���H��H�	+`aKi��@s�mthe��<6,%��b�#ep+Zv��S=�r@����=5�w�S ��s�Z�#�"���
v������$���T%&q1�����b��C�p����\���������tW�n
:�k��hAp���������Q�l@
��O}�9@3dQU��
(+�/���������l�h�c'�\s��\���`��V��Z1B�!�9�f�jX����F\\Y+��
��(]5�V��<�W&�f��q�o����!o
���U��TXS��\��4�%������"�;w��'����iD��t0q��.
���?��������7�"9�]��4^]��mui��e?t���B������Z����"&���L&���*�T(m)��z����0��3���(cdL�j���������������������dXb�L(���� �\��R��Q����i��6�t�m�e4�����L�������PU���AB]�M�
��`�_e������
4�
q�Dm�6����P5�jZ�������Y��R1jL��S]VU�X	b9��U���X��
,��5�P�tP�0�+�����K3���F�r��r^����#R�R'��de�����&��L���.�1}04�ls0��eg����(!K������������U.#=�LiS���42����#2��T,l��0�c{���!���������(�@�p�I)O��Y"Q�"=��B�($���p�a��K��a@�

��1��fX������m����P���$D���1���7@iyeU]]Uee��F� ��;��t�W=����>��D��b=��;�a�v|`���X�nn>�I*��*�J�xR7\wT����D>g�`��9�C��5�4*?"�����4�4=�A��M��{MA�1����)
�����1��!�G��@T�(/W���PX�<�m*.� ����#+++y|������T��c�|��A3�p��*��yd�61�[#�&��b��
�Q&��(G�u_[{�I�����v�y���JZ�����i��i5�U�7�.�#��q���]O��x���x���?4��lb�{RU�_6���r0�`�@���������
�#��������71$#*=�C��4�h���\�b���(W��H�1M	t���l�
y�EH9`�0$�P����k�f58��
��J�jdb	�H�+�\M����h������DL���\;�`���k����R���X�Or�4i�����3|&L���7��c����8��g�|7a�d�G+,~$�33>�|N��"d���LDH^C.����a��	R�����!�`�A/�|c{��@q�=�8������=a���*#L6q���>������V���-|�p�9
�a��(���\��w],Ob��3��P��imo�	�g�nr��1g��1�&(c�����c��be�f���!�"b�b#d�is�#4n��W&��hK(����C�c&9I����x������=h7����k�� ����������"(��:��d�X��x����C��~C�z��:�;vsssst�r�.�@��#2������N+pk}
�<����0�/�(p�-�i������������t��^[n_���Z�����`����~���77c|S\��|����(�.~�f��//lt=�8�B�	��o}f��G�2l����~��O�����4���.*����V�t�{�q�2�R����������@"���;�����O��\��v��~|�(��66K�^=;�|���`�
�+G�gse��V�I5�'�`�y��_��%�p2Rs]]������)sn��T�����i���7/
��S����QE��}�h���)�fXr���6)p���*�0$Z&�N�w.R�����}�yJ�i�H�l���0�[3&6.�7C�������>w)*u-6�����@�?�� ��/�G�7��	��y�A��h��U�
�]��gM"�J'P�IR�1y`���t?:�9Ms2<4
��=�{
LJ���ic����6W���qH.���1����?
��k��[vR^�c���L���<��S*��!��AW����p��I}z���������o?����A���/����������t���eW�z��9;�������2l����%*�������K?�s5h;p�����f�K:	\�n��/���g]���������H�v��O�6�E��sj���Mh!��}k^�]���]�����I�D�]S��]�������rW��nc�'�]���_��|���������
�i�.s����&��� q���z+{�Y����	a�L_k"����
,�����[��#&I�DG����o��!����2�3
�K3]�N�W���s�B��4R.:b[�_��In��1�!A����My����7]����g7����>���T�S.����yf�������&����Q��t2}lg���^���VY���]�����U��O@�-9��^%�0���+Z��u�����im���o��V'b�]�������%�v�*{�8j�gb��3�������������~����}S����Q�������`�6�]�2�������������r��G�mJn��`�����8�[�U�s�i�����C�������=�yw��X���F�H*��7��y�k=����7&�_
���)���+�v.5��C��X�9���v�q��a�g���M������.]'�9������������[wg�wWYd����Kd�W��x�������������grV�5�K������\�+�w�p��iw��0=��nE�~��@3���DZ;n��M���qg���cdk��#NW��88����/����z�����!���ej4��bBw;��X�:Yy<r��c/9�l��I���S���j�y�`�������-���0��..�7,�Vd������*�d�k���e�[�Ww2i����4�;�$v~����]���#�����?<C��FJG��+��9��\�9��l���}���S���2Z6�����7+����{S���	�6���5X��%F�����eE�t�h����������_���i���M�G��{�����PeT�V��f�cz0pAR���W��i{�$���
�m��{�8�u|JY
����V�:��~G��69��7a���sI����l���	?���y�]�="��xR�S���}|��?\Z9|s��-&^�z��Q&}����-.4��a��A��'��|6ja����eY����rk��/�cj����|���TYc�?��p#������?r�/4�L�j�6����>=G�V+_E�����N����_�q�]nq|�x���uY�c!�Y���]�%�F���"��]�a�'� ������p*�t����C��5X��-���L��X�va�i��fH\�'0�������IR�������!�i7���&sv�F�pw��d7&�w{�����i��o�tN��������l���]� [�W�n�u��DZ��%�����������?��x����5�|��D~]f���n����=������J�O�6��9����%���{/y?n��c�ovw��e\~l���iW��a!�G\��S������F��>���W��*on��w��!1!��7b�"[t������*��o���m-L�
��������LB���g���q������d�+�T���ca������k����m��+=-5�}'��6�HK���x���%���A���%�@�&��aF���lN����0&�������w0n�|_8d��G��7������$�n]��`��3F��7v�D��B_�C>^}��}�V�$Q���zB �O��-���������?�]K�����v�������/&��sJ�>&Fl�����q��rcV��##8���������8�w�j��x��cIo_����zpr��W�$�A��Vw��y�t���L����d����;�Z�~��(i��������A��N&�8R�cm�����/&���:�������JY�Y��]D/�{7-W��-���H��A���q�2~��������6$��}�X���c�6B�+�L���89������2*�.>w����O>��i�D��������&���>�RQ':�p��O���5s>��=��H���6-;�y�dM��cr���h`T��&&��Y���|��r�Y�zk��>��<?�{`���*gf@����F��z��U����;��)�l�j���5��NC���;�pV���B���~
�k�e�������+�l�E�Xg��e����Q��<�z|��|�����k^N��r��$�,.X9��f�B���]��S�c�:�*x�~��������~�W���)�S]�$�����2�&�3c�V
lWX����>N�ZO=g������
�����0�
Z
�����
��i�uH���!u'X6��i����%e4L��O,
���\;8`�>���S���iwU�O
�@���i���]���G�t�t7#��<�T�������U�:�������q���L����ssCm��]�w�����T��_�P����.{R{[9�|��y��g�![v}��
�T�{x\U�w����.���9�����?��#v�F�&m-7r�*����=�����[��?58Y`<�dn���v-�8�ncJ��e��}�W���,����?d����/k���A�fo��z���qO��g{6:_-����o���+��?��/�r�P�#6b�����nk�~����K>L��r*+w�q�Ub�	�v�*;��w�������}9{�m����	��r����	��]���RvpD[n��)c�^�y?�h����z���]����V�'�]��i�B��6}���M�{TR���Nxj[}������G����f`_z����5c6�m�[��$/���������fl�u��(����I�wM�m����f�����)�[��L�6�����2��3���tM\�!�>I:��x\��������*R���>���E����_Q�Pfi��Z�v>����[<�{.i����C���!�
>������e�O�cO�,���	/���mge��d�����
>BT0��F�����[���Y���	t�	��h�������t��2���-sS"��v^J���i�t�r����t�|�Oss�u�:f��{��E��%$���p��C��ph���Rl�C!���	�]��M�Aj���+����q�Z|� �6cK�zh������(�5��-�����^9�J�8�d�'����_c�7���Lx������E�?e��%o����9����U�sR���7���8�r���=�N�k��n���A���=<�8��;)�����������~�#����E�c�w��kj'�����s��SG�
�D���_�h���e����k�u�����{��T{��;���������2oX�:�>�;�xq��}{����w�y����V��~#���w2����N��zJ��������pb��Gn�;�y*�����q�<��Vs���z,�<iS��������2�o����pu���l���j\���1��=��������&'����)oG���:kh�����5y�����n���.o���dcy�?�o��>~����_��qCJ�����;?�0��?5�p��/g&���p[�l���F����O�~jX�R��������%,���������
:���.Lj�l���K���������*���><(�O�}n��1B&p���
%'bb���al���LA�
b���������u?)H+�����+����B��i}i+�Fw#!h����m��77�ZV����N?����w�X6�Z�����+����)*�����g���W���L(plH���������-��X�`~��E���C��rs1r�\i;�����������8�)~�d/�`�/��0'`ZyND�����_�2~�w�����K����'�9~�h���/gb��[��J����N������.��O��B�}��E�e���~y0���gg�T��[��Z����~R��wW�k��m���Tf�|��v�,S����4Z�k��)��ou�.9�}�R�S��q�<}f��,7�����/��*�����8y��eu��U��X-*Tp����V7G�L�i�Z �\�?;��'mz69�6�7I��'�8h\+�������2Wg��d�>��{��;��cV��k��k%sd��"2����5�//��p~?�nf�g����������kl�9G�|)~p����S:n�^��_�x���I&;n>�;�u��6��6��z��o���!��[��;����������rX�\*yn^����Ic�x��g�WD���*������_��z�E��8|�����3�.�2}��?f�U�%��xq��o��G��6��]���{K���^���>e���q^�>�|����g�n�����v�{�7�^�/�
T�����|�s����f�����;s�"Y�;K|���F��
endstream
endobj
1648 0 obj
[ 0[ 507]  3[ 226 579]  17[ 544 533]  24[ 615]  28[ 488]  38[ 459 631]  44[ 623]  47[ 252]  58[ 319]  62[ 420]  68[ 855 646]  75[ 662]  87[ 517]  89[ 673 543]  94[ 459]  100[ 487]  104[ 642]  115[ 567 890]  121[ 519 487]  258[ 479]  271[ 525 423]  282[ 525]  286[ 498]  296[ 305]  336[ 471]  346[ 525]  349[ 230]  361[ 239]  364[ 455]  367[ 230]  373[ 799 525]  381[ 527]  393[ 525]  395[ 525 349]  400[ 391]  410[ 335]  437[ 525]  448[ 452 715]  454[ 433 453]  460[ 395]  853[ 250 268 268 252]  862[ 418 418]  876[ 386]  882[ 306]  884[ 498]  890[ 498]  894[ 303 303 307 307]  917[ 498 221]  923[ 894]  951[ 498]  1004[ 507 507 507 507 507 507 507 507 507 507]  1081[ 715]  1089[ 498] ] 
endobj
1649 0 obj
[ 226 0 0 498 0 715 0 221 303 303 0 0 250 306 252 386 507 507 507 507 507 507 507 507 507 507 268 268 0 498 0 0 894 579 544 533 615 488 459 631 623 252 319 0 420 855 646 662 517 673 543 459 487 642 567 890 519 487 0 307 0 307 0 498 0 479 525 423 525 498 305 471 525 230 239 455 230 799 525 527 525 525 349 391 335 525 452 715 433 453 395 0 0 0 498] 
endobj
1650 0 obj
[ 226 0 0 0 0 729 0 0 312 312 0 0 258 306 267 430 507 507 507 507 507 507 507 507 507 507 0 276 0 0 0 0 0 606 561 529 630 488 0 637 0 267 0 0 423 874 659 676 532 686 0 473 495 653 0 0 551 520 0 0 0 0 0 498 0 494 537 418 537 503 316 474 537 246 0 480 246 813 537 538 537 537 355 399 347 537 473 745 459 474 0 0 0 0 498] 
endobj
1651 0 obj
<</Filter/FlateDecode/Length 55512/Length1 141116>>
stream
x���	\�������~3�,0 ������n�lB��06��+�����ejV�i��/Z�����^��{��Y�b��J[T����3 �������8����{���������|��E������]?�Z����.*.^8?��`G0�#���[����5��n�����Un����O�?������k�(�Lk�>���
�\�����Y����G�n:���
S���<n�J��A9
h�>�:��0��0{�B��#�b�K��g5N��Lu�=�Vb9�g�/lJ����
X�0{����+���O������Q�����pL�k���s��J��B^�i���_l/�X|
@���c��q���&:���������V��Q'���������~ 	�SB'����'�=q�*R��C"�s��/�:X
l�+�h~�~E,���X�����B�1D�5xL��>�(I� �{$\�n����j�'%���B�������|�,��
{���?(�X/
���*S�a}���E��o�"�~�\�?_g�8��m���^Q|�2���#}H�G���v�_!����4����u��s����d�Y�Wo����\�R|	����Z��z���w��Dt�����8�����������g�WP����*K����]&���wO;���!�%�qh��s������W�oR?�+[�|�]�D��w�?������w9,k��\e>K���J��T�u�{�����a
D(��??��^��n���i�|����jS�'�
g��})BO�|&@�}���������+�~���7���\v��^~�YG�������T�{�~������vo�b���h���v����������u6�iAM�����s�����U�����A|B���)����r���a{
��[!\8	c�b�G
�0=�H���/��!On��rb'����}���������Qp_���=
�x�+^!n���l��+��6���i#,�W�ClA������\��W��r����2��Z��Z��:�����OA�����w�t?�>����a�x
�:���H|�6M|��
*0=���^��@��F�����V�F-���Cx���P%��
�uH[`��
8��`��}(�@�E�����i��P�Q���r=k|E�=��s�O��`A��."8�y���
���wX0?�9=c���
�a������$4A�0�5�N�2���W�G�k!F���]�m������
���%P�g}y�+�k"=
������*X(�B�8
��B���~�����"L���f�{�`���5���g�B������j;��1z�+^��W���x�+^��W��)S�yo��g�u��c����3���x�+^��W���x�+^��W������=�x�+^��W���x�+^��W���x��M��Z�jC-BMB-G�A�)��}O���4�E�E�z��
u0jjO�����?��W���x�+^��W���x�+^��W���x�+^��W���x�+�.����#��W��"z4�������G����

��
����)���AT�ja"��E���C������hKtFtVt������D5�t�"C�a��*����OJn��F�O�L�T�9T�9���>�a�����g��3W�i��9����}�d��R�8g��+�����z�(�O1�=YxV,�@��W�������Oz��W|��#�I��Z#�_���^�d�;�����A�?/���M�]�#:�Y�7b���[��������xL<b��qM�@��.�8O�)���t,���dV����lkd�l>[��bk�:v
��v�}�i�{^%���R��|%?I%�%������Y�0���A�~�G����M����	��B-����D�}��\�o��ju���$�Zek�8EB�/������>;>h��u������?/�oK�D�����M�^���r}��d�������u�k�GUU�Q1�����a%�E�Cm�C8 /�N?�%-59��`��
�h�*?_�B����:�3��)%KK�x�X����:�M%��8
ur5C��6�9���6�i���t�A0(-�Pl48
mlL������g�����R���b&.[���NVg(v��oh)�+B�jU��p�*-ZUjL�1�L66���!LN��Z���n����~����Q\�����mP(�r*�>�/�>fXkhM����M����)�)��N����--���fg������p8Ny�3�XT�4�Y����Sa�
-�ol�����cQ�t��'�����]i���q~qq|,k�l0	3�U�`��6���)���}]%!v^�����y�1�/Uq��{~C�s�$CZ*F_�6�7��bb���
��S[�EE���V�	[�g����V�__�����P�pZ�M�`cU@�����j�����\�����VNkq�������}�{ �}�5�����P���-�EI,nqL�����O��9����9m��Z�cj-_%���r���{�[�����U�����kpz���%�b,�:\.9�W�`�����U
{����^~0#�
Ky������j�H~cHz��&�o_:4t������Qm>�C�����T������)�Xx:��|9K��D^�h��l��npB��a�j�5��U:��x���-�6�W�q����%5�rT�K9'�aqWF(�=Xb�w-��&����g�uZ|���-����x����e�ks����,���XRo�{XIK}�{���V������a�a,��b�v��c�X�_��
�rV^S���gOA����j��5�c{t�55��������,s�1��.[n�F�1��4
3�r}��
�T�
r~r���ec0�M ���&�M"�M�q�E
o��q[l���gImCK]-�� ������S0ie�R�T�8��n���|�+��7e~&�����
�=��(r��6���w@�^�[m�����g��t������+&��q�������&����r�U��~����k��m�v�F�qmp��+0�\Q��5�N3j���sB�q.;�T$����-A�L���KAeZ���c�jY����j)H>�d#M�3`�%�\�[��R��,S�H�����{
�OK4��*���7O�-��T�|jki�rn����s�qD�=B�i����2>�^�C�U��n��`�q!�,|��',vjMe�x�S{5Z��]�}�����OV>s
�]4����3.��!i�F~s��{pcCm���XsZ���V�lni������W�M4��O�zh�X�[~�8_���p��h�)��P�Jb��r!VL��,�u��&<���Qw���}h4&���Q�����A�+�8���E�G� �!��YH#2�4�S�Y���L�*�2�
D[��5��������J,&A�aTG��u�2�
��Q�C������l��a[�6`mz4`�0�R���&���1#~r��"~$'��(�=�;�Q���P�v�7d�����K���	��b��R����� �!Wt�#W��!���������{��6�-���7	��N��5���Wh/^"�H��@5�Jx���Y�~�_��&�#<E>�$<A����	����!�&�"�$����L������x��a;a�AWT�����>���{w�"�I�� l%l!�N��p+������o"�L��p#�jw=a#�:���kW����|a-��pa
5XM����p�r�J�>qaa9aa)a	�R�b�"�B��|B3aa.a�B���q1a6aa&�"�Ba:aa*a
a2a��PG�H�@OGKC�uE�G8F��B5a��PIIA� '�. �J	�%�bB��P@J��	C�	�	y��<D.�?!����M�"d2�2D�
�`�JF!��J0�R��$B"��
�H ]a|C��� ��h �b��(��I� �����!�z�C� B AG ��
AMP���/���J�� D�@`����NB��$�������n�����q2#�@����(�[�?��o_�"!|I�����jD|F8�
�
�>%|�
�E|L8�
-D|�
-B|H����+���+��.�����-����
rv��:����o��5���W/^�v/����?Ox��{�Z��O
�B=C�~���#<Ex��a/�q�c�z�n#����G�	�������uv&���$<�
�s���
���p�+�q�+d�nW�H�]��Q�;]!6�Te+U�BUn�*�Q��T��m���7S��7�B*7P��		�������P�
��]!U��Tsa-���@\�
�E�q�C�v�G\�
���<q�]N5WR��l;�G�c��/�=������P�R��u���:Qw�>���v�m��>�z?�}�����z7�]�w����u��������PoF�	�F�P�G��z�����5�n@�u=�:��~�)���X�$�b�rW~9.s��5�0�����%�&B#�b�l�,�L�E�A��.�B!����C�G�&d2]|�f�	A�@��@�'h]�(mLCPT?�/����K���E�����Q�B=����������������������O��E}�1��p)nEmc+(��]�|�/��,$, �'4
	��!�0�0��B&���#��������(��;���"�X.%T�����U*	#	#���r��2B)a��PL("��h�B,!�M�"�	��B8M3�j���z
�$�	�_p�F�	�G����P�U��;�/P?G��0�����~��{�e��P_D}�����>��,�~�����>�+��n�]�;Qo��/tP���f��Q�5�SX���&&�	u���	���q���1�Z��p!a4�N�!X	
u!�`&�%��	I�D���&�`$(A$FW$��D�Q;Q�������A��Q���7��P_E}�u�h��B��^�,�+KW�/�����t�}���v���K�����z��K�-}o�rI�b�����������������f~i����p��f1���yJ����������]����6�>[Ps�����4�X.@3���f����9������9�s�����Cs��>�U���#`��s�Kx�~sB#Kts�������6���5�G666.o���T�by��Fa�[���������f3�+�A��Op�DU��B'0�V����L�E�����m���,S�S�M�O�L��[��-�����������6�^kq�/���-5v��{���>j[�}�e�}�+,������XJ�e�J���l���^,���b��)fE��I]�-4E�>-6E���Y@���
�b���!bK��E��5MA+�����Bz�-���C�n
6l	� ���m�;@��v�?����8����������o�(	��jm��Zq�U���7h�Mk�,�i�J�5#55�
�iSJ�U�U�M��������C���6�XHl����^Uc�@�����=���[9���8M���V5��\�����V���meBa�3��l]��Z��������uktAm�sO�lr����Uj��6��;�<��/���e^3~�`��l��K���b��5�r4���6OlFX������&�U~��T~u&�	a�����p#�]=��F��������$�������.��[a����ix^��
�`<��W�=�������(�r��~�]��\1��>PB����H��#x<���l�\��x��r��i�������R
:��Nx	�GY��������V��������;:��N��fX(~�RX
�`9\W�jXWa,�cz-���p5l�k�Z�6��p�7���	6�-���v��)�����F����	���y�
��}p?���o���F�?���pZ�E+��m;��	��������]�6�����=����^x��u��+��l����������~x����������e8����T����{
���v��7�-x���#8�����������X�O����gpk�cM�Gu��K��=����0���L����_���$�#_=�:w�q����|���^��0��z�Oo����X�#��sG���P��b^r���=+��<���%��%�{����������{��3�\�E�JOG��8�ux�������R�y[n�������#x:|����Z^��������v�|������'?�1������;�z��'��~������G������5�&0:O�N[e���)�L�e~L�4L���OY��Q��.	<�Ds�2?����`</�X8�dz<7�Y�eq,�GYDw�K�,��<e�r�����X#�G����+�<��,��c�YZ�0���X�.�*a���/���0�*�����xB`��gwA��{�GX
{#�n\���
�*&�LE��G��N1���t��
�p��U������pX�O��9W|Ol| *`��-�
����]EE�i>ObV{	|q�n����^�o��\'V����j ������Ay���A�����������3X`\�������Ri����s��2�������l���?D�����.��������bqG��(n`u���Ma�}|}���)�P^a�I�TH�JQ����S`�/� �UxRTtR�
��xF��{������{�/�C���jA��{[rLHBF��rm�V������
�W�-���i
S��L�Q&���1#�>!�E���gd�c$����`��m�/m�<e�h��Ze���j�1^��M���cS���MRt������1>�'(zT�]a�����������@��	��"�3Y�u����YKW�����OO��|����n��}�g�����PZ�$1��_4�'&��g�Pa>F1Nj�(Cs3��b4�����$mt?�%;X�a�:����%I��g���qRB�����2����ZR��5JKC�����\����H9�+c��p[Wlc���#�!!j��sLM����SG&���>	m�y'��u���*;�����z�O
�/����n����1k{���gP�O}�?�%=���d��O�����	�{<Fd��h�������X8�i��������X��{.n)4�_���=<�l���yEi��1����d���������C��$�J
�I�UKj��E�:U����GICRB;�zk~�/i��#;�	K+�����Z�(�xM_E�sE��I�y��pVq���e"��K�j���y�_�<�t����h>����|���:�L���l��2�������[G�����#��-%iD�*4ls��7o�t�����)�"�J)J*�K����E�T���V�L+�jU_i�1"(,!.t�]������c���AQ�Q1}#5F�1����9����l�
7�O��]�wY�B#E�)�#��'l�p�A�g��m�b���������+��w�3/��l@�F��k=��������%y���{Fw5O�a���f]?9C����ZN��j�Ww��}��S��^r?�:�H\�3J��i>|[�����pF��ZP�c��-�HLTFt��y���$y�I�'f6e�o���K&�j���A����#m
����xF'�
����##\���R(pCtf��~<������������ �"�b��Ww�W��� 1L��Q����2�W�-���w(^;����G )�qad�%��*�qF;�2���p��]��y�!�3+�GU�	���X�_^u�<�`��L�u�g��:�Gk;����v�tm�K/];��E��?Op��*�7��`�o���O��P��1��Y^t������A�<�l�.��*dh���n9���:�/�jZ��5Rqt�UB�����������,5-'*1L��
���I��
V�>���-�������u�`7�]4R��y��mS�F�=�Y#����+��]��=O!�mLfRbV�V�����MP�TJ%�H�u�<WH
�	e]������/`T���S����Y��m�V�{�zFC��7��cG\�6&K�	
��cR���7u
���������z�[46�Z��f��,���m��]	�
�BBNU�F�8KX�Gw�����66O�����^�p�Y���U����<���L�>3'y�=30K�_�[���p������Z7#���@�����?����r$�����M	�Q��*)(6=>>=6H��QP�X���I�n)H7hX�����)��V}RD���>yX�j�Z�:�i�����c^�S"�; !�[��������V����%  �$����1��Rx���)��x���4je@[�(�9�Uv�I�V�h���G�/�<3�#n
�`�����S`����!g��O����c�Jm:��O�1����d��0A��zK������>1;%���PsrbEM�%!�����7�g
�s�,�0�c�J�V*���V�6�_Rg����2���ba�J�Q(4x
P�>��P��$�����}7_Uq��1/�jc8=��O1����l��4�z[!�v{<~("*o?����o,Gn���M��*V�����3_Q�)�xGg���w��v�I��w�����-[|���X�_���l��H�k9
R`��^��|\��-<m��@�<H|27�T*5����vf�e����P$=h��n��q�P��)����x�z>p�E+�X1�s�j2�Y��z���������������(q�������}���Z}��L\>���W�Y?�.�m�b#�m���{�*.(�\���yfWP�J��5�4�b��j��_�Vy���5;x@~H���Y����o�APx���y�����gu;sl������\rY���]����6�K�������s���4!5�'��>?$�����vr��w�x���V�J�����d	oO�w����w��� �)��y��>��5&(�=]w���M�225z�>4~�K{$�?~�����6�f��+R�C���_�����|0�����o"���]~���k���������z���?�A_)������s#�>��gU6����^1�aREzq��Z���!�>
A�����%���m���)
SF��+�m���a����K��;$15?��>�>�j������`�0cdtr�&*Nb��6S�\������(��2��m�=���?��J*��Z,�TRU��K���[yMb;�c�1��,@IL���gh���8H4�t3!��	4��L��4�K�{����af�rJ*Y���������������[8?`�6;bF'�+�fP@�5�57gT�\�U�W�P��7����r/�-GT�A���3T��R�����O�}z��K���d�>����i�e��\y��*�0��z���(����,C����F�����z���+�5F6�Lm^�������{�?w*�Xa���������>�f��b���g��>����>�y�|��f��t����a��#��c�9�'����,�F�:�jq�������u:i�������
�"�����D����v	���;�Q�T8���%���A4�C�J�L%R;�g�Q����H�*���m���%3*F��*�R��=�H�&���5����;4����N��O�d��J���H�����p�+����+k�����i����!���Mt�&~P��-~����!��o�]�T�q���q>7E����+h�O/�Gn�+[F��h�T�1H��U�je*�EIk{���M*xT�S�J�����dz����J��$���n��T��jO�
c]2%C��~��)8�2\����������������>u�n�V����a�i����Ae
W�2�<�D�{����y�SdU�
�BMg:�n�����n��B1���F�rWO���3�a��sO9�j!�����@��r��))#���R��v��?~G�5�j�6q�����!R�$�
���<���?��*��a������d;s���`7�������C�&3(BW5).�_][R/����9{�M������|l���w����+2���|S���?I:�qu`���3����R��%.�x�uY�w<�2~F��]��a�d���He�-�RV�V8b�/7^�w��d'h�gu�s�)`�k���1�������a���Xo����g{A������5%����\I">,2'
���|�T���w&��&�'W�x&c�wh�'Yn�0�O|�5�t��_x����h-�X���(���xgi�}�Y/���GB��"��<H�������,zV����^�Z���v�����<Zy�C���)V�d/���
��Y�q�'�vV������r�6�����
���)�$�9����E6�M	D�9Lr^�b������w��!?�ox�!xc/6$���b)%�L8��Q��$I��r�J(��H"�\���o���Z�z�"
k2���������*�����)�O)x�N�8�����{�D�F`2��+���Zrk�������n�s�"t������B��!(����� �5��3�%������7�e�w��J�6�[��KU?F����������]����X��s�Y��>�?	��Spb��zX������������D����`P����j�����l&zTy��*���y"�/O��<<|bC<��X��o� �V�h���Z�\�aK����[����,��1���{�-r����W�����X�J�2\j��^��b�LTx������ce���
�P!50�F@���A�\�.�3��if�)+r���FaZs�*��/��&����FpT�B`�W 7|��^���JU��>Y�SM��P�Na���k��=�]
�H��,%���C�u�a�D�K��������Db�b��+@��q�Wp���|�����_���A����4
i���w@�H�w
��_d�@��v�r���a�g0�|�R��ykl��M�8�a��W�5��%mrA��%��
�(�MZ��#�+���BF@p�"�21����:7��)�z��R��2R!���E(S��*&�N!�(���S�vb*�'����3gH.�T�.7��K-��J�|�������V�����0��-|�"���,�{F�vP�
�C8�6dEq�
�I���@�h������i�#���FU'Td�u�!]
Uq��Q-�/')JN>��y�������(��S���2I���8|�e#F*�~-�
[����[��z�M��]��\J
!�����w���2G�;��;����>��Ht�J��]�����(|�cJM�~-
|��V���0����UU�]<���D���?�E����1���p���0������=`�1�Dz���������~��}�\!o����)������
���1����s�Z5O���tRh��)4_A

��]N�7��Kr�U��d���w��N���'��Py��3=T��k4��*���p���>V����?~�O>^����O���)��:�j���"���Q����i�]f��bgX�8G����9L���q�g�N�%����UK%��B����e�|���C_�u7+f&�bAo�������>V����$��J����ME+��Z�/������G�<^��!�8{&�_C{�A�gD69��a#�����m"&�2�fWz"h�����"�T5�_F�����,�2�_d���������|�d��o�3��G&��n��`�c����>���~�?#^|d��op��r~�{3��f�]�������������gc��<�����r�|L9��3�������)��$�	��FQO�X�^����>�bo2��P i�e��q��A6�f�c{:m#I�������������m7'z�j���$%c~�����]
0��������6e	u{~n2�?6E����R��h�\�-���������sl�MJ�h�,�D/`�����X��]�F�3��t��U�OM�]*��������OO�m�(�|�2�o�qw����57���v���Xx]..�B@�h�X��p)|k7���a�#Z���
����zu��a�Y�&G�c�&�;������^&�:��J�5he&��e�x~����V�6`=HH8f4RR6��g�;)n:����Z�u.J��eWP2����+B1BD,$p!��w�C��o��]��@=x��a���x.��������Q��l��Z�����o����r�W�v3�����%3��&�e}3�=y�37v$v}�|_=�ybM�hN�������!M�� vhaf��:��V=���R��<�������3t�!81N���I�zO4������Dw���*��y�t,�2��_�?�bs2�+�t@���V+�O:}*��!���X��I�������
�_X�E(tE�L|��R]�V'v;���������:���V	x���T��d������ ��J�Y80�O��It���W�/�d�l�m�-�U�$"�P�H���H2�x �����Y�6%�AT>T���U�M�Sj���K/����U4I�J�<�_�CS�T~��a�/�h�X�p���E���q|	���C�rOee�aY&4�2J����#"MjYI�Ex����������w���7k�I�s45����\�3�������H�*|YB?���b�DFE������|R$c%��2��(�/w���]>eU�z�b�\���Q��#���������n>+fh'�K`4����������4�	O�q�^��L-nZK�&q��R��J��V1���ybs"��P3
�T�S��a7���B��R�r$�R!�SAO�A��[�m��(�`�Z��H���Y����9|UV&'����v�Ib��X�S1��d��?����_90�C��:��c���AHV��i�������b%�N���������K�~#��I����������In	>�D�C0�\�
]�|�����n�9������D[�8��!��'��
*n��T�'Ot]���h���G��=}�������|���r�s����?o�^���{�6���@ ��;��C7Fnv���~��a�X��evZ����6�B�J����$�w^y������5T){���������T���E�:|t����1'+e8e;�6�ey��������DP����h��=�������W �h8��?�)4�_���~����s������`��������r�@BM�,m��b�!�G�=U���m��"IJJ���e����JFO����I)o�g����.8w����W@�L����,~����k�G��6+*��7��AW�i�r���YQ���h4�]���>�-���~���(�v�'�u��w�����/n���4"����d!�}�!QN$��Y���_78uR�� �~h��������,�����@����
7�V�J��h)�;��[gvO���BG4������gJ�,>���S �����|�?;z��YX{��X.Km,�(I�"t�Q�
��/_�M��	Q���/s=:0RV�ck���_�|�ek��\�<�F�hY�fNBM���S�M*;S������������Ltr(�1�)>����U��-�m.hK!I�����u�����_%sH����g��7����
���=�1�U-)l���f�
�������k��#�8���P����������J5v���o��x�l���u1�(�U��� BG��3�F�BV6�9�,�����h�.������.�OC�����y��s�-y����Oj� ���
��%^	%�&J����������5[N��5!��`n�D�^o���W^Y��m��m���		T���$�T���c�%>��b���w1T�,{=P]���`�W�R����JHe���{�w�	N���{G�9|�l!=*��w*K'���qT�n��g���?���x�p�i'��r3���Z��H�&�;T��Lg�9N{�M\�7��e:��I�}�/��Tk�W�k6G���&�^����5c�]�k�{>�{�cS���]U����*�N!�SB����K���c�R�L�C�������%��*���-�g��/��$�8�����F6��n��z��H����@$�]�`T.a���]��H���R����F��8=.�E���Hho���v>����]���	���}��Ghc��v�E�[�����3#)���(�0CY���h���J�p�U���)�ZyU	��:��������	�H�Dcjr���(��b���U�y�G7x��>�Es���������j�o0i�%R6G=��(��L����N�7�<�,������M����Y��vr�HbD�F����E�$,����DP����I���b�H�����$�$�u'ISd�����6���lQ�' �Ip9!��,�����5��k�e��.�����-��K[&���Z��4+�zi�,��������P�}��3vN����Ja`#pR(��x���������n��������4�G�8�N8���s(�J����X5R�N+�:�����\��'���������(����I�E���&0�$��)�"fRy;�lk���F�����	����/&�/&.r]g����������=��R��R��Z���|A���/0���3��J|���|t�'�Eu��f�R 
����x5$������7=�),������+o��B�L�c��Y�_�� sd�����j�:�k�m�d�C7����"�n+��Y����psV4XH�\Y�0��g ]F��<��G�/&8�u�S�[�Z84������%�87�hb�������qn�Pn_P�rJ������9�����'�{�:	���1��`����w{�����X<mMA��*�G'O��B�pw�V�����M!R 1"��W'�U�����x|~��'<�P:wbj��azu@y�7@��[����x>N�:����Sw�x��Y��Grw�@��On+�-m(�*��T��JP��Q�!|�Ewn~��>�!y����
i��H���#.�f���\�[�v�
�����W�$���h�`N��S��mQ���>�tP'&1�1�mT_�?���Z�Y-5_a���HOS��V|�N�*��+w����G��9=��X*P����8��E@���Mr����w�i;-E�t���pJr���v�#-u���n
�����B��Wc�K�;����|�1�B�
�_�����DX�\dc~�0�^�������3_�Tb
���@x9@�[��M���?t����/�1�z��#c ���-,���B0�������TH��B���mL>����jW���7����DzZ���F�e�F�)��O����i�^���=���E��|z�C�\�.�
�Z����y1Q]A6�r�{�"��#g���qf�Zh�������3�]0��r�����'��U
��
���T�/|��i�����xb'��'&z��Q&�qzG���=L��8@xX�>�D�Q��w�v�?�p�/�+M���!Sr#%�<k(�syy�0�T���������R���B����E���8-��j���VW� �"gp�h���FJ��n�����#Q��q�k���U��M@+��kW��o�����2��]y["?��J1�v�y^�����4|����Rc��aU�OK���k@f���FMA$!��}�v�Xlo���6��*9�txT�Zh�e|���n�b��,�#w�*[nW�&����&�f�j�R�D�4D=$�C��U����	p���X���lp2x#$���������=�9���h�;s�P���7p��������\��Aic��(�
�HH�����.��7w����6����e��R��b{��K�%{��O�!�YP�}�_��\^oWePv��#�&DfU*���z���["Y�w�A����0��Fz
H�\��V�.#�#�9��yy>�8���g�[nX��f��|a
��FJ=��b,%f��W�����X7���t��:�z��:rq+�%%P��z�U,G�U�fhs�S�M�����+P/�T������_�T��(B]��2�CK �_,����U�R��9�	��o����s�������Q!���������/@��4�X��5��Q� \Y��
������DG��go��/�F/n�uE
@f��_j+���hL����h�+����R��u����b3.���Q_���W^�OF�(��"z�)��0#U�`�
���1���IK�����,�����������Ma�����������x�Z���(�����$����h����
R�/������^�MJ�f����>�������Q@��6�R5��o�s�(�9+���F�V�j���H�M��={t*���e�n��4�+?U��^�rQg��s�T��b���0[�gC������=��
�E�(�_��!o�l����L������D
���l�f����i&wC����1#sX>KM���D��CB���E� E_S��@
h���j���\D�(��W�r��V��{����/Q�J���;%B�o|���E[���jmdf��~����x��I�����q����[��P2���6�AM��jj�
)�gG��c?�ZX�B�|x���0��s����7JO3�n��]-�Il��] ��^��p���]��k�#�"�������(c{IUz�rFP���5��[�mZQ7���N�����!'`����ah���V�H�\Hm�P�aw��q6�A�
h���1�����j�W/�l���q�Y�EXWgM�����6����k69�0�e�.���]�Xy�x��)�#`
��F�Q��,&��	k+��L�`�'0����&~����1���q�Z����KW�o�?���~�����t&#6����r]
S[&{/MN���kA�"���W�'����]�za����������'�1��.��y�?Y����P����	�F����j!�V�R���F�>�d�+h���H��o�S�^�2���5�5�L�^:�:�q�kGn����=j
DR�?���q����������n�X�mw+;�3��?�93U[�a]�}����Z"vi�n\H��t�Ywe7����q�v4�����>4r��%
V�R�2��Q�?=�.�yJ��"�'�l�[�����*���`�l�26�o�	��e$������=c��������UY�Y���R���r����V�J�_x�hV�����+O$Z{������-���kst��Z���pc���69Z��3&X��X�2���Vy��m�|����^�������~vk�V��2�Uma��V�����s�����_�v����2W��>�*��c�w&����K)�\�;!�z�w�����I���_��������l�,�eCEW���c+����ndt0�5}|��$���FsPM"^��/�k�
S�i5W��Z�����_y�(o����V���&�"����\V/���+�K�S����'�(�As
�v_|���5k;����j�9q]x�p�RG�P�3SE;N|���g�,�]w��=K�������� �I��>��0��8��3�����$Y�q�<��)'o`�UJ/8�5~p[���T�m����Z�j�
�X_]<T��c� �2� y��d���&��u����AO-kH�j��1����IH����ur��T���'�3�w�?bge�7V����1�w�����C*h���EM�u����eV�.���V���{r���PI�j���n�,:�
:������+Mfu�����E�	m62�rg�;�9�	��LrG�������g���T����
��royx������aN\Z4�������cc�@�;��Qv�~����8����_�Ln�� ����R4V��S�k4�k~�k�a"g���#u3��?����~���0��&�#&.0
4�6W�"Ih�E����2Yx��!+����ZZ?W8m�jp
���d���;�~�p{����8��[�+���Do;L��������{��g,��9w�-�z��w��=�����y���@����$�cl�����z�������F?h��{|Ek��2���/��O_w���z�.���pk���}q}��nM�V�����>U�Z�����iD���b����*�:{4�w<&E�����������B�w:Y��t����Q����������smJ�b;7�v�MIV�����t�_��L�k�c(A��.n��+�������� �_6�r���������c�Ei)�d�5��j=O��k=O\��/p}�7�"+EKo0H)+�1A��"<�9l"KeEV��
�K��)��1�z�.�b
��_�Z�\p����_ �bJ��[j��E�����n[l:�G���+l�P���
���4����RB ���;E�-�m8�!�f��p���,dd���I�zS������q�;wG��u������;o�sw��4����]�Q�:��w&��bS�Z��\p�O���b�q��vf�#�M���!���d��m��������oy`�4T
I�0B��:�6�3�z���,��#�V��N�@o1H^�%`39������%?�PK�j���P	�:���TY�6�=��P���Y�!����A;�`3>��A|`��S�q�SQ�z�M}�}J�T�7M���;�����5��o����~s<�o�������OJQ�����SD���	����>t<�5�|��Hm��v������Jy�+7���-��*--�+�)���|��F�U��W���R��'�vv�4z����R+W�m��i�C��������m�,'��j��!���SC5����
���/���:�=���������z��ez���Y��f�pX��k�c4��&�$��`�e�}W��Y�1���/Fo���nLBn6�3��wNu�t�0��{������:������w��.��Z>����}�D���]7�����T�f~C�L�Fp����y^���9i�SS�1���)�_(0
�I�x���Y��"(���:���u<k��{Q�����>�0���ba;���;�G�p�1&;GS7F��yD����H+�I����Cmm����v�Y�)�r��b�os���t��Ol�;�&v��7���-��	��W��ESD�:W�'�/e�������v�0��'j��Ef}����8�G��ROH�M���!q�����@W�&�j%gR��)�+dXVq?��`f�W�I�~���R�'�X�.I m���65��F���
����H[�:��)��C����2b����\�����KJ�M������x��f#�{��j�W$��}����P��2�R&����Q4[�����`I�
+z�td��n���nQ��O��5X�|��*�kQ������9�g0��/�bN�Z�,y!:|B(��qZO,k�a��mUd")��[�P�����{<��7Fc:��D�*Z]�J��?,����
O�OH���'7����][U��v�����?����?�>H�m��������t�������ww��1��uwa���"6t�l����^�=h-����^\��6JR�Iy���_��%���~�
 X�f��I]���U\`�����j|��S� 6;����!?)�ZE����|J��0���T�)�7E�2]�)T�lS�g���['���M�����;����E{��M�,�x�N���{���dw}}�X0�}��n��)*��UWL��-2:.�������
~��b��]�'D.��X�y�vE>������~esW�u�(>=C�R`{�'���3����'�Q�����{��X)9������{�1�Jd���1�����KZws�K$r��������wuLe�=[��/
�%b����
3!�������G�]�<q���E�<yb4���+����=��r�BS�l|�R��rz�(9����U.p���� � II
��@G&������h)���J��������
k����Y���������J�W��SI��r������L�r��$�
��3��M4����~��4�hT�9���W�%dc��9ODr��n���Q'8Uwc~��)�E�9%���&
��rfy��F$)�L0W
%��&�Zr*|�~�T�����)�K	��W��RU�7i�o)��%�\�QK&�{%*��d�]����X��)+GN������b��$�g1����c��^�$ag�K?�����X���b�������*��.�t%aTXg�Hz�GT���*�@�+��sL�������9��
)pq�����'sJ��B=��g_	EVI&�Q�.>�$��\������eAK�+�D���\�((I�%���DM���?.Q����:#d>�_47$�i��:�()7��i���C�G����-��XCRJ����W4&%M�
��6���*M��NL��9�h���*�C�Q�%O�O{����'�s�����1��v��,+�+�%�n��'._��������j����UUq����w�bz�	#!=��S��
cC<��*�?$_���l��"+�3	R�����$�<>������%6�y�L��2�Y�Z�j�w�hR�o����$���sZ��i�����/�fb�����T�o +y���	��SW(mP���+��#5���X_�+�}�5������;s�[��L��E��L���*r��4 �5��Fo�;��F�'��]0������&��n����A�;C��dA�$�H��'p8������*�/���v+�kOP�KnSq�ByN��d��"W�����;%��r���u���M�.I����|�,���mM�����g��z�i��g��������1��	�qRRM�4I�����C;��3�3�E���C�XO|�0SV��I#�������?�}\�k���qd�^��������������b]�YfT���"G�If���G��B��X�6�	\���L�L��57{2~9�H ���xNo�����{�h!Rq�A��U�Y�;0���R�x����4�t(
R��)�y@
���y�nC�����W�XU�t����7E��ew���Ik66w�6�]�
���>\<����g���c���C�G����DrpW3�I���zi���R���X����)��^�5>m1��NL[x�����r`Q*��'���	��z���y���R4TT���X��9{JS��aQ��&q��RxJ����j]�p�fI�c���K���t�%k�,A�� ?t�mQ�������D|;�H���H�kc���E���,Y�o��i3����F�BN����-{<q���p<��:��#���a�R�*���!a���L%	�NPR�vix��^�T)�����|�]�.�Lu�YS��������X��f���!Wa����N
q���Xf��&���W6�?(3h��Zp����� 4������-�H��F�Ri2B�z�&�e�nN���
{������bR2sB����������Wq^?�1��_x�BCk�
it����Xt��d2�%������dh�tEPr�����7��2�f�;��6��-���Y��|v����.k��r���A�]�Fz��u�=��|]�6���xr�*��2�d2�S�EC�����M�O�|fy��n1�Zb��:M;��	*�6��/�[
�J�V���jx5\C|�xY��O5X���C��V>hPND��w���b�l���)����4
�1��6+��`��v���f�l�����j�?f�#^���O���
���]���m?���A8G�y���BC��i��_&��8��.+�8;�lu���E�t��g��\M����2*B'cB������`m](X��QD�:��$6	���N��/�������>�Yw�eJ�Y�xZ�4�P1[���j�8� ����8�����^��6[��������)�^.���/�g��������6����fCqN���T�*A��U��
��Z���B���J�����.����o��/
��bN�*'��:'�6gB������^�����d���������q��e ����E24�e��B���|z�	wbc�{�����L��I��et�5�W�?��X*lt8o�PCIZ���GA����h�X�'{��Xy��B�R>f���aM;�kK[�EM�#�l-�$�I\�iN�� �F��e#�J��.�Ec����}��)���0
�����z�(��+�,����G�4������}���M��������<��]�Yb��6��%�C�����%����{�{��&C���o
�c�_? ��a��t$XL@��u@�&�	�A�eL}��s�x�������[La?B�%��i��'��%���,�@K�mO#Jc�3>ex3�;�Q�Sg���"5@��%����0t���0D�����9�'m\����p*��r��M�����3��{^��:PS���f���X��=mA/����5� �Y����7��7,2D�	�T�0��E<e�cO������.#*F�1�����\<U���^��m*��9������n��>�%u��K�4
s���Kv��|��X��U�
��l��uT���M'%����S��!�1W���/0�L�D�lF�M�Q���/������3*7��3�@>��E=4��;�9bj�|{�#�p�A�X�c�*�V�$f��$��1S��v	Aohp/J:�h�2����O4���)����8��6D]ZF��`���
�t�����]	�T�mv�QI�x��j0�
��L��� ����5($r���3��B�M��g���1��)�q0G�Z��8n��k�D��u��x����s�V��y�
&��|]�0����rJ�R��M��{��r��erP�-6�[�H���\�p?�!������O}^9�2�2t�x��!D^3�x�*a�v��0dh�0���u-fM-�XT���9Q;�1w����b[��C)c�-&6e�H�E���I4$�E�
�<�~~�����U�k��
�:qq$'5.GNNk]2Z�^@���R���`�h�q�*L���UG�����&"���J��p?�h$mW��I)�����%�l%Y�G�)�6
�(�LI�j�%�����)<�t����&q|�x��+mXy��J
���;�K�i���--�V��y�.3o�����e�<�bi��������(p��U}$�������@�"'V@b9�3�-r��T�6G����H0:�n{t�� F�P��`���C�(��B��B��H�1<Z���-�sb'����
�#gx��D������<�Z�=\h��Q�1~v�)��
���d�
H�����DC	�I���4!��=#K�\K��E��@���OC}��	Yrh3����!u'�k�����/���L�A�_X�TB����c4C�/����~�#q?��4c���)��5 �v	�����99+X���j��G^��_��f���V��*uV��t��j��,�������K\C0B��� M��B��Alj�`4|��8�I�,^������9��<�W�i���4�;_��UX�_N�.�K�W��.<��#�����*���x��W��MM��
�������b��0����g~Oj��	�� ��	���S�������w����r_I������ k�8�a�rG:��P
WF<�.�r����rFy�B���
�����z�J���<*�G���?�>�7�u���WD<.���S�:�F�������ld�N��7YC��
3cMG"�6Fn�
��g���Z�nax�yo)�hhr��b��xZ�����8�e(�>	]��j�'Oi8��������.�U��<)�X����F���UC�F����I�T�`[c��*��Q�f�u��T�Hh���e�E�]��5���!�A���#��T ~�������CW~�����L������R�[u����LV7�����N:U����T��p���V{/�zV������g�
��3Dt	"�d&r9h=_lh0+�O��X�k[9�;2�kY�k��
Q����>���,�o/�u@�M�j
��c1��-M��D/�H�mW��"��U����5K��%���djO?�����=��J~^~B�����O��=jH�5���m��$Jk=�DR��e��*j;��C�O�	M���*�O)�	��J�]E�M3��M�`|L��)�ac,�P�]2ff@�@w�@H�})p����b�(��$��H�A�w$�V������S����T$�JE1_v�V���+���n�����)����"2\z����D�Zc�}b�x��H��b�UC�P�-*23�6[2�iBn9�1#I��V���w
���6�em{�G��{�Gom�dRR��x6��h�x\��m���FO�A�R�������#v-���u���M���]c
�1��3����jz��U+)�T�s��ma���������
�<�������
FF��0���{F�$�.����&D3a@T"����>�� ��UE���7�:���K����[�fQ���#mQD���Wc�H�j���''����Iz��S��M�x~agR��a����W�x<���0�W�3�N���T����~2y�$��.�^C*�hD1�n����)c��0rD����GOa��H���p��I8�Ud�E�>���)$�;�] h�3`]oD���0�F�@���������E�yD;-<?eM�\g`(A��(A@
W��r��sV���� oi���Xte��al��t��.r:���+p5q�W/�\�iO��N�K0z
\"b��(��_��W[�a�U���Z5������,��Z��xx������������PO��N��T�����]h-<����rS�{���*��w�-��������]��c���2h����$yx�Y�+���B����z4����%k$-��+�y�b���I�:��;&�����+k:���m���M�w�������5�ku���N�x���+������}��o����5���n^y��Xt��R%�FI��c������/����)�Zj=����b���s\R�.F���R �h,5�R�!�x)�wP@N�����.lmIF%'���G�f�A-���$$e������iD�E��%�><U�C/�p��12[���K�1��@��u��<<8A���������B�w�l����R���j���|����)��9����Z�V���nX����`����B�`��3����z�>�)i�<�f�sx�
Y+�1��$�V�k����-��Y���Gg���%��{�����\��Q�!��f�l{	���+�����`;���|0	��xIM<x�E�^��<<0�7���t��e��yx����v$~�A=��H1�A�W8*�[��\-���v��������d�����N�%�]��n����u�Ti��+v5�iv
��XzmW��9��}��C���lJ���jh��=2x��6���T�j\&���:����H�+e�	~����ZS�������sJ�~��%�:��>�����AI������g�vI����0s������Dj����Q���,D���Qv�d��F�gC���"�-�fcr�Y�)�1�@��]�<+�'�8���o�&������mMt�zB�g��B�(<�����]6
���^g*���-�i�Fq��$����a-���i���@���������L�3��������
��5H4n<J��]��"��A����g��+�w)m��G����i�>\�|}F.���Ww���y�-���&��L71�`��+t�{�������'�5��F!������j0���K1�#��.���#��t����p��������)��L�Z��������<���������)o�NjW��a[�DV�����:��D �w"�d>������uk���T����L
.NjYl��@Sr`��
MK��[��>R�H�L��I�)�4��I��7|�`G���{_����������?`X��k�~�R�U��l9��~���jL����xZ��3
��z	����i�'�������k�
��:�N�l��(�E�{����zZ�N��6|��/p{AK|�UR��5�uk/z��`dM�Qn	�e�����E�����;
��� F�:$��X��c���9C<����4[�DE�P;iZ����
c����6E�^��E�5kTp�S�
�!��
P���3�Zo������m��M��CM��=�w������q��eS�������uB����U.k�P&��0���q\����xh�3+��{\�\����[G������wX�����������L2f
o�4�kI�,��L��ohP�F�� ���i��Db)�o:C<��C��vK3X<���MH7V�g�(~9�9W�s~�]�3�O���[O�Y��#~)Yy(�*��P�_���GC�M�����`�#SjY*�7
��������g��u9�F���������mw�(�:yf��7��c��?���$������	w_i@2�v5JyQD)/�(�@�\�7�����S��s��@���[�{�������nYB������@���B)�=t��?b�r��;F�����B���J�R�	�	I-����@�x>���<�^���x��xL�x�����8��v���v����K��4�>��
���	�R���1�����7�����bk*�
��k�9����L#[�sZdOsF�v��YD�F��q��9��HS�5~���q��tx�:]
ipp�����/<s��=���S��j��MhQZ���vIB��V��7���ttC�����I����yX&���`k\lu[��:��u
�����k��k+�����B3�^I��s��a_������J�v�����N55	_z���8�T�:��]�faL��L��0�������u��Z�&�oi��h�yo�Sh������kZ|������������u�����^Sko���s�������Y����&|���������K�+�j���A���KP�~��^�t��H��h5�P�.����a��6`A����^��P��C��^3.�%}8����
�-)S0�	�9����h�U��v��%�c.t1���x��!~�(	����������������(1vAx��]�3�zdf���\V���������k��*�3������xu������xuX�.���K�g��{�n?���7���lE:m��u��Y��Q[�.$���0F�+��|a�:�
~^>'�t���*�,���<����j��_&n�y�����PE��G�1�I$(�u����]�)��w�an2�l���>�v����F#�Hh4b9���"�?=Yx�����i������������H�M/����T� �z���$��8�����t��p��	�������������������b1J�B���!�P���~n��6��M[0����
�{�0�� ��3b_Qe��,=BY?�zJ�Q���P(V�����c��+�^d#d�;�~��:j��� V��9�	ZDy��*��9kYbzf^=���&�,��_��	[	�����l���>���1���E>"K����$����2$+��2��i��d%����H/�t'����@:
�:H�B&��D!�L�.H:!���"���e
�UJ��@���OB�������-O�_��X��m#����������������
���{�Hkd��>�$��o7���%1���7m�kG��:]y�`0������2���8�_=�ec��>���13G6]:�k����TC��.�#+���;A���+b�<q��O�5f����<�P���b��R��b�9	�� �+�����n��\�2��ig���`|��n7�}��#kB���[s��6���Wc�;������#���?�{�����0�*^����[$�����Ol�]w����������8���F<����%�j��� �d��c8����6���?�>=�j����o=����gGv ������S����0������>��;?��#���=x���\����z��;:r�>��a��g�\_�r�q�[�����8�����\��7�*�����i�����{Oe�!p3���X�G���M�D��<&������RN&Z$�WJ�A3^%�w�8����8��S����\;'��eD�����q���FJ���Wk��*���$��B����Z���v6��{u��m|��u��Z�R$����D��=��l4�"�d7!�����j���/��R�q�-C��E3�����t����e���0.<5�	0�>'�h:�D4�M<������_��p�b��E���l������X����K��(�u���ZZ�����ZZ����e4�4�g��x��xo����v�� ` p�$�@ �c�`;1	��^�M���w���$��yU��x�!�y>�[������:�������.v��
{�w~iu���D�R���M�������}M�hs�������+�����(�$�7o����4a��%�B,h\�g8��m�`�8�n��l����B��6\�_P��7����B�4�D�s9M��
��)�P�+�L���8�����i�a�x,L�Xj��q1,�'_�T	/D��
Tm?O8�#�8/���.\�������=?V�5�X5Tr��;������_����Q��*��L��Tm{_��jk"$E|>T�AC�r�n�%�3��'��Z�����&\�B�VL��;�����s?<PJ��h�(��2~�)D?B�����C�]�BCHI\C��\������iV!�F���Ms�Cy��mR"�&�w�Xkfu���h�Fn+��K>�*C�f��v�Qm0r�X���W�V�����F�a��@�s�<g�A��IZ�YG��Pl���m9��p(�O&�F���&w�8V�L}U2�~��0�&>��nk�v��{V�8.��������N��z��isc��+�d�����b*�,��P�9lk�~zM���b������jy���/��Z-���o��:����R35�����l���|�H�Eh�P���A�~��uT��Fh)�����|�P��I��u�,o4q�c3�&����������� ��p����js"�f�w�~@m�����2 �x: ;���&��U��Rg[!Q��7lc�o(R�D~������i����u��.���&{iIK�?k/n�gt���Ikf0����Q}�~�9$p���`���]�@�n�u%�=Q8f���B�628�,Y
��d���X�g]9:�\�H��C���6���������
���(5xx���J��^(�<����U��qX�D�f{�I�@���Mr�>G4?���eLu��[K�����vxSvIW�h�Sm��)�j�c����d��X���G��^Q�����0TR�@��ND�r�
w�l��f�v�2k�-���^=w��������s]Rq�m��A
���7��_����a����P<�>��$�=��'@���F��+d�F��u��e�!>5��m��;��+����Vc6�bM^?��[�k��}��}fp���������K7���e�.*�$�+�U�=O�x������f����qu4�����1��Y��%�
I1�3w���=��Y�N����qW���yH����R=����H�������������*����Rbq&��:/�2U/M���P������&��KYi���\����W��KB�C{�'�����.��qd��C_cEF����^j���f��5�X=�W� �I?R�����!�N�<�7K�V�>skT�i�7��[��r<�������l���|���K}��������%G����-��-���<��SVKz�v|���5�q9�JCy�[2��:O�t,,���kk�\[��Z,�u],[x��#��5���
o���(�A&�Je���CcgE�������|��,e� [�v$���Z$��n!�u�����$��������l������qV����{���&4k���B}MZ��M�uZ_�+���Y�W�]�sN�����.��#h�	`l��G������n���U+�Z����]���JO%G�b�a��5����n�6�nO�������l����X�O�X��q���i�����S����p��Ih��j-�	<~�`f4u3�����-E��d���+�^����2X�t1|&`|a�r0n�������aX+~I��f[S����a+�I\����k������������h+,hJ��m�u��Yw�������E������.�ko3#JL�����������u�@l��VI,����#�����{���A��lkK�
����U�����B(�v4�[�>"I�P������%GY��$��h�P|*%}�a���b������>\tGu�3(�
62k�}�#s�	�Bk��@�?����	�TS��5��cw��������m	E[����_�W�n8x��"�	�7����6��������\5g�����^z�>�,)��
�"&V���Ak��2���S7~7���`�Q�0D�����/����CD9���7<���y����:��T������sU��[;:�����GG�>�_�}����,�o_`�?�%������(��j��BZ���x
���*:�����"De}A�I���P�z��?��;
%(�-�1�T4`u
�P;6����<���N��/z�/5�\����>�\�eG��������(r��[�����q���-.�L�:VG���>r��
w�fl�����7��Q�^ki�{SA��=�����4�\�^J�E��BA{n����5�yx��(���(��%�V/E\���{|���|�%����	���/�sTr0gz����@.��w���:uM���%�1p�t�����h�<�i����7�jd�\+��|�{�����(����n)�.��h�&��G��g��=Y�F��A�V���|P���[k
9�^'�P�h�Wb	2=O����
�&������X����E�T��'��q�
i��1�?XTA����Bh�+����$�E0#.���b����+�F��u��FFI$$�0������2�_u�Iy,@���d!bA�&��byjC����G�&�C�|��Sj
p����G)(����2��iA���	���;Ww\���{<^#b���lzp{+'���qK!��"Y�����LJ��x6�";��b�- ��f�@^
�&��s.���FrP�����]d������E�D���>g<�5�d���,<�'��h+��B�S��~��0���:�
�����1G��"��k��+~D�x������?3��n��%D�g��.�����
-�����r
ZW��/L���~��K��3bq��z�$�DN�t��&_�$���E&�R�J����VX�p�
K�r|"���
TY9>!����%��o��p���K�������_=_&i�4��"~��~���������?y��c���(����kJb�?nM����{�y|��G�v�����������������7��eG���'H����<��.
t�
 �4z�P�WI�N���]V=?1��qX�#�qOE�_5p@OrP��V��S���\����������cr$G�X����v't��s(B4��B������j�C��	"p(�
���\��j�8�5]�W&=�q�].�@pa���J����c0���{�gc�Q�D=���)��k�"�B�zN�(�Ur�L�gTV�@�o|6p���j�����kA�7G�c-+f%�]����u����@�u��w��AM�M;R>���[�~�j������*�8k<�0
(+��������.[2����:'������c�����,$@��@5y�����*|
�)A�E����)A�HH�S 	u����H���R�Q��m ���r���}|�V'�4�t#�V���N���~v�`t�8�i�F�������4�����!T�hL!��+r�P��0�s����y�H�;����Get�4I���h8�a��zyz�#5HC������;��0q�O~(J��,�0���;V����A�3�g!J��y���H�T)�
 ��89@�a�p�*�5��7����x����+DR��U�Et!
C�C�5�����M�qDz,��
o��������T�g��C��Q�w�����y���-�I,���C�d��Sj�D����	�Y�	!�Cm<6+t���m�����4Ze�X�)��8B��,E���4��1c�b�@��<~����/��~0����~B���/oo5X,�������<�|�v�h:�{�T���UMU~�qg��;����H�����8~ GlN��K�z����P�F�/����k�-�������g�;�_6��9��9� (��u��E5����1 �����;h��D��������ig���C��W7@d����cbc�e��d�"+JeD�=�te�Ut��)=�XH�x[F&y��,�t�W7��
��O�9��oWt���s����P��4��G����M2"E��N7��e>���"��s��/��@��d��Q�
l��u��xq���5�v���n�qM
�v?��6��g���FEB�v',q������`�Z)i������(%%���r�V)�����'KtJI�f�2,���X���-�R����T��VJZ�5G�k��K��KX�d��}���DPJza�N���J���M�?��d����7r����l7�v�#wO_���Hh�Y���_r������t�����yO ��L��;��r�X������?;���KJM�\?g]�(�)f��<�Bs�s2
���1/�|
��*��2��U�.�9#/��`���tbD�qo�I\M��F�&���T��9,��f-��lFD~�*��Duh�,�{�d�X�S�	o=��&�Xn_^t��^������Ev���W�B\��/(�<)?q����"��1T���U�:��'m�t����uj
�?S�cN�{C��M-�1Z����n�:��'��L�J
fAk�JN��&�M�R�Y���PN�����5s�����0=�=�@OO�D��	�����������ZG��.��S�8/>���,Uz*����s1[qb
��^��V��I��:�
��g�5X�"m�6=�6��u���t�����2N�tC��T�h��o ���;+#m.=g�1F���`y6��6Z�;E����%1N�\a�j�h�����7��p��jCs�2����48V���T1�ak�Q%�w���B~�G�6��8D���^KYBr.��w P!qR��!�r�H�kX����������~R�9-~��7�
���L%VW�[[S
���2�7�Oa���:��g����/�������-O������,}h�N\������z/#+�o�����bV~�{[z���@Wc�Q�p��Gc����������V��`��;�������s�E=�|e�����h��M��
-E��U�,`�$j�n1�$.5-�w��-�v�`�#��	��1�j�R(�����b��!~g�C�lvnto�������=�������������+��rq7zJ\����IG�3f��;W�:�KY���de�����������
7���_��;����$��	�W~V��{++��Q���q�wF`�&�~^
{�����8��o���E��W�w}0kE*��I������;��j UO�
��wh�R����v{���e���Bq�m���E�Z���B�����}
�#�y��
e�;���&N�/E#����x���g��|?��n����9�<Q$���f�������c~|`�+�������z���~�Ig��=�\��)�=d��[��vX�s+k��P����ktI����op-��!��Y�T��F�C�8�����F+�v�
��Y�m��2�e���������6��~E��|Of��'��Y�$���������W�w�d�4DzV��6��q3��#Ik��v'T���+���������C�.��at�����6�)��&=z�r�f�����W4�o�
7o�!V��yW���8X�$C�����������6�����b�g�e��3��?���B��#�����?��8_��SF��0��#;�>x����*�(�jy��]]��� HX gN����)�&�6�E�o�����q���U%8�#��	�����`'���q��-z��*��T��a�u~�=�u`��2���z}�_�?���_�$It���@j���u���"�c2b:V���?���N$���/��hG��7����.����F$TVj�3d����{�g��v�Y�+����q(-�S���	��������('(BL'*����(������3G�7�9��Z"nW�&A�U@��>���+`$M�F�~T��T��VAO��q��Zk��������@&<��L	�Z*:����^&�/�B;\��W~�����.�N���g�AK�
W:���:�Jc��5�!�b}����`�	yT�W�f3�U��}Fg��m�l��HK������/Z	������6���@H��o(����44DJ�J��(����)}Wb���`0an��7��
w����{���	7}NV����X5"[�U#��LO;?�R]`��8S@g� ��K_��-E���I����AbX_c�L1y���J{,���]Z(2j}�}^�m���}9_�5=��Sp�X^�=&�ch�� 8M���F+������7���h0�Z�,�b�qd#.����q�M��o�o�U�D���IK��/&��]
����V��#�J����tEgUp���Cy4/�7�}L��������������u-ip��.���UP*
4�7���
�!���+!����*�h��	�fE��/�A���]�%��L:V�{�����{N�/���Y����j�����U��j���b�.h���i-���u�p)�l��-�v�d0Y�������$���;�Rn�������
G��f���#�#�4��+�l���@.��j?��CG��	0~�E
v����^K��m�G���!"�q���s�a����%�tg��H�G�
}���c��C����c���gy�s��dbnZ��F+�c�xK�����/*����Y���cFF���K�}����3t��&R��E��%��l�Fo/�iJ'Y:U��-P�6��[a;k���
'8��)|��G��|�jjK����Zq�y�`�QW<k#������_��D�N�&�l5����K����1��� �rA�o��G,�.l�),�LXa*��[�gO���%���:�x.��D��?�I�����p�t�V��ZW4��k�E�����P����"�%8p�@���!���K	���uI_��.OGc��'�t"���&=����0���\l<9;+��]	[*�W(�
wf����P�_r������A��7���
6�LSf��e,���4S�����A��{�G�	F�X�h%�}�4�v����5�p�?O���3i<}{H��M���rN��;��\�zOZ�!�����b���G���;����7���hAyl��|�����6��p�4g�����/�)�1�"-��EKn�Q*�sK����O4e*P8����P��$�����#��Fe�B������K�JI�����?�V�\bQJ8|?�e�TY������z��)%1X�R��])q�o���HJ�^�E����H�dH.q�L&'_%�Ff�j�	�A
<13�Zc
�k��(i�JgtD��0O��{t�S>���6�������r"����&_%�{��\(xWx$D�s�*c��z&��lTu���6h�7`+N�z�
�0=��f
�ytP�U�2���}p����X��
�Yrxh����M���a.8�^~�-��\CvSJM�i�����!�Aa�/"1J��/��<���:�pK����20��b��z��Y�X<&�O�����%��%��Xq�^��H�H��	N�zt�������
��;.������m>�����>�,�yw�*�V��c��]�Ap[�Z���i0fA�*���2�����`����m���^�����.��ps�����#�^�p�g�h`T�7Hb�nU7�����w_>:���w�^�{���t������9Wt^}M�5�K�����r�,����v*�d�W\�u���d�}����\�	�������xeg9e#���
��|M$~_!�����Q9Z�c�s�9��=���F��>x����G�����w����2�L�U�����L&�/@�g��?0u�����l62�|<�>��D�CW�����f������e^�o�]�d1��]�|#�*���gw��y��\T������~�O����]+�C�����Z;���|��m�W��q9�����5����sE�8G��������	|���49�W����B�o=�v�3*[T�9Q\(\����:��������������e0�U�5�3�p���O�kW�!�4A�7�Dsx�����+�����u�cK{��L�sy����Uk=i��b���.?vg���(����vo��O����,�'�����AI������������|�����'��o��o)��mbP��g������?'�����-P�AY����&����5=�x��(&~�|�w�,�f����[/�]�y������\vj���/b�m�8��[TZ����;�s,��x*�������d��_����T����s�Yq�����{b<~�A�*����������_�_�����9�l��J ��4�����F�67C=����u>�J���  �n���Ng
����,���d����a��L�X�*��D���1m�{
��Um��*|N|��6O�1��������=�������3
��KRk>�Q�U@�j��f�+d�e���g��#mn5�Q:���B�������P����EM-+Pj���B.�����������l���{�y���B-�d���<����Q�u8//B9��s<�0��_oQ���E�$xC�#U(1e�[�u�����H$�\����M���:!�Z2���g�(��3�Kz��p-4q�
#%OD�������DZ�g'A�~�'j����lcKC����������.��8�f}�r��z������6�U����q�4�5���,����FmFA57}6�@��m��P�qm��U�<�d��M���%W,�O���O�^��4!�|� t���
�6f���vBz�����v��qTc�x�����y����P����2?�
��J�dXV�z=����SL�M�j7�
#���F=`es���K��;�����1��Z�@N���I2\�E,Z"]�&�)������UzK�J����. ��Em���[n�1ZBo"-~�f58�e����2�����{ �7��I�����qA����
�Qf��}�[�x0�q�`����(����T��=�%��,��5P��n�2��|��Fc��v1����E��g}����Dy����x�)�jj�$N�W:�v�����nca����{C��\��|�[Q��\
%��Bh��58$����j+�R�k;}�oZ��z������������N�*}��;r��w����t�uIIR�m�g]�Vh"oZ_\����-+�+V�a�GL�	�J����Lc:��u�w�T�SzT'�)�	BS���
��]QP� �ZX�.�
���n�h��o�.������@��{_�Vu�{���K��E���I�����%��-q�bY��������	�-S�0�Gi�)�t��na�Im]`J�.��0�.��h�t�8��s��e�I����f~#}�|�{��������5�J�^b���[���3pF��G�MI)I�B(��$�I���?���.���r��
,j��
EQ/��	?��d�X������X����G2H�{U�{�Z$Z�sK����d�M��QG
��F����bu����@M������I��������j�����*����(#-e�dt|�^�\5}l��=��4R����H����[H� R��Bu�YLM~Mm^~MM>�kCt���qTttm�����-�u���dH0�o��ol;X�vSfaU���e�d�yN��/>�S)��*���D|�*##A�c���'2�N��L��i	�r���_9}���y�c.~X�b������ef�T��T�����y2'��>�(-A}�}�����`~������1:��5\������&d�V���$e%j�2E��
�D�G��I%O3B;�y��$CAB�H2� U��{J�3�*T��%��_&3���V�VH�r�E���Ls���B�T������|���������(��ar*+29=O���9�5�t������Q11Q�S���8�������dc\�1]������Z�\k5�'F�,����}T�B���|�|:�Z��5����������;���#{l���}�]����@���'�>P��k�n�n�:z$��:��� d�Ld;��|�W��h� }�7
M�7�q�_�']�O���p�rI�_-�>�_�WW��m~�z������������z���fz�NL
@E'������?i�v�Y���?���?7je��X,����c�/��*s-���O����?/�j���f_|�0��m��8&�s!dF9��io�Px�U��D�
'��<d�y*4�H
y��k_�;<Z\TYa�m���H,�i��`���s�j�����;$�YT�X�&�8��,dz%qm�����-�����t�H!��SSz��������"tos�'}6*���T��[m��&��k\�[���q}k�'o���|_Y1Ml�l�!�q]�3�\�R�=G��5����x��wr� ^.�����Z=����%6F��S��]����y>)��2��!]b����R)�����}���D���G���9`�s5*�+2�����W��w��T�'�*I^4��?�VQYc���(�:�Q�[�%�n��]�[d����-�E����z����8#�NM�����i�[�r��5�����d��O+�=���$��hV���Y���'f���
����
�&����T��
��%>!��X�]��[l<��Z�RP���;��F����+kH/�3�4�A�]�T0���u��Mn����E�*T���tC���������j,
���3�����h�^���8�F��'��`{d}(?��8${�����d8�J~
TSb��!��8���'��8�	������|��;
�Z����0$�[
[����o��F7������"3�M���}�F�Z���\�����d�9�vT�������+�`go�U��n�\v�$'o������d�J�V���K�{�zv�m���<fDo4D�~�X�X��9��58�rUZ�\������k�����	I��55++�q�3���O@����53�m�����X�h��2,���M�c��v�H�r<a�m����}
�Bm��d�:�kTPgtrcj�����d�+���o��sZS��%����(���\����V�>������bR�Vu�.K�]8����6^	g���)`L1[!��:���6���xA�����o�*	�l�P��U�YSy��Q��6����be�8�)Q��D�J�a��m�p%P���@*	�b�Q��DuQQNg~VL{�$��L\������Vm��|�/=e���r��7����k���M�jS:�m�����Zg.�,���������Y��F��4e���j�M)�(���o���X+*-��|Ztj��=��,��V^SA>���[��s�'W��`6+K�y��*��
?;,�I/����V���:u��{���������kjy�5���Kj��Z�5lj�?�65dX�����N�[��������������+K�J��m:�j�O/�O��H������=�Z9�Z�)4A�)_m���c5o��u��5���]�jo�J-47��{�����������c['j���w��qt��q�����u	��d��2�
����*������V�V���������������������j�g���kL2�v�9���J�0]�6��EV�up��o�fCZ�����O����b�a��VC��t4�(Y[V�b�NV��C���z�:�X�]�a��.�$��'��WZscrK
����5f$Y�r����� �V�U6,����7f=�?��"/.xd����l��� f�_Zn�����&�h�P�UgMW��29�4���w�Bk0�kZ�������j�Q��{'����{�������(W�����
���������Z[N�1�h4����������e�Q���[��+MZ6�������J����gW������������r��ETR^z}~�F�&������F��e:S|�-���3��h������z7����q������*
�� ��W�,����v
��Uy&�J�V�L�Vsg_�hJqbLT��3��.$X�*�y������.<��q���
e���&5/�lN��e�J�q�i���)@������*MT�����*M������o�*V�X�no�U�:�VnL�K)i)�c���	�n���jJ6�Y�����+w4f����y�����Z�2&����n�/�����T�F��Zc:��:��p��*SU�	�L�fU&6Ter��d�Q�XR��������2���r���X�jt�N�5�Ye]m���Z�0�m���J������22��g�$�A���}�*����$l�1���>�u��{��������������f�U��V]^�PS��8����y������)���;�D�Z-��efS��Kyu�)u��6����u��TB���b�r��n�3�UkK���%_m*��l+���@�����[[a<�>b��.�iW]��*�f��6����5d��������,�F��h�_�n�+�5;�����������������K��d�]��������\6��������Q��1Qjm�n������
�e|QV<b.=(����(G:�:��#��*|�T����?�n0
�rsn{QR2���KO)��v�J����8��5�����z1�*���}��mW�_�7���)q7mF��
4U:�F�O*������2��D�#a�%u����i�/��]��I��0�%]�&��V"}��
ts����GY�����0�}z���1�����0: ���Q�6�3�F�Nl��mW�_�d��������R��&���Z���7�K�NG���W"���oeR�J�zS�ws^�(�@�~�)��2�	�_`�/�_\�����
o.��(��J��Y�(kEz����T���]�Om�m����lJw�w�����+�#�)���?�j�z�zO�S��/�L�����_�h����u?����i����������W?�P���iK�:�]����j���7����o��_Rj.k>��[��b[R�@G���oPkr�����nl;�����s��:�;g;��U��V����{>����'�5o�����������7�����
|}
���?��������k�8�o�#�#����~q���F�8v`�n�C;?>~���������K�}m���w�����w�~`�������3�_ie��{,{<{^�+�=��m���.L�O�����~���I��~G��z��:�����S�)aYxM���s����)vJ;��'��Z���4��')k�h�����|	�k@'=7����3?��_������z�����P�"�E��#P�=���n���K���;�VfW<�@;=wF(B�P�"�E(B�P�"�E(B�V�q�i�|i��[�G]������?�?����:0�j�l	~.x	�\��E(B�P�"�E(B�P�"�g��E��/��+����k�q�5��|�Z�q�[�T���6��e_�2d���rh�Dl+��ob[�����
�o�j�+�Nlk��C��hL�����B�Nl���*�������UB��f�2�]l�Hf����P�����!���b[�����'����N�m�Oh�jd4��m
3��EEIQb[���j��^�%���(�k8��g$�1���6���)��Mq�m�3mS�i��L�g��8�6���)��Mq�m�3m��L�v�Mq~���P)��Vr!�#/
��
�X���G��0����Lr�h����HO���>�NX�G�������{����Q/�����nhMIx����8V���d��r�]g�^5��v��ky�k>x: ��
���s _ ������+�3Ep�Q+�'a��	
�u��xEMy�efD_	�Cp�����*'A���2��z@&����!�����B@����$��(���'�bS�"YoM<)\pdPh#���&��v���#��T;����v��xW;��{-@�����o�n"��`��u�����5Ht�<=D#��C���z�U�`��A0@�D[��N����j�[�'�K\fa7��'J���Y��� H�I�9��.46$l��n�5�fD��R��Z;�����Z�k��B����l'��5��5��������_Lb7��yd�Y���aN��p�%�������v�o�|T ����iCe���wX�=ZPY�N|G��:����I���C�_�I���LO�^�;���5��W�e�5�����_���D'�N,���]�����9-��/�{3���?�����H�/��{A�'�W ����x��H�9��9	�������X��h/�&^�m��v��b,�J�t�SDZ���^��h�����NQ���V'<h�Y HSd�!kK��\��9�|����^��}W��3�.����yZ Y�E4��M9$+o�XP<���������k��R8	�A�"���|-!>5�����A�i3����H��IL���{|�6��E�k��7�����b���b��9�����U��r������B� �\���8I=��<b������������N���9/4?9Ims�����W�I�����,�-���!��3���\"�8��I�D��	��^m!����I�\���H��������>���0B��B�+���!w����-��$i���t���O��G��oy�~�v�������*���W�p�W^��a�
�"'v�B�M�@y����n!:���#�W�s�i���S����A������������Z�����
�!d'�c�\b�w�����=D����"g�����W�-����y�vAF�����x��������z��f���$�7�&W
�
zKr����E�Z%�lhA��2��B����5����LX��ROY�R��l�K�
KD�H��C2Hq������
O��4�}z
�C�����R5�#W�!L'y�<�p�+a�#x�|L3��h U��uY���$����=�FHU&��M�����GH����������`QH��R��F��W��H��u���	�P-��H���E�`fz�0�#y�bX��#��A�P7�%5��1���'9����{�`}?����@;	��m��"{��h/����#�`d���E� ��G�k��&RIG`�i�^��Q��zC��8�{������'i����%m!����m Q/���Q�=��	��3�����	�T�"�\,�J�a|��l#,_/��V-�n"�~m�{$��w���pd;�t���!b���%�5�����6U�A;����+��y������d~m��E|o#�
��F��[�Y�h�!��F�;�'v�U-D����t���K�Iy�IB�a���"y5���H����/���B0�r
�8_ig����2[i
��r���T�o��}^�=��z����rM����g��[������Y�	|�}�;���i��wx}~|�w�����W����}3|����:��V�����s0��W�w��3�����I��aw�"GX��|�;�w<���/�s����3��3����' ��A���I�������N!���|X=��)�.w����vM�]����������	�.~�?e�u��C��������.�4K��,�q~���=A~J���B����+<��������9w���-=s��V� � ���^��vw�����w���� ���A�5H�������']�dc�((��`�������v����R�1|�o]��FT���s>�v�����0,zA��X%;�����8f�~L���N���z��1�C_U\V.�[��:��~�S���`]�YC�9
�����x\B��w��o�%�.���	}�����(���,�J�>�����Y(�O������=����V�1��|n8�+���s��?n���10oP��NW�NL����`�K�mS
�YW0�M.�$���w�~�1�9X.�|�9�Z�K�c-�������1&�!`��8�s��k�{=�-��a�a��IK#	�l�]�����^��|p���������=�q{�����)T�]��7��>�N����n�zD!7����� .+3�IW�(��<���E�����������d�|�O�!��Op���^�t	����}b^)�� q���<n���/���+^�0���N�'7$7��T��\�,��Al�	$� �(pl@�i�����p�@0N��c�
,
���IHx��$k���],�=�:\v�g��<A;��.7 ��w\�-?,f�W�DN��6]Gr-s7��nXzi��?���^~Z��	"���s��-@|s�P`�,l=9��7�E/
K@��������Y������4hD���f��W������d��(�e��J�����N	�:����
aE��
���	�%�1�q*0�k���.r�a��1�@��&
U����������2�����Cc=��|^�0��,�������V������|K�8������w������������o�m����[�����=������[�t���:���������32n�;{F�����i?�24��6��2��
w�v�����s�t�u��@���1�c:�pwKo/a�2
�����z��G�����l��ZZ{;(+P�����������tu��`�!�L�nGw~-��m�g���6�?2]h94:tG�p��o���t
��N8b�l��w�]0��:����X��������/.�Gn
Dn
��Fn
��n
h�O������z�[�[�[�[�y�6���:�[�[�[��n@l��A@��	}m�b�O�#&~��'������u:�����^���9���7�z��k]o4���'�u}t4^�x�Z����z����\���2�i#��sJF&Hd�����0�t�lq����,w�+������,C��:�~�����V#�e��&��b��bf��gv1��f�q3����9�(s�����$s�i�a�V�I�6�s��8�����9��c�0d^`�V6�eyn�-�v�U��m��lw��nd������w'{#�k���o��r���s�e���c���c_��g�}�����}��=�G�CN���b�?p��Z.�[�#�����0J�r#`�0��F������c�����NF����7��F�F�3l`��F�Q#`�0���i�h0:
�-`t0�{��!��	��`�
`�c��������8�{.0J���2��0�Y��b1�D�(0*�Z��h`t0Z�n����=������aF��L
c`��*��0������
������NF��^��0z���E�m����51����[��d�F��=��0�0�0�0�0z0:��^�~�
�0:���)��rq�9.�{��q�s[��^�h`4��OF��W�c���a��F��Q'`4J>��Nd��G�{����FoF@NF�f�D���F��Q;`4MFn��`�	�0�`tF�F�F?���e��'�h��,�F��Q`4
��Y�h0�0�0z0z0z0�6`�`�`�K��"wg�����_&��J�f�h0��FG�OF�FFOF��W#�[���c��Q
`T�Fc��`���3���oF?��G�U{���0j�vF��[�c�����NF�FoF�1{X�q�1���b��
� ��F�(}0�
0�`�e������M����yn�Ssc�?vr<���8?W��Z��n�&n7`����F�FF�F�F���0z�����~/��>�er�e��du�ZY'�h�z�b>��0��vFA��o��G�������*���F��0�y�[������7�_F����*f�Mfv���Q`�0rF7FwF_��F�F�FoF��.1K��9�e3g�z�����MF���G#�5b��`�`�`���_����odz�]Y�[Y	wN���'���/��>����;�����������Ji25dO�85�R@���2�[>��#�����^���H�:�|^����%��t��y~i��3g��X�����J�Tr�}N�F*���w�>\|m����"�����^���������������98X�
��$������a�l�E��v��,/��l���dt��2�)�L�++�d/�nsR��B������	/�Ed�o��W�.�d��s��*GVV|K�+��wZQ�H
0 :`vB
Rh��M��B��E~��r����>
�dRpH!;Kw�r����*e��2*UseMh�|���MMM��>`�
-��^,����Aw��`�%wI
�p�eZh�H-W��F������,�@�4R���f�O�O����W��"��|��������6%��P��|_���@g�2�lK����$���f���.�L�K�3�L������M�6�����Q8����������R������R�_R����������L�f55n�{����	��?�U>=�;*<L��	<������!d�2�}^�fT��C�Q��B:��%Y>OB&l�j!�E�2.�c��g�n��)~6��Z��J,�+,f��N�;�'�( 8���������������f�f���F
����q#�I��A
���$"��."g�Ph0!*�NG�>�1|H�����_�����B��`��!���H�2jE���lc

���a��>(��#����)��8�&V����;�@"3��tI���Nw+]
$���_�� ���:�D�"�3��'�B qH�M���2�����='�aH����$t��H�N�*\���	�\<��V0j8�G��^�$�����A!�+��$�.���5[�KK/�:���?q&F-��y��Q�������<�&�w&^"�z��W����/�"���S�5����V�����T����C�"���X;���`#.��28%m�Bj����S����P�R�I	f�=�Q0U(H�+�����}dJ���#J�T�!.@����F��	X�TY!q&[����e4�P�-�Y��$��Y���0J]x�-/��^����4~���T����J��p	�$
mgq�C�'�����E��HQ���Z����y��/��T�X�H�a��&l)�,�A:!��XA5��$��dX�T�@ZD�I,KK���g%��\��`+�/�:po9n�Lp8 ����GrB��`^���'���������#�Iz&)6�3�������������o>47���Z��Z���e;�� �vb�3�v 'R/�4�4g6�,�	����o"��d4�P
����U�Gqg���3��T�$k��geI�	i�2�$K�!1�R��J�$����e4����T�XJ�����=gS����������q����y��y����8�MJR��bC!c(�Aa-������Ad@�Z��v^c#�3��'EVC��i�� �K��7�5 Qd0j;�-HA�R-H|Za����J��T`�����o����9�+����*�@2m� �B�(d�He�@G0d��`�H�5�>�m���j���&SI�2�VT�V��Ilt�F����G���&5�\GK�d���#���,3�����h�H5b��`�-
5T,�*5��m�>���m�@f�7�<*�")����$3$/���g��v?����M��D�C{�$C���"� 
Y�(V��g���mK5
�b������H�D�2o��������Awb�8�C�=2qO����95r@
9Y<Q�� z����|p�6����P(�&����Y5��+�B
�
�@���Z���CY���K���,�E�`�H8�&6w���h�0y�\Np)T�B��=�<�C��,o<k<����D�{������J���R�fT_8��eSF��T;�"�T!$����&)�5�����
���P*G��5rA�v�{���E��&[���b�������GC�[����kUjU�^G�pC�T
�JCn�.�u;�!b?#d��R��)@,D�B*�J�A,D%��$f2�2�d�����.�[P5�
q�0�����4�&e��Vq�d�U
��Z	:��ca��w]�L#�E���LN6�y�)@y�:��� ���C������	�#�0d�V�����8�R�9,j&TqM�0G����`hH�2 �M@�:�2��g%�%��h&�>%2�dD���
c%
`��Y0j�ts`T4,�&�.�	)��B�+�G�"�������D1��������om�Th(TP}�
2]D,Uf�*+C���!�H�� �@�RNc������H�� J���$����8/��"�?����!�#x��d�C{�|Y$<GBu�����TD�E�Zx-(b��R��q ���� 9h<��M\*/����c�L�� ���H��s��F���B�1�X�r%M���VM�J!��8��a���5��A@�����	��A�[�@# >��r�h��@�����ee�Jd�U��v+����������Y-�VV��U���*���c�pBo��`����V�"9�_��h��~Ff�r��F��`@�$E#t��?b��d��III�4RHRA#1D�RU�%cd)bnt#;!a+01��@YW2���cj*Z�:U^�t�E���ETGL,���*J�l x��$C��=u��^C�$�i����v��@��Z�,#��J�dh��QU#�$=KE7x4$'$�H�dT`�h8,:
��? ��+H�fJ���5h<��w $�2r��K����A�@��UX*,t�V3��v��_]KI$Y,Vvg�$p\(��8�=h�h�HM���8QB5q%C$��ce �0B�(<X6)��8C��6�]�@�K��d��>T��+�[E�=�-�0�-�
�( 
�
m!��<�7"Z �9�p��C-4RXb���H�FSSa��,� E��?O���E)#�!+J�Ar����@N|�@P�&����7�nq�@86�����F?M����`�VF������}�s��c-���Hc-�����Z��Q��kx�	����Z���"����T���~+�ER����{�2��n��e1$l>W}=(�`1:�!��pXU�)FD���b����\�X�D��G
�*������U �7kvH���:�+��[c��^�;����y���o�+��*�`.�sq'�q@�
�@�*6������_�e%���;'�b����
�y�tx4�!+P���.���������B��+8("*2���!%�_>RI��5�z���H�OXD��w�D�����1V�>����V�L�E ����i���d�0��*�g�������Y�H��h�1+'og-go�)VNgs+&�d:�bE���G���y�V���	�3�!@8.F�,��r�4��Sq��~�r����������}�uwM:�1��������C{���.���g���� -�P��p��
��F�w���K3�)�rh��}	���=v�>l��n���n��6����?Z���]�����C����GV%�J�H�U�O������{mOE�8~�����k7�r�>���,,��}�(�w+k�n�y�]�M�x_���PxY�����"�hi��$_���;R��]l�~��L�����\���dY{�dgLr�Y����\��s�z�Y�w�#f���)U��+�w]K� p�����1,[�m��t.�C�:xX���c1���%��;�[��r�R�e�<��4����g��������C����������0��l���X3�2"��."�JS�^@$�$H~<v���t��$$$���1���8X�w"�S����2H�%:��Y�7�/�e�5@����X���+�u�,g��mp������tE�5��Ow7�o1�kx�����L,���q���{������9�������f��sd=�q������*e��)����:�2������������6�8����1����������z�P��0;���A�����3|2mIN�W��*�?��]�`]��j��������M
=5L]k���&��u�=�|�1��\ec��Q���k����4���n�k��J�"���\"`����^f$�y�����5*�X��XOl����A�Z�a����`a�'��(��`&���A��l$��+�
�����oi�i�]>i�~N���k0f���G�6�s9�~Y��sS�]�����\���z����.��s>����@M�,xj[����@�#>�!������^���O��>QS{N9����Ww�9����c�*'���"���A�W����U�q������^�g�}=��d{������k���v��_bt�����[�v?���
�O%��	F�u���s�u�d�l�����u�un�x����c]��V��������V���<K��1p]�0t����u���������=
���4��I�f"��g�E�H,'�����Y��D��7�U�8�W������x���������f Y������@&l`�oba`ab�0g�0�'3L������"$��E=�$���6��j����7Wc��5I���X��@]�-
���2���2��(��8�):��
�)�_n��0C?�3���@De����{���9<���w���+�xo������s%���������l4��GpV�O�t9������y:�mu���B�2s6�5��m���*�^r��hd�%o�+��Z����n�pG[���P�C�X��6}���;S�|����u���g��<}���������u�O��7��\�Xg��n�{�N��lT��gd���F��~�+�?�)|�A������8�m�H;����=�0��_3��Y��UL-{{��?�cEn����"������&�m+��y�
}���tU��m`�7}i���P1�{[�r�M���r<��������4���/k����E�O��u��n,���n����^~�{�z�z0�F��Y���S��\Z����h���1���}�6Ut�g3��B�(���y��s��[��,�T����������Q��<����O�pD�
��/���]!W	�~<kE<��8�n������qV`��>>-�����)T�6�	�r/2k���viS���.���h�\����<v����'�F�a�H�	����_y��~�����P�Rdv�e�yg����H���PF��@
������WT�'P����@��`-��q��b����V�lNgX����:�5����o�w|�+XY���m�ar����+��9su<O�~���+����?f����F�&������9~�����a�GP��5Wz2H������
����s�}U3\��I�U����J����:7�4--i*��?����=�Z]�K�������L�3����3��b�N8r���p����Y�/��R����{Gx�u����p
�?)�xV�b���_��vU�������b��{�7@�`���Ou\*�L|�J4����mS��)���S�-�|�����>�/�u-�0�O�)��%�!�8�K����]"�V���������2C>A	��@pZ���i;��mE��\�4_��g
�)�{v-<����'wT�)�����f�.�@R0
�=�N�~(�1�!�����%���� (G���Cp�]`')���_���8�����0��QY���qN���<{*������1e�q��"h
'n_]�����1��F\�����R��y�4�c~E���j������w[�z�b�t\�A!�ms���Vz2�����m+�q*�g�#�m���e�;!����.v���gRG�e�������
���T��{��^9{��}I����>#�nkC�{m+�@Qh�N��������[R�-������Z��5w�c���7+����0c)�r���?�\YPnb��eCz�\����+OZ���)��o*I���dALP
�a������W4:�?����\���q�<�W������xm]���Q�ryy������"N�6���b����u�&h79�4zY�����n�3K��Mw��W�}������u%M?�8�M����J}���V�f����,P8v�7��1Q�73V������s#�J���1AX[���;.ti?;Wz3�,��p����dfi!�Dy����woTX=��q����u�����7�����sc�w�gLpTum]X���WEY���Dr�~�[f�/h�jv��t�r�
�#�a.)	��]���f��~�6�+(f��� '�����m0����iLF�t���#.�����,���N���fz�%�7�������@;o�x����<�@d���/��iO��k��P���s��u��0���w2d�6�d�tW�r?�r�����g(�����F��dJ�[x��Jx����7�����������*S�����G����@���%3�dd�%�X~����i{:��d�-N,t��-{5_!�j�m��N�E�4g�N���r�}iKo���u��Y�>��]��%/2��W}W`��k�T������	w�q�egJv3S�s9�=|wi[N�[G�z_:(j�����p��>/�o���5s�e�v��e*�7�0'f[����\����NQqH����v��e�:&.��=�����|{]\�I~f�t:����uI(��/��r�=�)�_w��n���:��U.8%;wU�c�1��m�\r������n[�lw���������'�!�Yikz����v�T�`p�hm(�����3f��?��Z�����������T�op����U}���,n�����c�%�������T�i��C��^v�_��,��i��w�����o�y�2.'�@��iP o��JG��G��'�,���L��w:^T$x��*�����a/;7�-�9���g���Oo��V�U�,���1��[*���`O�C��M�gn�����!�k�����L8m�d�Lqp�:�~�vX�����Y�7���E��$F��.����I:��-uh����.C�c?��r�COdK�K7�R�U �.�0[���jz�%.t�������s��[��H�Y����n��G�:�U���S��y�8�����%��w����9���;�Yu��{��[��J��w��L'xn;���9��L���A7�l�I:����n��z=������*�hz�]n�7���0|�{��LX���}]�>�v�w1z:8���e_����J�{����+�1���-gK}G����g�t�W���;t���RC�^s���3+m�����X���Z�A������33�GF���O#�a
���
�C�����w��E)�u.�6�0��W@:_=prVz���������*�jz�$����tp�1���l���U�s��5��W�j��<
6���T�x_t������Y�23�F�p4��I�e
��4,k��KJ���m���o�:����������z5[���UN��L��r�����|����y6�d�������:"p����s�q�O(`�fh'�9P���o8���`����|=��y�o���p���'�U^��Xq�+VA6����/����XN���v\�~;p�����*>CJ,����?yR���t9X�V	��Oh�n	�w��Y/��i��3��U
p��%4�/��o�1�������1(�����e�H��O�'���b ����8Ljb~gmB@���QB������=Y�m~7���A3�_<u�l��s�n{�E���%n'�v�f:��oQ}����oJ�������U�;���������[�q��_H�>�#p���~OmpzO���+����O���{����^e������F�2�X?���=�9���mf9h�f�Z~��3oM��p�@������������X��Z�%eR�.�E1�qG��j����7X}�=��z�������{��U�:���2~\�x���I��r��Y����p�`{�;�FD:��F���������6)�\�H�$u�C����@�G�8�L�d��c��4�v���L���zkz��H	sje~
�]Qj�9s��"���������KBJ�������]����9���� �8�|?�=}Z`}��������L�m%c"wM[�|�AGx�|����j��eV��~�Y�����9z��w�3����H_k'�������^���}�T��i�?{��[���Q���)�����R��*�l��.���?~_����h�������NM���[+h���Fj3��������[�u%���x����o
vWC���}�i��t�w����+����Q�������(���
endstream
endobj
1652 0 obj
<</Filter/FlateDecode/Length 408>>
stream
x�}�Mk�0���>v��8����M[�a��i��:jX�������R����y�^�����\���Y�b[���mS[8�G��m`���V7�?�����������}i�m��,|u���l4��
<����6f�F������u����(P���u=V�S��l\�������\#��X��)��p�*
�2;��-���[*S���T��5<q�nK�p{!�\}z�9��g���������$$���fD1R��H��D��G1R��8�g	���DK"�����n9�e���mH3�Q��:��OS�>��������Vpy2\���eDDm��
>�%8��%8DT`A
�XD"��b��7�D`�����W��]������XR	������2��h�L|8�~�����W��9r��
endstream
endobj
1653 0 obj
<</Filter/FlateDecode/Length 32462/Length1 101444>>
stream
x��}|\���9��hz/IwFW��4�]��X�����%��dI��r�����f�p�)�$��d'��$��n����o�M!������;3*���&o����;�����|�+W�yF���A�E��O-�{!<�=+�{����=�8l��Wl�d/�|�>�nG�zlp���u_����A�;�r��A\�E���������^{"��<�CP�J��7�?�	C����~��n(�|�,^�vE������0�!BI������T���:�o������[^����kz���WZ.�"�����7��h�a��}���u���Bh��~"{��u�}�\�.��`��/�ZG�����_�t���m����(��������>�$�mg��h#��=����h�a�e�A��(w
J1L2NE,����90e\0��A�(D�i���b>A���Q��"��v;Y��A�4��# m�1VEv��
�����`�{Q��`��Ixn������2�����@�|�~�O76���w�=�C'Y��ml	Z�U�`4��0W���4']�����1k��_��f��c���c~�|�O��9�Y�|��n�]���M��l�zN�z<��/����<�
��bp����������Bk�C�,��O�C�����8�Z3k����7��y��P��m��?W�4����`���J�?W����~�b������nl=���;�[z��G!W(�@����X��|�?�1�a�z
���7]'�!7�:��j���u!�U8�O���G���~~��yH��?k��t�0s��hI������\���yi��5�}|v=�M��5�9�b����\�0�K��\�K��O���^�_�](�}����%��<��&��h3�� ]L��B|%S�\�C����qO�}j���f���������3(���>r�1�F$���lq��(����+@��'�"���PO��k�uA�u�_��pJ��o�3��0�A��6���
���y�������>=���������9�����1{>�
���m�r#� �{�������u��Y�����Q�_=�*��17��z2��U�� ���`����t"J����>E�t;�/���
��
�y���2_G{�]�<�Z�
�Q_�c<�w��Ih���PDADADADADADADA�?�k�+�"� �"� �"� �"� �"� �k�Z�:���1�z�k#/�M	� �"� �"� �"� �"� �"� �"� �"� �)F��D��t��	~��%,~�����#��J��Q�@Y�����
��uh+�����	�E�����k�����n�?��KL@�/oGI���a|j���h��c`�N�G0����j�������3��/��~D���@���}W��'}|��w��R�B�bV<��|qW5�YG��lNt�R�/jh��~x�E:d>9at�F��&���c
��G�8���N��W��x-������n|/>�������7�a�����q-.����W�����e*��l�rG��>�]��Pn<���p�yf�� C���-
���s�q>@������������2z9|N��L���i���p�"w���n!�g�������x��M�
�u�5�U��=�e�%���
��2��R���	B<o1h5j�\%�p,CS�V5=v����8���4Rz��wFE��U5����=b7����9xEOO��g�'��KPIZ��Z���V	�������U	K��b~��g�bA	�F��-CUv��W�j������F��J�r@���Fdr��!�K����2,f����
I�dY�X���[��Q]es8��u�R���U�$�\�U�ft�}$���������_��]���{a�A�����>���,T��w|b�-�R��j�[��[��>6Q#�~��x���f���j�D��d�����<��B���Ah����A����#X��>�1��p/�Q=���p��KZ��[���rT�=��-C��>{Z*p_�M�_h��hgO��!�����|k��y� ��
��z$3����&V64w�2�u>�P�vr�Z;�!�a>C�����eTW���{�������('��H���j�EK>S%���`G�������|�;l�g	�o��1������%�9��Q��+z�;��K���F/!���*J�A�%��V��;�
���*�$7k(���u��&C+�l�%� ��$[�&6�'�1�*�h
�sM���	A��������
�mn:)����0BJ��.�D'���:
���)Z�>���!K�!���7�k�|[������CR�6�l/�j�|T%`��>S�\+���uW4�����Bc�A2�������9g}�=��\��5����^L�����=}G<����{���B}�A����&�������,�C����"-O���4�x��������:�Q����X2�m'����Z���JR����
R�����=b+#V���1���:�V��u�puL��#��	Y����k����ln[2t�g	�Y������2����Lq
�L����
R_N�������T`nV	a^	�C���F�P8��6?
�De�W�T��p>*D<��`Z������/�2�u�/����*H+!��t>�H�!��4�4HS��c�������E0F��`�q)���s������0��q��*m
"s�S.4��w�a�s������8&S�'�Zxv���:���q��y>:�5���z�_�O���*�_�}���T���+������M�g'^������'�0?�!����^������������?��O�-�O�
�o�
�������f�>F��2�(��H
5�>�8���q��?9���`t�mt#b����v~�(>x���{k�thM0�v���z��5�i��;���:��?6r3����2��y����z��#;�g���?�T,��SE�7�z��1/�����A|��.���I��{��xo?����k�w������8cg���;��yng`'��k�w���^~�[��Jo�;��u���zt�w�|�k�w��}^~����G��7xo��Bm����a�o/^;�{���M]��n�z6�lX�a�f����k�w������o�V���]�������������x==xim���h���{tl�m�z��y[k��-G���\
��v�w!��/�m�6m�6��y���y�(7_[[���n>!^���XJ��D�X��C���x\hi�Iw��_b+��4w��>�������%��A[��ZP,�u�}(�����q�>|8�bI�o�{<b>@���V,Y��}56^�����C��4���A������v�y��M��k���"XFS��-�(��"��G��n�,CH�4;L�?D�gV5���9f���-�Y����n�$�GG��7p1p��<<�Y���N&jB��
�;���uj��g���8]�8q�|�M���$�HGAY�����~���<�����*g
����V�A�C��{������
��C; �B��f�UBd���hx�}��:������b���G����(:��!�F���� �&:���z��t/z��C���:��������	h��6��"��
�8�n����������`;��%P�����DZ���=��m��\D}����������������pV��HF�FC<K���>'H#	*BP�'!MB�R�d��
�_�2�����8����/g0������������>8��224�����,�uh����$'��Sy������2*/�)��(�.7�������hC����g/�����L��0�#�5e*�D
Y���}��L����4+�$T:�+�?�Tf����8�E�5�@:/��U]*d�^�E�J�y�m��bXf�l��{/��t�Z��K%z��Y�9y[x�PJ��$.2���x�v�{�)`�x��BS�
X�%�����P�m�`�\&��A���LP�2�
�[��1Av
�Bj8B>ay�� 8O(��xK� S�0��G����^T^^.�Tk."���h.dks�5�;�r�;}V�����i���������I4�����&�d�Y"����J��hUI��&��RR�Uo�Qb��V�mZ�M/��������zMs2	^���L�02��9'�sP+�O��w.'����8�'���������F��J�G�F��$	����QI*�P~AW�q����E�"��l ��bG�3O��j�D��h:7�-�}0��,��-���������a��{���)�����I������u���8�-y���Oqt��z7���oq��dK`%��������
�����!�u������i����"�/�K������Q�6�#.�!�O��-�9���Vf4/���vT>���`����)$T�a��!�&bo4�Qd�,�H���]M;~|O��rk���z���h��*�u[k�c���7cE���5'ei�\�R��)q�#x��K�t��Sm
�^�2�������������� �*MDz�l�Mp6:���J
E�EYFU��Q6x8���D�kgp_2�vfS��{6 2Z����/��h���{s������z���|�=����~���7�]t������MD26���(�c�v�E��z;��u:9����5�M��8�#�"��dAph����(�d�[�2�+QHF
��-i0/��S1E��4�$A*%*�JiPJ�E%��
n�2�Dz�&���������(.V2��[���f+��!R3Nk�BB{]a]��n"4����1��J���8��@�}O$��*j���1'��|�&�
�2"����L�"�	8���ke���KH�f���aj���s���Y���#3��(u�1t(pf�Dh� �kw'������%����j�����\�<�[����FO�23���!K�X���
�J�o�[��KV:�'5���&,���r�%JP�������{3�.���saa��z�
(�������k(?
�O�T�R���P���:uVg�A������l���G4�i>p�f���B|����g%7��/]f��a�C�9����R�����j�X�,=={�����.�Jd��}s���f�w����L�����9���3b����(����za|5����e[��V5�v�+,���y����):i��y���_��3�^�M~�����9m�o�>�@�����Y�=3���I�P�Gm8��N�����Va�m���zf������L���/���������ob����G�o{neZr��[w<?�J=�������������������g��]����-?y�������&?������O������6q�����Y1�.�;e�����23� �|��#�^��$��V��s�^�|e�q�w��d���GV����.���kss��"��������@_>�B��M�6Yv
�J.����kF=s�����a�y!�]�"�	�E�E����2o2��!�i��Lt���t��[A�52f�1���r���>�,�nXS'$T,-p��%�+%�7�[Upx�qu��D��5���e�I�'f��e���[�<
�����l���h����V)(�DO;�
M�����Q=j����%d'd+l���A����V ��p��@k*�7��oc�[M!)"��Zb'���	%�w/����t1a�
����5�1G7��{��lM�<����9UN�������\�������}w�$�5���`��(N�V��d^Gqln�����\�����l��a��Z���GYWqI������u���j��>J�W)�F�&���K�+��J[�~hA�'��=
��Y��n46$��8����)33u���
u�Yb����&&���%��'���D�(Z�K���>	T��-�pk�$ �������
�yLi�[,�.hgt����i���'�[�R��!�m2q�.��T4
��j�b�z�RB��a�����F�M�9q%1�pR�9v6/��l�Bg��=p�UF��P��n1�N�9��r��r&�����=�������!��H2��H���&�����������2�2����6������i�:�`�V�x-�E��&�"�����=a���� �5����������G�BQt�C����9r~���s��q���5�9���3��w��W���+���Z��Y�[�p���k�B�V�R?4e-�<1^V�f�Nn�O�\�P��)��������/�����2gUq�F�Z����<i�q9�������%��������n&��D���\�����3�Aj<+��d��s��������oL��Xs������U�~g(����*/�����q���������$+Y����B������w���Q�`��(�J���������k��.�2y"��s8��O|EODY�U��qv�5}���>1���;���5a����<�R����sk�dR���:Pdd����:][~����G�c�6���C�D8�z����G<��w�s+��&i��,�����/�;����������Dr�%~��0�0@�����9����/p���@7Y�D�\nC16v\�53��~��l�8#$���9�&�#������i;	�����R��F[�1"Y�p����8�g�5Z����fR�<���8�e%���	�V~2eg�
g2W�)�P�@nN� �R�_0S�����$Ao2,��'eT��MK�6�_r����7���S��/��J�cas����}5M��\>�Z�������;��M�,[���E���"�ps��6Z��5��{z�moZd�;t�����������9A>��)	g��tA�qZL�&B{UTu�,�]�N������0�E��<�c�5�3���O�_�`��/��p>H1N��eB�)�V��^VBRXV���;(a�Rv�D/#�8���C�3��J"q��U	�3��`�2��fy��o7ey�[��6���J�(c���p[d�w�L9�-lH��Ur�J'��SiT�����g���:
Q�':��x��Ix'-�${��������I]{Q�Of����qV�W6������}���3�tr��v���G8���Sr��^�$�P����'��������'�S ���d2�FEE���zvf!�O �r�0���rs��[p����)����%tnt4��
/|�n@�{l6h�B4��H5��,��YZZ|�l������t��?vHb�t���Xj!�Lw�D�!��e����f5Z3�����cTz���o�8%8� m����b/�{d&��X��O�S���r��^^1SA������i�������y(�cp���0�gQL��<�L(�r�t��1GO0�3�^�����NO��>������7SoJ�Vi�7�Pk���8:}y:%���Z�I����*s�3�	MMM	����yO��Z�}SJGG���qr���X�����_��5�Cq�d�q��0���,��E{��������M�e;��e�K[J�$4��|��-N�����;[���g\Xw�����������,}zse���;��RQ��0���a�Q,�W�9�	m�9�u���"q�����h/5If�J���wt��N��vO\^�<���5�,���+[\C}�����a3��I�M}U���X�_�(i	Q����>��l����=��f���7�\$Z�����@8���/\sD|S��L��iGe#�����j)��q�7�n���������7���':�]2�?Z�Zk�?\�^`%��*eS����y3w�T��8�[��V���>���O]�6�=��5��3�RQ#�m+&���\U�Re�)*;[��eo����8��=�G�*��n4��>*���0{ic��c)��Ie�5=%��2!�����>5�}cU����`;�����\wM�-�����>
���mJ��bT2�V��X��Q�N��IJ���-���6Ic��d*�BeRKm���R����II��\J�'�����3aR3�q[�lhv,�];�s+��sD�	� r��_��O��//�:�o�n&�La���+�=�&w�>�L2��f����W��i�m��>�%������-�DX r�������Y�Ik6)Y�������Y��.jcX�O>2������v�y��L��]�b|io���rs���
y	��/��^��W��h�&Zo�id��.��E�MV�\p3��i��j0Zd4k
���>����[���@
h+�>�V�z�2*��TdO�(�e� (�',�����LAN����
Ml� ��E����ogB��=����s����o�
clIE���Lk3��j����f2��XCa�b�������LITV��,�����\�r�2����;��E&��E&H�]'F��=�JE�D�M(�����1Y��8�7�)!��|����k�;�?*��6���[(�#�c4E���y��e��<H����B�?
��(>r���kC��CP>&����=Z�B!�E��'S�%���L���j����#��"f���v~���B����gM����#�L�����'����n1h�#s/�&=�8�yy���qr�B�+T���DT?&@��Q��(KZT�~aJ���}�,��e%=u2ry9���'�{��������Vt��Zm��YV.��K��+m\����xII�F�U���X]Z��ODg��-��o"�x�q���RP)Z���>��y�����)6��D
Ea�6��T��1��!�,?����G�
�MS�v������B�v�����x���l��K�&�h_Nf�xpYJmU-x����dI����^�P����������:��n�����������{k�����A��I������IZi��i�����_SiH���o���\}����,��O�`���})�`���}��\c�P��_�S���l)rS��G��G��,OU�/��,>�OM8�oH}.JdE����FQ�e��!�����y\��p��W�#�.m��X�`_N�����Z���Q,����=]�{�s�zJ�V����~B�W�����������l+��$X��2���N?�����t�D-�����f�������ry��#��(f��Zn�V[�>���X(?�����P�\�C�7���G����s9�Iy<A������_%�m^��N��E�t��2���0�����'�1��,mx�q��F��XC���8���(�R�pi+���&e��R8K;J�`����d����e��8��������d�@��/�/8)k�������[�^y^����������j�"��p9f�Yu��H��5��
m����/�n�������t�����Zl��j���I�$_[�;$i�8m�8��f���.4MK��R��)T��
Iy�{g8�0�����w����af�s��q�����d[v��U�#��������'	)W�".*����l,��d�f�3�R���WY��� �����<g.��\��cj����*W��	�{*tH���o�	
�����^h�hP�&�n�����:�fWvm �7Y%����!�GH������R�v	}A)�@�1���Fb��I�	1>�/�crSbR9I��k__���2��o'�G��Lr��q�4�>�j(%{���q��{���v�vG��t��zo!�PT�p�b�0P�ey]�N�&�D�sFB#��1?���
LZ�`(�
r���r0���� �L?l�A�:�qKy��$-�0���
�&Vs�R��5
���<�b$2�|����B����~��_������w~|��T�S�L7���	JA�4�
�ts��1��S����i���E�{;o�|�f�����I!�e�~=���%.$�0��b�B��t�����0�g���Q��d�����d�%UR�$H������S>�D,6��UB���|����]�����A�erR�z+����X�W3%9����<��������E�v��'���A��.���1{|lO�����40r\s��cl������^�����<
,Zi���\��f��(z)�����+~L�2�0��d�d��s$��9���j�\v���,�Xc��Nv��V`��cSv�wf�s`�?��[rx�k�Q ��������r����2)�2��9��S��S3w����C�3�<�����L_�����  Q�J��L&}����?���O/O��)9��
d5t�}�<���H:�K�e�a�a@�Gp�R���h0�A=T�-g���=�p|T��l��l�����u��Za����(�r��8 ��4;��wI
���]�V�	��*��������C��h���P ����yAF>��HT<%����^��f��!`-L`�����LK0hp�J)jf�4i0�_F{S������/�=��lSq$�,r-y7���T�4~���v-��n��R|��&�]��I�����C<<�)q����Ri�=<	IP�--����uA7]�
y�
~��f��q?���_)
�!���a��C(�
�*e������������W"%�2��}���SEVT�"_D�������K���F�(-��!�4bY��������"�r������������n(����$�*v�b|���*�r<���G ZB��&���7 �F>>���it+�n���-	.�T����F������3H�:��&w���o
����S�
�^��������Zp9-kpE�S��?m�������&�l������L�Uid�d�}t���$����g��s�]�*���S��"�dTV�+7���Y+��@��W�z�Lq���%�A�'�TnYF�h$R9�oK^�h��7�� ���`�A�b�n�n�O+S@
WF�������H�
�<���D���C��]MMt�d��T�
���������8�k(��|�3�}����>����������u�r����
a���i}�RW�m��:*�>�:���&w����������,���/^<������ToaW��g��z��c1T�=xf�E��������r����g4���|�h�dsEB�f0���.�����s����L�9p���yq�u�/�-�mCOD�r���[+��[ I��d�6W�?�����}�Y@u����������s�^j��'�B�;g��_#�8o����)S��7�Z}ss�����`���4�MZ���5�v#0x �]&�@��A�XS�5��F�N&���F�^&����A.H��Rj,K�e@�����}����L��D�~{)<>`
�G�V:?)
�����1p�x 7��.r��=U��y��n�������K�����m ��Q��q&�/�+���q��]�0�l��!' ��Agd�d��2�bV�&����>S��#����i.q:�H���{��c8����\q,��-s�P�4.�����5��@��@#�����(W�w�#38�:��y]��P���[�.G��|������{�G��M����[g�2�?��g�|���J�N�d)�Qzn_�����������4�L�������Y�Y�P~w~��1��U���;� ���.z9����;�����w"'���������(�]�kH0}-�4&O�#�`��N=�u��l��������A������$e$�g7��m�Q���
�����i����
W��Rl>F^���c�u��o�tG�`nx�����r����h��jVl6����[���
-�����J$�~4�OC���f����)����(�rbb�HL*U���\��:�f�+�^�����:�BY�����`e���-��K�����W������.
�B�'V�:���+��K�lxSW���O��3�#Y[���<w��;�.�����y������N�}���H����m�Kt�V}ST��m`
������O
�n��� d�W'��K�a��1�L9�%U?�|Ga������QeN��0�td�l6W�*M�\�T26���9�.�8Bt)*^�[eq�|� p{���������#?"h�o�u�����T<f�����X�I�t�\���t�@9�������������5�T�����4��������.��~DNX�j��[��NYU��?H������`f�t��BZRdw'��(��s���:;�H��AE�������[����4�I�;|���6#+���4W/��$k�.����bY�k�R!�VoP�"xa��Z	��M��+���-@�9"@f�RN���<��k��#�2��^������Q�P�����()O*����8��=���N�n>\�������',*_��s������X�OT���g��6���[w��Q���$�F������Y��oYO0���K�+�d�v��=Q�Q�~��Gj��^"f Y��!�~��*��SN3���<��C���f��Z1E�����X��`�!k�����p
�0�+�����F���5a�|�{����H��L_&�K�:�'�H��X�d����.=��q�)*�������������E�x��=
������2�nb���p��?y���Y~\(��D�����5���D�2���r&k�9���������������Z�sAG�]�G{N��f������o>������1�f{2w6�{����QZ������-���m�-��������Z��>9��m�?neC�:
�O��&}����QXS��:�OM5{�v']�-�H�w���+�`:"'�k���w��������������E�#yG[F������{��q&�&J�J�d/��\�w�I��+FFF��BC�`��|t�zMl
��TZ�
`�����*��Cs"<7�n��(������CGs�J-+�F�$6�����i�A�R*I5�&�&�u`i��H/��uF�����@�3�Yu�j_����H��x���X6�#1&��r!4�E�R"�/5M�����b�	M�����1[%�6���'�%����uC�n��hmW�����o���	b@�%�d�Z
%��5#8������t��C���6m\��	;�WeE��������x�!`�6��&JT��[M:l�M��C����������pJz�������zy�m)�=����������R��h�?!Y�� f���>ojm�.�-��9~2���#��
��:0th��@����8��3�Fq�l|��)=��F��Q���g�k�5������<�vz4�G.���
��9>�Er����h#~!����}h��8�x��+S{��H�dB���m�����e� \!b�)��7v�-]sCf+��|��c��8��Vu���w��$e�R!�s?����f�BP~�q�[n��Q
K�V�Y^e�y�%~���_MYW����el3���SQb]W���\1��a�N�H�N��s�l%rLO�-��`��j�7#������I|O�hRd�[����c����b�KQ��w(��[��J����@��L�
���V��S��!��l���� ��i����b@�fu���U�M�O`�:��+%������7���tp������E���Y8����m��c;
���6�{x�� J��V���d���PS
��E.��=;��3W��F���gs��J�d6����T�����,����\Ht}�b��	�/���8&cc�~�m����<��	���]�"B���DF�g���.������_�0��n[������������L���v�q�����`������%
*�\�`2%zQ����I��Q��2<��2�7�#�W��h��U��wv]#�V�7��;�r��+���_Y8����G���vO�u����:������//�x�����/?Xxv>�y����O��<�,�E6��~`/2���z�����>��QGm��6d���
y}r�l���<��>v`�S!�U����|~"-��
��a���Xh�������?���sw�w-"�J�D��P:�}�g��:�_�M8�����
��������x�L���
v��G�B����B���Q��s����
F�Ry��k���������N5)�>�B�����$���������f�{.���p-'i~�w���c�k�������/
cf/J{Q�J0��5�;�PXJ�W#�b;��f�b;6�)��P�G1��e���mh�T~"�����|���������K������T4�a��M�����G�N8����b��[�c�\�����5������Wm4��L������������������N4C+4
�NE�f������a�3�9Y���"N"/ �3���)�-���CO��&���C��tj��/<��l��������WwD�cW����#��������{�X�.O�M��r�]�����i�.�
}A�!��4����6g���>�-�{b
�H�F��6yR�&}����B�I��������
����q���L��q����fTp��������V���R��uf�G����������E|7����|�Dvf:(F���.j���T#��"%#�d��R�	���Ik�5�&Rtq����k.goV�C!�7{
R16�\��B�z�����[M�<%��u��r����6I
~���A��
�2�}}������[J 	'x(w�����&�c��\{��y+������9*��BcBo���B�h��9�r�E2v�i#��5td���SI�>aT��$T�e5�l
!~��������-Z9���m���M�)�}��y�X��<���(�\��W����D���k�7�����XS
55��%�5��1�bW+��=���|(4�w���/���;�)#S��)n$���O�6�=J�%�9r��Dc}�RhlwI������F��>���>��w�B�VN6=#J�/8���k0�\�cs�����z���t��'o�oI!O�eYk���Kj��>@���s�H�E������@$UJ��Q
t�R�T(�%E���R_n���O#K��=gJ������3���q�d��e����rt�C�:u�*^�=���|�}��\���*v*���	!�@�-$k��e�������x�ZC�JR�����]���l �5�L��I�����g��jO�]����[H�-
G-}��hA{HM���"h�o��ci~���=�$����]���w�t<q\u|v��
��������2U<X��r�\(d��w`�����$EC�)_���x�g�������-�%t��O�0� ���v�:b�����[Rl�/�)�� �Lg|����L����B���L[I�|4a��������'N6��l���U�l�c��e�@��g��RD���H�6�fa4y��~�UI��f�8=��kr87�����]�6;������ ����=+E}R�q�.����:\��q��s�����\�l�;�`�,?�]�0���r�bM:5���3(��k�+�&���U��9�J�,��� ��`�b2�I��
NJI���d ��1�������w��0>���%�������2{S��6��),=:5��a+�fE�ZtX�ThJ�Z��p�	���x��d�_[�?��1��A�>X
3�����|����l��'�g��o�p11�Ax�tpY���F��]W��>;��~���F8�Mc?��k�<�[u��;�~1������+F
��H��a�����zF"�W��
$��Lv��
+�"�������We� ����#wd��L�0��]���*�l)�����rL�q�R�����l�L���F%��h9_q�W����s�-.�G����0 �w��I�|����
J�xT����M�p|��	��^gRk���pd��U$~���npV%�%�F	�y�
,L����{2v��������Zw)���e���,�����3�ia�L#�%�������-f��W��86-z�\*��4���T;;��h��V+��w��n5�'�	������M�m� ��E#F�-fWk
`�m���>|Pir7���0��Y��g�F���#�Le���|+��F����%�I���2qF���j�.5SD��@��03��Y{f���DM�E��%��Z�vm{���:��\�``�52�����C��A�u5�Q44��K"�`�����/:~l0��rE��)�tu��{1F$.�,�Y����e���/�g6�,��W��&� ��7��3]O\r;
���������9����������3��r5-7�^�����*�ZJ�V�����;v��!U:��X�Q��G+����$����4�q�2��XVTXI4U��iD�=G�&�VA�������l����;�w�^A�LF�n���
�h��q�'�cG��K���������$���o����u�]��]���A����d���X�+�E"�������}I���/7��f��)+��M*���aM����>�j���p��7��d�����{��;�7���zL2V�w�YLB4"�f������2�g^Z(�C�-��)s����(�'�i#�)��"*v%�u�������rp�zZ�����B7����j�j%\e��Y�U;)�+`A87[<��	Z�����z�=1M�����\���dF�K�d��5;z�V��L)������c��A.���#-������#{��Lr�e�����Js<<��)�X�W<�K����a��:J�m
�=s�mi�%�9��@`�V�^M�����e���������%|��Z�?�U��B6�j��	wFO�C�����&Gv~${ ��79�
��=��D�)���^[������m�����,������30����p��VQ���I1�����O6�8�����&w����'-RB���^�#����s�N��a�����,�l^
�H c�kW��
b����������4qZ,q��%�@���ml�O1��t��#*Q�(��*���;'�Y����NL���=#��U����g�J�q�'��"�X�m�bEV(��M�"��Jm[�i]�5����W�������X=^Ku=������6������?��#�wM`�0�W
d����qQ�L���b�TWI=�	������H�A�L��2�\�j�~Qu)��2��Al$i�)p�������z6-Q�jP�Z��/8�����
��b����F$l��o����X9�Q��EHR��XV��Ur��F��!��5cPIe��8WU����:�mx��� ��\��JQ�N�����B1'�U�����}g�"�7���|U�@�d��LS�k%��Y�fW(�r��}9���H,Nk0*Wc�B%q	���V(��b���L��nw>B�:5t�S�������Vmr&::mll�E��B\���l|�Y(���[���M~3����e�(�w~R�[#��j���6����K�BEp�}V��������2"��[���\s��z��lM�]��X~���X�XZxr�
��B��_�*{f�vr^��/*�?�uxF�Fs�s����PK,a������[���FO���F��C�N��c,)��Gy��g�Z������������,R��MBX�s�@�u\�
�T�m�����c���
"@�����uW]�������.��z�a���a��!��D\tc]e�Q��
��we4N3�^1S��-9���E��jM��5�659���q�� B���Q�����Z�5�|n��i�IP�/Z�H��q�����������? �������'��V��pXaZA�K�m�L)��D]��<�O�Jk��jm ���O��_�����;�M���N�TBY����2��PS[!���i�%[���w����g����V3����`t�?bs���t�;`�
%�g��L(e���3��|�
�����������[��i��o�D/�B���&|_�������7�[�\2�����/KeK�jfL��DY\���]QK��	���:���{f�n�n=.:������;R��<�����:�5���qE����E�����W������$v�uyl^oUi)I��f�T|�>����n�O�&#PA��"���`bGk�_|
��n�����:l+:���Z!�yQyY4<����9zp��41������"���2�A�33r�
�`����W1�s�d	=0_-��F��|�oS0T����i�?J���u&�z�s��gw4���>���k��2��G�^!�m��=n�/ZN�+�v����gW�'W6n����j��j�R����j�No����Q�4&��tz����P���4�=�J���){%U������y��4�v�
�\!_E���W��n�q'2$�1!_��(������/P�u� W/�O������y�/�J��J�D�^>?PO�a�ngX*{���3�I�������������8��]��
��o�Z���R�\��"��|_%�(���U����{��VrB�{��~*y����v���sI#�����rp����!'�Z�)����1��v\m�����n�S�f_��EK�Ab�w��h&�j{�z��~�(Z�$�f����jI��dv����0��=b��9����_#V+����k��
^�����G�%��5Y2
�gv�Oh&Vukb������TJBe��TR%�i��16jH�,�H�)�\B�U��;\�����c�N�U-���]����� W��1��!��1X�|�y��r�]P}RB��j�!Q�'eZ�^g����9�����O����]`-j�P[�Y�-�������{�Y�����p-�v�1�� W�v�U���&���`���������k�u��s~O��V�~������������U����������� �����Wo�����V�������^�!sc���"*@u�W���WAw���;��=^��������
�����}P]���{�U���oJ4lr$X�[��	�8� X��6���e�jV)��F���RZCK�gP�5����\�}�F�W��N�k����3(%8�{���1*p�����ZM�����A���0�FTW�.J���&A{�������R�Q�������je	��1��W�F��"����k��%qq��y��0���/�k����e_�J�y�@���+AR�dq�V
I�HHb@�m�Z�{m��NQ�8�n���:��N�k�b��{�����i���s�{	�v�3������#/��w���|��������v>g�6^r18�a!�0�V������gr�w2��o�ZrI�F)U��9
�����\3��0����l|�����E2���������x�{o��S������A�� Iz���������o�F�Q@�I6��^WPX����R�����������Z���(���l���~K��Z��"���W�h��a���������
Hrx�I����|��0�(�8��:Bp�&���	�m3�k�9�H����bJ?5S�U�F��lY-�p�����G�n}P(r�4����������C
p�
�H@�U �&,D��#�0���w$,J�w�_$�Cn�e��S"�J���(NM��J�A�.>�Z�I�5o�����|�a�H���9��	@�4<�������|��j��VC�8�_���:N+�st4�'iwKv����+���8��uQ����*�\%��D�h�:Z)�sXQJ<g���|+�cF��2�������s���@*����J�0oX�~�|���<>����p,{�N~.R����
����K�$��L!����$�vm����N!S����P\���\��On=H�I��hR�)��.�D�Z�hKU2���Lyd	<H���F�?��Y�K�n�D����2~��r��������#W��J�r��'4i��0������&PK����Q\���B
�a��h���'�B�J#Q�}-h�M��(Ix�L)q���N�g�}�ZH�I�-Q��Sb��P0fH�G��`G�;�H`	<��7p�+�����l�J�AD�Q`S%�z���, �C��L�]DJ.�T�}��P$\�?����a}�hA�Hx^��H%1�&Q��GsO�g\��d��8d����A�L"�$�H}>.>�������T�U��SO�=��8��x��w4n�h2�]���������O9p�
l"DO��W��Q�YD� �J�U��Pb��*\����� ���d<5�cw��R�S1��Q<���&��^�)W��[�
����'�8��$25u�\%T�`,�zIp��E�;��5v���u���c����/*6�>V��2i�D�P��B*1?I���@��*�L%���=�{��i��q'^�bey�u�WO0�������R���)�U�{�Hq�)��y
L�

���C#E�:)?��d_�r&O�0��@���	����W��j"���~f��k����#�KM���:$G������5�ak��\�[l���F����H�0�'y�B�B����P�p�J�D&kP$�v1�(�V��y��Z���&��s���T�
a\[���Sb�}E��pS��g�a�����R0dF�����K��!����i�kT��=��+aLBb	d*��		3��]�m����{��|b2QA8K�O��C���=�	�$b/�G�x<����g��I7nM)-5L�Zl��m�������*5Y����)�	�U�C�-
�[�����?(�]P2v����������k�qpzR������=>
����=���Pf�t$%������������tS�Gt:��P�x���u$Q' HD1���tU�2Z,���+�QD�H%�'�������>Q�V ����r/��������5�v�V�����4�����~�}!�|M�v��~��i�fA�����I)*	��MN����k�$��h9���qXuB.���B�R�������'2k����6��/"6&�4�}))��>���>�g�nv�#����VC_�������"#��	����m��aQQa���(*31#j�������Q���w�6�X��]�X��Q�u���E3u|:{*�W\zV��9���4��2!z��0���)a �Kf"y�L�!��!3(5�O�&u��7�Zx������g"G��p�|�a|�!�pj_(J�Tg�_yEN�yMC�q|�\���r/�-�[�t]W���1"��S����2�y���}�B�&R&�j3u���:�ewn�E3���9"@^E���b�:�_@G����}�
8�"�=�g�/[�j��S��^�s�\�m�5��/�|��Gm���w�8�,>��E{������|�������8�>14��3��,������W�	��-�95S7�@���J�R��SdT�D2�w�Z�!�I4:���:��X��D�#
��D���Ohx}R����J����(r��]������N�
�&J��G�D�4|����)����:��������"�DD	N�C���'��|�j*�R�
�ED~�xE���;���Eb�h@�bXh7-y?!��������0�#��@7xSXbHf�8T r�[�H��iDJ�6���������q41�4^��������`e6�J�{c����N��.`z���	��u����������.��U���K��K��i�
�#�����0b�"�q	���������j�x�C�H.Qg�s�����i��cT��T���s19��0���
�{���N��M�:�K�KV������1<>N��H�������$I���FP��D�����2:�K��l���mp�{i��J29�@)��z�g*�s""�g���Bm�lM��kt�%.���T����i�a���������l2Y&EGLh��d��m����#���^|��TOCVN�{r��!+�i1��'��`aH����A�9������T��A?bO�av
����Z�Z�����*Z�;:W7����@��������e���I��cK��py*��b��K
�|9���{]QV����K�� �O�%�pt���.�/)f��}������L)�/c��*N
�C�Y��
	���vVA���(�e|N�,n\a�eL��E�)-����
���y�2r�)c�w�n�>U�����d���^MQam3j����&<>�1E���EO��P��
��DbbI|��x}�^�����ef�r�"
xIk�
-�9Y���e�������p��J]������
4����RB�X"J�3e_vyQSsSQJ��1�9)bO)����\QV�di*�l/l�'�Y�P��Rk"5�����R
��dW�E���"�R�Wj�%������Z�G�p-�0��K���/U(�\i�z���nS����	���^�C�s�Dr���G&�����{5�
���9������-�����5���&j��#_��/�I�����W��
���]���D��9�6F	���F�F�
�������R�F�sx��r!�	� [wG��������/�[m�A�������E�;���dMH������s�}������P�L�AC�N"\��I�[��%|�s<�B��I$&�5j�F����I��Z���3����.$�P<NK����vM�v��J�<�`��T"��*�@�����+W
�o(����0*zz���������ax�����BJ(�&(�h�T*W��?��x$F��+R�0�q�B��W-�r��:�6z~+oW�;��p��O��_�a����\&��=����
]�
����fR���0[���������_A9�	6Vq'Cd�Ei��!�rY�qzk$q���� z��D�����x�Xzq����7�W���� Qpu�cH��O�I���i-������,����Q�E�D�,��/��J+�� �y�w1
�vr���TE@/2�.��,��n�T�Q��)�|�6���w�������q� ���v���`�tJ84f����a@������}�46�"�	���o��kX���q���3�RSR_K{6����~c�w�����
���/2�2�d	�Z��b��f���g?�}:���:�"�}y�y+���������@��@�3B���a��B*�J,�-�����F�����y���J��d*��]��.A_�gh�~Z5
�t!M����B��B/_�^�pj��������h�r$MJ�9��5�;���oO	�r���0=������[JN�F�\:05hY�B��(D!
Q�B��(D!
Q�B��(D!
Q��O�S�2�0��0e�(D!
Q�B��(D!
Q�B��JB���DD5�@o]��q�$��d6Mr��Ms�����:<"��a�|(oa���Z���D���E��i1��7�MK����lZJ�	���L�V�i9q�!��)�a�$!���4EtN6�!"t3�47����:�4�W�i1Aw���&6-"����������%Dzd*����F6-p"�lZN�
�Q�l�k�O�����q�+T�?�4c&���I3�`���:�-�4c&���I3�`��-�4c&���I3�`�2y�`��-�����U�bA9�=D��	;a!<����_���2Hy7>���)'z/<QJ8�h��Z�68��9|��v�PSFTB�JlD7��n6��H��MT����[t@�KB�������6���"R��\!�����������A<,D;[�2��A):�	�y�4B�����<-��
�f8�J���:2|\��4n��Z��~t��Z.��ZV�
�m���0�L;���q�����6��D([��f%���q������7�:�)�p�P���c0c���#c
d�V���5��5�Z)�9`���g]?�3�������uD��1�-����[[q�t��F���<�;F#�����C~�_��U ��%^"�=O��W�.��p������e�Yo�cl���V��!��v�B���v���Ex����"�p�a��6�Xc����8dG�Z� ���{1_k+;��������{����n���J��>�"#�J:p�O/FrH�����?����U�S�X�FR1�����c�}�g0cZa��d�b�w3�9$q�F�%�:F�v�g^�R0��������`����d=��}��Z��a�5�� FF�V���W��}�c������GP���?6Y@3n�������a��'�����#�t\�2����l�V�
��(5��L�sex:pAZ�`�2Vgx�C�7F������
��6�{0^L������%��Hg=������Z�^��6q�h1\ll��z�
�k�2�5c9�vQ|���y.(i	���M#�G�S����iWhg��Ouc�,���Y7�������B��5L?�S�E���32����o&��l��a�Y�E�����rM��	�;�c�'��89qD2_TS��������.��h��;qa�O+��vv�c���.���,��Zf������bl�bvg4����������~��{�[�����h3r����6<���������j�2�P+����by.1����w(Z�F�4����7�V��<��<���7/�2�N~�ab����y��f`~���,Y�.�s�A�h���������d���:{����������~�����70-��,����{������g��-���7;��l_��3'�5xNg�s/�MV����
���`�� ��A�������Cs<����~Dt�c?�j�#�G���k�a���D~�	�\�I�y[����l����-h�e�n������3`��X��0����G@��K�����2x���CHtc;�M;�G�N<�f��I`�G��.���%h��]"3���5��x��Eqf6�����:��e�g��qb��2�*/����Y�Gs���'��{�sgz��������J�������e=.1A
Q����\9��CI
�h`��`K���P%�k�c���5���c\A�<�M��5�]k$f�6����������
>�l=tE�4A���(��WW1��&vLd$m�r:��p�L�E�d�������R�m������
��	�Y�JZ�1B��2��
�Pi|�A��~)�����P�]�X�r&�+S�3�=�l�����cP����>�@r��m�#D-\Y�5m��Y���U87�c�2�
BaP�j����Y���
�n>?T����=�a�jq��F�5b[��z���X������h��J��
����H��N��� I���m�e�{5}�>�p��ob-}!.�R�	��!���8C��F����j�����Z|t���vy�>���I�:t�������m^���f��U��=�n��ms6��mt�������V�����=t�8r�d�Q����w]ivZ\�v(�����+;�^�Nc��K;����<�T{��n1;h�E���Fi���c��H�n��Fw:�6�k����F��n�9��	��f�m�6��f�L)m�y-������|f���`<]fv��=v`j��Z;f�L���.)���e�g���� ���y�V[���N�Zy��z\�nTlqu��N���Y�ig����4��5�U�tU�N/(����-��������hog��a����jw�.P�g��+�Vh��y3i��n��}�����H���x���������F�tt:|v7�tvv�<P�k�a^��q�C ��������@q�jX|��I� \0;�-P����3
�lK|p������7�Kw��=�����!�;{�����EF��;h���P��_	�}.P��d��:����6��y2�|>�w|V��e�fv�-�	����]����'��N��BMG���mq9Au�5d<o������s��W'��Cw�?����b$�@������u�73��=v8k�*6�4�6O���v�=Xg�o��`A���hA-�/�%������#���k���Tw���$Y74jwZ����w9�f���L	�.%-�����^��na\���?�	�qvh���a����p����33P��A4�N�B����Du�l�pD!H�1��A��c���v
V�F������B����^����
����`sfv���n��n�tyZ�P.j.`L*���F�f��8Z${��Q�j��`^��4���r��1A9,j�du�8^�@o��W�c2V=����Bt�V�aX�E�r�����@1�����������,v3��g<�>3\�@f�8L[��
���b��8.1v��x�8�����!���v�S�m���[��DHC=����i���;A!o�����u^/*d�4���6,]n;�.**���I���Hc!��\��u�N���aVD3,�"���w�!?���q���8��.[���t�P�a����������m(27���\s������d��K��[��n��h�UZo�M
t]}�LS���N)m�|���ej��mj��F}iM����.��CO7���i���zcC][O����LF(3��U5��j��S���Z�M��ic-�dY��
�Y������SMU��9z���X�xV�R�����T�TUZO�5���6��r`[c����V����F�j��6���PYZU��*m���|e�us�M�*����r#N5�d�S��LS�TYU��ZO��V�N3��j�K=��J7������R�_�h��Aj���4�CVZ�7.�ej0���zS����#8��Z���12\��0�@�oj0�Rn,�^
�������Ah��_�6�F�����_h���s���^h� �VZ+���������~tBk�5�����wk�7��6��Y����#�O�+��p���L/�����s3��z:�z�������l�v|���c�4��IOW�}��6���r�_�&��0"��e����/J[U��')�6������IfK">/]���x�����I.�[H���
�}PI�}q�c��c@�80��T
����q5��Is��Y^�S�����};#6�F4z�/z9oF�6S�\��U9����182���<����i���TSC���D�Z<��m�;[}.g�� G����f�p9��q�T"VkG]��N0���u���F{�-��g�p�ue��8�,��Pl(�.�/����EAY��]��d2�������u��)��L6�Yfw��~�����f|aNQyFY^~EFE�qj�XC"�Q��50��z�1��<��K�P.�zI����Q~���v?���G�>���a�3�^��5g����X���x����>[��g��u_�9/BW|}u�=�i�D���wk�����t�f�b�m�s��pn\���]��<ci�^���z����RI�gU�w�����3K�j����o�:����m�Y^Y7vg����M�������y������X����e���r��������7������>z���O�N���^�+�5���u����~����w�������Ug���m 9�al�W_�������a������@���k��"�n[W�r��~t/)Dx�X�4V�
�jZ~z5�r����������{0z�!�C���C�rMb��o�W���K�v����������B<��0�`�<m�qU������R�n���,v���0#�"6"xe&T1���c�x��V.3T��j�D�������y.��gP#y�r���%G8�Cr��86���J��7?}����������u����Ic���+�l{����.Pw�feT�/K�KJ~�M���b�>��m/��g�������zZM}��+V}����1��)�{����>�����x�;wF|���H�:^����.u�m���Y����^s�7�S�W��=��q��;��wD����3,���������q�?O���������c����#-'9���OMH����-�|&�*����n�t���w�}���?����������-��2�[���/nK����f���]W����'!�}�|~��W����<c�Q�@��w	��L��>o��
�V���E���8����s@yL4�|��|�y�E��j4Zs�SI/n�{y��\��s�5�_�q�]k�W<y���g����[���e��{�=W���9T���~:����:��1�{�o��bJ�'����+�|��3��N�����������F�����n5l��z�'����'n���%�u������-~�1��W�Y���%�7����5��?�X�]~q���;�=�����o]|��������5�.����G���_������{���������a����o+u��<�)�|i��A���w%T���|�}��?5���:w�7E��?��J-����"��p3����\%$=���l^y|a��[������?����P�N+���f0�i�9(�S����9��"C^s����W������[�Q�[��a-��n1�����X���J���:�k��
����z�����!p��r{qw?/F��2��"�A!�����h���Q�M�R$��$�s)1�;sz)�������|�����f,y���G����wg�g�n8h��{����>:�a���E�������f_����~I5%>9)qIi�c?G\�n�
1GDw�rwL�a������6����7�:������c)���W�-�����5��+n|?%�o-��O�<?�S�����9_���U7�
��Mc-Oz���29,m����k'��<���x���K7|"��x1�������o}`M��q��<��~��Hs��=�Q�n�����3�/?��<Mo�������u-��_{o����uo����
D��4��i�=���7��=��T����%�^9�j����4�z����5m����&�SaB�e��;���{f.�}����n9��������������^�X�y��-g7�u��������O�]���}�?�����\9��jZ��	���x [�S�d��B���)O��V�Yr�������ZW�Om�;p����i>�����>n��j�i��������c�B����#_����C�c�_���}"z�w�ks�L?;��5_�0=����'���������J�z's���r�[*�|�.u�s�a�9�-����S�8��E�'_��z+�o��4����
n�������Y�rA�����-f��.����<v_
��BC�!7;'?�P�='gs
(��7������^���OU����=3���}����u�{7�&)����z�g��_
�h�SkZ=�������&�?[����a�S���C��q��%flz4��2��g�,�3�}Id��KC�T��!Y�P���%kH�$-��!)��������=<�g����������\��=���9K��s1��D�2����WO�E�O����f��+�OX:�(�%�9���/�=G����;G�X�D1���)���g��x�7�jd���%����.w���Uf>���Y:t��n��t�Ny��?��)�`�AnBV��a�W�~c�st�v�j��i�1���4�A������J~�����2�c=��O��
B�?�{mD
������������.��7?���'��L��_�����c����	��-$H�b�U�!�Y���5��5���<	��7w�e���?i��
�r�7���8����AT��y�f��V��P�
�[��`�[���W��������/��ic�I:�hC`o�D���W��M@����>�����N�U�z�r.�8V?4�������X^���kp�:��A����A[cu���x���G�b=����K�g�=G?� %*��yqu��b[�6�W�k�q���\MV��&�l�>�h��#�����68�S�P Ze�?�u�Oe�6�q�4����)�a����r��jC���{�����58k�on����������\Fv��9b���x�_�w��~�bp]!o���lN��� �c���� a�@���V����������W6�@p�B��^<q��][^^b�����,����kr��[3E�;�&vO�,�tH��a����tD��NZ���e�+���`��>P�.�P/a~��S%��P�K�����:��<�����I�ED����(.g���*�zDx�����g����2.N*���g����H�����\g����(V�=�.�;%`�v����%���hh�G�;���c��{��M��.r��l����6X���- 0:�������#������P��
�������i�P�����������x�B���:;�4v�|�hn���u�����i����mK�,�]R^G�����w�Y��y�p�A���JZ�����Oc�����&��d�^�p��L7+�H7R^��}+���<����q�5pc*�%;V�l���@�m_��T�;��U���c{���s���<.?��C������������J������KR���v�QNAdN��.�*]��������Nw�P-�h;c	�'�Z��+����l�R��yZN�RZFIT8��.���"AX^�Q��9���U��;���c���:M�.&E����E���@*��*���Li�->D�
�iA�!�n�}����Sa=U���#j��s]!���o�l9L�A��tC��`����<�ww��S$sL tky����o�y�?&:X�Mk�m�;�*"v�@((�48m�EY�/q�'�O������J����n���;���d#��jS�D~��iD�I���3�:w ��E��'OK6 y�w'd)
�=�|E�H.�eI�o��)%��l���7��7�9Cc��O���?�Kvn6�3�
@���)>Zt�����5���Q������h84Y����������Q�A�Q�/-������4g�LJ�9{����`�0�`N��0�?��Y�/��'�$��9H(z��q;0h���K���#������?���$�=�P�����������.���6��`Z���2���X���Y"�6O�K}cG=�pI��lrJL�L	�f��w�GI�.7�w�D|��6iW�;x#�����ND>�d�zk|��5�L7\���_��ig�����P���Tv��T�%)O8�P�� �����������`�T�y��k7Y�g�W�Q��RjR^��N�,�GSSF"��Q��V�;�������H�t�S�)iHR��
t�9�W��;<�,��:�BW�ih�yk���+�hmK�		xX��������Gj�U�������FuF�|�R��3p��/7�8�w�'T� pT]�%����������D|I]��A�"1�(�����I�����Q�LAL]L����98T��n��poP�;�����4i�l��G�K������X�t8��;�$�K������[�sr�BBDVt��>D��\����#
O����'�������#�O����^���Oz(SV��K���|�T�2,������q�1�!��R���;���������0������h
	����U������$�-����P�� �O���������H6�i=��n�,�h����)���hv`k-����Mmq-������&��+D���9�yKV�`�-���>��X ����W�6��;�����@�Es�0��C5���S�J$���8�vx_H9\+�������a�~6�@J�����g��d�D����a��3���6�N������B�"Q������S�{�����)������q'^�s�l/|?�;)L\��A���`��R^�^����m����-�Gc���W9�
�T+vi��������z]�][u�zp�6}A��5�|�
?�f�BB7MOEL���eeG�)�s.O�S�4\\�����wq\�x���]��c�0�6=��1�	`����L��?���\i�b���V��~WA��o�0�9�-�E�c����?X�\���R7��]*C����I�����2����}��/+
q7�=��A��B��U���������S����v�\�F�9�/���j�������j��������W���r��B�qO�X��6���z�������d��G�rM/�
�^�������@V$��]CilI���M��
���t�?7��";�_94!��JJKv�����������gI��	y=�u_�K�c	N�//����r��amS�WbX�Fr^�8V��g:L�^������������W���0���f�Z��
�Y]�
endstream
endobj
1654 0 obj
[ 0[ 507]  3[ 226 563]  17[ 535 535]  24[ 607]  28[ 489]  38[ 460]  68[ 845 638]  75[ 654]  87[ 508]  90[ 532]  94[ 453]  100[ 483]  115[ 554]  258[ 471]  271[ 520 425]  282[ 520]  286[ 494]  296[ 299]  336[ 469]  346[ 520]  349[ 221]  361[ 230]  364[ 441]  367[ 221]  373[ 791 520]  381[ 521]  393[ 520]  396[ 345]  400[ 387]  410[ 329]  437[ 520]  448[ 440 699]  455[ 441]  859[ 245]  862[ 409 409]  876[ 362]  882[ 306]  894[ 299 299]  1005[ 507] ] 
endobj
1655 0 obj
[ 226 0 0 0 0 0 0 0 299 299 0 0 0 0 0 362 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 535 535 607 0 460 0 0 0 0 0 0 845 638 654 508 0 532 453 483 0 0 0 0 0 0 0 0 0 0 0 0 471 520 425 520 494 299 469 520 221 0 441 221 791 520 521 0 0 345 387 329 520 440 699 0 441] 
endobj
1656 0 obj
[ 278] 
endobj
1657 0 obj
[ 226 0 0 0 0 0 0 0 303 303 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 488 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 498 0 514 514 416 514 478 305 0 514 230 0 0 230 791 514 513 514 0 343 389 335 514 446 0 433 447] 
endobj
1658 0 obj
<</Filter/FlateDecode/Length 36716/Length1 101196>>
stream
x��}	|T���y���e&�g���LfH��aI!C6 $��d'h�}4.(������Z�����%�/T�Rmk�b����j�J��������L6�R��_m?���y�{�=��s�]_���As]u��'^��n��^P/��^P���w����Q!�hIa��]����������_��*@u�m[7����g���[:��^���v�U?@���{Gg��_���`�c]-����|�����.���b���]k7o�}����@���um-�<��M��7����l_�q[j+�w����cs���:v6pkE�o?�em���'��������6m����f��~c�������nS1�X(*������W����*`����g�J���cS�?����jd"���!�j�86�����F�~;�I�/� `M#����.���,�]rP�o����LB�E���)r��e^�.�A�>��8m�(���2�Ay;�G���LxTnb=E��o����?A����O�W`�7��7E�'���:24}���e�`��8���h�9'��uD����:��M'+��:�
��v@�	����Q{�Ixj�?����
U�;��:����}��
�
'�??,��{�W�,�N��Qu*��c�8I�z��?&o_����~C�Rqq�R�X�R�w+.������<������w"��%{wD�t��m�U2��<�>���(�B������~��1���'��cc�
x����w������16�d[��NFr3lP��
�WOM���T�	������_����a��~�������R�������R�a�����I]n%�![w��!�����VI����"�x������R�]�����g��O&����4A4AD�m��I�����/_��/�+G�����_m��s�`��Qz{Q���js*������o��G��u_���{����	��o�=	��P�o�Y������T.[���|��	��	��	��	��	��R\�c����Gq���LpU���b�W��Lg|;LO���L��j�4A�J�x���	��	��	��	��	��	��	��	��&��o��	��	��	��	��	�?FB����#�l����CA��b���� ��4X����V�wp����Z<_������x\���z9	���n��Gi��Dm@m����pz����W�-na�
�~p��C�k���&�^�C���l(�"(��)�OA�'���\
g���L.�[����q[���n�2�
��V�~�	� �qoq�����q�A�}&Y�,��q�'�*��>F�>�
�y��_Q�d�>._��U������c# 97L���p�~M��u$���S7�u����\�g�����;gm��g��Z����j��3�/��CK�,nj\�����5��S_W[S=;X5�r������������}��,��b2����J������:o}��7Ge~����,�mAA�(AsTDQ�X���,��c5���9N3H��aM�(����yb�W�������0����F����>MJ��RF��k�u��Z1�5�u���]�u��h�_����th���_���S���~.g'%����~Tz�lT����G��u�.�'"��F�U�D��-q
�.�����b���]�����pTh�J�B]o��QS ������|��]���yk��/��x�.*��b����{�2V���(|�O�%Y�����4�o�!���a�\>�V�D{������`a ��Y��k���$K��7{=l�����]�hO������>>�`����m][:z������h���D_���
Q��;����)-���Z��������%a�J�Z�R���D�ha]-�K��m�%�-oS������"������0?��]o��3�nv�����.O4��E���%�1��6��Z�ja��i'�Y��>��]B��
�z|x�gb��K����)�9$����K����W3�	�j�\�'�!�.�>�}Q�([F�D���5�f��u��cT�p0a��~�,�����
��d�����2�H"6�1
�b����xq��o,����_����,,�vb�,���2�E����_�s�>�J���#���s�7$�������~|l*��9)!��<]�x����������gis
��z����-^�(����{Z{�����u�]�.z�
���%��.���������T���_Z��x���r{�����%���������5����l,q���<�2!��,�,-��J�w�H�2I ��8�d������dFj�/5�KS���J�Im�T$�!����
K���!���B�~`j�AUP��zC�D1�<��j��8=��G��%�����H�'4{P��z�e�9Se����FzZ���/=Q���BG�!<O��v6�vE�z�#l��U�pQ�;��wz��E5�����[��UL^Er�+q�s6�m���^��q�����Z�Iq _�r
F<���D^��x��}�Po�f�����0? fu�������ATi����:a5��:l�a�6�k-^)�b�:z"�H�5^���1
s�Q��l�����Ho��D�|p�k|�2P�o�$Lf��I�C���X��,�Y�k���$����k\�B`�|Z�&�.@��aim�s�>e$B�K�K
��1�E���B�������~.EW����L�,�n���9-YRbqT�kh����kQ�-KVV�MP��q��J�s�����=��Q�{;����\��/�.����K����W�?q��J?������S�M8i�y���r��������G����#L�m�v��*q���-�5�H��D���7�zl�k8[�����=`'�.���,W��dR����+�^��*�a���3� p��|c���M��4G�������r���X���9�1&qEp8m��N��Ql���x)�����!���7To;�?���KJK/���w��+��#�������#���>�Q�X0�����F�[��h����~��-������R�ztW������*�@�Knz�������+�	Soj�X����
<7d������bg�(
u�s����!RT��"M~���@�
�oD"}�HY%YE����Ii%���@���a!�<�xY8�C	����Y�b��(�4��~��JUC�tz$V��9�<��taLO*�W��X�,���`{,S���`k,�a�fR�����1�9a�z�u��J�s�R�n��c��"X��F��e� �&�$� h'h�
�T�����V����!� 8�`9�2�A����	BK	44,"XK�E8�r��#h �K0������6�j@����!T�&�\��f�\*	f� � XBPN6����iS	���R��WLPDPHP@�O���z��M��\��I��'�Q�l/��"M�H�&�$���-DH'p��!�8	Tf'���J`!0SY*���F��H�'�h	4���As6!(	r���	8����*p_P�s�c�C�w��|s,A�����c)�_	�|L��|Hp���!���}R�3�����?�7������&�[o�A�z�~:��	^���@8L�*	G�
	_&�-�K�!�_S�W�{����������4N�_$|������p_������"x2f[�����<J���Q�$|����	�#�O#��zQ�������'��c�>����T���}�Mp���$�.��n�Y[�C����
�������������f��n$���z����� \K6�!�W����$�WP��	zI�2R���.%c���=��Ed�B�~A����&�Ep.����dn���Lo#�J-l!_6l��6R�
�	��C�����l��Y�����u�j����B���������|�����k%aK�Dh&�*��Y�CX�\�pf��ay���0�,f�D��cf
����x�s���wn)���13�\S��;�H�(���^K�G8�`	��#a�\�9�T<7�zR�#a-AM�4�:fb�rv�F�L���i�,�������3*�c�BY���0=f*G�F05fb
M��J	Jb&�b��������|�'�#���d�\r)�`9�'�dx�Biz�%��pS{���N���iNi�	l����B~���T�3������6f\���W"�c�U*%��@N�2�H�p�#�Qo�8���#C��`��c�o��!��IJ����GS�����?B��� ��������?#�	�=�?�������;�@~������o ���{���V�_5t���
����E�K��A�5��0�"���D>��<�s��@�9���v?��v?�����A}��i�=��'�k�����������w���G�����B>����mt?���~@��}?�}��1C�G�(�O��E�	�������C�y�hw����p�n�.�]������N��"����v�� �6�[���[�o����Is��F���G��[�.�����hns_����*�>��(�y��s_,��/����zB��������;���iw�v��}�������*4�B;C����mm��z��:������}[B�-�-���l���p�[��-[�[�-�nshchS��ll���1�Q6#����<l�4�'�ote�#wm��7���������6t:��lu��ou���=���j+k
��5�V����[:�lYhy��P�,:�O/[
�-
-)k
-�k
-*[Z���������+�j���SV���C�1]L������	���"W����#�\Q�.!5%�����8��ENn��|��N!����:r��S�/���h������z�m�M����N[Z/aU-a�T��n��_�b�R�n+_����N�8���
u����z����MwM��%���e|����qy���-a�`���boB����9��������}�X�����~W
���K�1��;2�#��=,J�8K�D@����:X�i��@88K��L�����|J
��O��)������G� 
���S�n=�q�`�Q�z<I���>E����*�"-�V����E�c{.���+7mH���p[X>����i3���)�H���j���p�?��K�7���99V�?d��`��1?�n��`���%p%\��k�
a���?�(<	?�WN���_MC;�kA'<
0������< 7��\�9�L����#�dG����������_������1�����X���)R�����t��1����Bp:�X��	N����A;t@'��.X��:�a-���	�`=l������bzsBB���v����0��;��yp>F��a�pG$����y	\
{�2D�+�����
�������'���5p���-��1}3���p|[�^7�MRn���7��ee#����Q��.j~g�=�t��>x�9�<���qL=0��	o����	���i�8
���N�:��z�����#�
c���y��q��(�x^(�)Y�
5/���pT�^i����v���x�>������
KF�=�������*I�������wOZ�=��.|�q�K�~�+����~�)z��������S��a?���00��s#�1I��9��!xX�����?
%���:�(}<Q���~
��]�9x��p�<+�s�K���_���{x#1�^�f�����+�~'7pr�	x�_�1�
���^5`h�pw�0C��D������m���5� 4��kD�
P���=�*�}�B�KoZ��5�*��P����x��C�����_4y0������X�d����*
+$���#��c��b��J���U�O�>mZii�,~����e@�O�2��>K(-��%U��������/���+�s�u�,���.�E'�D����\T`N�L��	��/W)T��Y�++����Z�F���r�R�R�N�� ����>��u~�P<e��i��5*^�P<���f�{���b6�r�-U�4�j����_������Rg���J�Q�������0���>��C�n�<�U.�V��fS�LJ���a�����9�I9Q+���<^8�{�4��=Ks��*y!�Z�bC�<�|�+<�F��LI�q���P��6W\q�[%��h�X���M�f7�Lv'P�_Xk�[2�G�����Rl��8|��U���R.���t�L�5j������J�KkMQ��S���lPnu�Y�Cg�r��a�M
��i�)M.���In����1�A�w� ��y������E��h����\Q>���i����B^��4�������:�%��Sap��\!cHb�BJ���J�����A����Y��[��(w�p%��RS)~���D�)[da�
�����������^�-�$�+$E���z��K�DR)�
��x�1U���_��������i����,��2�����a9_����O���0�eN�N)�1R�V~�A����r���K�Y�O�I�""����2��V>�`��p�KY��=lt��,p�}:�_��f���`
_L���������ft�v�����������Z~��S\�K��Aga)�\����p��u�����v�)5�j
�I�<J-��t�����y��!��qg����`�\���gzuJ��HI�q{���j~��k&�&2�R��@�Q������� ���!
��|����L��i�O��&���[\l��&?�W���C��L�U���l��:Y#�����T�ag{�������X��r�����/
��K��LyQ��������{����Z|�{�-Z���@9-c^�)$V�?�pq�L�{����]'��dL����8�E
����?#Cu@���2n:m������G�KY�������.��p�%�y�&�gJ���HcmIz�(���pMY��d���9��-�,�J��;s���5����dEx�����_�<�������3>T���2�������M�t�oD��4�,�������
,�N/
3���aw3�`�dh*�t_���\fk�c�E�f*��eI��j��YxJ�0y��p�����B�W��������W�����UF�&�hUr��<ua�����S�`��k���*�2
K��jH���.Y�c�����Y.1]��x}szZjq�������53�����_�����
|����Jq��32
z�A��8jj�<����*��*-yi����4MG�M������/�����o��n3<Kd2KZe���^5tLk1���9�_t�9eQ�����������.���9��7^�1Cap��t(m�����z�G
���~���a��Q��V���XUV����9t��?���,Q�qn��c��Kf��yh���-����n�����v��q���e2����M�t6a�g�z�y���7�A5tKw���������z���r��7j�2Aiu�,���3U�-U;t�K�<�2�ad�0b���^����mR�3���D�[����>*z�n�%M���=m��'u>���B����bQ�N���>�_7��{��'O�D�W�25�����Z� WbG��
�Zk;^���\�����9�f��}��j�cw��*���R��'#��*���F!�%��������a��j|����'wd{q�m�B_�C��]v��_��?��K8�Y�9�S�@���t�A{�?a1pp�Q�}�u x�S�V���xEf��_>$.g�q������zXR�-��Z��f;�Tg�;+;_��3C���W�C����4��{cf�)f)`�j�V
��������\!^���T;6}\w��[���.�����xL�c�F����{�������
�L�3�>�p@�0�����~�1nK�|=�Li	sV�PT1����#��7�����_��T:�v��eK�INk�{���q�t��2�3�j�����N|q�	=�������n�<�
�
��X_T\T�(�;}i��l�]���n��ek�����i�}FwN���K��9�Q"S)}�����?gk�m�R����]y�:hS�u���p�\?�����i��RU�}|?��x�L����5i��S7��g��/���t�F|���J<�T
a��
���5�\�����~�F���l>�}%���PY�o����~}���#����<��_Z�uf�;�d���:
��S)����4'l�V�����E�@��5���������t�02o��s�:�2�?�9�
�0���9�9���+2�4Fg����Y�t���2%������n�-�W�%fd�Z_��&������?6:>nq(��������x�dnhx)���?"k�����",V��t68 �:��T1sg.�I�r�-�5���o�������J���[9�3�������=r������W�}`k�
��i���vc[�}i�g/>wye��Uqf���+}&���}�n�T_���es����
5%��]����=�����e�ggU��~�6�;v�����w���IU�%�sT�I�e1����j��0�C{��d��3���r��XT9G�5��>�_�K��w����;�?|��g���y�76�Z��i�r<�&��eUg,��+)2��m���k�K��5���2{�Z6���qV(59���E�:�|hm������(�Yp��R�32��+����tr�Q�`L� /?��"�]m���)��y�����]��[�����sV�k�`a!{1-�V��w�����c���}�}�����w��n��{m�%�@B���/���n���f��e*�?#�mTr>�=����^cMD�i�d'�7nm�3�V�Pw������buh������T�V�������N�-LZ4��<����<1u�^����o'�G#��z�zL|z:��i�����5������J^�;Z�����9E�\~	���{�j9��h��~����-�YS�}�[Y�;�]���-�N�nfj�[��3+c�x\u�D`e��������W(�Z�6{����yS2�������U����SW4���x��wQ]n���i��pW��O/��y���U�f��<^�l&��es��dO���\X���SLN�����:]�'�=���_5��vE]�����ia7���}8s���"����6;,f�{��Ymp�[t������I�C�����������6��V��e�M�OPH�����D)��C����R��)���N����4���px�=�d0�V����w���(�J���?������s��8 ��$� 	� x� ��H�)Q�EY���AY��#�%�dY�l'q_r�u��i����KM�[�ql����6��N�l���/��C���� Be9�����3����O�
, ��G���eBT�)�BK�R��3"�I�����!���%o)wN�
B�D�y����Z}�S��L�-w�c�BW��k����c)�1�3��5�@�c;Noon���m�����t�K��w��y������0�����L���Y^�T����,��nZ����D���9������)���dVL&�����v�"&}fX<�@���KjL���S�DIr��d���8��������I������;@�����E��XbE8lYZ�"�B�V�����8���i5�x�8�T�/��/�D6X��<�W���`�Rr��8ZH"�����Zx��?UUE|�A�?;�cJm
���C��NcB����i�I������h��w�Z��*��yQ�^k5��Bg`�	(=����G�hBc\A\��+d��e�Ax��9�v��:`�1�~;U��Co

e.�v����Y����V���HZ�Zz�F��$i�i&��;�a
x���r��F
/�������V��._��'��y���U��waw���0N��<����]u����6��'k8���f��W;C�����-�i�0>$g_m������7����n�9"��\�D9ok]h������eV���:�I����P��&s��A���(J %�"���yu������,3�+v�\���F
W�u�����-���Z��K������p���g(m�x����c'�/�tm�ipO���������j�})���Z��N��w�i��� ����i='8]n�?��r����G �u�j�tb�:���1�k�:f���\e���sA����������9n[:%a���N���3��YLY`~�1����
U���P>��Z���y�`)LP-���������n�c����@Y�n��	t��h���o����5�*�K�3Y�r`j�����M��^;���3'�t���
��lU�0������l��WtE=����{Gd��3���z�GwN�=Ve
�&��|��D���7�!�&Q��8��|F�:+������`��w(/81�O��N��o&�� �q�c1��z>����b�������r����h��]Ck�����-�u���^��{SwT�bFm�����3������v7;i|�>�59���������=�V�{�.k�ix���/������������#�"'QD��V�p*�9Ys2����d�������Sw �&���������������uS�)ljC�����
�7�1�24���#�96[������Y����?�l}#�7�o7���I����Dd%�<�&�E7����6�\���n�(n~d�C��64'�<@,�)���(��8 
���I���B��5Y�a&��'*�"��Q��,�_RL��7����������l��0 >��'U�#�n��eV�,V7�?�=��^`��9�)�a�����X�c�J��:V��hYo��s���y���L2H��[Ks�,a���M�^�����L�
��u[����e{[=?��L���3k���&Z��l��~�"��f�}��.�6�:|�����T�$��y���~��[%��G�I{8������&���oG���[���P[+<�����Z�_�
�:.�2EM���CM���J���m��������vANp57�~�R��A�[�q��9�L�T!��+��k�N�D�V�������sV�m�O�O/��R���j��L�RQ|v�R�T��r��~��������
�r�\�@"A"���e��-#E��*^��]`��*}�s���S_�H��Z�V��fN�l�.#g�����[W��1��w�Y�7���9*d����P�r#����������L�{���&��I�M&��g�MB"���U�]��p�%1��;=
zQm�=&�#"R�^�����Yom�|���J6����Bi����ro.��B����FQE�%O�m��d��~Ntd5m����EQ!�5��!4�)Q^��F���<�G�>�4����O��
-k2�@�*�^�z��8F��{<���b����]wS�`��v�'���uVt�V�j���ts_�a���}u[vv�~t������uH���\�^�1Is�������*�<�N?��d2����h��e�Upi�q��#��<P���0K��%I�H�
�6�'F�.����LVsS
��e��ee!���W����n���?�cE=�0�J��;'�5�{`]��e���n�8�����l��������W�������z����6�j�u8mo6���;'�����=c�DW���V[m���;���
�J�Ff*x;z��~
�D�Q�?��I��M��t;�Dyi���Cjw��L����6#' `jT��r��\V�}���*�^�K"���Del�E��bV��������P.h5Y�U/J%��k�D�
hO�8)*�����9�Y���795A1��~�[B������`���i����v���w�4��=f�u����TqSE&	�h��U����H��
��l�����;��5����qh��z`Mz�s���&�"���j�t�j�k��[}}9����������+vv�o�#q�?q�a�ay8�S<�A���aq'�Xs/TQu82Q�"��V����.i������Ht(^z���2���?8g�����$���'��)��/�V���~�=:��@r�no�D��vI"x���%��1��gE����9��QrJ
�~.�X_��b)��!hy��2�^��uV�\������D'PM�>1���������D�c(��w�i�r�yN
h�����k���M]�����8[��������M�^�rX�a�J��K����&��x��4K:\rA\
���s�c������T��6�G���r�����gV@-�9�~����w�k�E$����^}��������GD@^B?I7��4�f��]����S�'3����>�g���{�g�
�'��$�Yd��@T'��7_!���������Y�1����6�+������5����(@_.5���=K�zu�c�{	�Y{9�sMnB�a��'���yUW��z�g;?3'�<�Wl�����O�O�@~FE�K"|N���A���f���d�b�}u��"��tI0�([���k������=^�8����(��F�/��y�bx
�����5���<�A5��*�Q������4�1������ER��T�ssZ�q��l�i(Tq
'��8�L6���/�x=/���8�k����P�^]>���%'���R����P����5a�Ze 
&`�I��5�j��*J[�D����},�XX�!oIF��e9�~��Y$��e��r��GQF����u��~�L"�������H�l���Y5�g&�n��;�\!�ko���	�����m0�8��}�$
��s��F�����c����v�Z����m�UR�H�XY>�J//E��_f�o}%��)�;<���I�|�mk%Q�k�*I�	4waeYU*5�v�.!��rJ��U��%�,����8x�S��zA��zz 1�3%������{��;VhT$���5��(��=w=po������l����UK����_��	
K��7z[�X+�~m��1~G�������9�|I�Z����<�H��������z��@���"f�K��S�� ]�,�=]�Py�<'.�9X/���8%�� &V���F��kdkn��X�'1�H��(k��������4o�y��*�����Q����{/B��}����6��%eon��XP^�����F���9�_e�$���	������,��M��S;���� �����sL��7[>�9*�Y'eVV�Ke�q�����M1���|g���(���@����qTw�d�$-��U
�� �?�Rc�Q�$eL+c�Q�#8c�%��5@ P^�Z_Uy}V��s[���Y�u�������u	���\�"^���	����>�A�n\+�&������@Z�j�i���H[&,�FJQ�
�����Xn3������������N�V'���),��T pnm��*�����x~j��k�����������R�b��}�����$	� UE��^S�="u�>R��:]A#)�#�r�������������9;�R����E�hY�m@aV�e�!�#hek0�?����K��d=e<#�����9��j�46�+����������qI{Vd��D���D9��,b^A��{�H�����86P�+������-�3�4$uI�L��H"U�'QtRnu���P`�1MR�EE��R/�����vhT B(C	�e��2E��N^�h��{A�}u~�U4�G
EA=JR�=�r�nH
�a�lP��e����J:}����$�_j;rR�S���}��@���#��n�<�i�"#x��>�I���K��b]o!0�_�����}.���D@%�.��t�I�Cf���L�g�'
���#��< �������|����(�/ ��kS�LO��.�Ij@��_���N�7K'O���b`�M��,_�E�\��J��uzt`�~I����,�2����D����\�!Q��,y��t~� �Wx��q�|@��Y�*{�����7���� �"O(�r����6
���VH���?zd2�'%F�����%��w�)x�+M	a@d��$�d&WHb�D�$&F��Y�[^�,S��-��^��x���qI|����d~������j�Q�0�c*o,nJN$�I���]W��6��������-��'�CO�P��+w�c�_kY���*�K����1�qE�|@�A0<��)���f�M~��D^�����P�.|�e���G�;�Wd���"��{���=����Vil�WJ���339(�Kmy�O�,�O
���$���En0�x##	��$VV�H�p��rE�|
�|���*���a+�V����d�bK.Ns����:�t@l����?%��^����ER9`�6�x5�J�TR�_k�%�O�(.��2tqhm_�e���E�er�V�����������Y��&�4��SY��Eu�+��"����lf���F�\��m���R�]�2����G�E�m��D��
QJ�O���
�/'w���7�������B���7������g�Z�n)o�A0�q���8b@"���R�7��r!�����v���"`����{d:0h�5!|8|A�����r��^��X��Ln	G�*��������K���e���o�[����yz�-7`�X`�+���2��i���IC��>N+��,�O�?X��gD�N>�%H�`)��|
��+�����%O(���	����g8U�y~d���� :�tia��w��s�r��)/B��+>0����Qi]�6�����o!��Tx�Tomm]�l�yW�6 �c��m�q�Y!<����y�,b��|V��4�e��!GAVs�*o&B���MB�Q��f�X
�O�{��d�������x�i������BVin��R���l��R w��@a@)4t6����{��dcGb�r1�"�2g���%P��o��T���
���(�._�j)��&[z�s��5��.���������H��_�/|C��75�uEC=k����������Ta�C
6�=DhpD�V�ZM�8�n����m�9����b���g�r*��u4]{���b�+\��+%^��8��*z>��i���\O������N�u������������L�,���+/_��|���\�Y�[��f���pK��9��{C6"5l_����I#v��2����EC��H�%��lP��s������?_��M�?��~��0��q�p*�75�l���n�>�����uk��>y������w�oM�b�7�?��8���eo>p�.��?���?�t!?��,����W7��-j�Ei�����f/A����f��C������R��T�m��xe|�~N,�A���7�m�+[�x�6^�]�dZ
7[�n=�^��><7��M��v�/i���������
^-v��������m~4����i�n�oyWtG�&���2M�������'v<��6��$��*u9���w N��u����!�����7h.XE��3����G�k�OA�U��e�Jm���+iWx��j��`vU���FJ���z�]��j��2S�avy�z���w����Xd_��|$�P��������W�y?J���4a�t��~!~mS���N�C��i�������0�E1��?L�+���2p#��R �������F=sb��P�W�U��{2�]wO�m��AX�J��FqZ���/�����4�niN�|t}||y�ZEc�|��/��39�dm�8�������@G
��"����*k,�
7G#��]��M6hLv^%�ps�������o�9M��{���x��cskD�F�G�E5�g����������]��P��U���*� AL��H
}7�u���>u�����j��o�����	���;,�:����e}we���~@Z���/>�c$�'�����!_��K��[��l��.���|�
����K�c.s��]�%�p��kg������e�_<�����S?����9��{>�����\ �O��Ky��[�]�}�F�diwzq��.[
l5]Ck��=�5ml��IR���7B��l#�H#��i�7�������v���8�7h�j)<���:i��h/G�=9,b�Y���z�������w���i:������������2QUF��)��8������KH�z�i��%c=*���N2�X}��4#�=�dw���{Z2����`��*_h|[�n�����t3������J�<�%�j��?-�
y�F����~��IQ�{���X�=�4�������8��
�c���^OFNY��|����E�(ro�#��o�.^�c�p�P��u���i������NH�}�<I_�����\�*w������&I�S����	��\J-�Q%g�;`I�JXq~�:�_����R]������3���7yXd�5��/�MH�Y�!
�����|����V���^�a����WG�j�@{|69Q���w�ar��U
e�8I��,rr��ch=:��m��%u<��n�f��dY=y;��Mu�"��1�Vam"��V/���t[&h��g���2=6��i��,��]�Fm(-����������������%P����	�0,�b�
��r�nC��eKY�����#�#��4[��=���a�6������XC���sC�i��
�������F8@F&�aL�H�7����W,b����n�:����V���x2�S�=e~��bry"�X���O!P������!q���]UU)�Q��/�������.�Sy=�<{��������^���<��rf�Mm)�}��{����gv���>�7p�j]�NM�����	h��<|�$�oqt��xt���V���p�X�F�����)g'�h���&�C41�fV�����4�����XC������i�u���g��W�>�;���DN#� V�w4#f�dNB-��wsC=�����F"{p��L�lv�����v^�M;]�.��B�^�XPT�m^-�yz���l����lb]����@�:l�� l�1!+���{�����S�[����)I�_����T2S@��|�_����<sgA:���,�����!�������n�\���;X/�8�<An�3/a����
�����E��*�LN�h):��*�vQ�V��z�nP����M�S��RV�����(���US+�#�9�J���G���o�V����u��>?��O�\�����jTu���EO�/C��y�"~p�A9[�<��E�+�)��F��~�����A��>��%�`h8���/��S��Ru*\-N%�b
�]8B����#��Cu�"�����V��22��4z��;�D�����8g��,.���������?l�x#T_�	���C��K6�Z��_��<a'X�p�)��8�l�,�����x��������5ist�3�����T�?�
g�T,s����<z��\����<������pS����S��QjK��8����)��^=�j(���i�����3O�c����_�a����?@_�	�b����Q���A0d�L�P=r���U�!������^�X�fa�,C��_U��Z�)c�<:"����8N�S�W���@��#C	�9p����]�����%JoZ�~=����F
iw4z[�mb��q���������
o����3��
h0����)|?_Es��Xf/�&���n��������@�����������v�<�\�*Pk��>w��L������w���(�U6�%
gR9J&ZBgE�4_N,��p�rbz�sxL�cx��b����>�T��LuS_D�q5�mhK�{X�������"u=�*Z�����q�x^�m��	Z|��j����tu���S���k��l�6��@z]3��3�/8���c������hjv�_z!z}>mG����������z�|��^���_��bI��B�|Dy�Rdp�����b2t���BW=7������e�5�������s��F�T%��6-���zK��!�)Jp���muoo��vG��xp��U�P�:��;(��0��I�nsU�o��G��1�^�H����-@��a?��#�H�r��r6�lZ�CV�JV�g�����J���!���7�'��g���P�91T��X~��N�)9�BN<?G���r��BmQ��J6�[�V�&q���PC�{�^��)�����p��������h5���z
���M4k����'���
�4��IW/% !$�<*������:�@B�d����#D����r�&�(��JB&�� �h&$��;��UF���������\Q�
F�p�LN�0����ud�[������e���d7T�4$���t��'f��z���2����P�Vwxk!w�]h5�"�yS�[�5m��D��=�q�{{��H�H;rF9�����P-u�k�m@W��G�"|�W"-_S��@S;|�rF�������G��b���C� ����h*�i��H�|Y�	8��Eg)��A
���J�5\���W�u'6-G�����F�c4�0��_��e�����6�ly��&�s3������&c��4qNC-f���Dx���ek��gw["�n�
��{������G�!s�%�k��Mf�l����m:][��@�uf�N�@�����g�?�����A#�9���q������������q
��b�L4��zV$S��E@�yN,!��C'+H*�U�r���%�'���=��,�>[}=����(F�����N
7�4����q�->`�6�T�aW��PjE\�8�8��SV�Q�������f�4�4�X�b)���Q����S��vC/��'�+�A���������|^���D�&s�������=~q���:�^LL���m��%����\/%`j">���I�.���b	�E�JD�N�����������rI�v��v`��vc�+�H���5�'b�g�=�z��lZ��T*���C��ZO�D2AP�h�j�?������;XZ��8���"~��:�1���=�<���6�	b@��[�j[�+�<s�O�C��k2p�Z���:�����?���M��b��-V6�J�b%a�������������[��+���Mrl�Yl�{"/�NBH�%�Vr-���`J���)�h��Z�k�0����6C[���)�3C���������J�68�;������:�����������-�;��{���eZlVJ��T�+���R��!23����9���3�'�MJ"�B��{E�X�\)gU������e��0��hV#�Z�Tf�T���T�I�H�j��9-�J� P�}VT���-�E*5�WE$����������~���T���,��}��yc�#}��Heu��[����vm�y�E=��,{�+�������s�}��(0P-��<�_L[�I)AC��M7�5g����I)��*�w�DZ�t����K���2f��-.���hR���p�3�����vm���n�0Kt���
��*�T�n�����l����Sy�R��,#�u�5�UW��\�Ph����j6��Y�.�?�.��h��:�R���j�#M�6�rZ��ue�TE��#�H"����M��U�n�*Y��J��L��wc� �HN������
T�o�6�e)d����������)���8%��V�VR�*�IR��9�AE1"r�2[�29���?���>GCd��i(�/x���+�?��r�3��vgEK�G"5Tn���n�z�G�\����Vo������l��r�7�^v�2?>����2_}���c>���1oId��.��:=-�G���#�uVM��9���/T��EMmk�r=�T*6S�����m��1��<mhe>gT������
����f&�XQ+e�����@#c�x!X�����x��x'���p{m��������'���F��-�|��1�r�+mjil%���ZG�����8J�~�����
������c�U�:S;ot�+
e%��l7IV,�D����T�:��Y#��P.#��;��,y:_Ub��yj��GAq9�����m�T)���R������tM�2<f����3�z�=k�Y������#-G-��Y����!�^k����^�Z���2�=[�K���V��*��M������^����B��l�=�l��~�|e��|�}��|F+�=�#�|�'uc�5�q�Y�R���quE��2X^�U��;�
�
%~�L����������S��q�VGcM�R!}M���k5�e�b.���Z��3(�:��a6��
U�8����o)���JI��pT� ��o�k1��m��e[��v����-��O���s�2G}D��#{��o���!�6�3b2�h�{Pm�K^_�7��>�;��	[���?���n�
[���+W��	�QY�^u�f��^Z����W����*�V�d���J|z��Q
�mz��V�B�f^,`�������J��,����f����X�ze��|R)|��+�g��J�e�0-���j[;jp��l�z�
�me�
�N�J��Ek1$dYR�|9K����g�4�� �njU#���������M�&������f��&1���V���6+�;���|;�Q;YiQE���������Qj�[�+���V������cq�C��YU������yt��|�}�}dSOg�'���43��!��LK���S�_-U���L�"�g7�r�^��[��pP���/���&�.��K��[����bs��,g&Y���j�������.�2Y��X�1���w
�������+�r�>p�������H7aR�y<����������<�}�����>���G����).�Z`��}������3W���m���k����_�\�y������ZV����1�:,`���$0MYl�:V)y{�OyZ������d
+�CF�V�N)eer�7�!QJ%��<x��G4'W�����=it��Ug���=��.����JM]��'���XK�&�q�XD��k�H]���@m���������w��-���ZK��|z�e����,�]�M5F��K,E2Fn�k;v�����eJ]�i���
%X�P+��4fB�xfV��I9�m���k�G=�T�,0-cF������~�?����fF���]�a��e��������j�
+��Lf*�-*�0��/�*�6��V�pTZ�jk��x-u����ke�P]%��-���rK������� �;��?Y2l������4����%�1�ZW	���]��!V��>��/�}?$\`���E��V�(,3K%���z��U�(7+d*����Y\aV0����g�&�L�1i���UW���:�jzmUM����3�@��?��I\)�������T�g��H�`=�D9��'KB�������d~��|X�*,���cl�������0*K��Qf�1O���<��R;gR��:%$-\��iS��}<�LZ�'��4��Ts\�R��3h5e�%�T���C���=o^=�N;R�^Uf��`�J�Fiz��8�z�]�{H��_L�T��]�ba��WI���,�Qn.���f��ZQm����������Q	�37=���a^��2�<)s�oL�_;��9���X�N���*r�*:�*���<X>������������vc[Y_{�Rmg��eS�_>����+�o^8����2��8#�������t�����R�@o�������F}
bcg��S�^�� �}N�^�� _%\0�j)M�g�%v���d+j}�/�(x���f��v��O�����Jyl���?�����,?�����co>�gH��="��)�&���U�}�X��&�"��<K�Y�D���j��=�O�T�����&�A�p��_%U�9��X�H��i�O��=���w�N�SIY�Bv�K��~�#�� ����U������3�?�����B�h5�������N.
��-�c��Li�s���Ru~����nbT�)�M������wo�{m�R���G7��������KN��<��o���_g�<���i��l�\�#k�n5�v��RVy�*df�]y�j��b�0��?_�/�����������]����lcQd�?���f2Ya�9���e��r�r
�gp@�&-Sr��O�����������!���GY���NK�����9Gr�8����s'�O�\KNX�Y�eg�0��2_UtC��}6},e��?����.:	y�=i����M��a.��$2��;�L�T�0�k��5G�{��;@��,�y�{Gz��k�=)�|�!l�O��9����T�"�:%�GL\~��l��[K$�y�@V�j�� X��n6��**d��#�����]�~�{GzO��{����?��M������VT
6���`�;�F�>t��:��c�d�����|���n��&�p%����W�E]��^C���kooKK�7���W]~��hW��������?�>n
;*9d�k��v���Pmj�5;>�wGC�����'�)�����P���/S�%=��k0���bj��H�����B�F��B$�-/���qW�f��*|��Y�������������$/5��4}
��i�4W��r��}����a���{���J�}��M-nw�����|�\���K
�$w�G-���4�
�����[�|0�n��� ��ljja9������~����	�}[����%����]�����*S���4�����#��T��oX����K~��k���a��������~�}��.8TZk�7xZ����|�.��gyR�F����u�d�}<mmnK-���c����aAV��P�2��B���U���i���v�z*�������=������"��x�c��%�����������6�V�cY�@g�s����S�5��e���W����V��^�}������Tk�@xJ��'H4�t������.�����]���i4=;F��L�h��D��sn�����F��sn�f+"��QS�E5h���m��:w'��\0FG�s��G��UB��������IrVW�~@��S��� �#��dB�������R����VUO[�������z�Km�A6"�7�$�`5���%�Z�K<-
�B���F�u��1�x|��%��d��������(d���y������^c�yr�wy`I$^�<��>5�54��tm�8�#�vp�T:8�G�y�|��ZT��������
{hW[?��i@c_B���QQ���=lo/�Px�5�F�����;�s�*'M�����}�C��/���7�K<4Xn_�i���;���*W�JEX�bf�NJ�){�0��!���FT��4�e�eR6+D�B�0���;���K9��K�^i��+"������v����������+OS� ���GO��S���M�~�����n����K�������..��F}G�r78O�:�+���~h��7^��%�������9�������&�z�o��z�j��$�G�i5�L���r�s�>��r7_���������H�`<�7^}t�����o�z��}UM/�U2�$���Y�fC��:�;u�d}�WW5�5c�g��<i�+�m����>e_#��`lb�DmO��k)�~m�hn#�?�#�����#��6�U���{W����2��o��^27�����}����%�%!-
ty;�����vv6p�;���N/u�F���td��h�/m�r�c�i�du��C��Xitck�\s����&-������9f=��������r�*��B��H��SB����mGa:u����U�W �)�)a"����$tr�I���S�s��4D�WLZ

���5f�D|�s���*�>����XV&��P��0��0V6a|���r��H�$��Whd�����~l���o8���w,�:|���8:����:Vb��^�?V������������K��m����HI�{�����]���K��4qT��7�)<G&��t�IY�Kg�&������^�q����gtR��6�����?@<�q�w�e���o��	���\�FT�4��A�.:���M+���)i�j��/��.arY=n��J�?!=����
�Vue���&�
��K��m�������U
����M��%�@?�����Y*���1m/�_t��5
6��X4���h��
l�u������<f��
�=��D�H���@�A	���Uiq����SH�?�g!�6������u��<�4O��s��m�T�&��_�V67���F���������8�rm�V'��i+��_�+�JV��qNG>�*��{�
����S�[^��
|�ar�,��s'RM�f��R\!BSJ���{?R�$A�_*��H��0�����K;�����k�wn�#������������
�S�����C������%{�Ib1�pP�4�1
J���]���n�2&Fj���Mv�,���%��� ����)4)�x�m���B�,\� �,8�����H�G�~�r���A����Q�?�4�����x����b$Wqb��:�n�t�X�&V�j��mE�v�S�~���*���W�E	~��S?;��]-=�;�NP�]�����}`T��#j-��2��E�|\��440����O��������Y3��J��]����g�8�������]8n����-��?>;���o�z�q���=��������bT��1&r
��ktX���:��b�=�k`\��X�Z����$\~��A�/e���<���V��5jpD������������#��������'�
�~b^�RWGR�gE
g����d�;{p�->8LT
M`����H+�R/E���������S���BQ�Y��]�=/���Q��A��#�Z$�}��]��f����#��+G��qo������z��OcS���VS�z���K�������>���=�s����a���<�!�V����y+�Ef�����k��~�������-����K�����#=����f.A��k��11���!~���X���\j����D%O_�M���.z��$��K���*Q���E��Z�52�*�R�##c4��)?�����LQ,����ym�	�)mz�XcT|��j��gx\�o�Lg����jr�����o�����X���p�%
���_�e��z~K�����(U���w�
�sV&X"�������I�*.4�o�QY\lq8���m�K��
�E'	�&N��ZtJ��R[�@r���>�T�%f�l��c�X����`���}'�6G�S����������~WKD�N�$eQH����Z�$��YEi1W��G�������*[������0[���w:��9��������Zc����}�?YT�g.2/�Y�U�!�����`���O�o�F�d9E��~��dW������Y�+�J�bP�J���U�ELy�RRmJ��SRKsR(�����IkY�����������u����S2]�����[.��i������J*���Q*q�6a���E�QB��Me+�D���Et_n�<D�MJU�OQM�@?�T����VR�I���__I
w��FG=�4�
��|��M��z��g�����c��e��������������n�����>����O�;6h��?m3g�-8����O%��;����]�N�J��x~u�,�<�A�]����������D�6�n��
��
��������s}/�??X:���4424���l���F��������F~�A���W7h�6h�6h�6h�6h�6h�6h�6h�6h�6��	!T����$��+���f��IH��'BZ������I��2d��&������V�c2��V�M�{�t�i��d_j�W�
!�A���BZ��+wi�e�1JK��f����,���d��'��Hg{DH�������C��BZ���~(�����[H�!�]%�U�X�/5��;������V!��:T	e$���8��cB������i������i������i������i������i������iZ��qW
i*�G���	5�vH
�����8�M���@*����9!H�Q��F<��!o���89
�wJ�����AHMAN���Z��D�$�!/���.�yH�$�E��"�������RU��6�"����(���_������B�]p4�����'������_�4��v�����>"�Li;�S���g��_Q���n��,@����$o
&,��&r�J�I� ��>�����e9�':
Q{)>���A�8H��p"���|��ojP����t�q�>(��Z����J=���)H�S�����AR�����&�	�0�%N�4D�2
9�����������"N�"��{��
�b/s�O�P�!g��J��I����:6D�R�<�l	���bTsP��'�Q��Z�k*3��cX�+Bd;EJ��s��v5�G�>
�
d��k���6GZX$rXFi��E�����z�km4Ht�-7���b�������pA5t,�%�<�2�=���H�~����c�T?���
�#Z~+����Y�>���� A���blG��I���>tF��h�4�jj
a($v���U����7�?�!#�V8��b�,�}Yj
���s+��A��FH/�!V�u�S�;���*m�'0�i���@�V.�;����zX��I�-IS�$��K�>�/�u�\D�\T��t%r
����}��o"RtS���l�%��~b+r��<���	�� 2M3��_W��l��=N��'�)������H����#��q:��@����:�p��Mt����;A4���C�9H����������BW���%W%2/�����)�=_�UQ>)W4�@��O2���B��%y��W�Q����fR��#$����%�.$�{u-��A�q�!J9��]D3>� q�����GBM�_?}��6BD�X�>������F���Y��V�)o�Z)�h.fvZ�l�e�������|���D��+^�ER���'Z�����Xr����)T��
�B_�c����1a��t�4#�Y�cjWQa-D{��5���)Z��f�l�g�ERB>�;�[H��a���ux�`M�3Cd�'�)`\]������A��i2
�������C��X:�wsey7Q���y�ce�-�J��R�&5�:t!q��wh�q0�B�do�{�M�a)�)�%(�TI]����FA�q2J�$q\g�����>�S.�g�L�NI�8���%�Q�
���J&�� @>q�)�����#��?��?@8g��/NWc�H:��;L�q�I����D.��Y+N|����w�9���FcI���J��u:�V��/���m��������l9Nr� �/:g��Q/��BN5���WM�#�� ��C�8��8|���~���G���n(?m��}�r�G�6AJ����!��}B9\�r��1N/H��Zt1$���$�sI3Q
�Ed�p4�
g���!�����'��$�~i7�n���������1(7A��&<S�#��~8Oy�#p�
����^����Jq�Md0H�����c��?g'�1
5{	�Dz}��0�^r���j��p���e��a�H�n�|R,�i�e�n9�*E��>{��F��F9�$��g]�.�	���#��GJu�'��O������1�����u��E�jn�1B[��4�R.X��D&�D���Z���(�njn��C�X$�Np=�X4�%B�p����xhf6����`�X0��N�����h0<�
r^�bd!�������G��1\��-7y�*�����}|t�����Q���
s��8�gr6���v�#1ngh��}<'�e"�)�,��A�=����p ��Anxh�����p<����\pn*Os�@0�����=�G ����xC��M�B�7��_8��B���o.�/r�C�Y.�0���\,���3
�&�sP3���X��Jp�A_b!�s� pJ@������@�~_�����E����\0%��i �Ec�F��|�87��BsQ�?���\��A�1}E����i�v�^������N`�:������TJqc��A�1���D��9n!���g '��'"��1���������g}1�5$
�C������`��6�=��zz&C��/����b~�j�:���l��
��/^��b�Hb6���;�aN�����L��]l�M����P�_�����0J�:�/D�|�k��G@r���R-�����A��`�T��X���H�}��`l.�H@sS��+�,Ad`?�����=�V��X�'\�,�A]�#vz:>���!;���~~�@
}$S���#�8��Z:���A��D,���)v@�Qlk+�@Mz���]J��@�x������QQ��;�>�XHD���M\f6�G3%
�	l��
	��2�
%���N��5� j7���H8�1D%��7
E����!�i�G�P���[jA��,�X���v����?
%�������D�',S<88"�Lw�E��0��1��8J�7� ���A27������d��\d
�^�G�hg�����������p�G�j����3��&���Z�(@�"�C�r����4ss	�������)���3�@���}zh�@��P|�Xhzj�8��8l��A��#����B����F�4q|62��x,��&HD��,G���h`);������&n�X0m�
Gx�P��1��T|�S����Kc4���'��B������x��&F�'�u��qC��������^��{��]������=��������s�#���C#�.�������	nt���A��H�wO���������>#��p�BSC}������A8��9�������&Gp���h77�=>9����=�������{������q��o�od�������71��������q��gtl�����$78������}��{���vL�x���]\o�p�@�5
���b�}�}$����=�C�#������q8t�������&�\\���H��(4��	5FI#Po����E�eh���=}),�}�^hkWN/����=�q{�"d�q{��w{@E�6n���E@��q�`�6��m�����|�VA��Q:�6nl�.��v�M����?��
��r�c���S�7�����W$5k4�a����Z-.�6���^O�?���./i_oy�����z���P�Hr7��b���R�+DE������=�g��q�>���4�d ��-2t��>�C3'�c�-��c������{�+�S������s�<�]F%��X%�0���&�u���� ����L��a+`����0`����V��i��(`x0|0�^����w��`0�b�P�C7`�������R`����0x�A�0����.��y�p0�?����7����o1F�PjC;`���0`��db�V�a����z�`�
0���:�p`x0|0<^?�o0�00��0T7`���}��`8n�� K��0��v�0����7���G�3�������_����9`0�2����!��0���F�����L��4���0��a0|0�	�/����3�s�)`na���1-�a'`W�0`�0|0<�oqW�������S�$�X���X�c'K�HT�-Y-cH�T*{��K���Kr�J��*J����yfl���������s^�s>�������s>�s��
��a�2�2|G��@�z&���r�A"�|�>\g8e��Q�p��<�����	���Lf�
�{��q�
���-������>d3��F�c .�R�t:�n�yff&�@�d2����0�'x�'��k�M��@�j�9�/�T@�E��`<6R�70d���G,@�2����
�D !�������-T!���66,�	��ByP\?6�!aK���2I\��\
�E�8���x���]��tb:3ws��|8�����"��L�D����LTP��b���!J&�aD'!������S" 
1�9���f�[C��T~$8N���+PA,�@3�*R���bBJX1,o&�*~
U<�*�JD����"�`XG�X���N�J��#��P,�H�Gj����)Q�w��f��m�K���T&�%�H��Ix���� �!�A��D���h�xM�2�a.�#B#��F��M��%�5>
�(H��@���2-���[ �x[�=4�4��"�:��f�)"��[��"�
-���$�����o!��aN|��`��JU�I%	��,`��	qB#e��\%��`a"�>�:�)�E�����3�X,����/�5�<w�$�D�R�*K��be�d�R�R���������OC_1��a��G��q�E�12����$�`��K�Oh���X�4����:?������P��*��8&����u�e�bQ�	%Li�,��a0�O�a�,��5@�X�`E�jH��$���x�1@!"�V�����6�G<�
�@.=Qi.13�rS9b�eJBu��/5u\�	P0eR!�N�I�L�D �Af�@�Ty���<���Z�(����6�C�$<�>,�'R���&���������D9d���f�����S�`�����z��z�bX�2��I�aQ��
h����I!�p�/�x�i���U�QE�P
���,[�\�a���y��.IA�H2��klll��A��h?�jyq�}���w|5qD�p9�Ee���P�����@����D�����2�4*��0�.���RPIj<A�G�{li���P(o��;N��')D�(�������I��L2���Ck�P������@�WX�G,�_�HCJ��x�/R��O��#N�IKK{{�{d�7��"��.J^DI�?z�@
L&"�ypZ�2�$P��KIIA�Q�����(q����(Rj��bz���1<U
�.A3�:-�����E���?�4~�-�g!��Q�l
�yBT��: "T�F���;
��O������d���B1ds!��<Q-�&mH ��x��0��"�q���4��I�x����r�4�Qd�qe`���L�\*B��y�i�$Ha3�x�GGT�����A���G�m�t?*��\!��aNa�M��E�8�!|K����+ �Ii,W�@L�+OC��0h�$�H+|��C�A1lE��9�?<�����d
�&���vH)��������h����G�Y����,�h!d�0�7��������Pr��"�DF��4��2�3C�+����C"��7.�tK�Ef���������CO�1R�H��#r`�x��t�
���0������#�u�?�s��mLM��������R�ok������qr������u�6ZT�)4[�0�b�z���Q�r>����k6��]
�V�����L�<��y6�y�b��<�+��R�F�7tv�����fK0v�j^[p���)&Cf��X��6������O�ol�~���r��}�����.RI�zZS�{�����;��������z��4P_��.�H�������-Qb��z �S%��H���S{���W:/��J��������������H}��PN��{?W�1�<:��0"8H�TV+�O�}M�����s�aG���5�7lHV	+Ir����o;[E���������m�/��#���d[d]d�f>~��/*���r��!U{��Y���-
�[�\r�����6'�Z
�L�!t�������A@����y�b)i�%�aBb+!���@�q��ohsj�������m���7��]�dS��K����pJ-5vl�EA�����5�)V;?�����,$��g�I������QF}�*�����V*�^-�lnd�^?���
���=������;��3o���������W�����f�K�R��fBc�f��j����!T@���yg��W�����5��(������I�O~_�9���nP���������jY�R���*I��/B����)���0?�V�e��M�����VZ��Q2�xxvc���XSo�pb2�wL�����F�����Y�B���^~z��%$Hp�V,r��!�f�!���i��Me��E��r�_���7�o=��H�Y�M���=�����P.?�����t�E�Zr�8_����W��1mVo����ae?l&�)��?5�H=W����2��{zA"��V���0ry�����D����N(�r%��Pt�d��mUY����~�c���5<�A�m����Mgt����|��������6��e��)U���������~!7��.��&0.�����{T���!����"���k}��t���1�G�����i+�.}�)����U��nP>�:w��-�=�3��!�;X�G#���F\��Bw3q7���&g*f�����������XP���������<�)���}�ak���H�� YM]G�:�~����~����B_=M=]CMC�:�����@}�@�?�@��s���,���cm����v�?�P��/�l��������	hB��3��B�je����&����"� ��`��W���@�%���h`�*-;����?����u���q�wi
���h���d$�3�k��j���� �_x�n���R���4���{�GV�V�vbvW��t�����������t7ht�)Ul��~�G=�?pF�u����mUd���k���8��S�t��R��r[�/��h�����<�I�'��cY��b�c��6^)}��zi�sP��Y����pm��������Mg^�X�j�uL�X.m�#�Hx[�eXE���~�\���\��d�����7����o�5��]@5��?^����W�U�\��f3'��K�~�����_n/V�/>~��8�%�`��}�	�
7�e7��f�cL�~��a���������)a[�N�>����t���������)e��]H��q;��~e��o����%M������o�I�E��"����?_
��s�@^Sk����uZY�e��B�}km�������M��?��6��s��}��SV������_�$G�_�T4Y�.���6���D���mFk���,v�V������5��,��w�|yw��9�	��G2XO����%��������������������
���8i��������t����'l�!�!~>1t����������C���������]�!��BH��[C��/,+{x�f���P���k�<m�wRb���/��,��j�U��1}������Ys�v��[���
}�T��N��
���_�U����� ����|��/�+��m�j�I�ZS�u�{����=A7U�Y��M�z�j��r*m��3�F����L��ep%�������W
���i�`�K��/���6����3T��}v
���������D^aj�+g(�e��QT������UU����R9�)#�}���){�}��"e#C{��v������	�~F���88������D����_D��c��PPPr��7'gB������x�L7���si#j�I.�+�.�;����'��pZ�xi������	4��%
�����L�L���x���	��a���`YA����gM���\����a���[Wa,����t���'�L+f�g8�v��b���Z�g��=�}��Nc��Od>YQU�^ �XH;U�����o1�����$\�v�'.���������7��/�>��7czw�)+F~�<����%2DxY-��G()*�|��}A��N��}W�H�m��<!H�|ig,�c�G�[_G�m&�6�|v�<?����M���k����@6K����j����f��E����}Z�[�{����/�����^���
;�������R�����;_/�{�H.\�7@��Q�i^��K���������wl^�4s^y�s�:+s�����A��fc��	(����5����{�{�>��k\������f�����������E�$���g��S����WY�k�Z���8�_�����'�fF|��vn��Sk�������[f���4KW�<�L�y��6�_8�q�M�X�����,/�����{-VQ[��.���[[�.�M������om
��
�!�/��Kr;��D�=W�r�S|���-W����C�!!��|'B�hf� `~�$��W�A�	9��L���:��Ba�X �2 $���Xx�?�4;�p�������QT����'xT�~uW(4�����:qo(������"w� �e�-��aC>�l���8��Z�H����Y��a�l����~���I����q�3����9+���q��q�t����~ydu�[T������=<x%|p��b��V��b��s���i^c�~�� rcU���/E.��/f���T��*���z01���E��kn����J��t.)m�m��������J(�^�,s��]��J�Q�k���s��
7v��j|rt���?�q!����}b�g+�^�x��}�k���6#kKu�B�\/)�_�UT
��-\��s}��32J%��|��>V����2w�5;c���&�������7��F�s��/F=�>��y���(��s�����b�^%�j���-��7F%>�z����*��]����)��l���;�y���?��>79������������
A�[}9^��So����6^U�}x��N��L�e��7[�7���1���bs��������tS����zp��c��j�����G=���������������}����h,E��r�3G��1��)(O��Di.B32��9>~]r�qU)]�7���"�"�4���Cx�������/H�KGG��Ls�r����Y���c��BDx:69J���wM�����S ��������m��o��#	��J����
�	�������]e�
@	�6z	�6
��&����S��g����?��}H;��hy����[1A�{�93���7����@�����0n���2z���mq���C�!w�j����H������z��{}������m�]g��>5&h����h�����Y���_�0�$*
�6��I����<��Z�Vj���8������Z�Ej&j���~���G^����u;���,�����{�y�J��A��|��<Q#�0�G�QG��@�f���_l�ZH
-��<?w������Pg������*�4�C���LfG�T�42q�����1���H�+w��x�&��
�%q��j�W;��5��-�M{xgp��dq������v�]�g����o�1>_+/~���b�^������DU�>����\�*�����f�`���
5�+����\����e��~K�������_m��O~�V�~:0\Zi���M,G���A~���XO�Rp����#�����C��@�X�����2����qY-w�b1g��B�;SR���c��W����X��hcYgvyo�'�k�=���&<|�e�h�H"�w{�aO�J�����N�)JnB���1��7/�S92C�^*)M5�2`�v���e��>����kK��!u@���Pn��"5��O�B�����xW�/g���os���X���u�_>�U~u�b�H��,&c ������U���S���_��l��.�������*�rnW��[�9*]B�Z�)�1h�3.�����(��4}�T���?G�t�������<v��	�<��y}�X�|��r����^�[{X�Qu��e��g��Y���C����D�����
�����m�z�Q�[�vU�~�HP2��|A�8����]
���w�K2t�S��^��*�ba�Q��{�&6�WE��j7���!r�����vnX���m���*Q�yhUxy�<�#<����I3���?����M�4�\
IM7I��C�|��<8^���a ���,2N�i6���A��y�9�lg���?l�[�|����o�4+dSp����� ���G�.���5�������CI��~��l�5b���z6���-tB����YX=������U���m��%��5c��3W�I{&=�����zU�"e��CsnY�0��9�}p��}Wk���n���L��ek��mJ�	�e�n���n�&��.*31mv������#3X���%�x�N���S��:=U�!>��	hp����pV0��8w7+Ue���Q�v�u��b�"��]OV�Q�J�|c��_��%�
��:6�g
endstream
endobj
1659 0 obj
[ 250] 
endobj
1660 0 obj
<</Type/Metadata/Subtype/XML/Length 3088>>
stream
<?xpacket begin="���" id="W5M0MpCehiHzreSzNTczkc9d"?><x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="3.1-701">
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""  xmlns:pdf="http://ns.adobe.com/pdf/1.3/">
<pdf:Producer>Microsoft�� Word for Microsoft 365</pdf:Producer></rdf:Description>
<rdf:Description rdf:about=""  xmlns:dc="http://purl.org/dc/elements/1.1/">
<dc:creator><rdf:Seq><rdf:li>Smith, Peter</rdf:li></rdf:Seq></dc:creator></rdf:Description>
<rdf:Description rdf:about=""  xmlns:xmp="http://ns.adobe.com/xap/1.0/">
<xmp:CreatorTool>Microsoft�� Word for Microsoft 365</xmp:CreatorTool><xmp:CreateDate>2021-07-30T15:25:54+10:00</xmp:CreateDate><xmp:ModifyDate>2021-07-30T15:25:54+10:00</xmp:ModifyDate></rdf:Description>
<rdf:Description rdf:about=""  xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/">
<xmpMM:DocumentID>uuid:A3D70E5B-DB7A-4F99-9A77-D3B58180CB72</xmpMM:DocumentID><xmpMM:InstanceID>uuid:A3D70E5B-DB7A-4F99-9A77-D3B58180CB72</xmpMM:InstanceID></rdf:Description>
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    
</rdf:RDF></x:xmpmeta><?xpacket end="w"?>
endstream
endobj
1661 0 obj
<</DisplayDocTitle true>>
endobj
1662 0 obj
<</Type/XRef/Size 1662/W[ 1 4 2] /Root 1 0 R/Info 54 0 R/ID[<5B0ED7A37ADB994F9A77D3B58180CB72><5B0ED7A37ADB994F9A77D3B58180CB72>] /Filter/FlateDecode/Length 3261>>
stream
x�5�u�T������KBQ@T@9���A���]�z������-*v`+*�b��������<����f���3k�Y��������y�R��SSS�I�.4;����B��m`�[h7��~(��'����B���NG:O���.��B�V����}�{�B�U
=w+���
�-,e��m�xa�C�U?/��'��������5���+��|a����Y����=�0mr��
��>����;�9���\�;�v���Z���<V��X	��h���Cj���Q��AK4G��V��5��-����=:�#:�+D7t�<W����)�6��B��^X�b�Ao,�~X[�=�q�%�������;���a `i,���*�����S�W��`����VDuB�`(V�(���Q�E�6~�p���Xc#��
k�w���}����.6���[`��l���6�����5��
���-n�������>�;av�.��aw��=����q���_���q�xL��pN��8��0�#p$���8��8�p"N��8��L��3p6���8��<\�p!��%���2\��q%��u���&��[0���>$��6���5��l�����;p7������>�V��xS��cxO�q<�'�4��Y<��0
30/�y���x/�U����f�
��7���?x��C|��0��c|�O�>�W���k��������~���?�W����
����
����6�#���FL#��Hk45
M��FS��Q�hjT4�1��FS����(l�4���Q��hT4*��FE�Q�hj3����F"#���Hd�5� ���hc�1�m�6F������G;��m�2G#�����6FZ#��Hd�/���6fChc6
��6F�"�Q��FL���
��m |��_�1���E#|��_d0����`�5.���`�/2��F�"���`�<����G�"���`�/2��F�"���`�/2��F"|���ad02��E7����_d0�m��E
����_3j��E#����C����b$2� �Q�hc!���P��a/j�������a/���F���Q��a/j��F���Q��a/j��F���Q��a!����E�����_�/����E��xQ�(^�0�5��E
�xQ�(^�0�5��E
�xQ�(^�0��z
k�(�!h����P��k����C�BK,�����m���	��� ��;��'z`a,�E���(�������%��?��@��X��`,���V�
���
V�0�jXk`u��p��H�����`m��u���F��`cl�M1�c��V��`kl�m���N�;cv�.��aO��}�/������������{{0&b��q&`<���Z!��P��q��8G�X��qN�	8'�T���q��8g�\���q.���R\��p.�U���j\�kq��M���f��I���TS�]���nT���x�c
��x��|�'�8���xS�,��s�����<^�L����*^��x
�����xo�=���>f��B�|���>��_�K|��1��{|��U?�'��_�3~�\����'������������`d0R��(F�"���b�/���(F�"���b�/���(F�"���b�/���(F�"����bd02�F
��Q����3�Q��E#|�_D1�Q��E#|�_\�#|�_D1�Q��E#���`d02,�L
��(F�"���b�/���(F�"���b�/���(F�"���b�/���(F�"���b��j�
Q�DF�xer��xQ�(^��Ln5�0�5��ES�xQ�(^�0�5��E
�xQ�(^�0�5��E
�xQ�(^�0�5�"D�����_�/����E��Q�h/j��F���Q��a/j��F���Q��a/j��F���Q��_�/����E�����_�/����E
�xQ�(^�0�5��E
�xQ�(^�0�5��E
�xQ�(^��j7f��Z�����_�/����E�R�O���Q��a/j��F���Q��a/j��F���V{)U�~��l6)�j?l���cZ�%��h��h�����������X]���=�0�"���X}��������%�4������-�e1+�������<���X�����&C�V�P�aX��z#0k`M��H���0�a]��
���&�asl�M��b��V��a[l��������v�.�{`w��}����q��P
w��@TC���A8c"��8�"���#qL}M��XG�(�������x�qJm�5;669'�T��i8g�L��sq��8��b\�q.�%�W�r\��q��u�7�F��I�7�v��[q'��d���q����~<�G0�a<�'��SxOc*��s�n�T���Y�G7���<^�+�Mnl�2^��x����[5�M��xo�-����.f�C|�O�1>�����+|�/�-�����a~D���~�O�
�b.�����o����b�`�����|F#��2�����g$2���|F"#����gt��z�����`F>���n�CT4*��FE������b1��F>#�qm��FE��Q�2���e�������i�5B[��M�|F>�����i
k��F7����T��g,����F7��Q��ft3
��nFa����(lt3���nF7�����f6���FE�����f45
�nF)�����h�2���nFZS��hu>u3�k�ht��"���i�4�1����6v���m45
W�Hk�k(l�6EE���f4�L�5����5��L�5T4*���FS#����g�3���|F>#����g�3���|F>#����g�3���|F>#����g�3���|F>#����g�3���|FL#��Hd�3���`F0#���f3b1�`FE�X[���-Nf�����%5�����M�ll2��,�a�����f��jj�3���|F>#����g�3���|F>#����g�3���|F>#����g�3���|V#bk/:�NO}`�iS���h�fh���m��c�F'tD;�Gt�������=P}�s!��"�E�}���������n��%�8�cI�,�A���e�,�]��1�[����(�zva���=���|�����O\Ux���T�6�����{�-�N�9�qs���oC��
endstream
endobj
xref
0 1663
0000000055 65535 f
0000000017 00000 n
0000000170 00000 n
0000000304 00000 n
0000000643 00000 n
0000002701 00000 n
0000002871 00000 n
0000003112 00000 n
0000003165 00000 n
0000003218 00000 n
0000003394 00000 n
0000003641 00000 n
0000003781 00000 n
0000003811 00000 n
0000003979 00000 n
0000004053 00000 n
0000004301 00000 n
0000004479 00000 n
0000004727 00000 n
0000004861 00000 n
0000004891 00000 n
0000005053 00000 n
0000005127 00000 n
0000005369 00000 n
0000005646 00000 n
0000005923 00000 n
0000006200 00000 n
0000006500 00000 n
0000010068 00000 n
0000010232 00000 n
0000010459 00000 n
0000010759 00000 n
0000014604 00000 n
0000014783 00000 n
0000015034 00000 n
0000015334 00000 n
0000028185 00000 n
0000028359 00000 n
0000028596 00000 n
0000028896 00000 n
0000041417 00000 n
0000041727 00000 n
0000044491 00000 n
0000044781 00000 n
0000047574 00000 n
0000047885 00000 n
0000050817 00000 n
0000051118 00000 n
0000054494 00000 n
0000054795 00000 n
0000057763 00000 n
0000058074 00000 n
0000060901 00000 n
0000061202 00000 n
0000063017 00000 n
0000000056 65535 f
0000000057 65535 f
0000000058 65535 f
0000000059 65535 f
0000000060 65535 f
0000000061 65535 f
0000000062 65535 f
0000000063 65535 f
0000000064 65535 f
0000000065 65535 f
0000000066 65535 f
0000000067 65535 f
0000000068 65535 f
0000000069 65535 f
0000000070 65535 f
0000000071 65535 f
0000000072 65535 f
0000000073 65535 f
0000000074 65535 f
0000000075 65535 f
0000000076 65535 f
0000000077 65535 f
0000000078 65535 f
0000000079 65535 f
0000000080 65535 f
0000000081 65535 f
0000000082 65535 f
0000000083 65535 f
0000000084 65535 f
0000000085 65535 f
0000000086 65535 f
0000000087 65535 f
0000000088 65535 f
0000000090 65535 f
0000069409 00000 n
0000000091 65535 f
0000000092 65535 f
0000000093 65535 f
0000000094 65535 f
0000000095 65535 f
0000000096 65535 f
0000000097 65535 f
0000000098 65535 f
0000000099 65535 f
0000000100 65535 f
0000000101 65535 f
0000000103 65535 f
0000069459 00000 n
0000000104 65535 f
0000000105 65535 f
0000000106 65535 f
0000000107 65535 f
0000000108 65535 f
0000000110 65535 f
0000069510 00000 n
0000000111 65535 f
0000000112 65535 f
0000000113 65535 f
0000000114 65535 f
0000000115 65535 f
0000000116 65535 f
0000000117 65535 f
0000000119 65535 f
0000069561 00000 n
0000000120 65535 f
0000000121 65535 f
0000000122 65535 f
0000000123 65535 f
0000000124 65535 f
0000000125 65535 f
0000000126 65535 f
0000000127 65535 f
0000000128 65535 f
0000000129 65535 f
0000000130 65535 f
0000000131 65535 f
0000000132 65535 f
0000000134 65535 f
0000069612 00000 n
0000000135 65535 f
0000000136 65535 f
0000000137 65535 f
0000000138 65535 f
0000000139 65535 f
0000000140 65535 f
0000000141 65535 f
0000000143 65535 f
0000069663 00000 n
0000000144 65535 f
0000000145 65535 f
0000000146 65535 f
0000000147 65535 f
0000000148 65535 f
0000000149 65535 f
0000000150 65535 f
0000000151 65535 f
0000000152 65535 f
0000000154 65535 f
0000069714 00000 n
0000000155 65535 f
0000000156 65535 f
0000000157 65535 f
0000000158 65535 f
0000000159 65535 f
0000000160 65535 f
0000000161 65535 f
0000000162 65535 f
0000000163 65535 f
0000000164 65535 f
0000000165 65535 f
0000000166 65535 f
0000000167 65535 f
0000000168 65535 f
0000000169 65535 f
0000000170 65535 f
0000000171 65535 f
0000000172 65535 f
0000000173 65535 f
0000000174 65535 f
0000000175 65535 f
0000000176 65535 f
0000000177 65535 f
0000000178 65535 f
0000000179 65535 f
0000000180 65535 f
0000000181 65535 f
0000000182 65535 f
0000000183 65535 f
0000000184 65535 f
0000000185 65535 f
0000000186 65535 f
0000000187 65535 f
0000000188 65535 f
0000000189 65535 f
0000000190 65535 f
0000000191 65535 f
0000000192 65535 f
0000000193 65535 f
0000000194 65535 f
0000000195 65535 f
0000000196 65535 f
0000000197 65535 f
0000000198 65535 f
0000000199 65535 f
0000000200 65535 f
0000000201 65535 f
0000000202 65535 f
0000000203 65535 f
0000000204 65535 f
0000000205 65535 f
0000000206 65535 f
0000000207 65535 f
0000000208 65535 f
0000000209 65535 f
0000000210 65535 f
0000000211 65535 f
0000000212 65535 f
0000000213 65535 f
0000000214 65535 f
0000000215 65535 f
0000000216 65535 f
0000000217 65535 f
0000000218 65535 f
0000000219 65535 f
0000000220 65535 f
0000000221 65535 f
0000000222 65535 f
0000000223 65535 f
0000000224 65535 f
0000000225 65535 f
0000000226 65535 f
0000000227 65535 f
0000000228 65535 f
0000000229 65535 f
0000000230 65535 f
0000000231 65535 f
0000000232 65535 f
0000000233 65535 f
0000000234 65535 f
0000000235 65535 f
0000000236 65535 f
0000000237 65535 f
0000000238 65535 f
0000000239 65535 f
0000000240 65535 f
0000000241 65535 f
0000000242 65535 f
0000000243 65535 f
0000000244 65535 f
0000000245 65535 f
0000000246 65535 f
0000000247 65535 f
0000000248 65535 f
0000000249 65535 f
0000000250 65535 f
0000000251 65535 f
0000000252 65535 f
0000000253 65535 f
0000000254 65535 f
0000000255 65535 f
0000000256 65535 f
0000000257 65535 f
0000000258 65535 f
0000000259 65535 f
0000000260 65535 f
0000000261 65535 f
0000000262 65535 f
0000000263 65535 f
0000000264 65535 f
0000000265 65535 f
0000000266 65535 f
0000000267 65535 f
0000000268 65535 f
0000000269 65535 f
0000000270 65535 f
0000000271 65535 f
0000000272 65535 f
0000000273 65535 f
0000000274 65535 f
0000000275 65535 f
0000000276 65535 f
0000000277 65535 f
0000000278 65535 f
0000000279 65535 f
0000000280 65535 f
0000000281 65535 f
0000000282 65535 f
0000000283 65535 f
0000000284 65535 f
0000000285 65535 f
0000000286 65535 f
0000000287 65535 f
0000000288 65535 f
0000000289 65535 f
0000000290 65535 f
0000000291 65535 f
0000000292 65535 f
0000000293 65535 f
0000000294 65535 f
0000000295 65535 f
0000000296 65535 f
0000000297 65535 f
0000000298 65535 f
0000000299 65535 f
0000000300 65535 f
0000000301 65535 f
0000000302 65535 f
0000000303 65535 f
0000000304 65535 f
0000000305 65535 f
0000000306 65535 f
0000000307 65535 f
0000000308 65535 f
0000000309 65535 f
0000000310 65535 f
0000000311 65535 f
0000000312 65535 f
0000000313 65535 f
0000000314 65535 f
0000000315 65535 f
0000000316 65535 f
0000000317 65535 f
0000000318 65535 f
0000000319 65535 f
0000000320 65535 f
0000000321 65535 f
0000000322 65535 f
0000000323 65535 f
0000000324 65535 f
0000000325 65535 f
0000000326 65535 f
0000000327 65535 f
0000000328 65535 f
0000000329 65535 f
0000000330 65535 f
0000000331 65535 f
0000000332 65535 f
0000000333 65535 f
0000000334 65535 f
0000000335 65535 f
0000000336 65535 f
0000000337 65535 f
0000000338 65535 f
0000000339 65535 f
0000000340 65535 f
0000000341 65535 f
0000000342 65535 f
0000000343 65535 f
0000000344 65535 f
0000000345 65535 f
0000000346 65535 f
0000000347 65535 f
0000000348 65535 f
0000000349 65535 f
0000000350 65535 f
0000000351 65535 f
0000000352 65535 f
0000000353 65535 f
0000000354 65535 f
0000000355 65535 f
0000000356 65535 f
0000000357 65535 f
0000000358 65535 f
0000000359 65535 f
0000000360 65535 f
0000000361 65535 f
0000000362 65535 f
0000000363 65535 f
0000000364 65535 f
0000000365 65535 f
0000000366 65535 f
0000000367 65535 f
0000000368 65535 f
0000000369 65535 f
0000000370 65535 f
0000000371 65535 f
0000000372 65535 f
0000000373 65535 f
0000000374 65535 f
0000000375 65535 f
0000000376 65535 f
0000000377 65535 f
0000000378 65535 f
0000000379 65535 f
0000000380 65535 f
0000000381 65535 f
0000000382 65535 f
0000000383 65535 f
0000000384 65535 f
0000000385 65535 f
0000000386 65535 f
0000000387 65535 f
0000000388 65535 f
0000000389 65535 f
0000000390 65535 f
0000000391 65535 f
0000000392 65535 f
0000000393 65535 f
0000000394 65535 f
0000000395 65535 f
0000000396 65535 f
0000000397 65535 f
0000000398 65535 f
0000000399 65535 f
0000000400 65535 f
0000000401 65535 f
0000000402 65535 f
0000000403 65535 f
0000000404 65535 f
0000000405 65535 f
0000000406 65535 f
0000000407 65535 f
0000000408 65535 f
0000000409 65535 f
0000000410 65535 f
0000000411 65535 f
0000000412 65535 f
0000000413 65535 f
0000000414 65535 f
0000000415 65535 f
0000000416 65535 f
0000000417 65535 f
0000000418 65535 f
0000000419 65535 f
0000000420 65535 f
0000000421 65535 f
0000000422 65535 f
0000000423 65535 f
0000000424 65535 f
0000000425 65535 f
0000000426 65535 f
0000000427 65535 f
0000000428 65535 f
0000000429 65535 f
0000000430 65535 f
0000000431 65535 f
0000000432 65535 f
0000000433 65535 f
0000000434 65535 f
0000000435 65535 f
0000000436 65535 f
0000000437 65535 f
0000000438 65535 f
0000000439 65535 f
0000000440 65535 f
0000000441 65535 f
0000000442 65535 f
0000000443 65535 f
0000000444 65535 f
0000000445 65535 f
0000000446 65535 f
0000000447 65535 f
0000000448 65535 f
0000000449 65535 f
0000000450 65535 f
0000000451 65535 f
0000000452 65535 f
0000000453 65535 f
0000000454 65535 f
0000000455 65535 f
0000000456 65535 f
0000000457 65535 f
0000000458 65535 f
0000000459 65535 f
0000000460 65535 f
0000000461 65535 f
0000000462 65535 f
0000000463 65535 f
0000000464 65535 f
0000000465 65535 f
0000000466 65535 f
0000000467 65535 f
0000000468 65535 f
0000000469 65535 f
0000000470 65535 f
0000000471 65535 f
0000000472 65535 f
0000000473 65535 f
0000000474 65535 f
0000000475 65535 f
0000000476 65535 f
0000000477 65535 f
0000000478 65535 f
0000000479 65535 f
0000000480 65535 f
0000000481 65535 f
0000000482 65535 f
0000000483 65535 f
0000000484 65535 f
0000000485 65535 f
0000000486 65535 f
0000000487 65535 f
0000000488 65535 f
0000000489 65535 f
0000000490 65535 f
0000000491 65535 f
0000000492 65535 f
0000000493 65535 f
0000000494 65535 f
0000000495 65535 f
0000000496 65535 f
0000000497 65535 f
0000000498 65535 f
0000000499 65535 f
0000000500 65535 f
0000000501 65535 f
0000000502 65535 f
0000000503 65535 f
0000000504 65535 f
0000000505 65535 f
0000000506 65535 f
0000000507 65535 f
0000000508 65535 f
0000000509 65535 f
0000000510 65535 f
0000000511 65535 f
0000000512 65535 f
0000000513 65535 f
0000000514 65535 f
0000000515 65535 f
0000000516 65535 f
0000000517 65535 f
0000000518 65535 f
0000000519 65535 f
0000000520 65535 f
0000000521 65535 f
0000000522 65535 f
0000000523 65535 f
0000000524 65535 f
0000000525 65535 f
0000000526 65535 f
0000000527 65535 f
0000000528 65535 f
0000000529 65535 f
0000000530 65535 f
0000000531 65535 f
0000000532 65535 f
0000000533 65535 f
0000000534 65535 f
0000000535 65535 f
0000000536 65535 f
0000000537 65535 f
0000000538 65535 f
0000000539 65535 f
0000000540 65535 f
0000000541 65535 f
0000000542 65535 f
0000000543 65535 f
0000000544 65535 f
0000000545 65535 f
0000000546 65535 f
0000000547 65535 f
0000000548 65535 f
0000000549 65535 f
0000000550 65535 f
0000000551 65535 f
0000000552 65535 f
0000000553 65535 f
0000000554 65535 f
0000000555 65535 f
0000000556 65535 f
0000000557 65535 f
0000000558 65535 f
0000000559 65535 f
0000000560 65535 f
0000000561 65535 f
0000000562 65535 f
0000000563 65535 f
0000000564 65535 f
0000000565 65535 f
0000000566 65535 f
0000000567 65535 f
0000000568 65535 f
0000000569 65535 f
0000000570 65535 f
0000000571 65535 f
0000000572 65535 f
0000000573 65535 f
0000000574 65535 f
0000000575 65535 f
0000000576 65535 f
0000000577 65535 f
0000000578 65535 f
0000000579 65535 f
0000000580 65535 f
0000000581 65535 f
0000000582 65535 f
0000000583 65535 f
0000000584 65535 f
0000000585 65535 f
0000000586 65535 f
0000000587 65535 f
0000000588 65535 f
0000000589 65535 f
0000000590 65535 f
0000000591 65535 f
0000000592 65535 f
0000000593 65535 f
0000000594 65535 f
0000000595 65535 f
0000000596 65535 f
0000000597 65535 f
0000000598 65535 f
0000000599 65535 f
0000000600 65535 f
0000000601 65535 f
0000000602 65535 f
0000000603 65535 f
0000000604 65535 f
0000000605 65535 f
0000000606 65535 f
0000000607 65535 f
0000000608 65535 f
0000000609 65535 f
0000000610 65535 f
0000000611 65535 f
0000000612 65535 f
0000000613 65535 f
0000000614 65535 f
0000000615 65535 f
0000000616 65535 f
0000000617 65535 f
0000000618 65535 f
0000000619 65535 f
0000000620 65535 f
0000000621 65535 f
0000000622 65535 f
0000000623 65535 f
0000000624 65535 f
0000000625 65535 f
0000000626 65535 f
0000000627 65535 f
0000000628 65535 f
0000000629 65535 f
0000000630 65535 f
0000000631 65535 f
0000000632 65535 f
0000000633 65535 f
0000000634 65535 f
0000000635 65535 f
0000000636 65535 f
0000000637 65535 f
0000000638 65535 f
0000000639 65535 f
0000000640 65535 f
0000000641 65535 f
0000000642 65535 f
0000000643 65535 f
0000000644 65535 f
0000000645 65535 f
0000000646 65535 f
0000000647 65535 f
0000000648 65535 f
0000000649 65535 f
0000000650 65535 f
0000000651 65535 f
0000000652 65535 f
0000000653 65535 f
0000000654 65535 f
0000000655 65535 f
0000000656 65535 f
0000000657 65535 f
0000000658 65535 f
0000000659 65535 f
0000000660 65535 f
0000000661 65535 f
0000000662 65535 f
0000000663 65535 f
0000000664 65535 f
0000000665 65535 f
0000000666 65535 f
0000000667 65535 f
0000000668 65535 f
0000000669 65535 f
0000000670 65535 f
0000000671 65535 f
0000000672 65535 f
0000000673 65535 f
0000000674 65535 f
0000000675 65535 f
0000000676 65535 f
0000000677 65535 f
0000000678 65535 f
0000000679 65535 f
0000000680 65535 f
0000000681 65535 f
0000000682 65535 f
0000000683 65535 f
0000000684 65535 f
0000000685 65535 f
0000000686 65535 f
0000000687 65535 f
0000000688 65535 f
0000000689 65535 f
0000000690 65535 f
0000000691 65535 f
0000000692 65535 f
0000000693 65535 f
0000000694 65535 f
0000000695 65535 f
0000000696 65535 f
0000000697 65535 f
0000000698 65535 f
0000000699 65535 f
0000000700 65535 f
0000000701 65535 f
0000000702 65535 f
0000000703 65535 f
0000000704 65535 f
0000000705 65535 f
0000000706 65535 f
0000000707 65535 f
0000000708 65535 f
0000000709 65535 f
0000000710 65535 f
0000000711 65535 f
0000000712 65535 f
0000000713 65535 f
0000000714 65535 f
0000000715 65535 f
0000000716 65535 f
0000000717 65535 f
0000000718 65535 f
0000000719 65535 f
0000000720 65535 f
0000000721 65535 f
0000000722 65535 f
0000000723 65535 f
0000000724 65535 f
0000000725 65535 f
0000000726 65535 f
0000000727 65535 f
0000000728 65535 f
0000000729 65535 f
0000000730 65535 f
0000000731 65535 f
0000000732 65535 f
0000000733 65535 f
0000000734 65535 f
0000000735 65535 f
0000000736 65535 f
0000000737 65535 f
0000000738 65535 f
0000000739 65535 f
0000000740 65535 f
0000000741 65535 f
0000000742 65535 f
0000000743 65535 f
0000000744 65535 f
0000000745 65535 f
0000000746 65535 f
0000000747 65535 f
0000000748 65535 f
0000000749 65535 f
0000000750 65535 f
0000000751 65535 f
0000000752 65535 f
0000000753 65535 f
0000000754 65535 f
0000000755 65535 f
0000000756 65535 f
0000000757 65535 f
0000000758 65535 f
0000000759 65535 f
0000000760 65535 f
0000000761 65535 f
0000000762 65535 f
0000000763 65535 f
0000000764 65535 f
0000000765 65535 f
0000000766 65535 f
0000000767 65535 f
0000000768 65535 f
0000000769 65535 f
0000000770 65535 f
0000000771 65535 f
0000000772 65535 f
0000000773 65535 f
0000000774 65535 f
0000000775 65535 f
0000000776 65535 f
0000000777 65535 f
0000000778 65535 f
0000000779 65535 f
0000000780 65535 f
0000000781 65535 f
0000000782 65535 f
0000000783 65535 f
0000000784 65535 f
0000000785 65535 f
0000000786 65535 f
0000000787 65535 f
0000000788 65535 f
0000000789 65535 f
0000000790 65535 f
0000000791 65535 f
0000000792 65535 f
0000000793 65535 f
0000000794 65535 f
0000000795 65535 f
0000000796 65535 f
0000000797 65535 f
0000000798 65535 f
0000000799 65535 f
0000000800 65535 f
0000000801 65535 f
0000000802 65535 f
0000000803 65535 f
0000000804 65535 f
0000000805 65535 f
0000000806 65535 f
0000000807 65535 f
0000000808 65535 f
0000000809 65535 f
0000000810 65535 f
0000000811 65535 f
0000000812 65535 f
0000000813 65535 f
0000000814 65535 f
0000000815 65535 f
0000000816 65535 f
0000000817 65535 f
0000000818 65535 f
0000000819 65535 f
0000000820 65535 f
0000000821 65535 f
0000000822 65535 f
0000000823 65535 f
0000000824 65535 f
0000000825 65535 f
0000000826 65535 f
0000000827 65535 f
0000000828 65535 f
0000000829 65535 f
0000000830 65535 f
0000000831 65535 f
0000000832 65535 f
0000000833 65535 f
0000000834 65535 f
0000000835 65535 f
0000000836 65535 f
0000000837 65535 f
0000000838 65535 f
0000000839 65535 f
0000000840 65535 f
0000000841 65535 f
0000000842 65535 f
0000000843 65535 f
0000000844 65535 f
0000000845 65535 f
0000000846 65535 f
0000000847 65535 f
0000000848 65535 f
0000000849 65535 f
0000000850 65535 f
0000000851 65535 f
0000000852 65535 f
0000000853 65535 f
0000000854 65535 f
0000000855 65535 f
0000000856 65535 f
0000000857 65535 f
0000000858 65535 f
0000000859 65535 f
0000000860 65535 f
0000000861 65535 f
0000000862 65535 f
0000000863 65535 f
0000000864 65535 f
0000000865 65535 f
0000000866 65535 f
0000000867 65535 f
0000000868 65535 f
0000000869 65535 f
0000000870 65535 f
0000000871 65535 f
0000000872 65535 f
0000000873 65535 f
0000000874 65535 f
0000000875 65535 f
0000000876 65535 f
0000000877 65535 f
0000000878 65535 f
0000000879 65535 f
0000000880 65535 f
0000000881 65535 f
0000000882 65535 f
0000000883 65535 f
0000000884 65535 f
0000000885 65535 f
0000000886 65535 f
0000000887 65535 f
0000000888 65535 f
0000000889 65535 f
0000000890 65535 f
0000000891 65535 f
0000000892 65535 f
0000000893 65535 f
0000000894 65535 f
0000000895 65535 f
0000000896 65535 f
0000000897 65535 f
0000000898 65535 f
0000000899 65535 f
0000000900 65535 f
0000000901 65535 f
0000000902 65535 f
0000000903 65535 f
0000000904 65535 f
0000000905 65535 f
0000000906 65535 f
0000000907 65535 f
0000000908 65535 f
0000000909 65535 f
0000000910 65535 f
0000000911 65535 f
0000000912 65535 f
0000000913 65535 f
0000000914 65535 f
0000000915 65535 f
0000000916 65535 f
0000000917 65535 f
0000000918 65535 f
0000000919 65535 f
0000000920 65535 f
0000000921 65535 f
0000000922 65535 f
0000000923 65535 f
0000000924 65535 f
0000000925 65535 f
0000000926 65535 f
0000000927 65535 f
0000000928 65535 f
0000000929 65535 f
0000000930 65535 f
0000000931 65535 f
0000000932 65535 f
0000000933 65535 f
0000000934 65535 f
0000000935 65535 f
0000000936 65535 f
0000000937 65535 f
0000000938 65535 f
0000000939 65535 f
0000000940 65535 f
0000000941 65535 f
0000000942 65535 f
0000000943 65535 f
0000000944 65535 f
0000000945 65535 f
0000000946 65535 f
0000000947 65535 f
0000000948 65535 f
0000000949 65535 f
0000000950 65535 f
0000000951 65535 f
0000000952 65535 f
0000000953 65535 f
0000000954 65535 f
0000000955 65535 f
0000000956 65535 f
0000000957 65535 f
0000000958 65535 f
0000000959 65535 f
0000000960 65535 f
0000000961 65535 f
0000000962 65535 f
0000000963 65535 f
0000000964 65535 f
0000000965 65535 f
0000000966 65535 f
0000000967 65535 f
0000000968 65535 f
0000000969 65535 f
0000000970 65535 f
0000000971 65535 f
0000000972 65535 f
0000000973 65535 f
0000000974 65535 f
0000000975 65535 f
0000000976 65535 f
0000000977 65535 f
0000000978 65535 f
0000000979 65535 f
0000000980 65535 f
0000000981 65535 f
0000000982 65535 f
0000000983 65535 f
0000000984 65535 f
0000000985 65535 f
0000000986 65535 f
0000000987 65535 f
0000000988 65535 f
0000000989 65535 f
0000000990 65535 f
0000000991 65535 f
0000000992 65535 f
0000000993 65535 f
0000000994 65535 f
0000000995 65535 f
0000000996 65535 f
0000000997 65535 f
0000000998 65535 f
0000000999 65535 f
0000001000 65535 f
0000001001 65535 f
0000001002 65535 f
0000001003 65535 f
0000001004 65535 f
0000001005 65535 f
0000001006 65535 f
0000001007 65535 f
0000001008 65535 f
0000001009 65535 f
0000001010 65535 f
0000001011 65535 f
0000001012 65535 f
0000001013 65535 f
0000001014 65535 f
0000001015 65535 f
0000001016 65535 f
0000001017 65535 f
0000001018 65535 f
0000001019 65535 f
0000001020 65535 f
0000001021 65535 f
0000001022 65535 f
0000001023 65535 f
0000001024 65535 f
0000001025 65535 f
0000001026 65535 f
0000001027 65535 f
0000001028 65535 f
0000001029 65535 f
0000001030 65535 f
0000001031 65535 f
0000001032 65535 f
0000001033 65535 f
0000001034 65535 f
0000001035 65535 f
0000001036 65535 f
0000001037 65535 f
0000001038 65535 f
0000001039 65535 f
0000001040 65535 f
0000001041 65535 f
0000001042 65535 f
0000001043 65535 f
0000001044 65535 f
0000001045 65535 f
0000001046 65535 f
0000001047 65535 f
0000001048 65535 f
0000001049 65535 f
0000001050 65535 f
0000001051 65535 f
0000001052 65535 f
0000001053 65535 f
0000001054 65535 f
0000001055 65535 f
0000001056 65535 f
0000001057 65535 f
0000001058 65535 f
0000001059 65535 f
0000001060 65535 f
0000001061 65535 f
0000001062 65535 f
0000001063 65535 f
0000001064 65535 f
0000001065 65535 f
0000001066 65535 f
0000001067 65535 f
0000001068 65535 f
0000001069 65535 f
0000001070 65535 f
0000001071 65535 f
0000001072 65535 f
0000001073 65535 f
0000001074 65535 f
0000001075 65535 f
0000001076 65535 f
0000001077 65535 f
0000001078 65535 f
0000001079 65535 f
0000001080 65535 f
0000001081 65535 f
0000001082 65535 f
0000001083 65535 f
0000001084 65535 f
0000001085 65535 f
0000001086 65535 f
0000001087 65535 f
0000001088 65535 f
0000001089 65535 f
0000001090 65535 f
0000001091 65535 f
0000001092 65535 f
0000001093 65535 f
0000001094 65535 f
0000001095 65535 f
0000001096 65535 f
0000001097 65535 f
0000001098 65535 f
0000001099 65535 f
0000001100 65535 f
0000001101 65535 f
0000001102 65535 f
0000001103 65535 f
0000001104 65535 f
0000001105 65535 f
0000001106 65535 f
0000001107 65535 f
0000001108 65535 f
0000001109 65535 f
0000001110 65535 f
0000001111 65535 f
0000001112 65535 f
0000001113 65535 f
0000001114 65535 f
0000001115 65535 f
0000001116 65535 f
0000001117 65535 f
0000001118 65535 f
0000001119 65535 f
0000001120 65535 f
0000001121 65535 f
0000001122 65535 f
0000001123 65535 f
0000001124 65535 f
0000001125 65535 f
0000001127 65535 f
0000082496 00000 n
0000001128 65535 f
0000001129 65535 f
0000001130 65535 f
0000001132 65535 f
0000082548 00000 n
0000001133 65535 f
0000001134 65535 f
0000001135 65535 f
0000001136 65535 f
0000001137 65535 f
0000001138 65535 f
0000001139 65535 f
0000001140 65535 f
0000001141 65535 f
0000001142 65535 f
0000001143 65535 f
0000001144 65535 f
0000001145 65535 f
0000001146 65535 f
0000001147 65535 f
0000001148 65535 f
0000001149 65535 f
0000001150 65535 f
0000001151 65535 f
0000001152 65535 f
0000001153 65535 f
0000001154 65535 f
0000001155 65535 f
0000001156 65535 f
0000001157 65535 f
0000001158 65535 f
0000001159 65535 f
0000001160 65535 f
0000001161 65535 f
0000001162 65535 f
0000001163 65535 f
0000001164 65535 f
0000001165 65535 f
0000001166 65535 f
0000001167 65535 f
0000001168 65535 f
0000001169 65535 f
0000001170 65535 f
0000001171 65535 f
0000001172 65535 f
0000001173 65535 f
0000001174 65535 f
0000001175 65535 f
0000001176 65535 f
0000001177 65535 f
0000001178 65535 f
0000001179 65535 f
0000001180 65535 f
0000001181 65535 f
0000001182 65535 f
0000001183 65535 f
0000001184 65535 f
0000001185 65535 f
0000001186 65535 f
0000001187 65535 f
0000001188 65535 f
0000001189 65535 f
0000001190 65535 f
0000001191 65535 f
0000001192 65535 f
0000001193 65535 f
0000001194 65535 f
0000001195 65535 f
0000001196 65535 f
0000001197 65535 f
0000001198 65535 f
0000001199 65535 f
0000001200 65535 f
0000001201 65535 f
0000001202 65535 f
0000001203 65535 f
0000001204 65535 f
0000001205 65535 f
0000001206 65535 f
0000001208 65535 f
0000082600 00000 n
0000001209 65535 f
0000001210 65535 f
0000001211 65535 f
0000001213 65535 f
0000082652 00000 n
0000001214 65535 f
0000001215 65535 f
0000001216 65535 f
0000001218 65535 f
0000082704 00000 n
0000001219 65535 f
0000001220 65535 f
0000001221 65535 f
0000001222 65535 f
0000001223 65535 f
0000001224 65535 f
0000001225 65535 f
0000001226 65535 f
0000001227 65535 f
0000001228 65535 f
0000001229 65535 f
0000001230 65535 f
0000001231 65535 f
0000001232 65535 f
0000001233 65535 f
0000001234 65535 f
0000001235 65535 f
0000001236 65535 f
0000001237 65535 f
0000001238 65535 f
0000001239 65535 f
0000001240 65535 f
0000001241 65535 f
0000001242 65535 f
0000001243 65535 f
0000001244 65535 f
0000001245 65535 f
0000001246 65535 f
0000001247 65535 f
0000001248 65535 f
0000001249 65535 f
0000001250 65535 f
0000001251 65535 f
0000001252 65535 f
0000001253 65535 f
0000001254 65535 f
0000001255 65535 f
0000001256 65535 f
0000001257 65535 f
0000001258 65535 f
0000001259 65535 f
0000001260 65535 f
0000001261 65535 f
0000001262 65535 f
0000001263 65535 f
0000001264 65535 f
0000001265 65535 f
0000001266 65535 f
0000001267 65535 f
0000001268 65535 f
0000001269 65535 f
0000001270 65535 f
0000001271 65535 f
0000001272 65535 f
0000001273 65535 f
0000001274 65535 f
0000001275 65535 f
0000001276 65535 f
0000001277 65535 f
0000001278 65535 f
0000001279 65535 f
0000001280 65535 f
0000001281 65535 f
0000001282 65535 f
0000001283 65535 f
0000001284 65535 f
0000001285 65535 f
0000001286 65535 f
0000001288 65535 f
0000082756 00000 n
0000001289 65535 f
0000001290 65535 f
0000001291 65535 f
0000001293 65535 f
0000082808 00000 n
0000001294 65535 f
0000001295 65535 f
0000001296 65535 f
0000001297 65535 f
0000001298 65535 f
0000001299 65535 f
0000001300 65535 f
0000001301 65535 f
0000001302 65535 f
0000001303 65535 f
0000001304 65535 f
0000001305 65535 f
0000001306 65535 f
0000001307 65535 f
0000001308 65535 f
0000001309 65535 f
0000001310 65535 f
0000001311 65535 f
0000001312 65535 f
0000001313 65535 f
0000001314 65535 f
0000001315 65535 f
0000001316 65535 f
0000001317 65535 f
0000001318 65535 f
0000001319 65535 f
0000001320 65535 f
0000001321 65535 f
0000001322 65535 f
0000001323 65535 f
0000001324 65535 f
0000001325 65535 f
0000001326 65535 f
0000001327 65535 f
0000001328 65535 f
0000001329 65535 f
0000001330 65535 f
0000001331 65535 f
0000001332 65535 f
0000001333 65535 f
0000001334 65535 f
0000001335 65535 f
0000001336 65535 f
0000001337 65535 f
0000001338 65535 f
0000001339 65535 f
0000001340 65535 f
0000001341 65535 f
0000001342 65535 f
0000001343 65535 f
0000001344 65535 f
0000001345 65535 f
0000001346 65535 f
0000001347 65535 f
0000001348 65535 f
0000001349 65535 f
0000001350 65535 f
0000001351 65535 f
0000001352 65535 f
0000001353 65535 f
0000001354 65535 f
0000001355 65535 f
0000001356 65535 f
0000001357 65535 f
0000001358 65535 f
0000001359 65535 f
0000001360 65535 f
0000001361 65535 f
0000001362 65535 f
0000001363 65535 f
0000001364 65535 f
0000001365 65535 f
0000001366 65535 f
0000001368 65535 f
0000082860 00000 n
0000001369 65535 f
0000001370 65535 f
0000001371 65535 f
0000001372 65535 f
0000001373 65535 f
0000001375 65535 f
0000082912 00000 n
0000001376 65535 f
0000001377 65535 f
0000001378 65535 f
0000001379 65535 f
0000001380 65535 f
0000001381 65535 f
0000001382 65535 f
0000001383 65535 f
0000001384 65535 f
0000001385 65535 f
0000001386 65535 f
0000001387 65535 f
0000001388 65535 f
0000001389 65535 f
0000001390 65535 f
0000001391 65535 f
0000001392 65535 f
0000001393 65535 f
0000001394 65535 f
0000001395 65535 f
0000001396 65535 f
0000001397 65535 f
0000001398 65535 f
0000001399 65535 f
0000001400 65535 f
0000001401 65535 f
0000001402 65535 f
0000001403 65535 f
0000001404 65535 f
0000001405 65535 f
0000001406 65535 f
0000001407 65535 f
0000001408 65535 f
0000001409 65535 f
0000001410 65535 f
0000001411 65535 f
0000001412 65535 f
0000001413 65535 f
0000001414 65535 f
0000001415 65535 f
0000001416 65535 f
0000001417 65535 f
0000001418 65535 f
0000001419 65535 f
0000001420 65535 f
0000001421 65535 f
0000001422 65535 f
0000001423 65535 f
0000001424 65535 f
0000001425 65535 f
0000001426 65535 f
0000001427 65535 f
0000001428 65535 f
0000001429 65535 f
0000001430 65535 f
0000001431 65535 f
0000001432 65535 f
0000001433 65535 f
0000001434 65535 f
0000001435 65535 f
0000001436 65535 f
0000001437 65535 f
0000001438 65535 f
0000001439 65535 f
0000001440 65535 f
0000001441 65535 f
0000001442 65535 f
0000001443 65535 f
0000001444 65535 f
0000001445 65535 f
0000001446 65535 f
0000001447 65535 f
0000001448 65535 f
0000001449 65535 f
0000001450 65535 f
0000001451 65535 f
0000001452 65535 f
0000001453 65535 f
0000001454 65535 f
0000001455 65535 f
0000001456 65535 f
0000001458 65535 f
0000082964 00000 n
0000001459 65535 f
0000001460 65535 f
0000001461 65535 f
0000001462 65535 f
0000001463 65535 f
0000001465 65535 f
0000083016 00000 n
0000001466 65535 f
0000001467 65535 f
0000001468 65535 f
0000001469 65535 f
0000001470 65535 f
0000001471 65535 f
0000001472 65535 f
0000001473 65535 f
0000001474 65535 f
0000001475 65535 f
0000001476 65535 f
0000001477 65535 f
0000001478 65535 f
0000001479 65535 f
0000001480 65535 f
0000001481 65535 f
0000001482 65535 f
0000001483 65535 f
0000001484 65535 f
0000001485 65535 f
0000001486 65535 f
0000001487 65535 f
0000001488 65535 f
0000001489 65535 f
0000001490 65535 f
0000001491 65535 f
0000001492 65535 f
0000001493 65535 f
0000001494 65535 f
0000001495 65535 f
0000001496 65535 f
0000001497 65535 f
0000001498 65535 f
0000001499 65535 f
0000001500 65535 f
0000001501 65535 f
0000001502 65535 f
0000001503 65535 f
0000001504 65535 f
0000001505 65535 f
0000001506 65535 f
0000001507 65535 f
0000001508 65535 f
0000001509 65535 f
0000001510 65535 f
0000001511 65535 f
0000001512 65535 f
0000001513 65535 f
0000001514 65535 f
0000001515 65535 f
0000001516 65535 f
0000001517 65535 f
0000001518 65535 f
0000001519 65535 f
0000001520 65535 f
0000001521 65535 f
0000001522 65535 f
0000001523 65535 f
0000001524 65535 f
0000001525 65535 f
0000001526 65535 f
0000001527 65535 f
0000001528 65535 f
0000001529 65535 f
0000001530 65535 f
0000001531 65535 f
0000001532 65535 f
0000001533 65535 f
0000001534 65535 f
0000001535 65535 f
0000001536 65535 f
0000001537 65535 f
0000001538 65535 f
0000001539 65535 f
0000001540 65535 f
0000001541 65535 f
0000001542 65535 f
0000001543 65535 f
0000001544 65535 f
0000001545 65535 f
0000001546 65535 f
0000001547 65535 f
0000001548 65535 f
0000001550 65535 f
0000083068 00000 n
0000001551 65535 f
0000001552 65535 f
0000001553 65535 f
0000001555 65535 f
0000083120 00000 n
0000001556 65535 f
0000001557 65535 f
0000001558 65535 f
0000001559 65535 f
0000001560 65535 f
0000001561 65535 f
0000001562 65535 f
0000001563 65535 f
0000001564 65535 f
0000001565 65535 f
0000001566 65535 f
0000001567 65535 f
0000001568 65535 f
0000001569 65535 f
0000001570 65535 f
0000001571 65535 f
0000001572 65535 f
0000001573 65535 f
0000001574 65535 f
0000001575 65535 f
0000001576 65535 f
0000001577 65535 f
0000001578 65535 f
0000001579 65535 f
0000001580 65535 f
0000001581 65535 f
0000001582 65535 f
0000001583 65535 f
0000001584 65535 f
0000001585 65535 f
0000001586 65535 f
0000001587 65535 f
0000001588 65535 f
0000001589 65535 f
0000001590 65535 f
0000001591 65535 f
0000001592 65535 f
0000001593 65535 f
0000001594 65535 f
0000001595 65535 f
0000001596 65535 f
0000001597 65535 f
0000001598 65535 f
0000001599 65535 f
0000001600 65535 f
0000001601 65535 f
0000001602 65535 f
0000001603 65535 f
0000001604 65535 f
0000001605 65535 f
0000001606 65535 f
0000001607 65535 f
0000001608 65535 f
0000001609 65535 f
0000001610 65535 f
0000001611 65535 f
0000001612 65535 f
0000001613 65535 f
0000001614 65535 f
0000001615 65535 f
0000001616 65535 f
0000001617 65535 f
0000001618 65535 f
0000001619 65535 f
0000001620 65535 f
0000001621 65535 f
0000001622 65535 f
0000001623 65535 f
0000001624 65535 f
0000001625 65535 f
0000001626 65535 f
0000001627 65535 f
0000001628 65535 f
0000001629 65535 f
0000001630 65535 f
0000001631 65535 f
0000001632 65535 f
0000001634 65535 f
0000086450 00000 n
0000001635 65535 f
0000001636 65535 f
0000001637 65535 f
0000001638 65535 f
0000001639 65535 f
0000001640 65535 f
0000001641 65535 f
0000001642 65535 f
0000001643 65535 f
0000001644 65535 f
0000001645 65535 f
0000000000 65535 f
0000086502 00000 n
0000086983 00000 n
0000155850 00000 n
0000156560 00000 n
0000156931 00000 n
0000157272 00000 n
0000212878 00000 n
0000213363 00000 n
0000245919 00000 n
0000246393 00000 n
0000246670 00000 n
0000246699 00000 n
0000246954 00000 n
0000283764 00000 n
0000283793 00000 n
0000286966 00000 n
0000287013 00000 n
trailer
<</Size 1663/Root 1 0 R/Info 54 0 R/ID[<5B0ED7A37ADB994F9A77D3B58180CB72><5B0ED7A37ADB994F9A77D3B58180CB72>] >>
startxref
290480
%%EOF
xref
0 0
trailer
<</Size 1663/Root 1 0 R/Info 54 0 R/ID[<5B0ED7A37ADB994F9A77D3B58180CB72><5B0ED7A37ADB994F9A77D3B58180CB72>] /Prev 290480/XRefStm 287013>>
startxref
323902
%%EOF
Sync-bytes-per-tx.PNGimage/png; name=Sync-bytes-per-tx.PNGDownload
Sync-bytes-total.PNGimage/png; name=Sync-bytes-total.PNGDownload
Sync-tx-throughput.PNGimage/png; name=Sync-tx-throughput.PNGDownload
NotSync-tx-throughput.PNGimage/png; name=NotSync-tx-throughput.PNGDownload
NotSync-bytes-total.PNGimage/png; name=NotSync-bytes-total.PNGDownload
NotSync-bytes-per-tx.PNGimage/png; name=NotSync-bytes-per-tx.PNGDownload
#52Amit Kapila
amit.kapila16@gmail.com
In reply to: Ajin Cherian (#47)
Re: logical replication empty transactions

On Fri, Jul 23, 2021 at 3:39 PM Ajin Cherian <itsajin@gmail.com> wrote:

Let's first split the patch for prepared and non-prepared cases as
that will help to focus on each of them separately. BTW, why haven't
you considered implementing point 1b as explained by Andres in his
email [1]/messages/by-id/20200309183018.tzkzwu635sd366ej@alap3.anarazel.de? I think we can send a keepalive message in case of
synchronous replication when we skip an empty transaction, otherwise,
it might delay in responding to transactions synchronous_commit mode.
I think in the tests done in the thread, it might not have been shown
because we are already sending keepalives too frequently. But what if
someone disables wal_sender_timeout or kept it to a very large value?
See WalSndKeepaliveIfNecessary. The other thing you might want to look
at is if the reason for frequent keepalives is the same as described
in the email [2]/messages/by-id/CALtH27cip5uQNJb4uHjLXtx1R52ELqXVfcP9fhHr=AvFo1dtqw@mail.gmail.com.

Few other miscellaneous comments:
1.
static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
- XLogRecPtr commit_lsn)
+ XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+ TimestampTz prepare_time)
 {
+ PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
  OutputPluginUpdateProgress(ctx);
+ /*
+ * If the BEGIN PREPARE was not yet sent, then it means there were no
+ * relevant changes encountered, so we can skip the COMMIT PREPARED
+ * message too.
+ */
+ if (txndata)
+ {
+ bool skip = !txndata->sent_begin_txn;
+ pfree(txndata);
+ txn->output_plugin_private = NULL;

How is this supposed to work after the restart when prepared is sent
before the restart and we are just sending commit_prepared after
restart? Won't this lead to sending commit_prepared even when the
corresponding prepare is not sent? Can we think of a better way to
deal with this?

2.
@@ -222,8 +224,10 @@ logicalrep_write_commit_prepared(StringInfo out,
ReorderBufferTXN *txn,
pq_sendbyte(out, flags);

/* send fields */
+ pq_sendint64(out, prepare_end_lsn);
pq_sendint64(out, commit_lsn);
pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, prepare_time);

Doesn't this means a change of protocol and how is it suppose to work
when say publisher is 15 and subscriber from 14 which I think works
without such a change?

[1]: /messages/by-id/20200309183018.tzkzwu635sd366ej@alap3.anarazel.de
[2]: /messages/by-id/CALtH27cip5uQNJb4uHjLXtx1R52ELqXVfcP9fhHr=AvFo1dtqw@mail.gmail.com

--
With Regards,
Amit Kapila.

#53Ajin Cherian
itsajin@gmail.com
In reply to: Amit Kapila (#52)
2 attachment(s)
Re: logical replication empty transactions

On Mon, Aug 2, 2021 at 7:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Fri, Jul 23, 2021 at 3:39 PM Ajin Cherian <itsajin@gmail.com> wrote:

Let's first split the patch for prepared and non-prepared cases as
that will help to focus on each of them separately.

As a first shot, I have split the patch into prepared and non-prepared cases,

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v12-0002-Skip-empty-prepared-transactions-for-logical-rep.patchapplication/octet-stream; name=v12-0002-Skip-empty-prepared-transactions-for-logical-rep.patchDownload
From 63c960849549f6c2d6d7bcc8901e0379563498c7 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Fri, 6 Aug 2021 09:48:25 -0400
Subject: [PATCH v12] Skip empty prepared transactions for logical replication.

The current logical replication behaviour is to send every transaction to
subscriber even though the transaction might be empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN / BEGIN
PREPARE messages until the first change is encountered.
If (when processing a COMMIT / PREPARE message) we find there had been
no other change for that transaction, then do not send the COMMIT /
PREPARE message. This means that pgoutput will skip BEGIN / COMMIT
or BEGIN PREPARE / PREPARE  messages for transactions that are empty.
pgoutput will also skip COMMIT PREPARED and ROLLBACK PREPARED messages
for transactions that are empty.
Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 contrib/test_decoding/test_decoding.c           |   7 +-
 doc/src/sgml/logicaldecoding.sgml               |  13 ++-
 doc/src/sgml/protocol.sgml                      |  19 ++++
 src/backend/replication/logical/logical.c       |   9 +-
 src/backend/replication/logical/proto.c         |  16 +++-
 src/backend/replication/logical/reorderbuffer.c |   2 +-
 src/backend/replication/logical/worker.c        |  38 +++++---
 src/backend/replication/pgoutput/pgoutput.c     | 113 +++++++++++++++++++++---
 src/include/replication/logicalproto.h          |   8 +-
 src/include/replication/output_plugin.h         |   4 +-
 src/include/replication/reorderbuffer.h         |   4 +-
 src/test/subscription/t/021_twophase.pl         |  47 +++++++++-
 src/tools/pgindent/typedefs.list                |   1 +
 13 files changed, 235 insertions(+), 46 deletions(-)

diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index e5cd84e..408dbfc 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -86,7 +86,9 @@ static void pg_decode_prepare_txn(LogicalDecodingContext *ctx,
 								  XLogRecPtr prepare_lsn);
 static void pg_decode_commit_prepared_txn(LogicalDecodingContext *ctx,
 										  ReorderBufferTXN *txn,
-										  XLogRecPtr commit_lsn);
+										  XLogRecPtr commit_lsn,
+										  XLogRecPtr prepare_end_lsn,
+										  TimestampTz prepare_time);
 static void pg_decode_rollback_prepared_txn(LogicalDecodingContext *ctx,
 											ReorderBufferTXN *txn,
 											XLogRecPtr prepare_end_lsn,
@@ -390,7 +392,8 @@ pg_decode_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 /* COMMIT PREPARED callback */
 static void
 pg_decode_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-							  XLogRecPtr commit_lsn)
+							  XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+							  TimestampTz prepare_time)
 {
 	TestDecodingData *data = ctx->output_plugin_private;
 
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 0d0de29..f6688da 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -884,11 +884,20 @@ typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
       The required <function>commit_prepared_cb</function> callback is called
       whenever a transaction <command>COMMIT PREPARED</command> has been decoded.
       The <parameter>gid</parameter> field, which is part of the
-      <parameter>txn</parameter> parameter, can be used in this callback.
+      <parameter>txn</parameter> parameter, can be used in this callback. The
+      parameters <parameter>prepare_end_lsn</parameter> and
+      <parameter>prepare_time</parameter> can be used to check if the plugin
+      has received this <command>PREPARE TRANSACTION</command> command or not.
+      If yes, it can commit the transaction, otherwise, it can skip the commit.
+      The <parameter>gid</parameter> alone is not sufficient to determine this
+      because the downstream node may already have a prepared transaction with the
+      same identifier.
 <programlisting>
 typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
                                                ReorderBufferTXN *txn,
-                                               XLogRecPtr commit_lsn);
+                                               XLogRecPtr commit_lsn,
+                                               XLogRecPtr prepare_end_lsn,
+                                               TimestampTz prepare_time);
 </programlisting>
      </para>
     </sect3>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 91ec237..9f346bb 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7585,6 +7585,15 @@ are available since protocol version 3.
         Int64 (XLogRecPtr)
 </term>
 <listitem><para>
+                The end LSN of the prepared transaction.
+</para></listitem>
+</varlistentry>
+
+<varlistentry>
+<term>
+        Int64 (XLogRecPtr)
+</term>
+<listitem><para>
                 The LSN of the commit prepared.
 </para></listitem>
 </varlistentry>
@@ -7603,6 +7612,16 @@ are available since protocol version 3.
         Int64 (TimestampTz)
 </term>
 <listitem><para>
+                Prepare timestamp of the prepared transaction. The value is
+                in number of microseconds since PostgreSQL epoch (2000-01-01).
+</para></listitem>
+</varlistentry>
+
+<varlistentry>
+<term>
+        Int64 (TimestampTz)
+</term>
+<listitem><para>
                 Commit timestamp of the transaction. The value is in number
                 of microseconds since PostgreSQL epoch (2000-01-01).
 </para></listitem>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 64b8280..8d47a35 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -63,7 +63,8 @@ static void begin_prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn
 static void prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 							   XLogRecPtr prepare_lsn);
 static void commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
-									   XLogRecPtr commit_lsn);
+									   XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+									   TimestampTz prepare_time);
 static void rollback_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 										 XLogRecPtr prepare_end_lsn, TimestampTz prepare_time);
 static void change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
@@ -940,7 +941,8 @@ prepare_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 
 static void
 commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
-						   XLogRecPtr commit_lsn)
+						   XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+						   TimestampTz prepare_time)
 {
 	LogicalDecodingContext *ctx = cache->private_data;
 	LogicalErrorCallbackState state;
@@ -976,7 +978,8 @@ commit_prepared_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
 						"commit_prepared_cb")));
 
 	/* do the actual work: call callback */
-	ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn);
+	ctx->callbacks.commit_prepared_cb(ctx, txn, commit_lsn, prepare_end_lsn,
+									  prepare_time);
 
 	/* Pop the error context stack */
 	error_context_stack = errcallback.previous;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 52b65e9..5f3c697 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -232,7 +232,9 @@ logicalrep_read_prepare(StringInfo in, LogicalRepPreparedTxnData *prepare_data)
  */
 void
 logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
-								 XLogRecPtr commit_lsn)
+								 XLogRecPtr commit_lsn,
+								 XLogRecPtr prepare_end_lsn,
+								 TimestampTz prepare_time)
 {
 	uint8		flags = 0;
 
@@ -248,8 +250,10 @@ logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
 	pq_sendbyte(out, flags);
 
 	/* send fields */
+	pq_sendint64(out, prepare_end_lsn);
 	pq_sendint64(out, commit_lsn);
 	pq_sendint64(out, txn->end_lsn);
+	pq_sendint64(out, prepare_time);
 	pq_sendint64(out, txn->xact_time.commit_time);
 	pq_sendint32(out, txn->xid);
 
@@ -270,12 +274,16 @@ logicalrep_read_commit_prepared(StringInfo in, LogicalRepCommitPreparedTxnData *
 		elog(ERROR, "unrecognized flags %u in commit prepared message", flags);
 
 	/* read fields */
+	prepare_data->prepare_end_lsn = pq_getmsgint64(in);
+	if (prepare_data->prepare_end_lsn == InvalidXLogRecPtr)
+		elog(ERROR, "prepare_end_lsn is not set in commit prepared message");
 	prepare_data->commit_lsn = pq_getmsgint64(in);
 	if (prepare_data->commit_lsn == InvalidXLogRecPtr)
 		elog(ERROR, "commit_lsn is not set in commit prepared message");
-	prepare_data->end_lsn = pq_getmsgint64(in);
-	if (prepare_data->end_lsn == InvalidXLogRecPtr)
-		elog(ERROR, "end_lsn is not set in commit prepared message");
+	prepare_data->commit_end_lsn = pq_getmsgint64(in);
+	if (prepare_data->commit_end_lsn == InvalidXLogRecPtr)
+		elog(ERROR, "commit_end_lsn is not set in commit prepared message");
+	prepare_data->prepare_time = pq_getmsgint64(in);
 	prepare_data->commit_time = pq_getmsgint64(in);
 	prepare_data->xid = pq_getmsgint(in, 4);
 
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 7378beb..5a707e2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2794,7 +2794,7 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid,
 	txn->origin_lsn = origin_lsn;
 
 	if (is_commit)
-		rb->commit_prepared(rb, txn, commit_lsn);
+		rb->commit_prepared(rb, txn, commit_lsn, prepare_end_lsn, prepare_time);
 	else
 		rb->rollback_prepared(rb, txn, prepare_end_lsn, prepare_time);
 
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ecaed15..2b8f59a 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -978,27 +978,39 @@ apply_handle_commit_prepared(StringInfo s)
 	/* Compute GID for two_phase transactions. */
 	TwoPhaseTransactionGid(MySubscription->oid, prepare_data.xid,
 						   gid, sizeof(gid));
-
-	/* There is no transaction when COMMIT PREPARED is called */
-	begin_replication_step();
-
 	/*
-	 * Update origin state so we can restart streaming from correct position
-	 * in case of crash.
+	 * It is possible that we haven't received the prepare because
+	 * the transaction did not have any changes relevant to this
+	 * subscription and so was essentially an empty prepare. In this case,
+	 * the walsender is optimized to drop the empty transaction and the
+	 * accompanying prepare. Silently ignore if we don't find the prepared
+	 * transaction.
 	 */
-	replorigin_session_origin_lsn = prepare_data.end_lsn;
-	replorigin_session_origin_timestamp = prepare_data.commit_time;
+	if (LookupGXact(gid, prepare_data.prepare_end_lsn,
+					prepare_data.prepare_time))
+	{
 
-	FinishPreparedTransaction(gid, true);
-	end_replication_step();
-	CommitTransactionCommand();
+		/* There is no transaction when COMMIT PREPARED is called */
+		begin_replication_step();
+
+		/*
+		 * Update origin state so we can restart streaming from correct position
+		 * in case of crash.
+		 */
+		replorigin_session_origin_lsn = prepare_data.commit_end_lsn;
+		replorigin_session_origin_timestamp = prepare_data.commit_time;
+
+		FinishPreparedTransaction(gid, true);
+		end_replication_step();
+		CommitTransactionCommand();
+	}
 	pgstat_report_stat(false);
 
-	store_flush_position(prepare_data.end_lsn);
+	store_flush_position(prepare_data.commit_end_lsn);
 	in_remote_transaction = false;
 
 	/* Process any tables that are being synchronized in parallel. */
-	process_syncing_tables(prepare_data.end_lsn);
+	process_syncing_tables(prepare_data.commit_end_lsn);
 
 	pgstat_report_activity(STATE_IDLE, NULL);
 }
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 13f0278..0153a9e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -56,7 +56,9 @@ static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
 static void pgoutput_prepare_txn(LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn, XLogRecPtr prepare_lsn);
 static void pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx,
-										 ReorderBufferTXN *txn, XLogRecPtr commit_lsn);
+										 ReorderBufferTXN *txn, XLogRecPtr commit_lsn,
+										 XLogRecPtr prepare_end_lsn,
+										 TimestampTz prepare_time);
 static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   ReorderBufferTXN *txn,
 										   XLogRecPtr prepare_end_lsn,
@@ -481,24 +483,46 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 }
 
 /*
- * BEGIN PREPARE callback
+ * BEGIN PREPARE callback.
+ *
+ * Don't send BEGIN PREPARE message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN PREPARE and COMMIT PREPARED messages
+ * to subscribers, using bandwidth on something with little/no use
+ * for logical replication.
  */
 static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	/*
+	 * Delegate to assign the begin sent flag as false, same as for the
+	 * BEGIN message.
+	 */
+	pgoutput_begin_txn(ctx, txn);
+}
+
+/*
+ * Send BEGIN PREPARE.
+ * This is where the BEGIN PREPARE is actually sent. This is called while
+ * processing the first change of the prepared transaction.
+ */
+static void
+pgoutput_begin_prepare(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
-	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
-														 sizeof(PGOutputTxnData));
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(txndata);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
-	txndata->sent_begin_txn = true;
-	txn->output_plugin_private = txndata;
 }
 
 /*
@@ -508,8 +532,21 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
 	OutputPluginUpdateProgress(ctx);
 
+	/*
+	 * If the BEGIN was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the PREPARE message too.
+	 */
+	if (!txndata->sent_begin_txn)
+	{
+		elog(DEBUG1, "skipping replication of an empty prepared transaction");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
@@ -520,12 +557,34 @@ pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  */
 static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-							 XLogRecPtr commit_lsn)
+							 XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+							 TimestampTz prepare_time)
 {
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/*
+	 * If the BEGIN PREPARE was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the COMMIT PREPARED
+	 * message too.
+	 */
+	if (txndata)
+	{
+		bool skip = !txndata->sent_begin_txn;
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+		if (skip)
+		{
+			elog(DEBUG1,
+				 "skipping replication of COMMIT PREPARED of an empty transaction");
+			return;
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
-	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
+	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn, prepare_end_lsn,
+									 prepare_time);
 	OutputPluginWrite(ctx, true);
 }
 
@@ -538,8 +597,27 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
+	PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
 	OutputPluginUpdateProgress(ctx);
 
+	/*
+	 * If the BEGIN PREPARE was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the ROLLBACK PREPARED
+	 * message too.
+	 */
+	if (txndata)
+	{
+		bool skip = !txndata->sent_begin_txn;
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+		if (skip)
+		{
+			elog(DEBUG1,
+				 "skipping replication of ROLLBACK of an empty transaction");
+			return;
+		}
+	}
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
 									   prepare_time);
@@ -731,11 +809,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	}
 
 	/*
-	 * Output BEGIN if we haven't yet, unless streaming.
+	 * Output BEGIN / BEGIN PREPARE if we haven't yet, unless streaming.
 	 */
 	if (!in_streaming && !txndata->sent_begin_txn)
 	{
-		pgoutput_begin(ctx, txn);
+		if (rbtxn_prepared(txn))
+			pgoutput_begin_prepare(ctx, txn);
+		else
+			pgoutput_begin(ctx, txn);
 	}
 
 	/* Avoid leaking memory by using and resetting our own context */
@@ -888,12 +969,15 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (nrelids > 0)
 	{
 		/*
-		 * output BEGIN if we haven't yet,
+		 * output BEGIN / BEGIN PREPARE if we haven't yet,
 		 * while streaming no need to send BEGIN / BEGIN PREPARE.
 		 */
 		if (!in_streaming && !txndata->sent_begin_txn)
 		{
-			pgoutput_begin(ctx, txn);
+			if (rbtxn_prepared(txn))
+				pgoutput_begin_prepare(ctx, txn);
+			else
+				pgoutput_begin(ctx, txn);
 		}
 
 		OutputPluginPrepareWrite(ctx, true);
@@ -939,7 +1023,10 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 		Assert(txndata);
 		if (!txndata->sent_begin_txn)
 		{
-			pgoutput_begin(ctx, txn);
+			if (rbtxn_prepared(txn))
+				pgoutput_begin_prepare(ctx, txn);
+			else
+				pgoutput_begin(ctx, txn);
 		}
 	}
 
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 2e29513..753d121 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -149,8 +149,10 @@ typedef struct LogicalRepPreparedTxnData
  */
 typedef struct LogicalRepCommitPreparedTxnData
 {
+	XLogRecPtr	prepare_end_lsn;
 	XLogRecPtr	commit_lsn;
-	XLogRecPtr	end_lsn;
+	XLogRecPtr	commit_end_lsn;
+	TimestampTz prepare_time;
 	TimestampTz commit_time;
 	TransactionId xid;
 	char		gid[GIDSIZE];
@@ -189,7 +191,9 @@ extern void logicalrep_write_prepare(StringInfo out, ReorderBufferTXN *txn,
 extern void logicalrep_read_prepare(StringInfo in,
 									LogicalRepPreparedTxnData *prepare_data);
 extern void logicalrep_write_commit_prepared(StringInfo out, ReorderBufferTXN *txn,
-											 XLogRecPtr commit_lsn);
+											 XLogRecPtr commit_lsn,
+											 XLogRecPtr prepare_end_lsn,
+											 TimestampTz prepare_time);
 extern void logicalrep_read_commit_prepared(StringInfo in,
 											LogicalRepCommitPreparedTxnData *prepare_data);
 extern void logicalrep_write_rollback_prepared(StringInfo out, ReorderBufferTXN *txn,
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 810495e..0d28306 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -128,7 +128,9 @@ typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
  */
 typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
 											   ReorderBufferTXN *txn,
-											   XLogRecPtr commit_lsn);
+											   XLogRecPtr commit_lsn,
+											   XLogRecPtr prepare_end_lsn,
+											   TimestampTz prepare_time);
 
 /*
  * Called for ROLLBACK PREPARED.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b40ff7..11e2e1e 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -442,7 +442,9 @@ typedef void (*ReorderBufferPrepareCB) (ReorderBuffer *rb,
 /* commit prepared callback signature */
 typedef void (*ReorderBufferCommitPreparedCB) (ReorderBuffer *rb,
 											   ReorderBufferTXN *txn,
-											   XLogRecPtr commit_lsn);
+											   XLogRecPtr commit_lsn,
+											   XLogRecPtr prepare_end_lsn,
+											   TimestampTz prepare_time);
 
 /* rollback  prepared callback signature */
 typedef void (*ReorderBufferRollbackPreparedCB) (ReorderBuffer *rb,
diff --git a/src/test/subscription/t/021_twophase.pl b/src/test/subscription/t/021_twophase.pl
index 19f0962..6a8c295 100644
--- a/src/test/subscription/t/021_twophase.pl
+++ b/src/test/subscription/t/021_twophase.pl
@@ -6,7 +6,7 @@ use strict;
 use warnings;
 use PostgresNode;
 use TestLib;
-use Test::More tests => 24;
+use Test::More tests => 25;
 
 ###############################
 # Setup
@@ -320,10 +320,10 @@ $node_publisher->safe_psql('postgres', "
 $node_publisher->wait_for_catchup($appname_copy);
 $node_publisher->wait_for_catchup($appname);
 
-# Check that the transaction has been prepared on the subscriber, there will be 2
-# prepared transactions for the 2 subscriptions.
+# Check that the transaction has been prepared on the subscriber, there will be only 1
+# prepared transaction for the 2 subscriptions.
 $result = $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM pg_prepared_xacts;");
-is($result, qq(2), 'transaction is prepared on subscriber');
+is($result, qq(1), 'transaction is prepared on subscriber');
 
 # Now commit the insert and verify that it IS replicated
 $node_publisher->safe_psql('postgres', "COMMIT PREPARED 'mygid';");
@@ -339,6 +339,45 @@ is($result, qq(2), 'replicated data in subscriber table');
 $node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub_copy;");
 $node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_copy;");
 
+##############################
+# Test empty prepares
+##############################
+
+# create a table that is not part of the publication
+$node_publisher->safe_psql('postgres',
+   "CREATE TABLE tab_nopub (a int PRIMARY KEY)");
+
+# disable the subscription so that we can peek at the slot
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub DISABLE");
+
+# wait for the replication slot to become inactive in the publisher
+$node_publisher->poll_query_until('postgres',
+   "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'tap_sub' AND active='f'", 1);
+
+# create a transaction with no changes relevant to the slot
+$node_publisher->safe_psql('postgres', "
+   BEGIN;
+   INSERT INTO tab_nopub SELECT generate_series(1,10);
+   PREPARE TRANSACTION 'empty_transaction';
+   COMMIT PREPARED 'empty_transaction';");
+
+# peek at the contents of the slot
+$result = $node_publisher->safe_psql(
+   'postgres', qq(
+       SELECT get_byte(data, 0)
+       FROM pg_logical_slot_get_binary_changes('tap_sub', NULL, NULL,
+           'proto_version', '3',
+           'publication_names', 'tap_pub')
+));
+
+# the empty transaction should be skipped
+is($result, qq(),
+   'empty transaction dropped on slot'
+);
+
+# enable the subscription to test cleanup
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub ENABLE");
+
 ###############################
 # check all the cleanup
 ###############################
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 37cf4b2..75639ab 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1606,6 +1606,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

v12-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v12-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 19048129914418b7f26856b930597e81ed5e0ddb Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Fri, 6 Aug 2021 08:50:03 -0400
Subject: [PATCH v12] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN / COMMIT messages for transactions that are empty.

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/pgoutput/pgoutput.c | 100 +++++++++++++++++++++++++++-
 src/test/subscription/t/020_messages.pl     |   5 +-
 2 files changed, 101 insertions(+), 4 deletions(-)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 286119c..13f0278 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -132,6 +132,17 @@ typedef struct RelationSyncEntry
 	TupleConversionMap *map;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN or BEGIN PREPARE. BEGIN or BEGIN PREPARE
+ * is only sent when the first change in a transaction is processed.
+ * This makes it possible to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+	bool sent_begin_txn;    /* flag indicating whether begin has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -396,15 +407,40 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(txndata);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -419,8 +455,26 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !txndata->sent_begin_txn;
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
 	OutputPluginUpdateProgress(ctx);
 
+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
 	OutputPluginWrite(ctx, true);
@@ -433,6 +487,8 @@ static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
@@ -441,6 +497,8 @@ pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
+	txndata->sent_begin_txn = true;
+	txn->output_plugin_private = txndata;
 }
 
 /*
@@ -630,11 +688,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
 	Relation	ancestor = NULL;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	if (!is_publishable_relation(relation))
 		return;
 
@@ -668,6 +730,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/*
+	 * Output BEGIN if we haven't yet, unless streaming.
+	 */
+	if (!in_streaming && !txndata->sent_begin_txn)
+	{
+		pgoutput_begin(ctx, txn);
+	}
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -770,6 +840,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -777,6 +848,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	Oid		   *relids;
 	TransactionId xid = InvalidTransactionId;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	/* Remember the xid for the change in streaming mode. See pgoutput_change. */
 	if (in_streaming)
 		xid = change->txn->xid;
@@ -813,6 +887,15 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/*
+		 * output BEGIN if we haven't yet,
+		 * while streaming no need to send BEGIN / BEGIN PREPARE.
+		 */
+		if (!in_streaming && !txndata->sent_begin_txn)
+		{
+			pgoutput_begin(ctx, txn);
+		}
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -845,6 +928,21 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		Assert(txndata);
+		if (!txndata->sent_begin_txn)
+		{
+			pgoutput_begin(ctx, txn);
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index ecf9b19..7a586b0 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
-- 
1.8.3.1

#54Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#53)
Re: logical replication empty transactions

On Sat, Aug 7, 2021 at 12:01 AM Ajin Cherian <itsajin@gmail.com> wrote:

On Mon, Aug 2, 2021 at 7:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Fri, Jul 23, 2021 at 3:39 PM Ajin Cherian <itsajin@gmail.com> wrote:

Let's first split the patch for prepared and non-prepared cases as
that will help to focus on each of them separately.

As a first shot, I have split the patch into prepared and non-prepared cases,

I have reviewed the v12* split patch set.

Apply / build / test was all OK

Below are my code review comments (mostly cosmetic).

//////////

Comments for v12-0001
=====================

1. Patch comment

=>

This comment as-is might have been OK before the 2PC code was
committed, but now that the 2PC is part of the HEAD perhaps this
comment needs to be expanded just to say this patch is ONLY for fixing
empty transactions for the cases of non-"streaming" and
non-"two_phase", and the other kinds will be tackled separately.

------

2. src/backend/replication/pgoutput/pgoutput.c - PGOutputTxnData comment

+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN or BEGIN PREPARE. BEGIN or BEGIN PREPARE
+ * is only sent when the first change in a transaction is processed.
+ * This makes it possible to skip transactions that are empty.
+ */

=>

Maybe this is true for the combined v12-0001/v12-0002 case but just
for the v12-0001 patch I think it is nor right to imply that some
skipping of the BEGIN_PREPARE is possible, because IIUC it isn;t
implemented in the *this* patch/

------

3. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin_txn whitespace

+ PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+ sizeof(PGOutputTxnData));

=>

Misaligned indentation?

------

4. src/backend/replication/pgoutput/pgoutput.c - pgoutput_change brackets

+ /*
+ * Output BEGIN if we haven't yet, unless streaming.
+ */
+ if (!in_streaming && !txndata->sent_begin_txn)
+ {
+ pgoutput_begin(ctx, txn);
+ }

=>

The brackets are not needed for the if with a single statement.

------

5. src/backend/replication/pgoutput/pgoutput.c - pgoutput_truncate
brackets/comment

+ /*
+ * output BEGIN if we haven't yet,
+ * while streaming no need to send BEGIN / BEGIN PREPARE.
+ */
+ if (!in_streaming && !txndata->sent_begin_txn)
+ {
+ pgoutput_begin(ctx, txn);
+ }

5a. =>

Same as review comment 4. The brackets are not needed for the if with
a single statement.

5b. =>

Notice this code is the same as cited in review comment 4. So probably
the code comment should be consistent/same also?

------

6. src/backend/replication/pgoutput/pgoutput.c - pgoutput_message brackets

+ Assert(txndata);
+ if (!txndata->sent_begin_txn)
+ {
+ pgoutput_begin(ctx, txn);
+ }

=>

The brackets are not needed for the if with a single statement.

------

7. typdefs.list

=> The structure PGOutputTxnData was added in v12-0001, so the
typedefs.list probably should also be updated.

//////////

Comments for v12-0002
=====================

8. Patch comment

This patch addresses the above problem by postponing the BEGIN / BEGIN
PREPARE messages until the first change is encountered.
If (when processing a COMMIT / PREPARE message) we find there had been
no other change for that transaction, then do not send the COMMIT /
PREPARE message. This means that pgoutput will skip BEGIN / COMMIT
or BEGIN PREPARE / PREPARE messages for transactions that are empty.
pgoutput will also skip COMMIT PREPARED and ROLLBACK PREPARED messages
for transactions that are empty.

8a. =>

I’m not sure this comment is 100% correct for this specific patch. The
whole BEGIN/COMMIT was already handled by the v12-0001 patch, right?
So really this comment should only be mentioning about BEGIN PREPARE
and COMMIT PREPARED I thought.

8b. =>

I think there should also be some mention that this patch is not
handling the "streaming" case of empty tx at all.

------

9. src/backend/replication/logical/proto.c - protocol version

@@ -248,8 +250,10 @@ logicalrep_write_commit_prepared(StringInfo out,
ReorderBufferTXN *txn,
pq_sendbyte(out, flags);

/* send fields */
+ pq_sendint64(out, prepare_end_lsn);
pq_sendint64(out, commit_lsn);
pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, prepare_time);
pq_sendint64(out, txn->xact_time.commit_time);
pq_sendint32(out, txn->xid);

=>

I agree with a previous feedback comment from Amit - Probably there is
some protocol version requirement/implications here because the
message format has been changed in logicalrep_write_commit_prepared
and logicalrep_read_commit_prepared.

e.g. Does this code need to be cognisant of the version and behave
differently accordingly?

------

10. src/backend/replication/pgoutput/pgoutput.c -
pgoutput_begin_prepare flag moved?

+ Assert(txndata);
OutputPluginPrepareWrite(ctx, !send_replication_origin);
logicalrep_write_begin_prepare(ctx->out, txn);
+ txndata->sent_begin_txn = true;

send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
send_replication_origin);

OutputPluginWrite(ctx, true);
- txndata->sent_begin_txn = true;
- txn->output_plugin_private = txndata;
}

=>

In the v12-0001 patch, you set the begin_txn flags AFTER the
OuputPluginWrite, but in the v12-0002 you set them BEFORE the
OuputPluginWrite. Why the difference? Maybe it should be consistent?

------

11. src/test/subscription/t/021_twophase.pl - proto_version tests needed?

Does this need some other tests to make sure the older proto_version
is still usable? Refer also to the review comment 9.

------

12. src/tools/pgindent/typedefs.list - PGOutputTxnData

PGOutputData
+PGOutputTxnData
PGPROC

=>

This change looks good, but I think it should have been done in
v12-0001 and not here in v12-0002.

------

Kind Regards,
Peter Smith.
Fujitsu Australia

#55Ajin Cherian
itsajin@gmail.com
In reply to: Amit Kapila (#52)
1 attachment(s)
Re: logical replication empty transactions

On Mon, Aug 2, 2021 at 7:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Fri, Jul 23, 2021 at 3:39 PM Ajin Cherian <itsajin@gmail.com> wrote:

Let's first split the patch for prepared and non-prepared cases as
that will help to focus on each of them separately. BTW, why haven't
you considered implementing point 1b as explained by Andres in his
email [1]? I think we can send a keepalive message in case of
synchronous replication when we skip an empty transaction, otherwise,
it might delay in responding to transactions synchronous_commit mode.
I think in the tests done in the thread, it might not have been shown
because we are already sending keepalives too frequently. But what if
someone disables wal_sender_timeout or kept it to a very large value?
See WalSndKeepaliveIfNecessary. The other thing you might want to look
at is if the reason for frequent keepalives is the same as described
in the email [2].

I have tried to address the comment here by modifying the
ctx->update_progress callback function (WalSndUpdateProgress) provided
for plugins. I have added an option
by which the callback can specify if it wants to send keep_alives. And
when the callback is called with that option set, walsender updates a
flag force_keep_alive_syncrep.
The Walsender in the WalSndWaitForWal for loop, checks this flag and
if synchronous replication is enabled, then sends a keep alive.
Currently this logic
is added as an else to the current logic that is already there in
WalSndWaitForWal, which is probably considered unnecessary and a
source of the keep alive flood
that you talked about. So, I can change that according to how that fix
shapes up there. I have also added an extern function in syncrep.c
that makes it possible
for walsender to query if synchronous replication is turned on.

The reason I had to turn on a flag and rely on the WalSndWaitForWal to
send the keep alive in its next iteration is because I tried doing
this directly when a
commit is skipped but it didn't work. The reason for this is that when
the commit is being decoded the sentptr at the moment is at the commit
LSN and the keep alive
will be sent for the commit LSN but the syncrep wait is waiting for
end_lsn of the transaction which is the next LSN. So, sending a keep
alive at the moment the
commit is decoded doesn't seem to solve the problem of the waiting
synchronous reply.

Few other miscellaneous comments:
1.
static void
pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
- XLogRecPtr commit_lsn)
+ XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+ TimestampTz prepare_time)
{
+ PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
OutputPluginUpdateProgress(ctx);
+ /*
+ * If the BEGIN PREPARE was not yet sent, then it means there were no
+ * relevant changes encountered, so we can skip the COMMIT PREPARED
+ * message too.
+ */
+ if (txndata)
+ {
+ bool skip = !txndata->sent_begin_txn;
+ pfree(txndata);
+ txn->output_plugin_private = NULL;

How is this supposed to work after the restart when prepared is sent
before the restart and we are just sending commit_prepared after
restart? Won't this lead to sending commit_prepared even when the
corresponding prepare is not sent? Can we think of a better way to
deal with this?

I have tried to resolve this by adding logic in worker,c to silently
ignore spurious commit_prepareds. But this change required checking if
the prepare exists on the
subscriber before attempting the commit_prepared but the current API
that checks this requires prepare time and transaction end_lsn. But
for this I had to
change the protocol of commit_prepared, and I understand that this
would break backward compatibility between subscriber and publisher
(you have raised this issue as well).
I am not sure how else to handle this, let me know if you have any
other ideas. One option could be to have another API to check if the
prepare exists on the subscriber with
the prepared 'gid' alone, without checking prepare_time or end_lsn.
Let me know if this idea works.

I have left out the patch 0002 for prepared transactions until we
arrive at a decision on how to address the above issue.

Peter,
I have also addressed the comments you've raised on patch 0001, please
have a look and confirm.

Regards,
Ajin Cherian
Fujitsu Australia.

Attachments:

v13-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v13-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 6061bce8886b135cc0108dff772b2415fbe42c3c Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Fri, 13 Aug 2021 06:33:26 -0400
Subject: [PATCH v13] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN / COMMIT messages for transactions that are empty.
This patch does not skip empty transactions that are "streaming" or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 105 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 ++++
 src/backend/replication/walsender.c         |  20 +++++-
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 137 insertions(+), 16 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 64b8280..009bd00 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -671,12 +671,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keep_alive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keep_alive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 14d737f..0289ac9 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -132,6 +132,17 @@ typedef struct RelationSyncEntry
 	TupleConversionMap *map;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN.
+ * BEGIN is only sent when the first change in a transaction is processed.
+ * This makes it possible to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+	bool sent_begin_txn;    /* flag indicating whether begin has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -396,15 +407,40 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(txndata);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -419,7 +455,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !txndata->sent_begin_txn;
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, skip);
+
+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -433,6 +487,8 @@ static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
@@ -441,6 +497,8 @@ pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
+	txndata->sent_begin_txn = true;
+	txn->output_plugin_private = txndata;
 }
 
 /*
@@ -450,7 +508,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -464,7 +522,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -480,7 +538,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -630,11 +688,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
 	Relation	ancestor = NULL;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	if (!is_publishable_relation(relation))
 		return;
 
@@ -668,6 +730,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/*
+	 * Output BEGIN if we haven't yet, unless streaming.
+	 */
+	if (!in_streaming && !txndata->sent_begin_txn)
+		pgoutput_begin(ctx, txn);
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -770,6 +838,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -777,6 +846,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	Oid		   *relids;
 	TransactionId xid = InvalidTransactionId;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	/* Remember the xid for the change in streaming mode. See pgoutput_change. */
 	if (in_streaming)
 		xid = change->txn->xid;
@@ -813,6 +885,12 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/*
+		 * output BEGIN if we haven't yet, unless streaming.
+		 */
+		if (!in_streaming && !txndata->sent_begin_txn)
+			pgoutput_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -845,6 +923,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		Assert(txndata);
+		if (!txndata->sent_begin_txn)
+			pgoutput_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1011,7 +1102,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1032,7 +1123,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index bdbc9ef..7aa96a3 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -330,6 +330,18 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 }
 
 /*
+ * Check if Sync Rep is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	if (SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+		return true;
+	else
+		return false;
+}
+
+/*
  * Insert MyProc into the specified SyncRepQueue, maintaining sorted invariant.
  *
  * Usually we will go at tail of queue, though it's possible that we arrive
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 3ca2a11..8d2532d 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -170,6 +170,9 @@ static TimestampTz last_reply_timestamp = 0;
 /* Have we sent a heartbeat message asking for reply, since last reply? */
 static bool waiting_for_ping_response = false;
 
+/* force keep_alive when skipping transactions in synchronous replication mode */
+static bool force_keep_alive_syncrep = false;
+
 /*
  * While streaming WAL in Copy mode, streamingDoneSending is set to true
  * after we have sent CopyDone. We should not send any more CopyData messages
@@ -247,7 +250,8 @@ static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keep_alive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1346,11 +1350,16 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keep_alive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
+	if (send_keep_alive)
+		force_keep_alive_syncrep = true;
+
+
 	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
@@ -1447,7 +1456,14 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
+		{
 			WalSndKeepalive(false);
+		}
+		else
+		{
+			if (force_keep_alive_syncrep && SyncRepEnabled())
+				WalSndKeepalive(false);
+		}
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index e0f513b..eb54ce9 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keep_alive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 810495e..4ab8455 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -243,6 +243,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keep_alive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 4266afd..d9fc991 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index ecf9b19..7a586b0 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 37cf4b2..75639ab 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1606,6 +1606,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#56Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#55)
Re: logical replication empty transactions

On Fri, Aug 13, 2021 at 9:01 PM Ajin Cherian <itsajin@gmail.com> wrote:

On Mon, Aug 2, 2021 at 7:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Fri, Jul 23, 2021 at 3:39 PM Ajin Cherian <itsajin@gmail.com> wrote:

Let's first split the patch for prepared and non-prepared cases as
that will help to focus on each of them separately. BTW, why haven't
you considered implementing point 1b as explained by Andres in his
email [1]? I think we can send a keepalive message in case of
synchronous replication when we skip an empty transaction, otherwise,
it might delay in responding to transactions synchronous_commit mode.
I think in the tests done in the thread, it might not have been shown
because we are already sending keepalives too frequently. But what if
someone disables wal_sender_timeout or kept it to a very large value?
See WalSndKeepaliveIfNecessary. The other thing you might want to look
at is if the reason for frequent keepalives is the same as described
in the email [2].

I have tried to address the comment here by modifying the
ctx->update_progress callback function (WalSndUpdateProgress) provided
for plugins. I have added an option
by which the callback can specify if it wants to send keep_alives. And
when the callback is called with that option set, walsender updates a
flag force_keep_alive_syncrep.
The Walsender in the WalSndWaitForWal for loop, checks this flag and
if synchronous replication is enabled, then sends a keep alive.
Currently this logic
is added as an else to the current logic that is already there in
WalSndWaitForWal, which is probably considered unnecessary and a
source of the keep alive flood
that you talked about. So, I can change that according to how that fix
shapes up there. I have also added an extern function in syncrep.c
that makes it possible
for walsender to query if synchronous replication is turned on.

The reason I had to turn on a flag and rely on the WalSndWaitForWal to
send the keep alive in its next iteration is because I tried doing
this directly when a
commit is skipped but it didn't work. The reason for this is that when
the commit is being decoded the sentptr at the moment is at the commit
LSN and the keep alive
will be sent for the commit LSN but the syncrep wait is waiting for
end_lsn of the transaction which is the next LSN. So, sending a keep
alive at the moment the
commit is decoded doesn't seem to solve the problem of the waiting
synchronous reply.

Few other miscellaneous comments:
1.
static void
pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
- XLogRecPtr commit_lsn)
+ XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn,
+ TimestampTz prepare_time)
{
+ PGOutputTxnData    *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
OutputPluginUpdateProgress(ctx);
+ /*
+ * If the BEGIN PREPARE was not yet sent, then it means there were no
+ * relevant changes encountered, so we can skip the COMMIT PREPARED
+ * message too.
+ */
+ if (txndata)
+ {
+ bool skip = !txndata->sent_begin_txn;
+ pfree(txndata);
+ txn->output_plugin_private = NULL;

How is this supposed to work after the restart when prepared is sent
before the restart and we are just sending commit_prepared after
restart? Won't this lead to sending commit_prepared even when the
corresponding prepare is not sent? Can we think of a better way to
deal with this?

I have tried to resolve this by adding logic in worker,c to silently
ignore spurious commit_prepareds. But this change required checking if
the prepare exists on the
subscriber before attempting the commit_prepared but the current API
that checks this requires prepare time and transaction end_lsn. But
for this I had to
change the protocol of commit_prepared, and I understand that this
would break backward compatibility between subscriber and publisher
(you have raised this issue as well).
I am not sure how else to handle this, let me know if you have any
other ideas. One option could be to have another API to check if the
prepare exists on the subscriber with
the prepared 'gid' alone, without checking prepare_time or end_lsn.
Let me know if this idea works.

I have left out the patch 0002 for prepared transactions until we
arrive at a decision on how to address the above issue.

Peter,
I have also addressed the comments you've raised on patch 0001, please
have a look and confirm.

I have reviewed the v13-0001 patch.

Apply / build / test was all OK

Below are my code review comments.

//////////

Comments for v13-0001
=====================

1. Patch comment

=>

Probably this comment should include some description for the new
"keepalive" logic as well.

------

2. src/backend/replication/syncrep.c - new function

@@ -330,6 +330,18 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
}

 /*
+ * Check if Sync Rep is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+ if (SyncRepRequested() && ((volatile WalSndCtlData *)
WalSndCtl)->sync_standbys_defined)
+ return true;
+ else
+ return false;
+}
+

2a. Function comment =>

Why abbreviations in the comment? Why not say "synchronous
replication" instead of "Sync Rep".

~~

2b. if/else =>

Remove the if/else. e.g.

return SyncRepRequested() && ((volatile WalSndCtlData *)
WalSndCtl)->sync_standbys_defined;

~~

2c. Call the new function =>

There is some existing similar code in SyncRepWaitForLSN(), e.g.

if (!SyncRepRequested() ||
!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
return;

Now that you have a new function you maybe can call it from here, e.g.

if (!SyncRepEnabled())
return;

------

3. src/backend/replication/walsender.c - whitespace

+ if (send_keep_alive)
+ force_keep_alive_syncrep = true;
+
+

=>

Extra blank line?

------

4. src/backend/replication/walsender.c - call keepalive

  if (MyWalSnd->flush < sentPtr &&
  MyWalSnd->write < sentPtr &&
  !waiting_for_ping_response)
+ {
  WalSndKeepalive(false);
+ }
+ else
+ {
+ if (force_keep_alive_syncrep && SyncRepEnabled())
+ WalSndKeepalive(false);
+ }

4a. Move the SynRepEnabled() call =>

I think it is not necessary to call the SynRepEnabled() here. Instead,
it might be better if this is called back when you assign the
force_keep_alive_syncrep flag. So change the WalSndUpdateProgress,
e.g.

BEFORE
if (send_keep_alive)
force_keep_alive_syncrep = true;
AFTER
force_keep_alive_syncrep = send_keep_alive && SyncRepEnabled();

Note: Also, that assignment also deserves a big comment to say what it is doing.

~~

4b. Change the if/else =>

If you make the change for 4a. then perhaps the keepalive if/else is
overkill and could be changed.e.g.

if (force_keep_alive_syncrep ||
MyWalSnd->flush < sentPtr &&
MyWalSnd->write < sentPtr &&
!waiting_for_ping_response)
WalSndKeepalive(false);

------
Kind Regards,
Peter Smith.
Fujitsu Australia

#57Ajin Cherian
itsajin@gmail.com
In reply to: Peter Smith (#56)
1 attachment(s)
Re: logical replication empty transactions

On Mon, Aug 16, 2021 at 4:44 PM Peter Smith <smithpb2250@gmail.com> wrote:

I have reviewed the v13-0001 patch.

Apply / build / test was all OK

Below are my code review comments.

//////////

Comments for v13-0001
=====================

1. Patch comment

=>

Probably this comment should include some description for the new
"keepalive" logic as well.

Added.

------

2. src/backend/replication/syncrep.c - new function

@@ -330,6 +330,18 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
}

/*
+ * Check if Sync Rep is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+ if (SyncRepRequested() && ((volatile WalSndCtlData *)
WalSndCtl)->sync_standbys_defined)
+ return true;
+ else
+ return false;
+}
+

2a. Function comment =>

Why abbreviations in the comment? Why not say "synchronous
replication" instead of "Sync Rep".

Changed.

~~

2b. if/else =>

Remove the if/else. e.g.

return SyncRepRequested() && ((volatile WalSndCtlData *)
WalSndCtl)->sync_standbys_defined;

~~

Changed.

2c. Call the new function =>

There is some existing similar code in SyncRepWaitForLSN(), e.g.

if (!SyncRepRequested() ||
!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
return;

Now that you have a new function you maybe can call it from here, e.g.

if (!SyncRepEnabled())
return;

Updated.

------

3. src/backend/replication/walsender.c - whitespace

+ if (send_keep_alive)
+ force_keep_alive_syncrep = true;
+
+

=>

Extra blank line?

Removed.

------

4. src/backend/replication/walsender.c - call keepalive

if (MyWalSnd->flush < sentPtr &&
MyWalSnd->write < sentPtr &&
!waiting_for_ping_response)
+ {
WalSndKeepalive(false);
+ }
+ else
+ {
+ if (force_keep_alive_syncrep && SyncRepEnabled())
+ WalSndKeepalive(false);
+ }

4a. Move the SynRepEnabled() call =>

I think it is not necessary to call the SynRepEnabled() here. Instead,
it might be better if this is called back when you assign the
force_keep_alive_syncrep flag. So change the WalSndUpdateProgress,
e.g.

BEFORE
if (send_keep_alive)
force_keep_alive_syncrep = true;
AFTER
force_keep_alive_syncrep = send_keep_alive && SyncRepEnabled();

Note: Also, that assignment also deserves a big comment to say what it is doing.

~~

changed.

4b. Change the if/else =>

If you make the change for 4a. then perhaps the keepalive if/else is
overkill and could be changed.e.g.

if (force_keep_alive_syncrep ||
MyWalSnd->flush < sentPtr &&
MyWalSnd->write < sentPtr &&
!waiting_for_ping_response)
WalSndKeepalive(false);

Changed.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v14-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v14-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From fc88a42df802e6915865e7b40b090a724c808f99 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Fri, 13 Aug 2021 06:33:26 -0400
Subject: [PATCH v14] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN / COMMIT messages for transactions that are empty. The patch
also makes sure that in synchronous replication mode, when skipping empty
transactions, keepalive messages are sent to keep the LSN locations updated
on the standby.
This patch does not skip empty transactions that are "streaming" or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 105 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 +++-
 src/backend/replication/walsender.c         |  22 ++++--
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 135 insertions(+), 20 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 64b8280..965ca6d 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -671,12 +671,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 14d737f..c491212 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -132,6 +132,17 @@ typedef struct RelationSyncEntry
 	TupleConversionMap *map;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+	bool sent_begin_txn;    /* flag indicating whether begin has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -396,15 +407,40 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(txndata);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -419,7 +455,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !txndata->sent_begin_txn;
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, skip);
+
+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -433,6 +487,8 @@ static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
@@ -441,6 +497,8 @@ pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
+	txndata->sent_begin_txn = true;
+	txn->output_plugin_private = txndata;
 }
 
 /*
@@ -450,7 +508,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -464,7 +522,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -480,7 +538,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -630,11 +688,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
 	Relation	ancestor = NULL;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	if (!is_publishable_relation(relation))
 		return;
 
@@ -668,6 +730,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/*
+	 * Output BEGIN if we haven't yet, unless streaming.
+	 */
+	if (!in_streaming && !txndata->sent_begin_txn)
+		pgoutput_begin(ctx, txn);
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -770,6 +838,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -777,6 +846,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	Oid		   *relids;
 	TransactionId xid = InvalidTransactionId;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	/* Remember the xid for the change in streaming mode. See pgoutput_change. */
 	if (in_streaming)
 		xid = change->txn->xid;
@@ -813,6 +885,12 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/*
+		 * output BEGIN if we haven't yet, unless streaming.
+		 */
+		if (!in_streaming && !txndata->sent_begin_txn)
+			pgoutput_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -845,6 +923,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		Assert(txndata);
+		if (!txndata->sent_begin_txn)
+			pgoutput_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1011,7 +1102,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1032,7 +1123,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index bdbc9ef..5fe275e 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -330,6 +329,15 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 }
 
 /*
+ * Check if Synchronous Replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Insert MyProc into the specified SyncRepQueue, maintaining sorted invariant.
  *
  * Usually we will go at tail of queue, though it's possible that we arrive
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 3ca2a11..c945698 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -170,6 +170,9 @@ static TimestampTz last_reply_timestamp = 0;
 /* Have we sent a heartbeat message asking for reply, since last reply? */
 static bool waiting_for_ping_response = false;
 
+/* force keep alive when skipping transactions in synchronous replication mode */
+static bool force_keepalive_syncrep = false;
+
 /*
  * While streaming WAL in Copy mode, streamingDoneSending is set to true
  * after we have sent CopyDone. We should not send any more CopyData messages
@@ -247,7 +250,8 @@ static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1346,12 +1350,19 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keep alive to keep the MyWalSnd locations updated.
+	 */
+	force_keepalive_syncrep = send_keepalive && SyncRepEnabled();
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1444,10 +1455,13 @@ WalSndWaitForWal(XLogRecPtr loc)
 		 * otherwise idle, this keepalive will trigger a reply. Processing the
 		 * reply will update these MyWalSnd locations.
 		 */
-		if (MyWalSnd->flush < sentPtr &&
+		if (force_keepalive_syncrep ||
+			(MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
-			!waiting_for_ping_response)
+			!waiting_for_ping_response))
+		{
 			WalSndKeepalive(false);
+		}
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index e0f513b..eb54ce9 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keep_alive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 810495e..4ab8455 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -243,6 +243,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keep_alive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 4266afd..d9fc991 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index ecf9b19..7a586b0 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 37cf4b2..75639ab 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1606,6 +1606,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#58Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#57)
Re: logical replication empty transactions

I reviewed the v14-0001 patch.

All my previous comments have been addressed.

Apply / build / test was all OK.

------

More review comments:

1. Params names in the function declarations should match the rest of the code.

1a. src/include/replication/logical.h

@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite
LogicalOutputPluginWriterPrepareWrite;

 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct
LogicalDecodingContext *lr,
  XLogRecPtr Ptr,
- TransactionId xid
+ TransactionId xid,
+ bool send_keep_alive

=>
Change "send_keep_alive" --> "send_keepalive"

~~

1b. src/include/replication/output_plugin.h

@@ -243,6 +243,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext
*ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx,
bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext
*ctx, bool send_keep_alive);

=>
Change "send_keep_alive" --> "send_keepalive"

------

2. Comment should be capitalized - src/backend/replication/walsender.c

@@ -170,6 +170,9 @@ static TimestampTz last_reply_timestamp = 0;
/* Have we sent a heartbeat message asking for reply, since last reply? */
static bool waiting_for_ping_response = false;

+/* force keep alive when skipping transactions in synchronous
replication mode */
+static bool force_keepalive_syncrep = false;

=>
"force" --> "Force"

------

Otherwise, v14-0001 LGTM.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

#59Ajin Cherian
itsajin@gmail.com
In reply to: Peter Smith (#58)
1 attachment(s)
Re: logical replication empty transactions

On Wed, Aug 25, 2021 at 5:15 PM Peter Smith <smithpb2250@gmail.com> wrote:

I reviewed the v14-0001 patch.

All my previous comments have been addressed.

Apply / build / test was all OK.

------

More review comments:

1. Params names in the function declarations should match the rest of the code.

1a. src/include/replication/logical.h

@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite
LogicalOutputPluginWriterPrepareWrite;

typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct
LogicalDecodingContext *lr,
XLogRecPtr Ptr,
- TransactionId xid
+ TransactionId xid,
+ bool send_keep_alive

=>
Change "send_keep_alive" --> "send_keepalive"

~~

1b. src/include/replication/output_plugin.h

@@ -243,6 +243,6 @@ typedef struct OutputPluginCallbacks
/* Functions in replication/logical/logical.c */
extern void OutputPluginPrepareWrite(struct LogicalDecodingContext
*ctx, bool last_write);
extern void OutputPluginWrite(struct LogicalDecodingContext *ctx,
bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext
*ctx, bool send_keep_alive);

=>
Change "send_keep_alive" --> "send_keepalive"

------

2. Comment should be capitalized - src/backend/replication/walsender.c

@@ -170,6 +170,9 @@ static TimestampTz last_reply_timestamp = 0;
/* Have we sent a heartbeat message asking for reply, since last reply? */
static bool waiting_for_ping_response = false;

+/* force keep alive when skipping transactions in synchronous
replication mode */
+static bool force_keepalive_syncrep = false;

=>
"force" --> "Force"

------

Otherwise, v14-0001 LGTM.

Thanks for the comments. Addressed them in the attached patch.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v15-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v15-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 904ae00e8565d44f2403d1a212fc41dade0277a5 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Wed, 1 Sep 2021 06:52:17 -0400
Subject: [PATCH v15] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN / COMMIT messages for transactions that are empty. The patch
also makes sure that in synchronous replication mode, when skipping empty
transactions, keepalive messages are sent to keep the LSN locations updated
on the standby.
This patch does not skip empty transactions that are "streaming" or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 105 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 +++-
 src/backend/replication/walsender.c         |  22 ++++--
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 135 insertions(+), 20 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 64b8280..965ca6d 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -671,12 +671,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 14d737f..c491212 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -132,6 +132,17 @@ typedef struct RelationSyncEntry
 	TupleConversionMap *map;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+	bool sent_begin_txn;    /* flag indicating whether begin has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -396,15 +407,40 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(txndata);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -419,7 +455,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !txndata->sent_begin_txn;
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, skip);
+
+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -433,6 +487,8 @@ static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
@@ -441,6 +497,8 @@ pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
+	txndata->sent_begin_txn = true;
+	txn->output_plugin_private = txndata;
 }
 
 /*
@@ -450,7 +508,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -464,7 +522,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -480,7 +538,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -630,11 +688,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
 	Relation	ancestor = NULL;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	if (!is_publishable_relation(relation))
 		return;
 
@@ -668,6 +730,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/*
+	 * Output BEGIN if we haven't yet, unless streaming.
+	 */
+	if (!in_streaming && !txndata->sent_begin_txn)
+		pgoutput_begin(ctx, txn);
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -770,6 +838,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -777,6 +846,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	Oid		   *relids;
 	TransactionId xid = InvalidTransactionId;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	/* Remember the xid for the change in streaming mode. See pgoutput_change. */
 	if (in_streaming)
 		xid = change->txn->xid;
@@ -813,6 +885,12 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/*
+		 * output BEGIN if we haven't yet, unless streaming.
+		 */
+		if (!in_streaming && !txndata->sent_begin_txn)
+			pgoutput_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -845,6 +923,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		Assert(txndata);
+		if (!txndata->sent_begin_txn)
+			pgoutput_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1011,7 +1102,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1032,7 +1123,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index bdbc9ef..5fe275e 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -330,6 +329,15 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 }
 
 /*
+ * Check if Synchronous Replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Insert MyProc into the specified SyncRepQueue, maintaining sorted invariant.
  *
  * Usually we will go at tail of queue, though it's possible that we arrive
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 3ca2a11..f304b5d 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -170,6 +170,9 @@ static TimestampTz last_reply_timestamp = 0;
 /* Have we sent a heartbeat message asking for reply, since last reply? */
 static bool waiting_for_ping_response = false;
 
+/* Force keep alive when skipping transactions in synchronous replication mode */
+static bool force_keepalive_syncrep = false;
+
 /*
  * While streaming WAL in Copy mode, streamingDoneSending is set to true
  * after we have sent CopyDone. We should not send any more CopyData messages
@@ -247,7 +250,8 @@ static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1346,12 +1350,19 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keep alive to keep the MyWalSnd locations updated.
+	 */
+	force_keepalive_syncrep = send_keepalive && SyncRepEnabled();
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1444,10 +1455,13 @@ WalSndWaitForWal(XLogRecPtr loc)
 		 * otherwise idle, this keepalive will trigger a reply. Processing the
 		 * reply will update these MyWalSnd locations.
 		 */
-		if (MyWalSnd->flush < sentPtr &&
+		if (force_keepalive_syncrep ||
+			(MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
-			!waiting_for_ping_response)
+			!waiting_for_ping_response))
+		{
 			WalSndKeepalive(false);
+		}
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index e0f513b..41e013c 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keepalive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 810495e..905140f 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -243,6 +243,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 4266afd..d9fc991 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index ecf9b19..7a586b0 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f31a1e4..38d1c7a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1608,6 +1608,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#60Ajin Cherian
itsajin@gmail.com
In reply to: Ajin Cherian (#59)
1 attachment(s)
Re: logical replication empty transactions

On Wed, Sep 1, 2021 at 8:57 PM Ajin Cherian <itsajin@gmail.com> wrote:

Thanks for the comments. Addressed them in the attached patch.

regards,
Ajin Cherian
Fujitsu Australia

Minor update to rebase the patch so that it applies clean on HEAD.

regards,
Ajin Cherian

regards,
Ajin Cherian

Attachments:

v16-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v16-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 858fc8976c5099f4833aebc54f8337e78fdb6222 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Tue, 11 Jan 2022 04:35:09 -0500
Subject: [PATCH v16] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN / COMMIT messages for transactions that are empty. The patch
also makes sure that in synchronous replication mode, when skipping empty
transactions, keepalive messages are sent to keep the LSN locations updated
on the standby.
This patch does not skip empty transactions that are "streaming" or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 105 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 +++-
 src/backend/replication/walsender.c         |  22 ++++--
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 135 insertions(+), 20 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 9bc3a2d..fb1c26a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -672,12 +672,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index af8d51a..75296a8 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -132,6 +132,17 @@ typedef struct RelationSyncEntry
 	TupleConversionMap *map;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+	bool sent_begin_txn;    /* flag indicating whether begin has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -396,15 +407,40 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
+	Assert(txndata);
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -419,7 +455,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !txndata->sent_begin_txn;
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, skip);
+
+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -433,6 +487,8 @@ static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
@@ -441,6 +497,8 @@ pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
+	txndata->sent_begin_txn = true;
+	txn->output_plugin_private = txndata;
 }
 
 /*
@@ -450,7 +508,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -464,7 +522,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -480,7 +538,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -630,11 +688,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
 	Relation	ancestor = NULL;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	if (!is_publishable_relation(relation))
 		return;
 
@@ -668,6 +730,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/*
+	 * Output BEGIN if we haven't yet, unless streaming.
+	 */
+	if (!in_streaming && !txndata->sent_begin_txn)
+		pgoutput_begin(ctx, txn);
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -770,6 +838,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -777,6 +846,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	Oid		   *relids;
 	TransactionId xid = InvalidTransactionId;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	/* Remember the xid for the change in streaming mode. See pgoutput_change. */
 	if (in_streaming)
 		xid = change->txn->xid;
@@ -813,6 +885,12 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/*
+		 * output BEGIN if we haven't yet, unless streaming.
+		 */
+		if (!in_streaming && !txndata->sent_begin_txn)
+			pgoutput_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -845,6 +923,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		Assert(txndata);
+		if (!txndata->sent_begin_txn)
+			pgoutput_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1011,7 +1102,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1032,7 +1123,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index ce163b9..b945165 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -330,6 +329,15 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 }
 
 /*
+ * Check if Synchronous Replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Insert MyProc into the specified SyncRepQueue, maintaining sorted invariant.
  *
  * Usually we will go at tail of queue, though it's possible that we arrive
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 4cf95ce..43a480f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -170,6 +170,9 @@ static TimestampTz last_reply_timestamp = 0;
 /* Have we sent a heartbeat message asking for reply, since last reply? */
 static bool waiting_for_ping_response = false;
 
+/* Force keep alive when skipping transactions in synchronous replication mode */
+static bool force_keepalive_syncrep = false;
+
 /*
  * While streaming WAL in Copy mode, streamingDoneSending is set to true
  * after we have sent CopyDone. We should not send any more CopyData messages
@@ -248,7 +251,8 @@ static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1448,12 +1452,19 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keep alive to keep the MyWalSnd locations updated.
+	 */
+	force_keepalive_syncrep = send_keepalive && SyncRepEnabled();
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1546,10 +1557,13 @@ WalSndWaitForWal(XLogRecPtr loc)
 		 * otherwise idle, this keepalive will trigger a reply. Processing the
 		 * reply will update these MyWalSnd locations.
 		 */
-		if (MyWalSnd->flush < sentPtr &&
+		if (force_keepalive_syncrep ||
+			(MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
-			!waiting_for_ping_response)
+			!waiting_for_ping_response))
+		{
 			WalSndKeepalive(false);
+		}
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..9f59855 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keepalive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 41157fd..6f84614 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -243,6 +243,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 27be230..7086532 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index 9bb31ce..1d52e47 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 5015fa7..c407d54 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1607,6 +1607,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#61osumi.takamichi@fujitsu.com
osumi.takamichi@fujitsu.com
In reply to: Ajin Cherian (#60)
RE: logical replication empty transactions

On Tuesday, January 11, 2022 6:43 PM Ajin Cherian <itsajin@gmail.com> wrote:

Minor update to rebase the patch so that it applies clean on HEAD.

Hi, thanks for you rebase.

Several comments.

(1) the commit message

"
transactions, keepalive messages are sent to keep the LSN locations updated
on the standby.
This patch does not skip empty transactions that are "streaming" or "two-phase".
"

I suggest that one blank line might be needed before the last paragraph.

(2) Could you please remove one pair of curly brackets for one sentence below ?

@@ -1546,10 +1557,13 @@ WalSndWaitForWal(XLogRecPtr loc)
                 * otherwise idle, this keepalive will trigger a reply. Processing the
                 * reply will update these MyWalSnd locations.
                 */
-               if (MyWalSnd->flush < sentPtr &&
+               if (force_keepalive_syncrep ||
+                       (MyWalSnd->flush < sentPtr &&
                        MyWalSnd->write < sentPtr &&
-                       !waiting_for_ping_response)
+                       !waiting_for_ping_response))
+               {
                        WalSndKeepalive(false);
+               }

(3) Is this patch's reponsibility to intialize the data in pgoutput_begin_prepare_txn ?

@@ -433,6 +487,8 @@ static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
        bool            send_replication_origin = txn->origin_id != InvalidRepOriginId;
+       PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+                                                                                                                sizeof(PGOutputTxnData));

OutputPluginPrepareWrite(ctx, !send_replication_origin);
logicalrep_write_begin_prepare(ctx->out, txn);

Even if we need this initialization for either non streaming case
or non two_phase case, there can be another issue.
We don't free the allocated memory for this data, right ?
There's only one place to use free in the entire patch,
which is in the pgoutput_commit_txn(). So,
corresponding free of memory looked necessary
in the two phase commit functions.

(4) SyncRepEnabled's better alignment.

IIUC, SyncRepEnabled is called not only by the walsender but also by other backends
via CommitTransaction -> RecordTransactionCommit -> SyncRepWaitForLSN.
Then, the place to add the prototype function for SyncRepEnabled seems not appropriate,
strictly speaking or requires a comment like /* called by wal sender or other backends */.

@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
/* called by wal sender */
extern void SyncRepInitConfig(void);
extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);

Even if we intend it is only used by the walsender, the current code place
of SyncRepEnabled in the syncrep.c might not be perfect.
In this file, seemingly we have a section for functions for wal sender processes
and the place where you wrote it is not here.

at src/backend/replication/syncrep.c, find a comment below.
/*
* ===========================================================
* Synchronous Replication functions for wal sender processes
* ===========================================================
*/

(5) minor alignment for expressing a couple of messages.

@@ -777,6 +846,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
Oid *relids;
TransactionId xid = InvalidTransactionId;

+       /* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+       Assert(in_streaming || txndata);

In the commit message, the way you write is below.
...
skip BEGIN / COMMIT messages for transactions that are empty. The patch
...

In this case, we have spaces back and forth for "BEGIN / COMMIT".
Then, I suggest to unify all of those to show better alignment.

Best Regards,
Takamichi Osumi

#62osumi.takamichi@fujitsu.com
osumi.takamichi@fujitsu.com
In reply to: Ajin Cherian (#60)
RE: logical replication empty transactions

On Tuesday, January 11, 2022 6:43 PM From: Ajin Cherian <itsajin@gmail.com> wrote:

Minor update to rebase the patch so that it applies clean on HEAD.

Hi, let me share some additional comments on v16.

(1) comment of pgoutput_change

@@ -630,11 +688,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
                                Relation relation, ReorderBufferChange *change)
 {
        PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+       PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
        MemoryContext old;
        RelationSyncEntry *relentry;
        TransactionId xid = InvalidTransactionId;
        Relation        ancestor = NULL;
+       /* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+       Assert(in_streaming || txndata);
+

In my humble opinion, the comment should not touch BEGIN PREPARE,
because this patch's scope doesn't include two phase commit.
(We could add this in another patch to extend the scope after the commit ?)

This applies to pgoutput_truncate's comment.

(2) "keep alive" should be "keepalive" in WalSndUpdateProgress

        /*
+        * When skipping empty transactions in synchronous replication, we need
+        * to send a keep alive to keep the MyWalSnd locations updated.
+        */
+       force_keepalive_syncrep = send_keepalive && SyncRepEnabled();
+

Also, this applies to the comment for force_keepalive_syncrep.

(3) Should finish the second sentence with period in the comment of pgoutput_message.

@@ -845,6 +923,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (in_streaming)
xid = txn->xid;

+       /*
+        * Output BEGIN if we haven't yet.
+        * Avoid for streaming and non-transactional messages

(4) "begin" can be changed to "BEGIN" in the comment of PGOutputTxnData definition.

In the entire patch, when we express BEGIN message,
we use capital letters "BEGIN" except for one place.
We can apply the same to this place as well.

+typedef struct PGOutputTxnData
+{
+       bool sent_begin_txn;    /* flag indicating whether begin has been sent */
+} PGOutputTxnData;
+

(5) inconsistent way to write Assert statements with blank lines

In the below case, it'd be better to insert one blank line
after the Assert();

+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
        bool            send_replication_origin = txn->origin_id != InvalidRepOriginId;
+       PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;

+ Assert(txndata);
OutputPluginPrepareWrite(ctx, !send_replication_origin);

(6) new codes in the pgoutput_commit_txn looks messy slightly

@@ -419,7 +455,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
                                        XLogRecPtr commit_lsn)
 {
-       OutputPluginUpdateProgress(ctx);
+       PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+       bool            skip;
+
+       Assert(txndata);
+
+       /*
+        * If a BEGIN message was not yet sent, then it means there were no relevant
+        * changes encountered, so we can skip the COMMIT message too.
+        */
+       skip = !txndata->sent_begin_txn;
+       pfree(txndata);
+       txn->output_plugin_private = NULL;
+       OutputPluginUpdateProgress(ctx, skip);

Could we conduct a refactoring for this new part ?
IMO, writing codes to free the data structure at the top
of function seems weird.

One idea is to export some part there
and write a new function, something like below.

static bool
txn_sent_begin(ReorderBufferTXN *txn)
{
PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
bool needs_skip;

Assert(txndata);

needs_skip = !txndata->sent_begin_txn;

pfree(txndata);
txn->output_plugin_private = NULL;

return needs_skip;
}

FYI, I had a look at the v12-0002-Skip-empty-prepared-transactions-for-logical-rep.patch
for reference of pgoutput_rollback_prepared_txn and pgoutput_commit_prepared_txn.
Looks this kind of function might work for future extensions as well.
What did you think ?

Best Regards,
Takamichi Osumi

#63Ajin Cherian
itsajin@gmail.com
In reply to: osumi.takamichi@fujitsu.com (#61)
1 attachment(s)
Re: logical replication empty transactions

On Wed, Jan 26, 2022 at 8:33 PM osumi.takamichi@fujitsu.com
<osumi.takamichi@fujitsu.com> wrote:

On Tuesday, January 11, 2022 6:43 PM Ajin Cherian <itsajin@gmail.com> wrote:

Minor update to rebase the patch so that it applies clean on HEAD.

Hi, thanks for you rebase.

Several comments.

(1) the commit message

"
transactions, keepalive messages are sent to keep the LSN locations updated
on the standby.
This patch does not skip empty transactions that are "streaming" or "two-phase".
"

I suggest that one blank line might be needed before the last paragraph.

Changed.

(2) Could you please remove one pair of curly brackets for one sentence below ?

@@ -1546,10 +1557,13 @@ WalSndWaitForWal(XLogRecPtr loc)
* otherwise idle, this keepalive will trigger a reply. Processing the
* reply will update these MyWalSnd locations.
*/
-               if (MyWalSnd->flush < sentPtr &&
+               if (force_keepalive_syncrep ||
+                       (MyWalSnd->flush < sentPtr &&
MyWalSnd->write < sentPtr &&
-                       !waiting_for_ping_response)
+                       !waiting_for_ping_response))
+               {
WalSndKeepalive(false);
+               }

Changed.

(3) Is this patch's reponsibility to intialize the data in pgoutput_begin_prepare_txn ?

@@ -433,6 +487,8 @@ static void
pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
{
bool            send_replication_origin = txn->origin_id != InvalidRepOriginId;
+       PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+                                                                                                                sizeof(PGOutputTxnData));

OutputPluginPrepareWrite(ctx, !send_replication_origin);
logicalrep_write_begin_prepare(ctx->out, txn);

Even if we need this initialization for either non streaming case
or non two_phase case, there can be another issue.
We don't free the allocated memory for this data, right ?
There's only one place to use free in the entire patch,
which is in the pgoutput_commit_txn(). So,
corresponding free of memory looked necessary
in the two phase commit functions.

Actually it is required for begin_prepare to set the data type, so
that the checks in the pgoutput_change can make sure that
the begin prepare is sent. I've also added a free in commit_prepared code.

(4) SyncRepEnabled's better alignment.

IIUC, SyncRepEnabled is called not only by the walsender but also by other backends
via CommitTransaction -> RecordTransactionCommit -> SyncRepWaitForLSN.
Then, the place to add the prototype function for SyncRepEnabled seems not appropriate,
strictly speaking or requires a comment like /* called by wal sender or other backends */.

@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
/* called by wal sender */
extern void SyncRepInitConfig(void);
extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);

Even if we intend it is only used by the walsender, the current code place
of SyncRepEnabled in the syncrep.c might not be perfect.
In this file, seemingly we have a section for functions for wal sender processes
and the place where you wrote it is not here.

at src/backend/replication/syncrep.c, find a comment below.
/*
* ===========================================================
* Synchronous Replication functions for wal sender processes
* ===========================================================
*/

Changed.

(5) minor alignment for expressing a couple of messages.

@@ -777,6 +846,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
Oid *relids;
TransactionId xid = InvalidTransactionId;

+       /* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+       Assert(in_streaming || txndata);

In the commit message, the way you write is below.
...
skip BEGIN / COMMIT messages for transactions that are empty. The patch
...

In this case, we have spaces back and forth for "BEGIN / COMMIT".
Then, I suggest to unify all of those to show better alignment.

fixed.

regards,
Ajin Cherian

Attachments:

v17-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v17-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 3e1457d9a5a05ec3619991a750b180dfdc9a481f Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Thu, 27 Jan 2022 06:29:44 -0500
Subject: [PATCH v17] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN/COMMIT messages for transactions that are empty. The patch
also makes sure that in synchronous replication mode, when skipping empty
transactions, keepalive messages are sent to keep the LSN locations updated
on the standby.

This patch does not skip empty transactions that are "streaming" or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 117 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 ++-
 src/backend/replication/walsender.c         |  22 ++++--
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 146 insertions(+), 21 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 9bc3a2d..fb1c26a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -672,12 +672,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index af8d51a..19073c4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -132,6 +132,17 @@ typedef struct RelationSyncEntry
 	TupleConversionMap *map;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+	bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -396,15 +407,41 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -419,7 +456,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, skip);
+
+	pfree(txndata);
+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -433,6 +488,8 @@ static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
@@ -441,6 +498,8 @@ pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
+	txndata->sent_begin_txn = true;
+	txn->output_plugin_private = txndata;
 }
 
 /*
@@ -450,7 +509,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -464,7 +523,18 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+
+	/*
+	 * A BEGIN PREPARE will always be sent for a prepared transaction.
+	 * A COMMIT PREPARED after a shutdown will not have the txndata,
+	 * so check before freeing.
+	 */
+	if (txndata)
+		pfree(txndata);
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -480,7 +550,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -630,11 +700,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
 	Relation	ancestor = NULL;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	if (!is_publishable_relation(relation))
 		return;
 
@@ -668,6 +742,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/*
+	 * Output BEGIN if we haven't yet, unless streaming.
+	 */
+	if (!in_streaming && !txndata->sent_begin_txn)
+		pgoutput_begin(ctx, txn);
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -770,6 +850,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -777,6 +858,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	Oid		   *relids;
 	TransactionId xid = InvalidTransactionId;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	/* Remember the xid for the change in streaming mode. See pgoutput_change. */
 	if (in_streaming)
 		xid = change->txn->xid;
@@ -813,6 +897,12 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		/*
+		 * output BEGIN if we haven't yet, unless streaming.
+		 */
+		if (!in_streaming && !txndata->sent_begin_txn)
+			pgoutput_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -845,6 +935,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages.
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		Assert(txndata);
+		if (!txndata->sent_begin_txn)
+			pgoutput_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1011,7 +1114,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1032,7 +1135,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index ce163b9..7a372da 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
 }
 
 /*
+ * Check if Synchronous Replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+    return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Calculate the synced Write, Flush and Apply positions among sync standbys.
  *
  * Return false if the number of sync standbys is less than
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 655760f..9a83fc1 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -170,6 +170,9 @@ static TimestampTz last_reply_timestamp = 0;
 /* Have we sent a heartbeat message asking for reply, since last reply? */
 static bool waiting_for_ping_response = false;
 
+/* Force keepalive when skipping transactions in synchronous replication mode */
+static bool force_keepalive_syncrep = false;
+
 /*
  * While streaming WAL in Copy mode, streamingDoneSending is set to true
  * after we have sent CopyDone. We should not send any more CopyData messages
@@ -248,7 +251,8 @@ static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1448,12 +1452,19 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keepalive to keep the MyWalSnd locations updated.
+	 */
+	force_keepalive_syncrep = send_keepalive && SyncRepEnabled();
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1546,10 +1557,11 @@ WalSndWaitForWal(XLogRecPtr loc)
 		 * otherwise idle, this keepalive will trigger a reply. Processing the
 		 * reply will update these MyWalSnd locations.
 		 */
-		if (MyWalSnd->flush < sentPtr &&
+		if (force_keepalive_syncrep ||
+			(MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
-			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			!waiting_for_ping_response))
+				WalSndKeepalive(false);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..9f59855 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keepalive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 41157fd..6f84614 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -243,6 +243,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 27be230..7086532 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index 9bb31ce..1d52e47 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 89249ec..8ac18db 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1607,6 +1607,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#64Ajin Cherian
itsajin@gmail.com
In reply to: osumi.takamichi@fujitsu.com (#62)
Re: logical replication empty transactions

On Thu, Jan 27, 2022 at 12:16 AM osumi.takamichi@fujitsu.com
<osumi.takamichi@fujitsu.com> wrote:

On Tuesday, January 11, 2022 6:43 PM From: Ajin Cherian <itsajin@gmail.com> wrote:

Minor update to rebase the patch so that it applies clean on HEAD.

Hi, let me share some additional comments on v16.

(1) comment of pgoutput_change

@@ -630,11 +688,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
Relation relation, ReorderBufferChange *change)
{
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+       PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
TransactionId xid = InvalidTransactionId;
Relation        ancestor = NULL;
+       /* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+       Assert(in_streaming || txndata);
+

In my humble opinion, the comment should not touch BEGIN PREPARE,
because this patch's scope doesn't include two phase commit.
(We could add this in another patch to extend the scope after the commit ?)

We have to include BEGIN PREPARE as well, as the txndata has to be
setup. Only difference is that we will not skip empty transaction in
BEGIN PREPARE

This applies to pgoutput_truncate's comment.

(2) "keep alive" should be "keepalive" in WalSndUpdateProgress

/*
+        * When skipping empty transactions in synchronous replication, we need
+        * to send a keep alive to keep the MyWalSnd locations updated.
+        */
+       force_keepalive_syncrep = send_keepalive && SyncRepEnabled();
+

Also, this applies to the comment for force_keepalive_syncrep.

Fixed.

(3) Should finish the second sentence with period in the comment of pgoutput_message.

@@ -845,6 +923,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (in_streaming)
xid = txn->xid;

+       /*
+        * Output BEGIN if we haven't yet.
+        * Avoid for streaming and non-transactional messages

Fixed.

(4) "begin" can be changed to "BEGIN" in the comment of PGOutputTxnData definition.

In the entire patch, when we express BEGIN message,
we use capital letters "BEGIN" except for one place.
We can apply the same to this place as well.

+typedef struct PGOutputTxnData
+{
+       bool sent_begin_txn;    /* flag indicating whether begin has been sent */
+} PGOutputTxnData;
+

Fixed.

(5) inconsistent way to write Assert statements with blank lines

In the below case, it'd be better to insert one blank line
after the Assert();

+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
bool            send_replication_origin = txn->origin_id != InvalidRepOriginId;
+       PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;

+ Assert(txndata);
OutputPluginPrepareWrite(ctx, !send_replication_origin);

Fixed.

(6) new codes in the pgoutput_commit_txn looks messy slightly

@@ -419,7 +455,25 @@ static void
pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
XLogRecPtr commit_lsn)
{
-       OutputPluginUpdateProgress(ctx);
+       PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+       bool            skip;
+
+       Assert(txndata);
+
+       /*
+        * If a BEGIN message was not yet sent, then it means there were no relevant
+        * changes encountered, so we can skip the COMMIT message too.
+        */
+       skip = !txndata->sent_begin_txn;
+       pfree(txndata);
+       txn->output_plugin_private = NULL;
+       OutputPluginUpdateProgress(ctx, skip);

Could we conduct a refactoring for this new part ?
IMO, writing codes to free the data structure at the top
of function seems weird.

One idea is to export some part there
and write a new function, something like below.

static bool
txn_sent_begin(ReorderBufferTXN *txn)
{
PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
bool needs_skip;

Assert(txndata);

needs_skip = !txndata->sent_begin_txn;

pfree(txndata);
txn->output_plugin_private = NULL;

return needs_skip;
}

FYI, I had a look at the v12-0002-Skip-empty-prepared-transactions-for-logical-rep.patch
for reference of pgoutput_rollback_prepared_txn and pgoutput_commit_prepared_txn.
Looks this kind of function might work for future extensions as well.
What did you think ?

I changed a bit, but I'd hold a comprehensive rewrite when a future
patch supports skipping
empty transactions in two-phase transactions and streaming transactions.

regards,
Ajin Cherian

#65osumi.takamichi@fujitsu.com
osumi.takamichi@fujitsu.com
In reply to: Ajin Cherian (#63)
RE: logical replication empty transactions

On Thursday, January 27, 2022 9:57 PM Ajin Cherian <itsajin@gmail.com> wrote:
Hi, thanks for your patch update.

On Wed, Jan 26, 2022 at 8:33 PM osumi.takamichi@fujitsu.com
<osumi.takamichi@fujitsu.com> wrote:

On Tuesday, January 11, 2022 6:43 PM Ajin Cherian <itsajin@gmail.com>

wrote:

(3) Is this patch's reponsibility to intialize the data in

pgoutput_begin_prepare_txn ?

@@ -433,6 +487,8 @@ static void
pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn) {
bool send_replication_origin = txn->origin_id !=

InvalidRepOriginId;

+ PGOutputTxnData *txndata =

MemoryContextAllocZero(ctx->context,

+
+ sizeof(PGOutputTxnData));

OutputPluginPrepareWrite(ctx, !send_replication_origin);
logicalrep_write_begin_prepare(ctx->out, txn);

Even if we need this initialization for either non streaming case or
non two_phase case, there can be another issue.
We don't free the allocated memory for this data, right ?
There's only one place to use free in the entire patch, which is in
the pgoutput_commit_txn(). So, corresponding free of memory looked
necessary in the two phase commit functions.

Actually it is required for begin_prepare to set the data type, so that the checks
in the pgoutput_change can make sure that the begin prepare is sent. I've also
added a free in commit_prepared code.

Okay, but if we choose the design that this patch takes
care of the initialization in pgoutput_begin_prepare_txn(),
we need another free in pgoutput_rollback_prepared_txn().
Could you please add some codes similar to pgoutput_commit_prepared_txn() to the same ?
If we simply execute rollback prepared for non streaming transaction,
we don't free it.

Some other new minor comments.

(a) can be "synchronous replication", instead of "Synchronous Replication"

When we have a look at the syncrep.c, we use the former usually in
a normal comment.

 /*
+ * Check if Synchronous Replication is enabled
+ */

(b) move below pgoutput_truncate two codes to the case where if nrelids > 0.

@@ -770,6 +850,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
                                  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
        PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+       PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
        MemoryContext old;
        RelationSyncEntry *relentry;
        int                     i;
@@ -777,6 +858,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
        Oid                *relids;
        TransactionId xid = InvalidTransactionId;
+       /* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+       Assert(in_streaming || txndata);
+

(c) fix indent with spaces (for the one sentence of SyncRepEnabled)

@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
}

 /*
+ * Check if Synchronous Replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+    return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*

This can be detected by git am.

Best Regards,
Takamichi Osumi

#66Ajin Cherian
itsajin@gmail.com
In reply to: osumi.takamichi@fujitsu.com (#65)
1 attachment(s)
Re: logical replication empty transactions

On Sun, Jan 30, 2022 at 7:04 PM osumi.takamichi@fujitsu.com
<osumi.takamichi@fujitsu.com> wrote:

On Thursday, January 27, 2022 9:57 PM Ajin Cherian <itsajin@gmail.com> wrote:
Hi, thanks for your patch update.

On Wed, Jan 26, 2022 at 8:33 PM osumi.takamichi@fujitsu.com
<osumi.takamichi@fujitsu.com> wrote:

On Tuesday, January 11, 2022 6:43 PM Ajin Cherian <itsajin@gmail.com>

wrote:

(3) Is this patch's reponsibility to intialize the data in

pgoutput_begin_prepare_txn ?

@@ -433,6 +487,8 @@ static void
pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn) {
bool send_replication_origin = txn->origin_id !=

InvalidRepOriginId;

+ PGOutputTxnData *txndata =

MemoryContextAllocZero(ctx->context,

+
+ sizeof(PGOutputTxnData));

OutputPluginPrepareWrite(ctx, !send_replication_origin);
logicalrep_write_begin_prepare(ctx->out, txn);

Even if we need this initialization for either non streaming case or
non two_phase case, there can be another issue.
We don't free the allocated memory for this data, right ?
There's only one place to use free in the entire patch, which is in
the pgoutput_commit_txn(). So, corresponding free of memory looked
necessary in the two phase commit functions.

Actually it is required for begin_prepare to set the data type, so that the checks
in the pgoutput_change can make sure that the begin prepare is sent. I've also
added a free in commit_prepared code.

Okay, but if we choose the design that this patch takes
care of the initialization in pgoutput_begin_prepare_txn(),
we need another free in pgoutput_rollback_prepared_txn().
Could you please add some codes similar to pgoutput_commit_prepared_txn() to the same ?
If we simply execute rollback prepared for non streaming transaction,
we don't free it.

Fixed.

Some other new minor comments.

(a) can be "synchronous replication", instead of "Synchronous Replication"

When we have a look at the syncrep.c, we use the former usually in
a normal comment.

/*
+ * Check if Synchronous Replication is enabled
+ */

Fixed.

(b) move below pgoutput_truncate two codes to the case where if nrelids > 0.

@@ -770,6 +850,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int nrelations, Relation relations[], ReorderBufferChange *change)
{
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+       PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
int                     i;
@@ -777,6 +858,9 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
Oid                *relids;
TransactionId xid = InvalidTransactionId;
+       /* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+       Assert(in_streaming || txndata);
+

Fixed.

(c) fix indent with spaces (for the one sentence of SyncRepEnabled)

@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
}

/*
+ * Check if Synchronous Replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+    return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*

This can be detected by git am.

Fixed.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v18-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v18-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 6809393fd0f5eeeaffa1fd20f6a4232256c353e5 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Thu, 27 Jan 2022 06:29:44 -0500
Subject: [PATCH v18] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN/COMMIT messages for transactions that are empty. The patch
also makes sure that in synchronous replication mode, when skipping empty
transactions, keepalive messages are sent to keep the LSN locations updated
on the standby.

This patch does not skip empty transactions that are "streaming" or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 128 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 ++-
 src/backend/replication/walsender.c         |  22 +++--
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 157 insertions(+), 21 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 9bc3a2d..fb1c26a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -672,12 +672,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index af8d51a..0aa326f 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -132,6 +132,17 @@ typedef struct RelationSyncEntry
 	TupleConversionMap *map;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+	bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -396,15 +407,41 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -419,7 +456,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, skip);
+
+	pfree(txndata);
+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -433,6 +488,8 @@ static void
 pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin_prepare(ctx->out, txn);
@@ -441,6 +498,8 @@ pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
+	txndata->sent_begin_txn = true;
+	txn->output_plugin_private = txndata;
 }
 
 /*
@@ -450,7 +509,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -464,7 +523,17 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	/*
+	 * A BEGIN PREPARE will always be sent for a prepared transaction.
+	 * A COMMIT PREPARED after a shutdown will not have the txndata,
+	 * so check before freeing.
+	 */
+	if (txndata)
+		pfree(txndata);
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -480,7 +549,17 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	/*
+	 * A BEGIN PREPARE will always be sent for a prepared transaction.
+	 * A ROLLBACK PREPARED after a shutdown will not have the txndata,
+	 * so check before freeing.
+	 */
+	if (txndata)
+		pfree(txndata);
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -630,11 +709,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
 	Relation	ancestor = NULL;
 
+	/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+	Assert(in_streaming || txndata);
+
 	if (!is_publishable_relation(relation))
 		return;
 
@@ -668,6 +751,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/*
+	 * Output BEGIN if we haven't yet, unless streaming.
+	 */
+	if (!in_streaming && !txndata->sent_begin_txn)
+		pgoutput_begin(ctx, txn);
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -770,6 +859,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -813,6 +903,17 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* If not streaming, should have setup txndata as part of BEGIN/BEGIN PREPARE */
+		Assert(in_streaming || txndata);
+
+		/*
+		 * output BEGIN if we haven't yet, unless streaming.
+		 */
+		if (!in_streaming && !txndata->sent_begin_txn)
+			pgoutput_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -845,6 +946,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages.
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		Assert(txndata);
+		if (!txndata->sent_begin_txn)
+			pgoutput_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1011,7 +1125,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1032,7 +1146,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index ce163b9..11f7358 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
 }
 
 /*
+ * Check if synchronous replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Calculate the synced Write, Flush and Apply positions among sync standbys.
  *
  * Return false if the number of sync standbys is less than
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 655760f..9a83fc1 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -170,6 +170,9 @@ static TimestampTz last_reply_timestamp = 0;
 /* Have we sent a heartbeat message asking for reply, since last reply? */
 static bool waiting_for_ping_response = false;
 
+/* Force keepalive when skipping transactions in synchronous replication mode */
+static bool force_keepalive_syncrep = false;
+
 /*
  * While streaming WAL in Copy mode, streamingDoneSending is set to true
  * after we have sent CopyDone. We should not send any more CopyData messages
@@ -248,7 +251,8 @@ static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1448,12 +1452,19 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keepalive to keep the MyWalSnd locations updated.
+	 */
+	force_keepalive_syncrep = send_keepalive && SyncRepEnabled();
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1546,10 +1557,11 @@ WalSndWaitForWal(XLogRecPtr loc)
 		 * otherwise idle, this keepalive will trigger a reply. Processing the
 		 * reply will update these MyWalSnd locations.
 		 */
-		if (MyWalSnd->flush < sentPtr &&
+		if (force_keepalive_syncrep ||
+			(MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
-			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			!waiting_for_ping_response))
+				WalSndKeepalive(false);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..9f59855 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keepalive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 41157fd..6f84614 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -243,6 +243,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 27be230..7086532 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index 9bb31ce..1d52e47 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 89249ec..8ac18db 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1607,6 +1607,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#67osumi.takamichi@fujitsu.com
osumi.takamichi@fujitsu.com
In reply to: Ajin Cherian (#66)
RE: logical replication empty transactions

Hi,

Thank you for your updating the patch.

I'll quote one of the past discussions
in order to make this thread go forward or more active.
On Friday, August 13, 2021 8:01 PM Ajin Cherian <itsajin@gmail.com> wrote:

On Mon, Aug 2, 2021 at 7:20 PM Amit Kapila <amit.kapila16@gmail.com>
wrote:

On Fri, Jul 23, 2021 at 3:39 PM Ajin Cherian <itsajin@gmail.com> wrote:

Let's first split the patch for prepared and non-prepared cases as
that will help to focus on each of them separately. BTW, why haven't
you considered implementing point 1b as explained by Andres in his
email [1]? I think we can send a keepalive message in case of
synchronous replication when we skip an empty transaction, otherwise,
it might delay in responding to transactions synchronous_commit mode.
I think in the tests done in the thread, it might not have been shown
because we are already sending keepalives too frequently. But what if
someone disables wal_sender_timeout or kept it to a very large value?
See WalSndKeepaliveIfNecessary. The other thing you might want to look
at is if the reason for frequent keepalives is the same as described
in the email [2].

I have tried to address the comment here by modifying the
ctx->update_progress callback function (WalSndUpdateProgress) provided
for plugins. I have added an option
by which the callback can specify if it wants to send keep_alives. And when
the callback is called with that option set, walsender updates a flag
force_keep_alive_syncrep.
The Walsender in the WalSndWaitForWal for loop, checks this flag and if
synchronous replication is enabled, then sends a keep alive.
Currently this logic
is added as an else to the current logic that is already there in
WalSndWaitForWal, which is probably considered unnecessary and a source of
the keep alive flood that you talked about. So, I can change that according to
how that fix shapes up there. I have also added an extern function in syncrep.c
that makes it possible for walsender to query if synchronous replication is
turned on.

Changing the timing to send the keepalive to the decoding commit
timing didn't look impossible to me, although my suggestion
can be ad-hoc.

After the initialization of sentPtr(by confirmed_flush lsn),
sentPtr is updated from logical_decoding_ctx->reader->EndRecPtr in XLogSendLogical.
In the XLogSendLogical, we update it after we execute LogicalDecodingProcessRecord.
This order leads to the current implementation to wait the next iteration
to send a keepalive in WalSndWaitForWal.

But, I felt we can utilize end_lsn passed to ReorderBufferCommit for updating
sentPtr. The end_lsn is the lsn same as the ctx->reader->EndRecPtr,
which means advancing the timing to update the sentPtr for the commit case.
Then if the transaction is empty in synchronous mode,
send the keepalive in WalSndUpdateProgress directly,
instead of having the force_keepalive_syncrep flag and having it true.

Best Regards,
Takamichi Osumi

#68osumi.takamichi@fujitsu.com
osumi.takamichi@fujitsu.com
In reply to: Ajin Cherian (#66)
RE: logical replication empty transactions

Hi

I'll quote one other remaining discussion of this thread again
to invoke more attentions from the community.
On Friday, August 13, 2021 8:01 PM Ajin Cherian <itsajin@gmail.com> wrote:

On Mon, Aug 2, 2021 at 7:20 PM Amit Kapila <amit.kapila16@gmail.com>
wrote:

Few other miscellaneous comments:
1.
static void
pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
- XLogRecPtr commit_lsn)
+ XLogRecPtr commit_lsn, XLogRecPtr prepare_end_lsn, TimestampTz
+ prepare_time)
{
+ PGOutputTxnData    *txndata = (PGOutputTxnData *)

txn->output_plugin_private;

+
OutputPluginUpdateProgress(ctx);

+ /*
+ * If the BEGIN PREPARE was not yet sent, then it means there were no
+ * relevant changes encountered, so we can skip the COMMIT PREPARED
+ * message too.
+ */
+ if (txndata)
+ {
+ bool skip = !txndata->sent_begin_txn; pfree(txndata);
+ txn->output_plugin_private = NULL;

How is this supposed to work after the restart when prepared is sent
before the restart and we are just sending commit_prepared after
restart? Won't this lead to sending commit_prepared even when the
corresponding prepare is not sent? Can we think of a better way to
deal with this?

I have tried to resolve this by adding logic in worker,c to silently ignore spurious
commit_prepareds. But this change required checking if the prepare exists on
the subscriber before attempting the commit_prepared but the current API that
checks this requires prepare time and transaction end_lsn. But for this I had to
change the protocol of commit_prepared, and I understand that this would
break backward compatibility between subscriber and publisher (you have
raised this issue as well).
I am not sure how else to handle this, let me know if you have any other ideas.

I feel if we don't want to change the protocol of commit_prepared,
we need to make the publisher solely judge whether the prepare was empty or not,
after the restart.

One idea I thought at the beginning was to utilize and apply
the existing mechanism to spill ReorderBufferSerializeTXN object to local disk,
by postponing the prepare txn object cleanup and when the walsender exits
and commit prepared didn't come, spilling the transaction's data,
then restoring it after the restart in the DecodePrepare.
However, this idea wasn't crash-safe fundamentally. It means,
if the publisher crashes before spilling the empty prepare transaction,
we fail to detect the prepare was empty and come down to send the commit_prepared
in the situation where the subscriber didn't get the prepare data again.
So, I thought to utilize the spill mechanism didn't work for this purpose.

Another idea would be, to create an empty file under the the pg_replslot/slotname
with a prefix different from "xid" in the DecodePrepare before the shutdown
if the prepare was empty, and bypass the cleanup of the serialized txns
and check the existence after the restart. But, this is pretty ad-hoc and I wasn't sure
if to address the corner case of the restart has the strong enough justification
to create this new file format.

Therefore, in my humble opinion, the idea of protocol change slightly wins,
since the impact of the protocol change would not be big. We introduced
the protocol version 3 in the devel version and the number of users should be little.

Best Regards,
Takamichi Osumi

#69Amit Kapila
amit.kapila16@gmail.com
In reply to: Ajin Cherian (#66)
Re: logical replication empty transactions

On Mon, Jan 31, 2022 at 6:18 PM Ajin Cherian <itsajin@gmail.com> wrote:

Few comments:
=============
1. Is there any particular why the patch is not skipping empty xacts
for streaming (in-progress) transactions as noted in the commit
message as well?

2.
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
  bool send_replication_origin = txn->origin_id != InvalidRepOriginId;
+ PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+ Assert(txndata);

I think here you can add an assert for sent_begin_txn to be always false?

3.
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */

Have an empty line between the first two lines to ensure consistency
with nearby comments. Also, the formatting of these lines appears
awkward, either run pgindent or make sure lines are not too short.

4. Do we really need to make any changes in PREPARE
transaction-related functions if can't skip in that case? I think you
can have a check if the output plugin private variable is not set then
ignore special optimization for sending begin.

--
With Regards,
Amit Kapila.

#70Amit Kapila
amit.kapila16@gmail.com
In reply to: osumi.takamichi@fujitsu.com (#68)
Re: logical replication empty transactions

On Wed, Feb 16, 2022 at 8:45 AM osumi.takamichi@fujitsu.com
<osumi.takamichi@fujitsu.com> wrote:

[ideas to skip empty prepare/commit_prepare ....]

I feel if we don't want to change the protocol of commit_prepared,
we need to make the publisher solely judge whether the prepare was empty or not,
after the restart.

One idea I thought at the beginning was to utilize and apply
the existing mechanism to spill ReorderBufferSerializeTXN object to local disk,
by postponing the prepare txn object cleanup and when the walsender exits
and commit prepared didn't come, spilling the transaction's data,
then restoring it after the restart in the DecodePrepare.
However, this idea wasn't crash-safe fundamentally. It means,
if the publisher crashes before spilling the empty prepare transaction,
we fail to detect the prepare was empty and come down to send the commit_prepared
in the situation where the subscriber didn't get the prepare data again.
So, I thought to utilize the spill mechanism didn't work for this purpose.

Another idea would be, to create an empty file under the the pg_replslot/slotname
with a prefix different from "xid" in the DecodePrepare before the shutdown
if the prepare was empty, and bypass the cleanup of the serialized txns
and check the existence after the restart. But, this is pretty ad-hoc and I wasn't sure
if to address the corner case of the restart has the strong enough justification
to create this new file format.

I think for this idea to work you need to create such an empty file
each time we skip empty prepare as the system might crash after
prepare and we won't get time to create such a file. I don't think it
is advisable to do I/O to save the network message.

Therefore, in my humble opinion, the idea of protocol change slightly wins,
since the impact of the protocol change would not be big. We introduced
the protocol version 3 in the devel version and the number of users should be little.

There is also the cost of the additional check (whether prepared xact
exists) at the time of processing each commit prepared message. I
think if we want to go in this direction then it is better to do it
via a subscription parameter (say skip_empty_prepare_xact or something
like that) so that we can pay the additional cost of such a check
conditionally when such a parameter is set by the user. I feel for now
we can document in comments why we can't skip empty prepared
transactions and maybe as an idea(s) worth exploring to implement the
same. OTOH, if multiple agree on such a solution we can even try to
implement it and see if that works.

--
With Regards,
Amit Kapila.

#71Ajin Cherian
itsajin@gmail.com
In reply to: osumi.takamichi@fujitsu.com (#68)
Re: logical replication empty transactions

On Wed, Feb 16, 2022 at 2:15 PM osumi.takamichi@fujitsu.com
<osumi.takamichi@fujitsu.com> wrote:

Another idea would be, to create an empty file under the the pg_replslot/slotname
with a prefix different from "xid" in the DecodePrepare before the shutdown
if the prepare was empty, and bypass the cleanup of the serialized txns
and check the existence after the restart. But, this is pretty ad-hoc and I wasn't sure
if to address the corner case of the restart has the strong enough justification
to create this new file format.

Yes, this doesn't look very efficient.

Therefore, in my humble opinion, the idea of protocol change slightly wins,
since the impact of the protocol change would not be big. We introduced
the protocol version 3 in the devel version and the number of users should be little.

Yes, but we don't want to break backward compatibility for this small
added optimization.

Amit,

I will work on your comments.

regards,
Ajin Cherian
Fujitsu Australia

#72Amit Kapila
amit.kapila16@gmail.com
In reply to: Amit Kapila (#69)
Re: logical replication empty transactions

On Thu, Feb 17, 2022 at 4:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Mon, Jan 31, 2022 at 6:18 PM Ajin Cherian <itsajin@gmail.com> wrote:

Few comments:
=============

One more comment:
@@ -1546,10 +1557,11 @@ WalSndWaitForWal(XLogRecPtr loc)
  * otherwise idle, this keepalive will trigger a reply. Processing the
  * reply will update these MyWalSnd locations.
  */
- if (MyWalSnd->flush < sentPtr &&
+ if (force_keepalive_syncrep ||
+ (MyWalSnd->flush < sentPtr &&
  MyWalSnd->write < sentPtr &&
- !waiting_for_ping_response)
- WalSndKeepalive(false);
+ !waiting_for_ping_response))
+ WalSndKeepalive(false);

Will this allow syncrep to proceed in case we are skipping the
transaction? Won't we need to send a feedback message with
'requestReply' true in this case as we release syncrep waiters while
processing standby message, see
ProcessStandbyReplyMessage->SyncRepReleaseWaiters. Without
'requestReply', the subscriber might not send any message and the
syncrep won't proceed. Why do you decide to delay sending this message
till WalSndWaitForWal()? It may not be called for each transaction.

I feel we should try to device a test case to test this sync
replication mechanism such that without this particular change the
sync rep transaction waits momentarily but with this change it doesn't
wait. I am not entirely sure whether we can devise an automated test
as this is timing related issue but I guess we can at least manually
try to produce a case.

--
With Regards,
Amit Kapila.

#73Amit Kapila
amit.kapila16@gmail.com
In reply to: osumi.takamichi@fujitsu.com (#67)
Re: logical replication empty transactions

On Tue, Feb 8, 2022 at 5:27 AM osumi.takamichi@fujitsu.com
<osumi.takamichi@fujitsu.com> wrote:

On Friday, August 13, 2021 8:01 PM Ajin Cherian <itsajin@gmail.com> wrote:

On Mon, Aug 2, 2021 at 7:20 PM Amit Kapila <amit.kapila16@gmail.com>
wrote:

Changing the timing to send the keepalive to the decoding commit
timing didn't look impossible to me, although my suggestion
can be ad-hoc.

After the initialization of sentPtr(by confirmed_flush lsn),
sentPtr is updated from logical_decoding_ctx->reader->EndRecPtr in XLogSendLogical.
In the XLogSendLogical, we update it after we execute LogicalDecodingProcessRecord.
This order leads to the current implementation to wait the next iteration
to send a keepalive in WalSndWaitForWal.

But, I felt we can utilize end_lsn passed to ReorderBufferCommit for updating
sentPtr. The end_lsn is the lsn same as the ctx->reader->EndRecPtr,
which means advancing the timing to update the sentPtr for the commit case.
Then if the transaction is empty in synchronous mode,
send the keepalive in WalSndUpdateProgress directly,
instead of having the force_keepalive_syncrep flag and having it true.

You have a point in that we don't need to delay sending this message
till next WalSndWaitForWal() but I don't see why we need to change
anything about update of sentPtr.

--
With Regards,
Amit Kapila.

#74osumi.takamichi@fujitsu.com
osumi.takamichi@fujitsu.com
In reply to: Amit Kapila (#73)
RE: logical replication empty transactions

On Friday, February 18, 2022 6:18 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Tue, Feb 8, 2022 at 5:27 AM osumi.takamichi@fujitsu.com
<osumi.takamichi@fujitsu.com> wrote:

On Friday, August 13, 2021 8:01 PM Ajin Cherian <itsajin@gmail.com> wrote:

On Mon, Aug 2, 2021 at 7:20 PM Amit Kapila <amit.kapila16@gmail.com>
wrote:

Changing the timing to send the keepalive to the decoding commit
timing didn't look impossible to me, although my suggestion can be
ad-hoc.

After the initialization of sentPtr(by confirmed_flush lsn), sentPtr
is updated from logical_decoding_ctx->reader->EndRecPtr in

XLogSendLogical.

In the XLogSendLogical, we update it after we execute

LogicalDecodingProcessRecord.

This order leads to the current implementation to wait the next
iteration to send a keepalive in WalSndWaitForWal.

But, I felt we can utilize end_lsn passed to ReorderBufferCommit for
updating sentPtr. The end_lsn is the lsn same as the
ctx->reader->EndRecPtr, which means advancing the timing to update the

sentPtr for the commit case.

Then if the transaction is empty in synchronous mode, send the
keepalive in WalSndUpdateProgress directly, instead of having the
force_keepalive_syncrep flag and having it true.

You have a point in that we don't need to delay sending this message till next
WalSndWaitForWal() but I don't see why we need to change anything about
update of sentPtr.

Yeah, you're right.
Now I think we don't need the update of sentPtr to send a keepalive.

I thought we can send a keepalive message
after its update in XLogSendLogical or any appropriate place for it after the existing update.

Best Regards,
Takamichi Osumi

#75Amit Kapila
amit.kapila16@gmail.com
In reply to: osumi.takamichi@fujitsu.com (#74)
Re: logical replication empty transactions

On Fri, Feb 18, 2022 at 3:06 PM osumi.takamichi@fujitsu.com
<osumi.takamichi@fujitsu.com> wrote:

On Friday, February 18, 2022 6:18 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Tue, Feb 8, 2022 at 5:27 AM osumi.takamichi@fujitsu.com
<osumi.takamichi@fujitsu.com> wrote:

On Friday, August 13, 2021 8:01 PM Ajin Cherian <itsajin@gmail.com> wrote:

On Mon, Aug 2, 2021 at 7:20 PM Amit Kapila <amit.kapila16@gmail.com>
wrote:

Changing the timing to send the keepalive to the decoding commit
timing didn't look impossible to me, although my suggestion can be
ad-hoc.

After the initialization of sentPtr(by confirmed_flush lsn), sentPtr
is updated from logical_decoding_ctx->reader->EndRecPtr in

XLogSendLogical.

In the XLogSendLogical, we update it after we execute

LogicalDecodingProcessRecord.

This order leads to the current implementation to wait the next
iteration to send a keepalive in WalSndWaitForWal.

But, I felt we can utilize end_lsn passed to ReorderBufferCommit for
updating sentPtr. The end_lsn is the lsn same as the
ctx->reader->EndRecPtr, which means advancing the timing to update the

sentPtr for the commit case.

Then if the transaction is empty in synchronous mode, send the
keepalive in WalSndUpdateProgress directly, instead of having the
force_keepalive_syncrep flag and having it true.

You have a point in that we don't need to delay sending this message till next
WalSndWaitForWal() but I don't see why we need to change anything about
update of sentPtr.

Yeah, you're right.
Now I think we don't need the update of sentPtr to send a keepalive.

I thought we can send a keepalive message
after its update in XLogSendLogical or any appropriate place for it after the existing update.

Yeah, I think there could be multiple ways (a) We can send such a keep
alive in WalSndUpdateProgress() itself by using ctx->write_location.
For this, we need to modify WalSndKeepalive() to take sentPtr as
input. (b) set some flag in WalSndUpdateProgress() and then do it
somewhere in WalSndLoop probably in WalSndKeepaliveIfNecessary, or
maybe there is another better way.

--
With Regards,
Amit Kapila.

#76Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#66)
Re: logical replication empty transactions

FYI - the latest v18 patch no longer applies due to a recent push [1]https://github.com/postgres/postgres/commit/52e4f0cd472d39d07732b99559989ea3b615be78.

------
[1]: https://github.com/postgres/postgres/commit/52e4f0cd472d39d07732b99559989ea3b615be78

Kind Regards,
Peter Smith.
Fujitsu Australia

#77Ajin Cherian
itsajin@gmail.com
In reply to: Amit Kapila (#69)
1 attachment(s)
Re: logical replication empty transactions

On Thu, Feb 17, 2022 at 9:42 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Mon, Jan 31, 2022 at 6:18 PM Ajin Cherian <itsajin@gmail.com> wrote:

Few comments:
=============
1. Is there any particular why the patch is not skipping empty xacts
for streaming (in-progress) transactions as noted in the commit
message as well?

I have added support for skipping streaming transaction.

2.
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
bool send_replication_origin = txn->origin_id != InvalidRepOriginId;
+ PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+ Assert(txndata);

I think here you can add an assert for sent_begin_txn to be always false?

Added.

3.
+/*
+ * Send BEGIN.
+ * This is where the BEGIN is actually sent. This is called
+ * while processing the first change of the transaction.
+ */

Have an empty line between the first two lines to ensure consistency
with nearby comments. Also, the formatting of these lines appears
awkward, either run pgindent or make sure lines are not too short.

Changed.

4. Do we really need to make any changes in PREPARE
transaction-related functions if can't skip in that case? I think you
can have a check if the output plugin private variable is not set then
ignore special optimization for sending begin.

I have modified this as well.

I have also rebased the patch after it did not apply due to a new commit.

I will next work on testing and improving the keepalive logic while
skipping transactions.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v19-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v19-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 341359a89730a57d5087207df52ff754e45b9881 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Tue, 22 Feb 2022 07:47:22 -0500
Subject: [PATCH v19] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN/COMMIT messages for transactions that are empty. The patch
also makes sure that in synchronous replication mode, when skipping empty
transactions, keepalive messages are sent to keep the LSN locations updated
on the standby.

This patch does not skip empty transactions that are "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 230 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 +-
 src/backend/replication/walsender.c         |  22 ++-
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 254 insertions(+), 26 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..99b2775 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a04..4a530ac 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -67,6 +67,8 @@ static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   TimestampTz prepare_time);
 static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 								  ReorderBufferTXN *txn);
+static void pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+									   ReorderBufferTXN *txn);
 static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn);
 static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
@@ -166,6 +168,19 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+   bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+   bool sent_stream_start; /* flag indicating if stream start has been sent */
+   bool sent_any_stream;   /* flag indicating if any stream has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,15 +467,43 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -475,7 +518,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, skip);
+
+	pfree(txndata);
+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -506,7 +567,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +581,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +597,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -556,6 +617,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	bool		schema_sent;
 	TransactionId xid = InvalidTransactionId;
 	TransactionId topxid = InvalidTransactionId;
+	PGOutputTxnData *txndata;
+	ReorderBufferTXN *toptxn;
 
 	/*
 	 * Remember XID of the (sub)transaction for the change. We don't care if
@@ -569,9 +632,15 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 		xid = change->txn->xid;
 
 	if (change->txn->toptxn)
+	{
 		topxid = change->txn->toptxn->xid;
+		toptxn = change->txn->toptxn;
+	}
 	else
+	{
 		topxid = xid;
+		toptxn = change->txn;
+	}
 
 	/*
 	 * Do we need to send the schema? We do track streamed transactions
@@ -594,6 +663,20 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	if (schema_sent)
 		return;
 
+   /* set up txndata */
+   txndata = toptxn->output_plugin_private;
+
+   /*
+    * Before we send schema, make sure that STREAM START/BEGIN/BEGIN PREPARE
+    * is sent. If not, send now.
+    */
+   if (in_streaming && !txndata->sent_stream_start)
+       pgoutput_send_stream_start(ctx, toptxn);
+   else if (txndata && !txndata->sent_begin_txn)
+   {
+       pgoutput_begin(ctx, toptxn);
+   }
+
 	/*
 	 * Send the schema.  If the changes will be published using an ancestor's
 	 * schema, not the relation's own, send that ancestor's schema before
@@ -1141,6 +1224,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1183,6 +1267,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/* If streaming, send STREAM START if we haven't yet */
+	if (in_streaming && (txndata && !txndata->sent_stream_start))
+		pgoutput_send_stream_start(ctx, txn);
+	/*
+	 * Output BEGIN if we haven't yet, unless streaming.
+	 */
+	else if (!in_streaming && (txndata && !txndata->sent_begin_txn))
+		pgoutput_begin(ctx, txn);
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -1354,6 +1447,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1397,6 +1491,17 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* If streaming, send STREAM START if we haven't yet */
+		if (in_streaming && (txndata && !txndata->sent_stream_start))
+			pgoutput_send_stream_start(ctx, txn);
+		/*
+		 * output BEGIN if we haven't yet, unless streaming.
+		 */
+		else if (!in_streaming && (txndata && !txndata->sent_begin_txn))
+			pgoutput_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -1429,6 +1534,24 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages.
+	 */
+	if (in_streaming || transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* If streaming, send STREAM START if we haven't yet */
+		if (in_streaming && (txndata && !txndata->sent_stream_start))
+			pgoutput_send_stream_start(ctx, txn);
+		else if (transactional)
+		{
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_begin(ctx, txn);
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1511,28 +1634,62 @@ static void
 pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 					  ReorderBufferTXN *txn)
 {
-	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = txn->output_plugin_private;
 
 	/* we can't nest streaming of transactions */
 	Assert(!in_streaming);
 
 	/*
+	 * Don't actually send stream start here, instead set a flag that indicates
+	 * that stream start hasn't been sent and wait for the first actual change
+	 * for this stream to be sent and then send stream start. This is done
+	 * to avoid sending empty streams without any changes.
+	 */
+	if (txndata == NULL)
+	{
+		txndata =
+			MemoryContextAllocZero(ctx->context, sizeof(PGOutputTxnData));
+		txndata->sent_begin_txn = false;
+		txndata->sent_any_stream = false;
+		txn->output_plugin_private = txndata;
+	}
+
+	txndata->sent_stream_start = false;
+	in_streaming = true;
+}
+
+/*
+ * Actually send START STREAM
+ */
+static void
+pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+					  ReorderBufferTXN *txn)
+{
+	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+
+	/*
 	 * If we already sent the first stream for this transaction then don't
 	 * send the origin id in the subsequent streams.
 	 */
-	if (rbtxn_is_streamed(txn))
+	if (txndata->sent_any_stream)
 		send_replication_origin = false;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
-	logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+	logicalrep_write_stream_start(ctx->out, txn->xid, !txndata->sent_any_stream);
 
 	send_repl_origin(ctx, txn->origin_id, InvalidXLogRecPtr,
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
 
-	/* we're streaming a chunk of transaction now */
-	in_streaming = true;
+	/*
+	 * Set the flags that indicate that changes were sent as part of
+	 * the transaction and the stream.
+	 */
+	txndata->sent_begin_txn = txndata->sent_stream_start = true;
+	txndata->sent_any_stream = true;
 }
 
 /*
@@ -1542,9 +1699,18 @@ static void
 pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 					 ReorderBufferTXN *txn)
 {
+	PGOutputTxnData *data = txn->output_plugin_private;
+
 	/* we should be streaming a trasanction */
 	Assert(in_streaming);
 
+	if (!data->sent_stream_start)
+	{
+		in_streaming = false;
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream stop");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_stop(ctx->out);
 	OutputPluginWrite(ctx, true);
@@ -1563,6 +1729,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 					  XLogRecPtr abort_lsn)
 {
 	ReorderBufferTXN *toptxn;
+	PGOutputTxnData  *txndata;
+	bool sent_begin_txn;
 
 	/*
 	 * The abort should happen outside streaming block, even for streamed
@@ -1572,6 +1740,20 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 
 	/* determine the toplevel transaction */
 	toptxn = (txn->toptxn) ? txn->toptxn : txn;
+	txndata = toptxn->output_plugin_private;
+	sent_begin_txn = txndata->sent_begin_txn;
+
+	if (txn->toptxn == NULL)
+	{
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+	}
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream abort");
+		return;
+	}
 
 	Assert(rbtxn_is_streamed(toptxn));
 
@@ -1591,6 +1773,8 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 					   ReorderBufferTXN *txn,
 					   XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_begin_txn = txndata->sent_begin_txn;
 	/*
 	 * The commit should happen outside streaming block, even for streamed
 	 * transactions. The transaction has to be marked as streamed, though.
@@ -1598,7 +1782,17 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	/* If no changes were part of this transaction then drop the commit */
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream commit");
+		return;
+	}
+
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1617,9 +1811,21 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 							ReorderBufferTXN *txn,
 							XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_begin_txn = txndata->sent_begin_txn;
+
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream prepare");
+		return;
+	}
+
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index ce163b9..11f7358 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
 }
 
 /*
+ * Check if synchronous replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Calculate the synced Write, Flush and Apply positions among sync standbys.
  *
  * Return false if the number of sync standbys is less than
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 5a718b1..3024103 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -171,6 +171,9 @@ static TimestampTz last_reply_timestamp = 0;
 /* Have we sent a heartbeat message asking for reply, since last reply? */
 static bool waiting_for_ping_response = false;
 
+/* Force keepalive when skipping transactions in synchronous replication mode */
+static bool force_keepalive_syncrep = false;
+
 /*
  * While streaming WAL in Copy mode, streamingDoneSending is set to true
  * after we have sent CopyDone. We should not send any more CopyData messages
@@ -249,7 +252,8 @@ static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,12 +1453,19 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keepalive to keep the MyWalSnd locations updated.
+	 */
+	force_keepalive_syncrep = send_keepalive && SyncRepEnabled();
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1547,10 +1558,11 @@ WalSndWaitForWal(XLogRecPtr loc)
 		 * otherwise idle, this keepalive will trigger a reply. Processing the
 		 * reply will update these MyWalSnd locations.
 		 */
-		if (MyWalSnd->flush < sentPtr &&
+		if (force_keepalive_syncrep ||
+			(MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
-			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			!waiting_for_ping_response))
+				WalSndKeepalive(false);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..9f59855 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keepalive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..eb91d17 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 27be230..7086532 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index c6b302c..56761ab 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1608,6 +1608,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#78wangw.fnst@fujitsu.com
wangw.fnst@fujitsu.com
In reply to: Ajin Cherian (#77)
RE: logical replication empty transactions

On Feb, Wed 23, 2022 at 10:58 PM Ajin Cherian <itsajin@gmail.com> wrote:

Few comments to V19-0001:

1. I think we should adjust the alignment format.
git am ../v19-0001-Skip-empty-transactions-for-logical-replication.patch
.git/rebase-apply/patch:197: indent with spaces.
* Before we send schema, make sure that STREAM START/BEGIN/BEGIN PREPARE
.git/rebase-apply/patch:198: indent with spaces.
* is sent. If not, send now.
.git/rebase-apply/patch:199: indent with spaces.
*/
.git/rebase-apply/patch:201: indent with spaces.
pgoutput_send_stream_start(ctx, toptxn);
.git/rebase-apply/patch:204: indent with spaces.
pgoutput_begin(ctx, toptxn);
warning: 5 lines add whitespace errors.

2. Structure member initialization.
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
Do we need to set sent_stream_start and sent_any_stream to false here?

3. Maybe we should add Assert(txndata) like function pgoutput_commit_txn in
other functions.

4. In addition, I think we should keep a unified style.
a). log style (maybe first one is better.)
First style : "Skipping replication of an empty transaction in XXX"
Second style : "skipping replication of an empty transaction"
b) flag name (maybe second one is better.)
First style : variable "sent_begin_txn" in function pgoutput_stream_*.
Second style : variable "skip" in function pgoutput_commit_txn.

Regards,
Wang wei

#79Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#77)
Re: logical replication empty transactions

Hi. Here are my review comments for the v19 patch.

======

1. Commit message

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications).

SUGGESTION
"to subscriber even though" --> "to the subscriber even if"

~~~

2. Commit message

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN/COMMIT messages for transactions that are empty.

SUGGESTION
"if there is" --> "if there was"
"do not send COMMIT message" --> "do not send the COMMIT message"
"It means that pgoutput" --> "This means that pgoutput"

~~~

3. Commit message

Shouldn't there be some similar description about using a lazy send
mechanism for STREAM START?

~~~

4. src/backend/replication/pgoutput/pgoutput.c - typedef struct PGOutputTxnData

+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+   bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+   bool sent_stream_start; /* flag indicating if stream start has been sent */
+   bool sent_any_stream;   /* flag indicating if any stream has been sent */
+} PGOutputTxnData;
+

The struct comment looks stale because it doesn't mention anything
about the similar lazy send mechanism for STREAM_START.

~~~

5. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin_txn

 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+ PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+ sizeof(PGOutputTxnData));
+
+ txndata->sent_begin_txn = false;
+ txn->output_plugin_private = txndata;
+}

You don’t need to assign the other members 'sent_stream_start',
'sent_any_stream' because you are doing MemoryContextAllocZero anyway,
but for the same reason you did not really need to assign the
'sent_begin_txn' flag either.

I guess for consistency maybe it is better to (a) set all of them or
(b) set none of them. I prefer (b).

~~~

6. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin

I feel the 'pgoutput_begin' function is not well named. It makes some
of the code where they are called look quite confusing.

For streaming there is:
1. pgoutput_stream_start (does not send)
2. pgoutput_send_stream_start (does send)
so it is very clear.

OTOH there are
3. pgoutput_begin_txn (does not send)
4. pgoutput_begin (does send)

For consistency I think the 'pgoutput_begin' name should be changed to
include "send" verb
1. pgoutput_begin_txn (does not send)
2. pgoutput_send_begin_txn (does send)

~~~

7. src/backend/replication/pgoutput/pgoutput.c - maybe_send_schema

@@ -594,6 +663,20 @@ maybe_send_schema(LogicalDecodingContext *ctx,
if (schema_sent)
return;

+   /* set up txndata */
+   txndata = toptxn->output_plugin_private;
+
+   /*
+    * Before we send schema, make sure that STREAM START/BEGIN/BEGIN PREPARE
+    * is sent. If not, send now.
+    */
+   if (in_streaming && !txndata->sent_stream_start)
+       pgoutput_send_stream_start(ctx, toptxn);
+   else if (txndata && !txndata->sent_begin_txn)
+   {
+       pgoutput_begin(ctx, toptxn);
+   }
+

How come the in_streaming case is not checking for a NULL txndata
before referencing it? Even if it is OK to do that, some more comments
or assertions might help for this piece of code.
(Stop-Press: see later comments #9, #10)

~~~

8. src/backend/replication/pgoutput/pgoutput.c - maybe_send_schema

@@ -594,6 +663,20 @@ maybe_send_schema(LogicalDecodingContext *ctx,
if (schema_sent)
return;

+   /* set up txndata */
+   txndata = toptxn->output_plugin_private;
+
+   /*
+    * Before we send schema, make sure that STREAM START/BEGIN/BEGIN PREPARE
+    * is sent. If not, send now.
+    */

What part of this code is doing anything about "BEGIN PREPARE" ?

~~~

9. src/backend/replication/pgoutput/pgoutput.c - pgoutput_change

@@ -1183,6 +1267,15 @@ pgoutput_change(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
Assert(false);
}

+ /* If streaming, send STREAM START if we haven't yet */
+ if (in_streaming && (txndata && !txndata->sent_stream_start))
+ pgoutput_send_stream_start(ctx, txn);
+ /*
+ * Output BEGIN if we haven't yet, unless streaming.
+ */
+ else if (!in_streaming && (txndata && !txndata->sent_begin_txn))
+ pgoutput_begin(ctx, txn);
+

The above code fragment looks more like what IU was expecting should
be in 'maybe_send_schema',

If you expand it out (and tweak the comments) it can become much less
complex looking IMO

e.g.

if (in_streaming)
{
/* If streaming, send STREAM START if we haven't yet */
if (txndata && !txndata->sent_stream_start)
pgoutput_send_stream_start(ctx, txn);
}
else
{
/* If not streaming, send BEGIN if we haven't yet */
if (txndata && !txndata->sent_begin_txn)
pgoutput_begin(ctx, txn);
}

Also, IIUC for the 'in_streaming' case you can Assert(txndata); so
then the code can be made even simpler.

~~~

10. src/backend/replication/pgoutput/pgoutput.c - pgoutput_truncate

@ -1397,6 +1491,17 @@ pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,

  if (nrelids > 0)
  {
+ txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+ /* If streaming, send STREAM START if we haven't yet */
+ if (in_streaming && (txndata && !txndata->sent_stream_start))
+ pgoutput_send_stream_start(ctx, txn);
+ /*
+ * output BEGIN if we haven't yet, unless streaming.
+ */
+ else if (!in_streaming && (txndata && !txndata->sent_begin_txn))
+ pgoutput_begin(ctx, txn);

So now I have seen almost identical code repeated in 3 places so I am
beginning to think these should just be encapsulated in some common
function to call to do the deferred "send". Thoughts?

~~~

11. src/backend/replication/pgoutput/pgoutput.c - pgoutput_message

@@ -1429,6 +1534,24 @@ pgoutput_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
if (in_streaming)
xid = txn->xid;

+ /*
+ * Output BEGIN if we haven't yet.
+ * Avoid for streaming and non-transactional messages.
+ */
+ if (in_streaming || transactional)
+ {
+ PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+ /* If streaming, send STREAM START if we haven't yet */
+ if (in_streaming && (txndata && !txndata->sent_stream_start))
+ pgoutput_send_stream_start(ctx, txn);
+ else if (transactional)
+ {
+ if (txndata && !txndata->sent_begin_txn)
+ pgoutput_begin(ctx, txn);
+ }
+ }

Does that comment at the top of that code fragment accurately match
this code? It seemed a bit muddled/stale to me.

~~~

12. src/backend/replication/pgoutput/pgoutput.c - pgoutput_stream_start

  /*
+ * Don't actually send stream start here, instead set a flag that indicates
+ * that stream start hasn't been sent and wait for the first actual change
+ * for this stream to be sent and then send stream start. This is done
+ * to avoid sending empty streams without any changes.
+ */
+ if (txndata == NULL)
+ {
+ txndata =
+ MemoryContextAllocZero(ctx->context, sizeof(PGOutputTxnData));
+ txndata->sent_begin_txn = false;
+ txndata->sent_any_stream = false;
+ txn->output_plugin_private = txndata;
+ }

IMO there is no need to set the members – just let the
MemoryContextAllocZero take care of all that. Then the code is simpler
and it also saves wondering if anything was accidentally missed.

~~~

13. src/backend/replication/pgoutput/pgoutput.c - pgoutput_send_stream_start

+pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+   ReorderBufferTXN *txn)
+{
+ bool send_replication_origin = txn->origin_id != InvalidRepOriginId;
+ PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+
+ /*
  * If we already sent the first stream for this transaction then don't
  * send the origin id in the subsequent streams.
  */
- if (rbtxn_is_streamed(txn))
+ if (txndata->sent_any_stream)
  send_replication_origin = false;

Given this usage, I wonder if there is a better name for the txndata
member - e.g. 'sent_first_stream' ?

~~~

14. src/backend/replication/pgoutput/pgoutput.c - pgoutput_send_stream_start

- /* we're streaming a chunk of transaction now */
- in_streaming = true;
+ /*
+ * Set the flags that indicate that changes were sent as part of
+ * the transaction and the stream.
+ */
+ txndata->sent_begin_txn = txndata->sent_stream_start = true;
+ txndata->sent_any_stream = true;

Why is this setting member 'sent_begin_txn' true also? It seems odd to
say so because the BEGIN was not actually sent at all, right?

~~~

15. src/backend/replication/pgoutput/pgoutput.c - pgoutput_stream_abort

@@ -1572,6 +1740,20 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,

  /* determine the toplevel transaction */
  toptxn = (txn->toptxn) ? txn->toptxn : txn;
+ txndata = toptxn->output_plugin_private;
+ sent_begin_txn = txndata->sent_begin_txn;
+
+ if (txn->toptxn == NULL)
+ {
+ pfree(txndata);
+ txn->output_plugin_private = NULL;
+ }
+
+ if (!sent_begin_txn)
+ {
+ elog(DEBUG1, "Skipping replication of an empty transaction in stream abort");
+ return;
+ }

I didn't really understand why this code is checking the
'sent_begin_txn' member instead of the 'sent_stream_start' member?

~~~

16. src/backend/replication/pgoutput/pgoutput.c - pgoutput_stream_commit

@@ -1598,7 +1782,17 @@ pgoutput_stream_commit(struct
LogicalDecodingContext *ctx,
Assert(!in_streaming);
Assert(rbtxn_is_streamed(txn));

- OutputPluginUpdateProgress(ctx);
+ pfree(txndata);
+ txn->output_plugin_private = NULL;
+
+ /* If no changes were part of this transaction then drop the commit */
+ if (!sent_begin_txn)
+ {
+ elog(DEBUG1, "Skipping replication of an empty transaction in stream commit");
+ return;
+ }

(Same as previous comment #15). I didn't really understand why this
code is checking the 'sent_begin_txn' member instead of the
'sent_stream_start' member?

~~~

17. src/backend/replication/syncrep.c - SyncRepEnabled

@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
}

 /*
+ * Check if synchronous replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+ return SyncRepRequested() && ((volatile WalSndCtlData *)
WalSndCtl)->sync_standbys_defined;
+}

That code was once inline in 'SyncRepWaitForLSN' before it was turned
into a function, and there is a long comment in SyncRepWaitForLSN
describing the risks of this logic. e.g.

<quote>
... If it's true, we need to check it again
* later while holding the lock, to check the flag and operate the sync
* rep queue atomically. This is necessary to avoid the race condition
* described in SyncRepUpdateSyncStandbysDefined().
</quote>

This same function is now called from walsender.c. I think maybe it is
OK but please confirm it.

Anyway, the point is maybe this SyncRepEnabled function should be
better commented to make some reference about the race concerns of the
original comment. Otherwise some future caller of this function may be
unaware of it and come to grief.

-------
Kind Regards,
Peter Smith.
Fujitsu Australia

#80Ajin Cherian
itsajin@gmail.com
In reply to: Amit Kapila (#75)
1 attachment(s)
Re: logical replication empty transactions

On Fri, Feb 18, 2022 at 9:27 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

Yeah, I think there could be multiple ways (a) We can send such a keep
alive in WalSndUpdateProgress() itself by using ctx->write_location.
For this, we need to modify WalSndKeepalive() to take sentPtr as
input. (b) set some flag in WalSndUpdateProgress() and then do it
somewhere in WalSndLoop probably in WalSndKeepaliveIfNecessary, or
maybe there is another better way.

Thanks for the suggestion Amit and Osumi-san, I experimented with both
the suggestions but finally decided to use
(a)Modifying WalSndKeepalive() to take an LSN optionally as input and
passed in the ctx->write_location.

I also verified that if I block the WalSndKeepalive() in
WalSndWaitForWal, then my new code sends the keepalive
when skipping transactions and the syncrep gets back feedback..

I will address comments from Peter and Wang in my next patch update.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v20-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v20-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 0a2002d333d904f5eef605619d4052e7338918db Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Tue, 22 Feb 2022 07:47:22 -0500
Subject: [PATCH v20] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN/COMMIT messages for transactions that are empty. The patch
also makes sure that in synchronous replication mode, when skipping empty
transactions, keepalive messages are sent to keep the LSN locations updated
on the standby.

This patch does not skip empty transactions that are "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 234 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 +-
 src/backend/replication/walsender.c         |  31 ++--
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 262 insertions(+), 31 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..99b2775 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a04..47affa1 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -67,6 +67,8 @@ static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   TimestampTz prepare_time);
 static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 								  ReorderBufferTXN *txn);
+static void pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+									   ReorderBufferTXN *txn);
 static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn);
 static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
@@ -166,6 +168,19 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+   bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+   bool sent_stream_start; /* flag indicating if stream start has been sent */
+   bool sent_any_stream;   /* flag indicating if any stream has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,15 +467,43 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txndata->sent_begin_txn = false;
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -475,7 +518,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            skip;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	skip = !txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, skip);
+
+	pfree(txndata);
+	if (skip)
+	{
+		elog(DEBUG1, "skipping replication of an empty transaction");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -506,7 +567,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +581,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +597,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -556,6 +617,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	bool		schema_sent;
 	TransactionId xid = InvalidTransactionId;
 	TransactionId topxid = InvalidTransactionId;
+	PGOutputTxnData *txndata;
+	ReorderBufferTXN *toptxn;
 
 	/*
 	 * Remember XID of the (sub)transaction for the change. We don't care if
@@ -569,9 +632,15 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 		xid = change->txn->xid;
 
 	if (change->txn->toptxn)
+	{
 		topxid = change->txn->toptxn->xid;
+		toptxn = change->txn->toptxn;
+	}
 	else
+	{
 		topxid = xid;
+		toptxn = change->txn;
+	}
 
 	/*
 	 * Do we need to send the schema? We do track streamed transactions
@@ -594,6 +663,20 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	if (schema_sent)
 		return;
 
+   /* set up txndata */
+   txndata = toptxn->output_plugin_private;
+
+   /*
+    * Before we send schema, make sure that STREAM START/BEGIN/BEGIN PREPARE
+    * is sent. If not, send now.
+    */
+   if (in_streaming && !txndata->sent_stream_start)
+       pgoutput_send_stream_start(ctx, toptxn);
+   else if (txndata && !txndata->sent_begin_txn)
+   {
+       pgoutput_begin(ctx, toptxn);
+   }
+
 	/*
 	 * Send the schema.  If the changes will be published using an ancestor's
 	 * schema, not the relation's own, send that ancestor's schema before
@@ -1141,6 +1224,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1183,6 +1267,15 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/* If streaming, send STREAM START if we haven't yet */
+	if (in_streaming && (txndata && !txndata->sent_stream_start))
+		pgoutput_send_stream_start(ctx, txn);
+	/*
+	 * Output BEGIN if we haven't yet, unless streaming.
+	 */
+	else if (!in_streaming && (txndata && !txndata->sent_begin_txn))
+		pgoutput_begin(ctx, txn);
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -1354,6 +1447,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1397,6 +1491,17 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* If streaming, send STREAM START if we haven't yet */
+		if (in_streaming && (txndata && !txndata->sent_stream_start))
+			pgoutput_send_stream_start(ctx, txn);
+		/*
+		 * output BEGIN if we haven't yet, unless streaming.
+		 */
+		else if (!in_streaming && (txndata && !txndata->sent_begin_txn))
+			pgoutput_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -1429,6 +1534,24 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for streaming and non-transactional messages.
+	 */
+	if (in_streaming || transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* If streaming, send STREAM START if we haven't yet */
+		if (in_streaming && (txndata && !txndata->sent_stream_start))
+			pgoutput_send_stream_start(ctx, txn);
+		else if (transactional)
+		{
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_begin(ctx, txn);
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1511,28 +1634,62 @@ static void
 pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 					  ReorderBufferTXN *txn)
 {
-	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = txn->output_plugin_private;
 
 	/* we can't nest streaming of transactions */
 	Assert(!in_streaming);
 
 	/*
+	 * Don't actually send stream start here, instead set a flag that indicates
+	 * that stream start hasn't been sent and wait for the first actual change
+	 * for this stream to be sent and then send stream start. This is done
+	 * to avoid sending empty streams without any changes.
+	 */
+	if (txndata == NULL)
+	{
+		txndata =
+			MemoryContextAllocZero(ctx->context, sizeof(PGOutputTxnData));
+		txndata->sent_begin_txn = false;
+		txndata->sent_any_stream = false;
+		txn->output_plugin_private = txndata;
+	}
+
+	txndata->sent_stream_start = false;
+	in_streaming = true;
+}
+
+/*
+ * Actually send START STREAM
+ */
+static void
+pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+					  ReorderBufferTXN *txn)
+{
+	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+
+	/*
 	 * If we already sent the first stream for this transaction then don't
 	 * send the origin id in the subsequent streams.
 	 */
-	if (rbtxn_is_streamed(txn))
+	if (txndata->sent_any_stream)
 		send_replication_origin = false;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
-	logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+	logicalrep_write_stream_start(ctx->out, txn->xid, !txndata->sent_any_stream);
 
 	send_repl_origin(ctx, txn->origin_id, InvalidXLogRecPtr,
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
 
-	/* we're streaming a chunk of transaction now */
-	in_streaming = true;
+	/*
+	 * Set the flags that indicate that changes were sent as part of
+	 * the transaction and the stream.
+	 */
+	txndata->sent_begin_txn = txndata->sent_stream_start = true;
+	txndata->sent_any_stream = true;
 }
 
 /*
@@ -1542,9 +1699,18 @@ static void
 pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 					 ReorderBufferTXN *txn)
 {
+	PGOutputTxnData *data = txn->output_plugin_private;
+
 	/* we should be streaming a trasanction */
 	Assert(in_streaming);
 
+	if (!data->sent_stream_start)
+	{
+		in_streaming = false;
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream stop");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_stop(ctx->out);
 	OutputPluginWrite(ctx, true);
@@ -1563,6 +1729,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 					  XLogRecPtr abort_lsn)
 {
 	ReorderBufferTXN *toptxn;
+	PGOutputTxnData  *txndata;
+	bool sent_begin_txn;
 
 	/*
 	 * The abort should happen outside streaming block, even for streamed
@@ -1572,6 +1740,20 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 
 	/* determine the toplevel transaction */
 	toptxn = (txn->toptxn) ? txn->toptxn : txn;
+	txndata = toptxn->output_plugin_private;
+	sent_begin_txn = txndata->sent_begin_txn;
+
+	if (txn->toptxn == NULL)
+	{
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+	}
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream abort");
+		return;
+	}
 
 	Assert(rbtxn_is_streamed(toptxn));
 
@@ -1591,6 +1773,8 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 					   ReorderBufferTXN *txn,
 					   XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_begin_txn = txndata->sent_begin_txn;
 	/*
 	 * The commit should happen outside streaming block, even for streamed
 	 * transactions. The transaction has to be marked as streamed, though.
@@ -1598,7 +1782,21 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	/* 
+	 * If no changes were part of this transaction then drop the commit
+	 * but send the update progress.
+	 */
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream commit");
+		OutputPluginUpdateProgress(ctx, true);
+		return;
+	}
+
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1617,9 +1815,21 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 							ReorderBufferTXN *txn,
 							XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_begin_txn = txndata->sent_begin_txn;
+
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream prepare");
+		return;
+	}
+
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index ce163b9..11f7358 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
 }
 
 /*
+ * Check if synchronous replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Calculate the synced Write, Flush and Apply positions among sync standbys.
  *
  * Return false if the number of sync standbys is less than
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 5a718b1..e9ca57c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,12 +1450,20 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keepalive to keep the MyWalSnd locations updated.
+	 */
+	if (send_keepalive && SyncRepEnabled())
+		WalSndKeepalive(true, ctx->write_location);
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1550,7 +1559,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			 WalSndKeepalive(false, 0); 
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2077,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, 0);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3083,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, 0);
 }
 
 /*
@@ -3588,18 +3597,20 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * If writePtr is set, mark that as the LSN processed, else use sentPtr.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, writePtr ? writePtr : sentPtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3638,7 +3649,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, 0);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..9f59855 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keepalive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..eb91d17 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 27be230..7086532 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index c6b302c..56761ab 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1608,6 +1608,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#81Ajin Cherian
itsajin@gmail.com
In reply to: Peter Smith (#79)
1 attachment(s)
Re: logical replication empty transactions

On Fri, Feb 25, 2022 at 9:17 PM Peter Smith <smithpb2250@gmail.com> wrote:

Hi. Here are my review comments for the v19 patch.

======

1. Commit message

The current logical replication behavior is to send every transaction to
subscriber even though the transaction is empty (because it does not
contain changes from the selected publications).

SUGGESTION
"to subscriber even though" --> "to the subscriber even if"

Fixed.

~~~

2. Commit message

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there is no other change for that transaction,
do not send COMMIT message. It means that pgoutput will
skip BEGIN/COMMIT messages for transactions that are empty.

SUGGESTION
"if there is" --> "if there was"
"do not send COMMIT message" --> "do not send the COMMIT message"
"It means that pgoutput" --> "This means that pgoutput"

~~~

Fixed.

3. Commit message

Shouldn't there be some similar description about using a lazy send
mechanism for STREAM START?

~~~

Added.

4. src/backend/replication/pgoutput/pgoutput.c - typedef struct PGOutputTxnData

+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+   bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+   bool sent_stream_start; /* flag indicating if stream start has been sent */
+   bool sent_any_stream;   /* flag indicating if any stream has been sent */
+} PGOutputTxnData;
+

The struct comment looks stale because it doesn't mention anything
about the similar lazy send mechanism for STREAM_START.

~~~

Added.

5. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin_txn

static void
pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
{
+ PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+ sizeof(PGOutputTxnData));
+
+ txndata->sent_begin_txn = false;
+ txn->output_plugin_private = txndata;
+}

You don’t need to assign the other members 'sent_stream_start',
'sent_any_stream' because you are doing MemoryContextAllocZero anyway,
but for the same reason you did not really need to assign the
'sent_begin_txn' flag either.

I guess for consistency maybe it is better to (a) set all of them or
(b) set none of them. I prefer (b).

~~~

Did (b)

6. src/backend/replication/pgoutput/pgoutput.c - pgoutput_begin

I feel the 'pgoutput_begin' function is not well named. It makes some
of the code where they are called look quite confusing.

For streaming there is:
1. pgoutput_stream_start (does not send)
2. pgoutput_send_stream_start (does send)
so it is very clear.

OTOH there are
3. pgoutput_begin_txn (does not send)
4. pgoutput_begin (does send)

For consistency I think the 'pgoutput_begin' name should be changed to
include "send" verb
1. pgoutput_begin_txn (does not send)
2. pgoutput_send_begin_txn (does send)

~~~

Changed as mentioned.

7. src/backend/replication/pgoutput/pgoutput.c - maybe_send_schema

@@ -594,6 +663,20 @@ maybe_send_schema(LogicalDecodingContext *ctx,
if (schema_sent)
return;

+   /* set up txndata */
+   txndata = toptxn->output_plugin_private;
+
+   /*
+    * Before we send schema, make sure that STREAM START/BEGIN/BEGIN PREPARE
+    * is sent. If not, send now.
+    */
+   if (in_streaming && !txndata->sent_stream_start)
+       pgoutput_send_stream_start(ctx, toptxn);
+   else if (txndata && !txndata->sent_begin_txn)
+   {
+       pgoutput_begin(ctx, toptxn);
+   }
+

How come the in_streaming case is not checking for a NULL txndata
before referencing it? Even if it is OK to do that, some more comments
or assertions might help for this piece of code.
(Stop-Press: see later comments #9, #10)

~~~

Updated.

8. src/backend/replication/pgoutput/pgoutput.c - maybe_send_schema

@@ -594,6 +663,20 @@ maybe_send_schema(LogicalDecodingContext *ctx,
if (schema_sent)
return;

+   /* set up txndata */
+   txndata = toptxn->output_plugin_private;
+
+   /*
+    * Before we send schema, make sure that STREAM START/BEGIN/BEGIN PREPARE
+    * is sent. If not, send now.
+    */

What part of this code is doing anything about "BEGIN PREPARE" ?

~~~

Removed that reference.

9. src/backend/replication/pgoutput/pgoutput.c - pgoutput_change

@@ -1183,6 +1267,15 @@ pgoutput_change(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
Assert(false);
}

+ /* If streaming, send STREAM START if we haven't yet */
+ if (in_streaming && (txndata && !txndata->sent_stream_start))
+ pgoutput_send_stream_start(ctx, txn);
+ /*
+ * Output BEGIN if we haven't yet, unless streaming.
+ */
+ else if (!in_streaming && (txndata && !txndata->sent_begin_txn))
+ pgoutput_begin(ctx, txn);
+

The above code fragment looks more like what IU was expecting should
be in 'maybe_send_schema',

If you expand it out (and tweak the comments) it can become much less
complex looking IMO

e.g.

if (in_streaming)
{
/* If streaming, send STREAM START if we haven't yet */
if (txndata && !txndata->sent_stream_start)
pgoutput_send_stream_start(ctx, txn);
}
else
{
/* If not streaming, send BEGIN if we haven't yet */
if (txndata && !txndata->sent_begin_txn)
pgoutput_begin(ctx, txn);
}

Also, IIUC for the 'in_streaming' case you can Assert(txndata); so
then the code can be made even simpler.

Chose your example.

~~~

10. src/backend/replication/pgoutput/pgoutput.c - pgoutput_truncate

@ -1397,6 +1491,17 @@ pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,

if (nrelids > 0)
{
+ txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+ /* If streaming, send STREAM START if we haven't yet */
+ if (in_streaming && (txndata && !txndata->sent_stream_start))
+ pgoutput_send_stream_start(ctx, txn);
+ /*
+ * output BEGIN if we haven't yet, unless streaming.
+ */
+ else if (!in_streaming && (txndata && !txndata->sent_begin_txn))
+ pgoutput_begin(ctx, txn);

So now I have seen almost identical code repeated in 3 places so I am
beginning to think these should just be encapsulated in some common
function to call to do the deferred "send". Thoughts?

~~~

Not sure if we want to add a function call overhead.

11. src/backend/replication/pgoutput/pgoutput.c - pgoutput_message

@@ -1429,6 +1534,24 @@ pgoutput_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
if (in_streaming)
xid = txn->xid;

+ /*
+ * Output BEGIN if we haven't yet.
+ * Avoid for streaming and non-transactional messages.
+ */
+ if (in_streaming || transactional)
+ {
+ PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+ /* If streaming, send STREAM START if we haven't yet */
+ if (in_streaming && (txndata && !txndata->sent_stream_start))
+ pgoutput_send_stream_start(ctx, txn);
+ else if (transactional)
+ {
+ if (txndata && !txndata->sent_begin_txn)
+ pgoutput_begin(ctx, txn);
+ }
+ }

Does that comment at the top of that code fragment accurately match
this code? It seemed a bit muddled/stale to me.

~~~

Fixed.

12. src/backend/replication/pgoutput/pgoutput.c - pgoutput_stream_start

/*
+ * Don't actually send stream start here, instead set a flag that indicates
+ * that stream start hasn't been sent and wait for the first actual change
+ * for this stream to be sent and then send stream start. This is done
+ * to avoid sending empty streams without any changes.
+ */
+ if (txndata == NULL)
+ {
+ txndata =
+ MemoryContextAllocZero(ctx->context, sizeof(PGOutputTxnData));
+ txndata->sent_begin_txn = false;
+ txndata->sent_any_stream = false;
+ txn->output_plugin_private = txndata;
+ }

IMO there is no need to set the members – just let the
MemoryContextAllocZero take care of all that. Then the code is simpler
and it also saves wondering if anything was accidentally missed.

Fixed.

~~~

13. src/backend/replication/pgoutput/pgoutput.c - pgoutput_send_stream_start

+pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+   ReorderBufferTXN *txn)
+{
+ bool send_replication_origin = txn->origin_id != InvalidRepOriginId;
+ PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+
+ /*
* If we already sent the first stream for this transaction then don't
* send the origin id in the subsequent streams.
*/
- if (rbtxn_is_streamed(txn))
+ if (txndata->sent_any_stream)
send_replication_origin = false;

Given this usage, I wonder if there is a better name for the txndata
member - e.g. 'sent_first_stream' ?

~~~

Changed.

14. src/backend/replication/pgoutput/pgoutput.c - pgoutput_send_stream_start

- /* we're streaming a chunk of transaction now */
- in_streaming = true;
+ /*
+ * Set the flags that indicate that changes were sent as part of
+ * the transaction and the stream.
+ */
+ txndata->sent_begin_txn = txndata->sent_stream_start = true;
+ txndata->sent_any_stream = true;

Why is this setting member 'sent_begin_txn' true also? It seems odd to
say so because the BEGIN was not actually sent at all, right?

~~~

You can have transactions that are partially streamed and partially
not. So if there
is a transaction that started as streaming, but when it is committed,
it is replicated
as part of the commit, then when the changes are decoded, we shouldn't
be sending a "begin"
again.

15. src/backend/replication/pgoutput/pgoutput.c - pgoutput_stream_abort

@@ -1572,6 +1740,20 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,

/* determine the toplevel transaction */
toptxn = (txn->toptxn) ? txn->toptxn : txn;
+ txndata = toptxn->output_plugin_private;
+ sent_begin_txn = txndata->sent_begin_txn;
+
+ if (txn->toptxn == NULL)
+ {
+ pfree(txndata);
+ txn->output_plugin_private = NULL;
+ }
+
+ if (!sent_begin_txn)
+ {
+ elog(DEBUG1, "Skipping replication of an empty transaction in stream abort");
+ return;
+ }

I didn't really understand why this code is checking the
'sent_begin_txn' member instead of the 'sent_stream_start' member?

Yes, changed this to check "sent_first_stream"

~~~

16. src/backend/replication/pgoutput/pgoutput.c - pgoutput_stream_commit

@@ -1598,7 +1782,17 @@ pgoutput_stream_commit(struct
LogicalDecodingContext *ctx,
Assert(!in_streaming);
Assert(rbtxn_is_streamed(txn));

- OutputPluginUpdateProgress(ctx);
+ pfree(txndata);
+ txn->output_plugin_private = NULL;
+
+ /* If no changes were part of this transaction then drop the commit */
+ if (!sent_begin_txn)
+ {
+ elog(DEBUG1, "Skipping replication of an empty transaction in stream commit");
+ return;
+ }

(Same as previous comment #15). I didn't really understand why this
code is checking the 'sent_begin_txn' member instead of the
'sent_stream_start' member?

~~~

Changed.

17. src/backend/replication/syncrep.c - SyncRepEnabled

@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
}

/*
+ * Check if synchronous replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+ return SyncRepRequested() && ((volatile WalSndCtlData *)
WalSndCtl)->sync_standbys_defined;
+}

That code was once inline in 'SyncRepWaitForLSN' before it was turned
into a function, and there is a long comment in SyncRepWaitForLSN
describing the risks of this logic. e.g.

<quote>
... If it's true, we need to check it again
* later while holding the lock, to check the flag and operate the sync
* rep queue atomically. This is necessary to avoid the race condition
* described in SyncRepUpdateSyncStandbysDefined().
</quote>

This same function is now called from walsender.c. I think maybe it is
OK but please confirm it.

Anyway, the point is maybe this SyncRepEnabled function should be
better commented to make some reference about the race concerns of the
original comment. Otherwise some future caller of this function may be
unaware of it and come to grief.

Leaving this for now, not sure what wording is appropriate to use here.

On Wed, Feb 23, 2022 at 5:24 PM wangw.fnst@fujitsu.com
<wangw.fnst@fujitsu.com> wrote:

On Feb, Wed 23, 2022 at 10:58 PM Ajin Cherian <itsajin@gmail.com> wrote:

Few comments to V19-0001:

1. I think we should adjust the alignment format.
git am ../v19-0001-Skip-empty-transactions-for-logical-replication.patch
.git/rebase-apply/patch:197: indent with spaces.
* Before we send schema, make sure that STREAM START/BEGIN/BEGIN PREPARE
.git/rebase-apply/patch:198: indent with spaces.
* is sent. If not, send now.
.git/rebase-apply/patch:199: indent with spaces.
*/
.git/rebase-apply/patch:201: indent with spaces.
pgoutput_send_stream_start(ctx, toptxn);
.git/rebase-apply/patch:204: indent with spaces.
pgoutput_begin(ctx, toptxn);
warning: 5 lines add whitespace errors.

Fixed.

2. Structure member initialization.
static void
pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
{
+       PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+                                                                                                                sizeof(PGOutputTxnData));
+
+       txndata->sent_begin_txn = false;
+       txn->output_plugin_private = txndata;
+}
Do we need to set sent_stream_start and sent_any_stream to false here?

Fixed

3. Maybe we should add Assert(txndata) like function pgoutput_commit_txn in
other functions.

4. In addition, I think we should keep a unified style.
a). log style (maybe first one is better.)
First style : "Skipping replication of an empty transaction in XXX"
Second style : "skipping replication of an empty transaction"
b) flag name (maybe second one is better.)
First style : variable "sent_begin_txn" in function pgoutput_stream_*.
Second style : variable "skip" in function pgoutput_commit_txn.

Fixed,

Regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v21-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v21-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 06bbdf5b71248288f1b094a044941635589a4c68 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Mon, 28 Feb 2022 23:35:42 -0500
Subject: [PATCH v21] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there was no other change for that transaction,
do not send the COMMIT message. This means that pgoutput will
skip BEGIN/COMMIT messages for transactions that are empty. This patch
also postpones the START STREAM message while streaming large in-progress
transactions until the first change. While processing the STOP STREAM
message, if there was no other change for that transaction, do not send
the STOP STREAM message. The patch also makes sure that in synchronous
replication mode, when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 248 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 +-
 src/backend/replication/walsender.c         |  31 ++--
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 276 insertions(+), 31 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..99b2775 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a04..aafe805 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -67,6 +67,8 @@ static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   TimestampTz prepare_time);
 static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 								  ReorderBufferTXN *txn);
+static void pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+									   ReorderBufferTXN *txn);
 static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn);
 static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
@@ -166,6 +168,20 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. Similarly while streaming
+ * transactions, STREAM_START is only sent with the first change.
+ * This makes it possible to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+   bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+   bool sent_stream_start; /* flag indicating if stream start has been sent */
+   bool sent_first_stream;   /* flag indicating if any stream has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,15 +468,42 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -475,7 +518,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+
+	pfree(txndata);
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in commit");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -506,7 +567,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +581,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +597,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -556,6 +617,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	bool		schema_sent;
 	TransactionId xid = InvalidTransactionId;
 	TransactionId topxid = InvalidTransactionId;
+	PGOutputTxnData *txndata;
+	ReorderBufferTXN *toptxn;
 
 	/*
 	 * Remember XID of the (sub)transaction for the change. We don't care if
@@ -569,9 +632,15 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 		xid = change->txn->xid;
 
 	if (change->txn->toptxn)
+	{
 		topxid = change->txn->toptxn->xid;
+		toptxn = change->txn->toptxn;
+	}
 	else
+	{
 		topxid = xid;
+		toptxn = change->txn;
+	}
 
 	/*
 	 * Do we need to send the schema? We do track streamed transactions
@@ -594,6 +663,22 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	if (schema_sent)
 		return;
 
+   /* set up txndata */
+   txndata = toptxn->output_plugin_private;
+
+	if (in_streaming)
+	{
+		/* If streaming, send STREAM START if we haven't yet */
+		if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, toptxn);
+	}
+	else
+	{
+		/* If not streaming, send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, toptxn);
+	}
+
 	/*
 	 * Send the schema.  If the changes will be published using an ancestor's
 	 * schema, not the relation's own, send that ancestor's schema before
@@ -1141,6 +1226,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1183,6 +1269,20 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+   if (in_streaming)
+	{
+		/* If streaming, send STREAM START if we haven't yet */
+		if (txndata && !txndata->sent_stream_start)
+		pgoutput_send_stream_start(ctx, txn);
+	}
+	else
+	{
+		/* If not streaming, send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+		pgoutput_send_begin(ctx, txn);
+	}
+
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -1354,6 +1454,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1397,6 +1498,21 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		if (in_streaming)
+		{
+			/* If streaming, send STREAM START if we haven't yet */
+			if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, txn);
+		}
+		else
+		{
+			/* If not streaming, send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+		}
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -1429,6 +1545,28 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for non-transactional messages.
+	 */
+	if (in_streaming || transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		if (in_streaming)
+		{
+			/* If streaming, send STREAM START if we haven't yet */
+			if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, txn);
+		}
+		else
+		{
+			/* If not streaming, send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1511,28 +1649,60 @@ static void
 pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 					  ReorderBufferTXN *txn)
 {
-	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = txn->output_plugin_private;
 
 	/* we can't nest streaming of transactions */
 	Assert(!in_streaming);
 
 	/*
+	 * Don't actually send stream start here, instead set a flag that indicates
+	 * that stream start hasn't been sent and wait for the first actual change
+	 * for this stream to be sent and then send stream start. This is done
+	 * to avoid sending empty streams without any changes.
+	 */
+	if (txndata == NULL)
+	{
+		txndata =
+			MemoryContextAllocZero(ctx->context, sizeof(PGOutputTxnData));
+		txn->output_plugin_private = txndata;
+	}
+
+	txndata->sent_stream_start = false;
+	in_streaming = true;
+}
+
+/*
+ * Actually send START STREAM
+ */
+static void
+pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+					  ReorderBufferTXN *txn)
+{
+	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+
+	/*
 	 * If we already sent the first stream for this transaction then don't
 	 * send the origin id in the subsequent streams.
 	 */
-	if (rbtxn_is_streamed(txn))
+	if (txndata->sent_first_stream)
 		send_replication_origin = false;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
-	logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+	logicalrep_write_stream_start(ctx->out, txn->xid, !txndata->sent_first_stream);
 
 	send_repl_origin(ctx, txn->origin_id, InvalidXLogRecPtr,
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
 
-	/* we're streaming a chunk of transaction now */
-	in_streaming = true;
+	/*
+	 * Set the flags that indicate that changes were sent as part of
+	 * the transaction and the stream.
+	 */
+	txndata->sent_begin_txn = txndata->sent_stream_start = true;
+	txndata->sent_first_stream = true;
 }
 
 /*
@@ -1542,9 +1712,18 @@ static void
 pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 					 ReorderBufferTXN *txn)
 {
+	PGOutputTxnData *data = txn->output_plugin_private;
+
 	/* we should be streaming a trasanction */
 	Assert(in_streaming);
 
+	if (!data->sent_stream_start)
+	{
+		in_streaming = false;
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream stop");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_stop(ctx->out);
 	OutputPluginWrite(ctx, true);
@@ -1563,6 +1742,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 					  XLogRecPtr abort_lsn)
 {
 	ReorderBufferTXN *toptxn;
+	PGOutputTxnData  *txndata;
+	bool sent_first_stream;
 
 	/*
 	 * The abort should happen outside streaming block, even for streamed
@@ -1572,6 +1753,20 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 
 	/* determine the toplevel transaction */
 	toptxn = (txn->toptxn) ? txn->toptxn : txn;
+	txndata = toptxn->output_plugin_private;
+	sent_first_stream = txndata->sent_first_stream;
+
+	if (txn->toptxn == NULL)
+	{
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+	}
+
+	if (!sent_first_stream)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream abort");
+		return;
+	}
 
 	Assert(rbtxn_is_streamed(toptxn));
 
@@ -1591,6 +1786,9 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 					   ReorderBufferTXN *txn,
 					   XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_first_stream = txndata->sent_first_stream;
+
 	/*
 	 * The commit should happen outside streaming block, even for streamed
 	 * transactions. The transaction has to be marked as streamed, though.
@@ -1598,7 +1796,21 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	/*
+	 * If no changes were part of this transaction then drop the commit
+	 * but send the update progress.
+	 */
+	if (!sent_first_stream)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream commit");
+		OutputPluginUpdateProgress(ctx, true);
+		return;
+	}
+
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1617,9 +1829,21 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 							ReorderBufferTXN *txn,
 							XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_begin_txn = txndata->sent_begin_txn;
+
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream prepare");
+		return;
+	}
+
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index ce163b9..11f7358 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
 }
 
 /*
+ * Check if synchronous replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Calculate the synced Write, Flush and Apply positions among sync standbys.
  *
  * Return false if the number of sync standbys is less than
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 5a718b1..a492610 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,12 +1450,20 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keepalive to keep the MyWalSnd locations updated.
+	 */
+	if (send_keepalive && SyncRepEnabled())
+		WalSndKeepalive(true, ctx->write_location);
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1550,7 +1559,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, 0);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2077,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, 0);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3083,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, 0);
 }
 
 /*
@@ -3588,18 +3597,20 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * If writePtr is set, mark that as the LSN processed, else use sentPtr.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, writePtr ? writePtr : sentPtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3638,7 +3649,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, 0);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..9f59855 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keepalive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..eb91d17 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 27be230..7086532 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d9b83f7..77f33b2 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1608,6 +1608,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#82shiy.fnst@fujitsu.com
shiy.fnst@fujitsu.com
In reply to: Ajin Cherian (#81)
RE: logical replication empty transactions

Hi,

Here are some comments on the v21 patch.

1.
+ WalSndKeepalive(false, 0);

Maybe we can use InvalidXLogRecPtr here, instead of 0.

2.
+ pq_sendint64(&output_message, writePtr ? writePtr : sentPtr);

Similarly, should we use XLogRecPtrIsInvalid()?

3.
@@ -1183,6 +1269,20 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
Assert(false);
}

+   if (in_streaming)
+	{
+		/* If streaming, send STREAM START if we haven't yet */
+		if (txndata && !txndata->sent_stream_start)
+		pgoutput_send_stream_start(ctx, txn);
+	}
+	else
+	{
+		/* If not streaming, send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+		pgoutput_send_begin(ctx, txn);
+	}
+
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);

I am not sure if it is suitable to send begin or stream_start here, because the
row filter is not checked yet. That means, empty transactions caused by row
filter are not skipped.

4.
@@ -1617,9 +1829,21 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 							ReorderBufferTXN *txn,
 							XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_begin_txn = txndata->sent_begin_txn;
+
 	Assert(rbtxn_is_streamed(txn));
-	OutputPluginUpdateProgress(ctx);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream prepare");
+		return;
+	}
+
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);

I notice that the patch skips stream prepared transaction, this would cause an
error on subscriber side when committing this transaction on publisher side, so
I think we'd better not do that.

For example:
(set logical_decoding_work_mem = 64kB, max_prepared_transactions = 10 in
postgresql.conf)

-- publisher
create table test (a int, b text, primary key(a));
create table test2 (a int, b text, primary key(a));
create publication pub for table test;

-- subscriber
create table test (a int, b text, primary key(a));
create table test2 (a int, b text, primary key(a));
create subscription sub connection 'dbname=postgres port=5432' publication pub with(two_phase=on, streaming=on);

-- publisher
begin;
INSERT INTO test2 SELECT i, md5(i::text) FROM generate_series(1, 1000) s(i);
prepare transaction 't';
commit prepared 't';

The error message in subscriber log:
ERROR: prepared transaction with identifier "pg_gid_16391_722" does not exist

Regards,
Shi yu

#83Ajin Cherian
itsajin@gmail.com
In reply to: shiy.fnst@fujitsu.com (#82)
Re: logical replication empty transactions

On Wed, Mar 2, 2022 at 1:01 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:

4.
@@ -1617,9 +1829,21 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
XLogRecPtr prepare_lsn)
{
+       PGOutputTxnData *txndata = txn->output_plugin_private;
+       bool                    sent_begin_txn = txndata->sent_begin_txn;
+
Assert(rbtxn_is_streamed(txn));
-       OutputPluginUpdateProgress(ctx);
+       pfree(txndata);
+       txn->output_plugin_private = NULL;
+
+       if (!sent_begin_txn)
+       {
+               elog(DEBUG1, "Skipping replication of an empty transaction in stream prepare");
+               return;
+       }
+
+       OutputPluginUpdateProgress(ctx, false);
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
OutputPluginWrite(ctx, true);

I notice that the patch skips stream prepared transaction, this would cause an
error on subscriber side when committing this transaction on publisher side, so
I think we'd better not do that.

For example:
(set logical_decoding_work_mem = 64kB, max_prepared_transactions = 10 in
postgresql.conf)

-- publisher
create table test (a int, b text, primary key(a));
create table test2 (a int, b text, primary key(a));
create publication pub for table test;

-- subscriber
create table test (a int, b text, primary key(a));
create table test2 (a int, b text, primary key(a));
create subscription sub connection 'dbname=postgres port=5432' publication pub with(two_phase=on, streaming=on);

-- publisher
begin;
INSERT INTO test2 SELECT i, md5(i::text) FROM generate_series(1, 1000) s(i);
prepare transaction 't';
commit prepared 't';

The error message in subscriber log:
ERROR: prepared transaction with identifier "pg_gid_16391_722" does not exist

Thanks for the test. I guess this mixed streaming+two-phase runs into
the same problem that
was there while skipping two-phased transactions. If the eventual
commit prepared comes after a restart,
then there is no way of knowing if the original transaction was
skipped or not and we can't know if the commit prepared
needs to be sent. I tried not skipping the "stream prepare", but that
causes a crash in the apply worker
as it tries to find the non-existent streamed file. We could add logic
to silently ignore a spurious "stream prepare"
but that might not be ideal. Any thoughts on how to address this? Or
else, we will need to avoid skipping streamed
transactions as well.

regards,
Ajin Cherian
Fujitsu Australia

#84Ajin Cherian
itsajin@gmail.com
In reply to: shiy.fnst@fujitsu.com (#82)
1 attachment(s)
Re: logical replication empty transactions

On Wed, Mar 2, 2022 at 1:01 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:

Hi,

Here are some comments on the v21 patch.

1.
+ WalSndKeepalive(false, 0);

Maybe we can use InvalidXLogRecPtr here, instead of 0.

Fixed.

2.
+ pq_sendint64(&output_message, writePtr ? writePtr : sentPtr);

Similarly, should we use XLogRecPtrIsInvalid()?

Fixed

3.
@@ -1183,6 +1269,20 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
Assert(false);
}

+   if (in_streaming)
+       {
+               /* If streaming, send STREAM START if we haven't yet */
+               if (txndata && !txndata->sent_stream_start)
+               pgoutput_send_stream_start(ctx, txn);
+       }
+       else
+       {
+               /* If not streaming, send BEGIN if we haven't yet */
+               if (txndata && !txndata->sent_begin_txn)
+               pgoutput_send_begin(ctx, txn);
+       }
+
+
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);

I am not sure if it is suitable to send begin or stream_start here, because the
row filter is not checked yet. That means, empty transactions caused by row
filter are not skipped.

Moved the check down, so that row_filters are taken into account.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v22-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v22-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 7c1dbcd8a06665c46a83b87168307a814bae69a4 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Mon, 28 Feb 2022 23:35:42 -0500
Subject: [PATCH v22] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message,
if there was no other change for that transaction,
do not send the COMMIT message. This means that pgoutput will
skip BEGIN/COMMIT messages for transactions that are empty. This patch
also postpones the START STREAM message while streaming large in-progress
transactions until the first change. While processing the STOP STREAM
message, if there was no other change for that transaction, do not send
the STOP STREAM message. The patch also makes sure that in synchronous
replication mode, when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 273 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 +-
 src/backend/replication/walsender.c         |  31 +++-
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 301 insertions(+), 31 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..99b2775 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a04..84aa9c8 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -67,6 +67,8 @@ static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   TimestampTz prepare_time);
 static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 								  ReorderBufferTXN *txn);
+static void pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+									   ReorderBufferTXN *txn);
 static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn);
 static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
@@ -166,6 +168,20 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. Similarly while streaming
+ * transactions, STREAM_START is only sent with the first change.
+ * This makes it possible to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+   bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+   bool sent_stream_start; /* flag indicating if stream start has been sent */
+   bool sent_first_stream;   /* flag indicating if any stream has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,15 +468,42 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -475,7 +518,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+
+	pfree(txndata);
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in commit");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -506,7 +567,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +581,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +597,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -556,6 +617,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	bool		schema_sent;
 	TransactionId xid = InvalidTransactionId;
 	TransactionId topxid = InvalidTransactionId;
+	PGOutputTxnData *txndata;
+	ReorderBufferTXN *toptxn;
 
 	/*
 	 * Remember XID of the (sub)transaction for the change. We don't care if
@@ -569,9 +632,15 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 		xid = change->txn->xid;
 
 	if (change->txn->toptxn)
+	{
 		topxid = change->txn->toptxn->xid;
+		toptxn = change->txn->toptxn;
+	}
 	else
+	{
 		topxid = xid;
+		toptxn = change->txn;
+	}
 
 	/*
 	 * Do we need to send the schema? We do track streamed transactions
@@ -594,6 +663,22 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	if (schema_sent)
 		return;
 
+   /* set up txndata */
+   txndata = toptxn->output_plugin_private;
+
+	if (in_streaming)
+	{
+		/* If streaming, send STREAM START if we haven't yet */
+		if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, toptxn);
+	}
+	else
+	{
+		/* If not streaming, send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, toptxn);
+	}
+
 	/*
 	 * Send the schema.  If the changes will be published using an ancestor's
 	 * schema, not the relation's own, send that ancestor's schema before
@@ -1141,6 +1226,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1216,6 +1302,19 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 &action))
 				break;
 
+   			if (in_streaming)
+			{
+				/* If streaming, send STREAM START if we haven't yet */
+				if (txndata && !txndata->sent_stream_start)
+					pgoutput_send_stream_start(ctx, txn);
+			}
+			else
+			{
+				/* If not streaming, send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+			}
+
 			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
@@ -1266,6 +1365,19 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+   			if (in_streaming)
+			{
+				/* If streaming, send STREAM START if we haven't yet */
+				if (txndata && !txndata->sent_stream_start)
+					pgoutput_send_stream_start(ctx, txn);
+			}
+			else
+			{
+				/* If not streaming, send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+			}
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1324,6 +1436,19 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+   				if (in_streaming)
+				{
+					/* If streaming, send STREAM START if we haven't yet */
+					if (txndata && !txndata->sent_stream_start)
+						pgoutput_send_stream_start(ctx, txn);
+				}
+				else
+				{
+					/* If not streaming, send BEGIN if we haven't yet */
+					if (txndata && !txndata->sent_begin_txn)
+						pgoutput_send_begin(ctx, txn);
+				}
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1354,6 +1479,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1397,6 +1523,21 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		if (in_streaming)
+		{
+			/* If streaming, send STREAM START if we haven't yet */
+			if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, txn);
+		}
+		else
+		{
+			/* If not streaming, send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+		}
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -1429,6 +1570,28 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for non-transactional messages.
+	 */
+	if (in_streaming || transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		if (in_streaming)
+		{
+			/* If streaming, send STREAM START if we haven't yet */
+			if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, txn);
+		}
+		else
+		{
+			/* If not streaming, send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+		}
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1511,28 +1674,60 @@ static void
 pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 					  ReorderBufferTXN *txn)
 {
-	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = txn->output_plugin_private;
 
 	/* we can't nest streaming of transactions */
 	Assert(!in_streaming);
 
 	/*
+	 * Don't actually send stream start here, instead set a flag that indicates
+	 * that stream start hasn't been sent and wait for the first actual change
+	 * for this stream to be sent and then send stream start. This is done
+	 * to avoid sending empty streams without any changes.
+	 */
+	if (txndata == NULL)
+	{
+		txndata =
+			MemoryContextAllocZero(ctx->context, sizeof(PGOutputTxnData));
+		txn->output_plugin_private = txndata;
+	}
+
+	txndata->sent_stream_start = false;
+	in_streaming = true;
+}
+
+/*
+ * Actually send START STREAM
+ */
+static void
+pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+					  ReorderBufferTXN *txn)
+{
+	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+
+	/*
 	 * If we already sent the first stream for this transaction then don't
 	 * send the origin id in the subsequent streams.
 	 */
-	if (rbtxn_is_streamed(txn))
+	if (txndata->sent_first_stream)
 		send_replication_origin = false;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
-	logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+	logicalrep_write_stream_start(ctx->out, txn->xid, !txndata->sent_first_stream);
 
 	send_repl_origin(ctx, txn->origin_id, InvalidXLogRecPtr,
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
 
-	/* we're streaming a chunk of transaction now */
-	in_streaming = true;
+	/*
+	 * Set the flags that indicate that changes were sent as part of
+	 * the transaction and the stream.
+	 */
+	txndata->sent_begin_txn = txndata->sent_stream_start = true;
+	txndata->sent_first_stream = true;
 }
 
 /*
@@ -1542,9 +1737,18 @@ static void
 pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 					 ReorderBufferTXN *txn)
 {
+	PGOutputTxnData *data = txn->output_plugin_private;
+
 	/* we should be streaming a trasanction */
 	Assert(in_streaming);
 
+	if (!data->sent_stream_start)
+	{
+		in_streaming = false;
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream stop");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_stop(ctx->out);
 	OutputPluginWrite(ctx, true);
@@ -1563,6 +1767,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 					  XLogRecPtr abort_lsn)
 {
 	ReorderBufferTXN *toptxn;
+	PGOutputTxnData  *txndata;
+	bool sent_first_stream;
 
 	/*
 	 * The abort should happen outside streaming block, even for streamed
@@ -1572,6 +1778,20 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 
 	/* determine the toplevel transaction */
 	toptxn = (txn->toptxn) ? txn->toptxn : txn;
+	txndata = toptxn->output_plugin_private;
+	sent_first_stream = txndata->sent_first_stream;
+
+	if (txn->toptxn == NULL)
+	{
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+	}
+
+	if (!sent_first_stream)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream abort");
+		return;
+	}
 
 	Assert(rbtxn_is_streamed(toptxn));
 
@@ -1591,6 +1811,9 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 					   ReorderBufferTXN *txn,
 					   XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_first_stream = txndata->sent_first_stream;
+
 	/*
 	 * The commit should happen outside streaming block, even for streamed
 	 * transactions. The transaction has to be marked as streamed, though.
@@ -1598,7 +1821,21 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	/*
+	 * If no changes were part of this transaction then drop the commit
+	 * but send the update progress.
+	 */
+	if (!sent_first_stream)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream commit");
+		OutputPluginUpdateProgress(ctx, true);
+		return;
+	}
+
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1617,9 +1854,21 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 							ReorderBufferTXN *txn,
 							XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_begin_txn = txndata->sent_begin_txn;
+
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream prepare");
+		return;
+	}
+
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index ce163b9..11f7358 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
 }
 
 /*
+ * Check if synchronous replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Calculate the synced Write, Flush and Apply positions among sync standbys.
  *
  * Return false if the number of sync standbys is less than
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 5a718b1..33d6be7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,12 +1450,20 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keepalive to keep the MyWalSnd locations updated.
+	 */
+	if (send_keepalive && SyncRepEnabled())
+		WalSndKeepalive(true, ctx->write_location);
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1550,7 +1559,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2077,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3083,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3588,18 +3597,20 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * If writePtr is set, mark that as the LSN processed, else use sentPtr.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3638,7 +3649,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..9f59855 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keepalive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..eb91d17 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 27be230..7086532 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d9b83f7..77f33b2 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1608,6 +1608,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#85Ajin Cherian
itsajin@gmail.com
In reply to: Ajin Cherian (#84)
2 attachment(s)
Re: logical replication empty transactions

I have split the patch into two. I have kept the logic of skipping
streaming changes in the second patch.
I will work on the second patch once we can figure out a solution for
the COMMIT PREPARED after restart problem.

regards,
Ajin Cherian

Attachments:

v23-0002-Skip-empty-streamed-transactions-for-logical-rep.patchapplication/octet-stream; name=v23-0002-Skip-empty-streamed-transactions-for-logical-rep.patchDownload
From bf252c75fb5a5467d85c7c6356fa59d2be8a8424 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Thu, 3 Mar 2022 04:04:01 -0500
Subject: [PATCH v23 2/2] Skip empty streamed transactions for logical
 replication.

This patch postpones the START STREAM message while streaming large in-progress
transactions until the first change. While processing the STOP STREAM
message, if there was no other change for that transaction, do not send
the STOP STREAM message.
Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/pgoutput/pgoutput.c | 189 ++++++++++++++++++++++++----
 1 file changed, 167 insertions(+), 22 deletions(-)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index f071beb..84aa9c8 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -67,6 +67,8 @@ static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   TimestampTz prepare_time);
 static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 								  ReorderBufferTXN *txn);
+static void pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+									   ReorderBufferTXN *txn);
 static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn);
 static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
@@ -169,12 +171,15 @@ typedef struct RelationSyncEntry
 /*
  * Maintain a per-transaction level variable to track whether the
  * transaction has sent BEGIN. BEGIN is only sent when the first
- * change in a transaction is processed. This makes it possible 
- * to skip transactions that are empty.
+ * change in a transaction is processed. Similarly while streaming
+ * transactions, STREAM_START is only sent with the first change.
+ * This makes it possible to skip transactions that are empty.
  */
 typedef struct PGOutputTxnData
 {
    bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+   bool sent_stream_start; /* flag indicating if stream start has been sent */
+   bool sent_first_stream;   /* flag indicating if any stream has been sent */
 } PGOutputTxnData;
 
 /* Map used to remember which relation schemas we sent. */
@@ -661,9 +666,18 @@ maybe_send_schema(LogicalDecodingContext *ctx,
    /* set up txndata */
    txndata = toptxn->output_plugin_private;
 
-	/* Send BEGIN if we haven't yet */
-	if (txndata && !txndata->sent_begin_txn)
+	if (in_streaming)
+	{
+		/* If streaming, send STREAM START if we haven't yet */
+		if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, toptxn);
+	}
+	else
+	{
+		/* If not streaming, send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
 			pgoutput_send_begin(ctx, toptxn);
+	}
 
 	/*
 	 * Send the schema.  If the changes will be published using an ancestor's
@@ -1288,9 +1302,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 &action))
 				break;
 
-			/* Send BEGIN if we haven't yet */
-			if (txndata && !txndata->sent_begin_txn)
-				pgoutput_send_begin(ctx, txn);
+   			if (in_streaming)
+			{
+				/* If streaming, send STREAM START if we haven't yet */
+				if (txndata && !txndata->sent_stream_start)
+					pgoutput_send_stream_start(ctx, txn);
+			}
+			else
+			{
+				/* If not streaming, send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+			}
 
 			/*
 			 * Schema should be sent using the original relation because it
@@ -1342,9 +1365,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
-			/* Send BEGIN if we haven't yet */
-			if (txndata && !txndata->sent_begin_txn)
-				pgoutput_send_begin(ctx, txn);
+   			if (in_streaming)
+			{
+				/* If streaming, send STREAM START if we haven't yet */
+				if (txndata && !txndata->sent_stream_start)
+					pgoutput_send_stream_start(ctx, txn);
+			}
+			else
+			{
+				/* If not streaming, send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+			}
 
 			maybe_send_schema(ctx, change, relation, relentry);
 
@@ -1404,9 +1436,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
-				/* Send BEGIN if we haven't yet */
-				if (txndata && !txndata->sent_begin_txn)
-					pgoutput_send_begin(ctx, txn);
+   				if (in_streaming)
+				{
+					/* If streaming, send STREAM START if we haven't yet */
+					if (txndata && !txndata->sent_stream_start)
+						pgoutput_send_stream_start(ctx, txn);
+				}
+				else
+				{
+					/* If not streaming, send BEGIN if we haven't yet */
+					if (txndata && !txndata->sent_begin_txn)
+						pgoutput_send_begin(ctx, txn);
+				}
 
 				maybe_send_schema(ctx, change, relation, relentry);
 
@@ -1484,9 +1525,18 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	{
 		txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
-		/* Send BEGIN if we haven't yet */
-		if (txndata && !txndata->sent_begin_txn)
+		if (in_streaming)
+		{
+			/* If streaming, send STREAM START if we haven't yet */
+			if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, txn);
+		}
+		else
+		{
+			/* If not streaming, send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
 			pgoutput_send_begin(ctx, txn);
+		}
 
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
@@ -1528,9 +1578,18 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	{
 		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
-		/* Send BEGIN if we haven't yet */
-		if (txndata && !txndata->sent_begin_txn)
+		if (in_streaming)
+		{
+			/* If streaming, send STREAM START if we haven't yet */
+			if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, txn);
+		}
+		else
+		{
+			/* If not streaming, send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
 			pgoutput_send_begin(ctx, txn);
+		}
 	}
 
 	OutputPluginPrepareWrite(ctx, true);
@@ -1615,28 +1674,60 @@ static void
 pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 					  ReorderBufferTXN *txn)
 {
-	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = txn->output_plugin_private;
 
 	/* we can't nest streaming of transactions */
 	Assert(!in_streaming);
 
 	/*
+	 * Don't actually send stream start here, instead set a flag that indicates
+	 * that stream start hasn't been sent and wait for the first actual change
+	 * for this stream to be sent and then send stream start. This is done
+	 * to avoid sending empty streams without any changes.
+	 */
+	if (txndata == NULL)
+	{
+		txndata =
+			MemoryContextAllocZero(ctx->context, sizeof(PGOutputTxnData));
+		txn->output_plugin_private = txndata;
+	}
+
+	txndata->sent_stream_start = false;
+	in_streaming = true;
+}
+
+/*
+ * Actually send START STREAM
+ */
+static void
+pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+					  ReorderBufferTXN *txn)
+{
+	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+
+	/*
 	 * If we already sent the first stream for this transaction then don't
 	 * send the origin id in the subsequent streams.
 	 */
-	if (rbtxn_is_streamed(txn))
+	if (txndata->sent_first_stream)
 		send_replication_origin = false;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
-	logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+	logicalrep_write_stream_start(ctx->out, txn->xid, !txndata->sent_first_stream);
 
 	send_repl_origin(ctx, txn->origin_id, InvalidXLogRecPtr,
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
 
-	/* we're streaming a chunk of transaction now */
-	in_streaming = true;
+	/*
+	 * Set the flags that indicate that changes were sent as part of
+	 * the transaction and the stream.
+	 */
+	txndata->sent_begin_txn = txndata->sent_stream_start = true;
+	txndata->sent_first_stream = true;
 }
 
 /*
@@ -1646,9 +1737,18 @@ static void
 pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 					 ReorderBufferTXN *txn)
 {
+	PGOutputTxnData *data = txn->output_plugin_private;
+
 	/* we should be streaming a trasanction */
 	Assert(in_streaming);
 
+	if (!data->sent_stream_start)
+	{
+		in_streaming = false;
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream stop");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_stop(ctx->out);
 	OutputPluginWrite(ctx, true);
@@ -1667,6 +1767,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 					  XLogRecPtr abort_lsn)
 {
 	ReorderBufferTXN *toptxn;
+	PGOutputTxnData  *txndata;
+	bool sent_first_stream;
 
 	/*
 	 * The abort should happen outside streaming block, even for streamed
@@ -1676,6 +1778,20 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 
 	/* determine the toplevel transaction */
 	toptxn = (txn->toptxn) ? txn->toptxn : txn;
+	txndata = toptxn->output_plugin_private;
+	sent_first_stream = txndata->sent_first_stream;
+
+	if (txn->toptxn == NULL)
+	{
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+	}
+
+	if (!sent_first_stream)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream abort");
+		return;
+	}
 
 	Assert(rbtxn_is_streamed(toptxn));
 
@@ -1695,6 +1811,9 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 					   ReorderBufferTXN *txn,
 					   XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_first_stream = txndata->sent_first_stream;
+
 	/*
 	 * The commit should happen outside streaming block, even for streamed
 	 * transactions. The transaction has to be marked as streamed, though.
@@ -1702,6 +1821,20 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	/*
+	 * If no changes were part of this transaction then drop the commit
+	 * but send the update progress.
+	 */
+	if (!sent_first_stream)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream commit");
+		OutputPluginUpdateProgress(ctx, true);
+		return;
+	}
+
 	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
@@ -1721,8 +1854,20 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 							ReorderBufferTXN *txn,
 							XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_begin_txn = txndata->sent_begin_txn;
+
 	Assert(rbtxn_is_streamed(txn));
 
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream prepare");
+		return;
+	}
+
 	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
-- 
1.8.3.1

v23-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v23-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From c271271321f32fe8bcc466ed8a85290f0fe7908d Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Mon, 28 Feb 2022 23:35:42 -0500
Subject: [PATCH v23 1/2] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message, if there was
no other change for that transaction, do not send the COMMIT message.
This means that pgoutput will skip BEGIN/COMMIT messages for transactions
that are empty.

The patch also makes sure that in synchronous replication mode,
when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "streaming"
or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 118 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 ++-
 src/backend/replication/walsender.c         |  31 +++++---
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 151 insertions(+), 26 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..99b2775 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a04..f071beb 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -166,6 +166,17 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible 
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+   bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,15 +463,42 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -475,7 +513,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+
+	pfree(txndata);
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in commit");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -506,7 +562,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +576,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +592,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -556,6 +612,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	bool		schema_sent;
 	TransactionId xid = InvalidTransactionId;
 	TransactionId topxid = InvalidTransactionId;
+	PGOutputTxnData *txndata;
+	ReorderBufferTXN *toptxn;
 
 	/*
 	 * Remember XID of the (sub)transaction for the change. We don't care if
@@ -569,9 +627,15 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 		xid = change->txn->xid;
 
 	if (change->txn->toptxn)
+	{
 		topxid = change->txn->toptxn->xid;
+		toptxn = change->txn->toptxn;
+	}
 	else
+	{
 		topxid = xid;
+		toptxn = change->txn;
+	}
 
 	/*
 	 * Do we need to send the schema? We do track streamed transactions
@@ -594,6 +658,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	if (schema_sent)
 		return;
 
+   /* set up txndata */
+   txndata = toptxn->output_plugin_private;
+
+	/* Send BEGIN if we haven't yet */
+	if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, toptxn);
+
 	/*
 	 * Send the schema.  If the changes will be published using an ancestor's
 	 * schema, not the relation's own, send that ancestor's schema before
@@ -1141,6 +1212,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1216,6 +1288,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
@@ -1266,6 +1342,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1324,6 +1404,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1354,6 +1438,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1397,6 +1482,12 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -1429,6 +1520,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for non-transactional messages.
+	 */
+	if (in_streaming || transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1598,7 +1702,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1619,7 +1723,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index ce163b9..11f7358 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
 }
 
 /*
+ * Check if synchronous replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Calculate the synced Write, Flush and Apply positions among sync standbys.
  *
  * Return false if the number of sync standbys is less than
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 5a718b1..33d6be7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,12 +1450,20 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keepalive to keep the MyWalSnd locations updated.
+	 */
+	if (send_keepalive && SyncRepEnabled())
+		WalSndKeepalive(true, ctx->write_location);
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1550,7 +1559,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2077,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3083,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3588,18 +3597,20 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * If writePtr is set, mark that as the LSN processed, else use sentPtr.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3638,7 +3649,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..9f59855 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keepalive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..eb91d17 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 27be230..7086532 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d9b83f7..77f33b2 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1608,6 +1608,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#86Peter Smith
smithpb2250@gmail.com
In reply to: Ajin Cherian (#85)
Re: logical replication empty transactions

On Fri, Mar 4, 2022 at 12:41 PM Ajin Cherian <itsajin@gmail.com> wrote:

I have split the patch into two. I have kept the logic of skipping
streaming changes in the second patch.
I will work on the second patch once we can figure out a solution for
the COMMIT PREPARED after restart problem.

Please see below my review comments for the first patch only (v23-0001)

======

1. Patch failed to apply cleanly - whitespace warnings.

git apply ../patches_misc/v23-0001-Skip-empty-transactions-for-logical-replication.patch
../patches_misc/v23-0001-Skip-empty-transactions-for-logical-replication.patch:68:
trailing whitespace.
* change in a transaction is processed. This makes it possible
warning: 1 line adds whitespace errors.

~~~

2. src/backend/replication/pgoutput/pgoutput.c - typedef struct PGOutputTxnData

+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData

I felt that this comment is describing details all about its bool
member but I think maybe it should be describing something also about
the structure itself (because this is the structure comment). E.g. it
should say about it only being allocated by the pgoutput_begin_txn()
and it is accessible via txn->output_plugin_private. Maybe also say
this has subtle implications if this is NULL then it means the tx
can't be 2PC etc...

~~~

3. src/backend/replication/pgoutput/pgoutput.c - pgoutput_send_begin

+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)

IMO there is no need to repeat "This is where the BEGIN is actually
sent.", because "Send BEGIN." already said the same thing :-)

~~~

4. src/backend/replication/pgoutput/pgoutput.c - pgoutput_commit_txn

+ /*
+ * If a BEGIN message was not yet sent, then it means there were no relevant
+ * changes encountered, so we can skip the COMMIT message too.
+ */
+ sent_begin_txn = txndata->sent_begin_txn;
+ txn->output_plugin_private = NULL;
+ OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+
+ pfree(txndata);

Not quite sure why this pfree is positioned where it is (after that
function call). I felt this should be a couple of lines up so txndata
is freed as soon as you had no more use for it (i.e. after you copied
the bool from it)

~~~

5. src/backend/replication/pgoutput/pgoutput.c - maybe_send_schema

@@ -594,6 +658,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
if (schema_sent)
return;

+   /* set up txndata */
+   txndata = toptxn->output_plugin_private;

The comment does quite feel right. Nothing is "setting up" anything.
Really, all this does is assign a reference to the tx private data.
Probably better with no comment at all?

~~~

6. src/backend/replication/pgoutput/pgoutput.c - maybe_send_schema

I observed that every call to the maybe_send_schema function also has
adjacent code that already/always is checking to call
pgoutput_send_begin_tx function.

So then I am wondering is the added logic to the maybe_send_schema
even needed at all? It looks a bit redundant. Thoughts?

~~~

7. src/backend/replication/pgoutput/pgoutput.c - pgoutput_change

@@ -1141,6 +1212,7 @@ pgoutput_change(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
  Relation relation, ReorderBufferChange *change)
 {
  PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
  MemoryContext old;

Maybe if is worth deferring this assignment until after the row-filter
check. Otherwise, you are maybe doing it for nothing and IIRC this is
hot code so the less you do here the better. OTOH a single assignment
probably amounts to almost nothing.

~~~

8. src/backend/replication/pgoutput/pgoutput.c - pgoutput_change

@@ -1354,6 +1438,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
int nrelations, Relation relations[], ReorderBufferChange *change)
{
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ PGOutputTxnData *txndata;
MemoryContext old;

This variable declaration should be done later in the block where it
is assigned.

~~~

9. src/backend/replication/pgoutput/pgoutput.c - suggestion

I notice there is quite a few places in the patch that look like:

+ txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+ /* Send BEGIN if we haven't yet */
+ if (txndata && !txndata->sent_begin_txn)
+ pgoutput_send_begin(ctx, txn);
+

It might be worth considering encapsulating all those in a helper function like:
pgoutput_maybe_send_begin(ctx, txn)

It would certainly be a lot tidier.

~~~

10. src/backend/replication/syncrep.c - SyncRepEnabled

@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
}

 /*
+ * Check if synchronous replication is enabled
+ */
+bool
+SyncRepEnabled(void)

Missing period for that function comment.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

#87shiy.fnst@fujitsu.com
shiy.fnst@fujitsu.com
In reply to: Ajin Cherian (#85)
RE: logical replication empty transactions

On Fri, Mar 4, 2022 9:41 AM Ajin Cherian <itsajin@gmail.com> wrote:

I have split the patch into two. I have kept the logic of skipping
streaming changes in the second patch.
I will work on the second patch once we can figure out a solution for
the COMMIT PREPARED after restart problem.

Thanks for updating the patch.

A comment on v23-0001 patch.

@@ -1429,6 +1520,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (in_streaming)
xid = txn->xid;

+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for non-transactional messages.
+	 */
+	if (in_streaming || transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,

I think we don't need to send BEGIN if in_streaming is true, right? The first
patch doesn't skip streamed transaction, so should we modify
+ if (in_streaming || transactional)
to
+ if (!in_streaming && transactional)
?

Regards,
Shi yu

#88Ajin Cherian
itsajin@gmail.com
In reply to: shiy.fnst@fujitsu.com (#87)
2 attachment(s)
Re: logical replication empty transactions

On Mon, Mar 7, 2022 at 7:50 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:

On Fri, Mar 4, 2022 9:41 AM Ajin Cherian <itsajin@gmail.com> wrote:

I have split the patch into two. I have kept the logic of skipping
streaming changes in the second patch.
I will work on the second patch once we can figure out a solution for
the COMMIT PREPARED after restart problem.

Thanks for updating the patch.

A comment on v23-0001 patch.

@@ -1429,6 +1520,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (in_streaming)
xid = txn->xid;

+       /*
+        * Output BEGIN if we haven't yet.
+        * Avoid for non-transactional messages.
+        */
+       if (in_streaming || transactional)
+       {
+               PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+               /* Send BEGIN if we haven't yet */
+               if (txndata && !txndata->sent_begin_txn)
+                       pgoutput_send_begin(ctx, txn);
+       }
+
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_message(ctx->out,
xid,
I think we don't need to send BEGIN if in_streaming is true, right? The first
patch doesn't skip streamed transaction, so should we modify
+       if (in_streaming || transactional)
to
+       if (!in_streaming && transactional)
?

Fixed.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v24-0002-Skip-empty-streamed-transactions-for-logical-rep.patchapplication/octet-stream; name=v24-0002-Skip-empty-streamed-transactions-for-logical-rep.patchDownload
From 86779b2911b87b9f92228fc4e1494c979ef1edbe Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Mon, 7 Mar 2022 07:38:40 -0500
Subject: [PATCH v24 2/2] Skip empty streamed transactions for logical
 replication.

This patch postpones the START STREAM message while streaming large in-progress
transactions until the first change. While processing the STOP STREAM
message, if there was no other change for that transaction, do not send
the STOP STREAM message.
Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/pgoutput/pgoutput.c | 191 ++++++++++++++++++++++++----
 1 file changed, 168 insertions(+), 23 deletions(-)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index b0a98f2..84aa9c8 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -67,6 +67,8 @@ static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   TimestampTz prepare_time);
 static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 								  ReorderBufferTXN *txn);
+static void pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+									   ReorderBufferTXN *txn);
 static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn);
 static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
@@ -169,12 +171,15 @@ typedef struct RelationSyncEntry
 /*
  * Maintain a per-transaction level variable to track whether the
  * transaction has sent BEGIN. BEGIN is only sent when the first
- * change in a transaction is processed. This makes it possible
- * to skip transactions that are empty.
+ * change in a transaction is processed. Similarly while streaming
+ * transactions, STREAM_START is only sent with the first change.
+ * This makes it possible to skip transactions that are empty.
  */
 typedef struct PGOutputTxnData
 {
    bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+   bool sent_stream_start; /* flag indicating if stream start has been sent */
+   bool sent_first_stream;   /* flag indicating if any stream has been sent */
 } PGOutputTxnData;
 
 /* Map used to remember which relation schemas we sent. */
@@ -661,9 +666,18 @@ maybe_send_schema(LogicalDecodingContext *ctx,
    /* set up txndata */
    txndata = toptxn->output_plugin_private;
 
-	/* Send BEGIN if we haven't yet */
-	if (txndata && !txndata->sent_begin_txn)
+	if (in_streaming)
+	{
+		/* If streaming, send STREAM START if we haven't yet */
+		if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, toptxn);
+	}
+	else
+	{
+		/* If not streaming, send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
 			pgoutput_send_begin(ctx, toptxn);
+	}
 
 	/*
 	 * Send the schema.  If the changes will be published using an ancestor's
@@ -1288,9 +1302,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 &action))
 				break;
 
-			/* Send BEGIN if we haven't yet */
-			if (txndata && !txndata->sent_begin_txn)
-				pgoutput_send_begin(ctx, txn);
+   			if (in_streaming)
+			{
+				/* If streaming, send STREAM START if we haven't yet */
+				if (txndata && !txndata->sent_stream_start)
+					pgoutput_send_stream_start(ctx, txn);
+			}
+			else
+			{
+				/* If not streaming, send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+			}
 
 			/*
 			 * Schema should be sent using the original relation because it
@@ -1342,9 +1365,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
-			/* Send BEGIN if we haven't yet */
-			if (txndata && !txndata->sent_begin_txn)
-				pgoutput_send_begin(ctx, txn);
+   			if (in_streaming)
+			{
+				/* If streaming, send STREAM START if we haven't yet */
+				if (txndata && !txndata->sent_stream_start)
+					pgoutput_send_stream_start(ctx, txn);
+			}
+			else
+			{
+				/* If not streaming, send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+			}
 
 			maybe_send_schema(ctx, change, relation, relentry);
 
@@ -1404,9 +1436,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
-				/* Send BEGIN if we haven't yet */
-				if (txndata && !txndata->sent_begin_txn)
-					pgoutput_send_begin(ctx, txn);
+   				if (in_streaming)
+				{
+					/* If streaming, send STREAM START if we haven't yet */
+					if (txndata && !txndata->sent_stream_start)
+						pgoutput_send_stream_start(ctx, txn);
+				}
+				else
+				{
+					/* If not streaming, send BEGIN if we haven't yet */
+					if (txndata && !txndata->sent_begin_txn)
+						pgoutput_send_begin(ctx, txn);
+				}
 
 				maybe_send_schema(ctx, change, relation, relentry);
 
@@ -1484,9 +1525,18 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	{
 		txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
-		/* Send BEGIN if we haven't yet */
-		if (txndata && !txndata->sent_begin_txn)
+		if (in_streaming)
+		{
+			/* If streaming, send STREAM START if we haven't yet */
+			if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, txn);
+		}
+		else
+		{
+			/* If not streaming, send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
 			pgoutput_send_begin(ctx, txn);
+		}
 
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
@@ -1524,13 +1574,22 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	 * Output BEGIN if we haven't yet.
 	 * Avoid for non-transactional messages.
 	 */
-	if (!in_streaming && transactional)
+	if (in_streaming || transactional)
 	{
 		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
-		/* Send BEGIN if we haven't yet */
-		if (txndata && !txndata->sent_begin_txn)
+		if (in_streaming)
+		{
+			/* If streaming, send STREAM START if we haven't yet */
+			if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, txn);
+		}
+		else
+		{
+			/* If not streaming, send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
 			pgoutput_send_begin(ctx, txn);
+		}
 	}
 
 	OutputPluginPrepareWrite(ctx, true);
@@ -1615,28 +1674,60 @@ static void
 pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 					  ReorderBufferTXN *txn)
 {
-	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = txn->output_plugin_private;
 
 	/* we can't nest streaming of transactions */
 	Assert(!in_streaming);
 
 	/*
+	 * Don't actually send stream start here, instead set a flag that indicates
+	 * that stream start hasn't been sent and wait for the first actual change
+	 * for this stream to be sent and then send stream start. This is done
+	 * to avoid sending empty streams without any changes.
+	 */
+	if (txndata == NULL)
+	{
+		txndata =
+			MemoryContextAllocZero(ctx->context, sizeof(PGOutputTxnData));
+		txn->output_plugin_private = txndata;
+	}
+
+	txndata->sent_stream_start = false;
+	in_streaming = true;
+}
+
+/*
+ * Actually send START STREAM
+ */
+static void
+pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+					  ReorderBufferTXN *txn)
+{
+	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+
+	/*
 	 * If we already sent the first stream for this transaction then don't
 	 * send the origin id in the subsequent streams.
 	 */
-	if (rbtxn_is_streamed(txn))
+	if (txndata->sent_first_stream)
 		send_replication_origin = false;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
-	logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+	logicalrep_write_stream_start(ctx->out, txn->xid, !txndata->sent_first_stream);
 
 	send_repl_origin(ctx, txn->origin_id, InvalidXLogRecPtr,
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
 
-	/* we're streaming a chunk of transaction now */
-	in_streaming = true;
+	/*
+	 * Set the flags that indicate that changes were sent as part of
+	 * the transaction and the stream.
+	 */
+	txndata->sent_begin_txn = txndata->sent_stream_start = true;
+	txndata->sent_first_stream = true;
 }
 
 /*
@@ -1646,9 +1737,18 @@ static void
 pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 					 ReorderBufferTXN *txn)
 {
+	PGOutputTxnData *data = txn->output_plugin_private;
+
 	/* we should be streaming a trasanction */
 	Assert(in_streaming);
 
+	if (!data->sent_stream_start)
+	{
+		in_streaming = false;
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream stop");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_stop(ctx->out);
 	OutputPluginWrite(ctx, true);
@@ -1667,6 +1767,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 					  XLogRecPtr abort_lsn)
 {
 	ReorderBufferTXN *toptxn;
+	PGOutputTxnData  *txndata;
+	bool sent_first_stream;
 
 	/*
 	 * The abort should happen outside streaming block, even for streamed
@@ -1676,6 +1778,20 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 
 	/* determine the toplevel transaction */
 	toptxn = (txn->toptxn) ? txn->toptxn : txn;
+	txndata = toptxn->output_plugin_private;
+	sent_first_stream = txndata->sent_first_stream;
+
+	if (txn->toptxn == NULL)
+	{
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+	}
+
+	if (!sent_first_stream)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream abort");
+		return;
+	}
 
 	Assert(rbtxn_is_streamed(toptxn));
 
@@ -1695,6 +1811,9 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 					   ReorderBufferTXN *txn,
 					   XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_first_stream = txndata->sent_first_stream;
+
 	/*
 	 * The commit should happen outside streaming block, even for streamed
 	 * transactions. The transaction has to be marked as streamed, though.
@@ -1702,6 +1821,20 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	/*
+	 * If no changes were part of this transaction then drop the commit
+	 * but send the update progress.
+	 */
+	if (!sent_first_stream)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream commit");
+		OutputPluginUpdateProgress(ctx, true);
+		return;
+	}
+
 	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
@@ -1721,8 +1854,20 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 							ReorderBufferTXN *txn,
 							XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_begin_txn = txndata->sent_begin_txn;
+
 	Assert(rbtxn_is_streamed(txn));
 
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream prepare");
+		return;
+	}
+
 	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
-- 
1.8.3.1

v24-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v24-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 57e23b5eb51ef9c619302b5be19acd8cf248a43f Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Mon, 7 Mar 2022 07:24:48 -0500
Subject: [PATCH v24 1/2] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message, if there was
no other change for that transaction, do not send the COMMIT message.
This means that pgoutput will skip BEGIN/COMMIT messages for transactions
that are empty.

The patch also makes sure that in synchronous replication mode,
when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "streaming"
or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 118 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 ++-
 src/backend/replication/walsender.c         |  31 +++++---
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 151 insertions(+), 26 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..99b2775 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a04..b0a98f2 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -166,6 +166,17 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+   bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,15 +463,42 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -475,7 +513,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+
+	pfree(txndata);
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in commit");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -506,7 +562,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +576,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +592,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -556,6 +612,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	bool		schema_sent;
 	TransactionId xid = InvalidTransactionId;
 	TransactionId topxid = InvalidTransactionId;
+	PGOutputTxnData *txndata;
+	ReorderBufferTXN *toptxn;
 
 	/*
 	 * Remember XID of the (sub)transaction for the change. We don't care if
@@ -569,9 +627,15 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 		xid = change->txn->xid;
 
 	if (change->txn->toptxn)
+	{
 		topxid = change->txn->toptxn->xid;
+		toptxn = change->txn->toptxn;
+	}
 	else
+	{
 		topxid = xid;
+		toptxn = change->txn;
+	}
 
 	/*
 	 * Do we need to send the schema? We do track streamed transactions
@@ -594,6 +658,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	if (schema_sent)
 		return;
 
+   /* set up txndata */
+   txndata = toptxn->output_plugin_private;
+
+	/* Send BEGIN if we haven't yet */
+	if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, toptxn);
+
 	/*
 	 * Send the schema.  If the changes will be published using an ancestor's
 	 * schema, not the relation's own, send that ancestor's schema before
@@ -1141,6 +1212,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1216,6 +1288,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
@@ -1266,6 +1342,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1324,6 +1404,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1354,6 +1438,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1397,6 +1482,12 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -1429,6 +1520,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for non-transactional messages.
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1598,7 +1702,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1619,7 +1723,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index ce163b9..11f7358 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
 }
 
 /*
+ * Check if synchronous replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Calculate the synced Write, Flush and Apply positions among sync standbys.
  *
  * Return false if the number of sync standbys is less than
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 5a718b1..33d6be7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,12 +1450,20 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keepalive to keep the MyWalSnd locations updated.
+	 */
+	if (send_keepalive && SyncRepEnabled())
+		WalSndKeepalive(true, ctx->write_location);
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1550,7 +1559,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2077,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3083,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3588,18 +3597,20 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * If writePtr is set, mark that as the LSN processed, else use sentPtr.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3638,7 +3649,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..9f59855 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keepalive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..eb91d17 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 27be230..7086532 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d9b83f7..77f33b2 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1608,6 +1608,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#89Ajin Cherian
itsajin@gmail.com
In reply to: Ajin Cherian (#88)
2 attachment(s)
Re: logical replication empty transactions

On Mon, Mar 7, 2022 at 11:44 PM Ajin Cherian <itsajin@gmail.com> wrote:

Fixed.

regards,
Ajin Cherian
Fujitsu Australia

Rebased the patch and fixed some whitespace errors.
regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v25-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v25-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 5e726e6baeff604449ea5ac6d004ab135b5b3225 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Wed, 16 Mar 2022 01:09:05 -0400
Subject: [PATCH v25 1/2] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message, if there was
no other change for that transaction, do not send the COMMIT message.
This means that pgoutput will skip BEGIN/COMMIT messages for transactions
that are empty.

The patch also makes sure that in synchronous replication mode,
when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "streaming"
or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 118 ++++++++++++++++++++++++++--
 src/backend/replication/syncrep.c           |  12 ++-
 src/backend/replication/walsender.c         |  31 +++++---
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 9 files changed, 151 insertions(+), 26 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..99b2775 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a04..b0a98f2 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -166,6 +166,17 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the
+ * transaction has sent BEGIN. BEGIN is only sent when the first
+ * change in a transaction is processed. This makes it possible
+ * to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData
+{
+   bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+} PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,15 +463,42 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set
+ * of tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
 pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
+	PGOutputTxnData    *txndata = MemoryContextAllocZero(ctx->context,
+														 sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+{
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -475,7 +513,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool            sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no relevant
+	 * changes encountered, so we can skip the COMMIT message too.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+
+	pfree(txndata);
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in commit");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -506,7 +562,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +576,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +592,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -556,6 +612,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	bool		schema_sent;
 	TransactionId xid = InvalidTransactionId;
 	TransactionId topxid = InvalidTransactionId;
+	PGOutputTxnData *txndata;
+	ReorderBufferTXN *toptxn;
 
 	/*
 	 * Remember XID of the (sub)transaction for the change. We don't care if
@@ -569,9 +627,15 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 		xid = change->txn->xid;
 
 	if (change->txn->toptxn)
+	{
 		topxid = change->txn->toptxn->xid;
+		toptxn = change->txn->toptxn;
+	}
 	else
+	{
 		topxid = xid;
+		toptxn = change->txn;
+	}
 
 	/*
 	 * Do we need to send the schema? We do track streamed transactions
@@ -594,6 +658,13 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 	if (schema_sent)
 		return;
 
+   /* set up txndata */
+   txndata = toptxn->output_plugin_private;
+
+	/* Send BEGIN if we haven't yet */
+	if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, toptxn);
+
 	/*
 	 * Send the schema.  If the changes will be published using an ancestor's
 	 * schema, not the relation's own, send that ancestor's schema before
@@ -1141,6 +1212,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1216,6 +1288,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
@@ -1266,6 +1342,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1324,6 +1404,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1354,6 +1438,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1397,6 +1482,12 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 	if (nrelids > 0)
 	{
+		txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -1429,6 +1520,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet.
+	 * Avoid for non-transactional messages.
+	 */
+	if (!in_streaming && transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1598,7 +1702,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1619,7 +1723,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index ce163b9..11f7358 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
 }
 
 /*
+ * Check if synchronous replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Calculate the synced Write, Flush and Apply positions among sync standbys.
  *
  * Return false if the number of sync standbys is less than
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2d0292a..21e7ac3 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,12 +1450,20 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keepalive to keep the MyWalSnd locations updated.
+	 */
+	if (send_keepalive && SyncRepEnabled())
+		WalSndKeepalive(true, ctx->write_location);
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1550,7 +1559,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2077,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3083,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3563,18 +3572,20 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * If writePtr is set, mark that as the LSN processed, else use sentPtr.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3613,7 +3624,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..9f59855 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keepalive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..eb91d17 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 27be230..7086532 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index eaf3e7a..4c575f3 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1608,6 +1608,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

v25-0002-Skip-empty-streamed-transactions-for-logical-rep.patchapplication/octet-stream; name=v25-0002-Skip-empty-streamed-transactions-for-logical-rep.patchDownload
From 09af8bb92e13d3f466e53eb58198f6c6a99d5ddd Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Wed, 16 Mar 2022 02:26:22 -0400
Subject: [PATCH v25 2/2] Skip empty streamed transactions for logical
 replication.

This patch postpones the START STREAM message while streaming large in-progress
transactions until the first change. While processing the STOP STREAM
message, if there was no other change for that transaction, do not send
the STOP STREAM message.
Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/pgoutput/pgoutput.c | 191 ++++++++++++++++++++++++----
 1 file changed, 168 insertions(+), 23 deletions(-)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index b0a98f2..221ef8d 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -67,6 +67,8 @@ static void pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 										   TimestampTz prepare_time);
 static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 								  ReorderBufferTXN *txn);
+static void pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+									   ReorderBufferTXN *txn);
 static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 								 ReorderBufferTXN *txn);
 static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
@@ -169,12 +171,15 @@ typedef struct RelationSyncEntry
 /*
  * Maintain a per-transaction level variable to track whether the
  * transaction has sent BEGIN. BEGIN is only sent when the first
- * change in a transaction is processed. This makes it possible
- * to skip transactions that are empty.
+ * change in a transaction is processed. Similarly while streaming
+ * transactions, STREAM_START is only sent with the first change.
+ * This makes it possible to skip transactions that are empty.
  */
 typedef struct PGOutputTxnData
 {
    bool sent_begin_txn;    /* flag indicating whether BEGIN has been sent */
+   bool sent_stream_start; /* flag indicating if stream start has been sent */
+   bool sent_first_stream;   /* flag indicating if any stream has been sent */
 } PGOutputTxnData;
 
 /* Map used to remember which relation schemas we sent. */
@@ -661,9 +666,18 @@ maybe_send_schema(LogicalDecodingContext *ctx,
    /* set up txndata */
    txndata = toptxn->output_plugin_private;
 
-	/* Send BEGIN if we haven't yet */
-	if (txndata && !txndata->sent_begin_txn)
+	if (in_streaming)
+	{
+		/* If streaming, send STREAM START if we haven't yet */
+		if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, toptxn);
+	}
+	else
+	{
+		/* If not streaming, send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
 			pgoutput_send_begin(ctx, toptxn);
+	}
 
 	/*
 	 * Send the schema.  If the changes will be published using an ancestor's
@@ -1288,9 +1302,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 &action))
 				break;
 
-			/* Send BEGIN if we haven't yet */
-			if (txndata && !txndata->sent_begin_txn)
-				pgoutput_send_begin(ctx, txn);
+			if (in_streaming)
+			{
+				/* If streaming, send STREAM START if we haven't yet */
+				if (txndata && !txndata->sent_stream_start)
+					pgoutput_send_stream_start(ctx, txn);
+			}
+			else
+			{
+				/* If not streaming, send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+			}
 
 			/*
 			 * Schema should be sent using the original relation because it
@@ -1342,9 +1365,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
-			/* Send BEGIN if we haven't yet */
-			if (txndata && !txndata->sent_begin_txn)
-				pgoutput_send_begin(ctx, txn);
+			if (in_streaming)
+			{
+				/* If streaming, send STREAM START if we haven't yet */
+				if (txndata && !txndata->sent_stream_start)
+					pgoutput_send_stream_start(ctx, txn);
+			}
+			else
+			{
+				/* If not streaming, send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+			}
 
 			maybe_send_schema(ctx, change, relation, relentry);
 
@@ -1404,9 +1436,18 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
-				/* Send BEGIN if we haven't yet */
-				if (txndata && !txndata->sent_begin_txn)
-					pgoutput_send_begin(ctx, txn);
+				if (in_streaming)
+				{
+					/* If streaming, send STREAM START if we haven't yet */
+					if (txndata && !txndata->sent_stream_start)
+						pgoutput_send_stream_start(ctx, txn);
+				}
+				else
+				{
+					/* If not streaming, send BEGIN if we haven't yet */
+					if (txndata && !txndata->sent_begin_txn)
+						pgoutput_send_begin(ctx, txn);
+				}
 
 				maybe_send_schema(ctx, change, relation, relentry);
 
@@ -1484,9 +1525,18 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	{
 		txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
-		/* Send BEGIN if we haven't yet */
-		if (txndata && !txndata->sent_begin_txn)
+		if (in_streaming)
+		{
+			/* If streaming, send STREAM START if we haven't yet */
+			if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, txn);
+		}
+		else
+		{
+			/* If not streaming, send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
 			pgoutput_send_begin(ctx, txn);
+		}
 
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
@@ -1524,13 +1574,22 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	 * Output BEGIN if we haven't yet.
 	 * Avoid for non-transactional messages.
 	 */
-	if (!in_streaming && transactional)
+	if (in_streaming || transactional)
 	{
 		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
-		/* Send BEGIN if we haven't yet */
-		if (txndata && !txndata->sent_begin_txn)
+		if (in_streaming)
+		{
+			/* If streaming, send STREAM START if we haven't yet */
+			if (txndata && !txndata->sent_stream_start)
+			pgoutput_send_stream_start(ctx, txn);
+		}
+		else
+		{
+			/* If not streaming, send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
 			pgoutput_send_begin(ctx, txn);
+		}
 	}
 
 	OutputPluginPrepareWrite(ctx, true);
@@ -1615,28 +1674,60 @@ static void
 pgoutput_stream_start(struct LogicalDecodingContext *ctx,
 					  ReorderBufferTXN *txn)
 {
-	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = txn->output_plugin_private;
 
 	/* we can't nest streaming of transactions */
 	Assert(!in_streaming);
 
 	/*
+	 * Don't actually send stream start here, instead set a flag that indicates
+	 * that stream start hasn't been sent and wait for the first actual change
+	 * for this stream to be sent and then send stream start. This is done
+	 * to avoid sending empty streams without any changes.
+	 */
+	if (txndata == NULL)
+	{
+		txndata =
+			MemoryContextAllocZero(ctx->context, sizeof(PGOutputTxnData));
+		txn->output_plugin_private = txndata;
+	}
+
+	txndata->sent_stream_start = false;
+	in_streaming = true;
+}
+
+/*
+ * Actually send START STREAM
+ */
+static void
+pgoutput_send_stream_start(struct LogicalDecodingContext *ctx,
+					  ReorderBufferTXN *txn)
+{
+	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData	*txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+
+	/*
 	 * If we already sent the first stream for this transaction then don't
 	 * send the origin id in the subsequent streams.
 	 */
-	if (rbtxn_is_streamed(txn))
+	if (txndata->sent_first_stream)
 		send_replication_origin = false;
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
-	logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+	logicalrep_write_stream_start(ctx->out, txn->xid, !txndata->sent_first_stream);
 
 	send_repl_origin(ctx, txn->origin_id, InvalidXLogRecPtr,
 					 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
 
-	/* we're streaming a chunk of transaction now */
-	in_streaming = true;
+	/*
+	 * Set the flags that indicate that changes were sent as part of
+	 * the transaction and the stream.
+	 */
+	txndata->sent_begin_txn = txndata->sent_stream_start = true;
+	txndata->sent_first_stream = true;
 }
 
 /*
@@ -1646,9 +1737,18 @@ static void
 pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
 					 ReorderBufferTXN *txn)
 {
+	PGOutputTxnData *data = txn->output_plugin_private;
+
 	/* we should be streaming a trasanction */
 	Assert(in_streaming);
 
+	if (!data->sent_stream_start)
+	{
+		in_streaming = false;
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream stop");
+		return;
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_stop(ctx->out);
 	OutputPluginWrite(ctx, true);
@@ -1667,6 +1767,8 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 					  XLogRecPtr abort_lsn)
 {
 	ReorderBufferTXN *toptxn;
+	PGOutputTxnData  *txndata;
+	bool sent_first_stream;
 
 	/*
 	 * The abort should happen outside streaming block, even for streamed
@@ -1676,6 +1778,20 @@ pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 
 	/* determine the toplevel transaction */
 	toptxn = (txn->toptxn) ? txn->toptxn : txn;
+	txndata = toptxn->output_plugin_private;
+	sent_first_stream = txndata->sent_first_stream;
+
+	if (txn->toptxn == NULL)
+	{
+		pfree(txndata);
+		txn->output_plugin_private = NULL;
+	}
+
+	if (!sent_first_stream)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream abort");
+		return;
+	}
 
 	Assert(rbtxn_is_streamed(toptxn));
 
@@ -1695,6 +1811,9 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 					   ReorderBufferTXN *txn,
 					   XLogRecPtr commit_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_first_stream = txndata->sent_first_stream;
+
 	/*
 	 * The commit should happen outside streaming block, even for streamed
 	 * transactions. The transaction has to be marked as streamed, though.
@@ -1702,6 +1821,20 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	/*
+	 * If no changes were part of this transaction then drop the commit
+	 * but send the update progress.
+	 */
+	if (!sent_first_stream)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream commit");
+		OutputPluginUpdateProgress(ctx, true);
+		return;
+	}
+
 	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
@@ -1721,8 +1854,20 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 							ReorderBufferTXN *txn,
 							XLogRecPtr prepare_lsn)
 {
+	PGOutputTxnData *txndata = txn->output_plugin_private;
+	bool			sent_begin_txn = txndata->sent_begin_txn;
+
 	Assert(rbtxn_is_streamed(txn));
 
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipping replication of an empty transaction in stream prepare");
+		return;
+	}
+
 	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
-- 
1.8.3.1

#90Amit Kapila
amit.kapila16@gmail.com
In reply to: Ajin Cherian (#89)
Re: logical replication empty transactions

On Wed, Mar 16, 2022 at 12:33 PM Ajin Cherian <itsajin@gmail.com> wrote:

On Mon, Mar 7, 2022 at 11:44 PM Ajin Cherian <itsajin@gmail.com> wrote:

Fixed.

Review comments/suggestions:
=========================
1. Isn't it sufficient to call pgoutput_send_begin from
maybe_send_schema as that is commonplace for all others and is always
the first message we send? If so, I think we can remove it from other
places?
2. Can we write some comments to explain why we don't skip streaming
or prepared empty transactions and some possible solutions (the
protocol change and additional subscription parameter as discussed
[1]: ) as discussed in this thread pgoutput.c? 3. Can we add a simple test for it in one of the existing test files(say in 001_rep_changes.pl)? 4. I think we can drop the skip streaming patch as we can't do that for now.
3. Can we add a simple test for it in one of the existing test
files(say in 001_rep_changes.pl)?
4. I think we can drop the skip streaming patch as we can't do that for now.

--
With Regards,
Amit Kapila.

#91Ajin Cherian
itsajin@gmail.com
In reply to: Amit Kapila (#90)
1 attachment(s)
Re: logical replication empty transactions

On Thu, Mar 17, 2022 at 10:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

Review comments/suggestions:
=========================
1. Isn't it sufficient to call pgoutput_send_begin from
maybe_send_schema as that is commonplace for all others and is always
the first message we send? If so, I think we can remove it from other
places?

I've done the other way, I've removed it from maybe_send_schema as we
always call this
prior to calling maybe_send_schema.

2. Can we write some comments to explain why we don't skip streaming
or prepared empty transactions and some possible solutions (the
protocol change and additional subscription parameter as discussed
[1]) as discussed in this thread pgoutput.c?

I've added comment in the header of pgoutput_begin_prepare_txn() and
pgoutput_stream_start()

3. Can we add a simple test for it in one of the existing test
files(say in 001_rep_changes.pl)?

added a simple test.

4. I think we can drop the skip streaming patch as we can't do that for now.

Dropped,

In addition, I have also added a few more comments explaining why the begin send
is delayed in pgoutput_change till row_filter is checked and also ran pgindent.

regards,
Ajin Cherian
Fujitsu Australia

Attachments:

v26-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v26-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 799ede2f30453aaa0ed379735125a4ab008f383e Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Wed, 16 Mar 2022 01:09:05 -0400
Subject: [PATCH v26] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message, if there was
no other change for that transaction, do not send the COMMIT message.
This means that pgoutput will skip BEGIN/COMMIT messages for transactions
that are empty.

The patch also makes sure that in synchronous replication mode,
when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "streaming"
or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   4 +-
 src/backend/replication/pgoutput/pgoutput.c | 142 +++++++++++++++++++++++++---
 src/backend/replication/syncrep.c           |  12 ++-
 src/backend/replication/walsender.c         |  31 ++++--
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/include/replication/syncrep.h           |   1 +
 src/test/subscription/t/001_rep_changes.pl  |  10 ++
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 10 files changed, 179 insertions(+), 32 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..99b2775 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,12 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid, send_keepalive);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a04..be2f438 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -166,6 +166,16 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the transaction
+ * has sent BEGIN. BEGIN is only sent when the first change in a transaction
+ * is processed. This makes it possible to skip transactions that are empty.
+ */
+typedef struct PGOutputTxnData {
+	bool		sent_begin_txn;	/* flag indicating whether BEGIN has
+					 * been sent */
+}		PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,18 +462,45 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set of
+ * tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
+ */
+static void
+pgoutput_begin_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
+{
+	PGOutputTxnData *txndata = MemoryContextAllocZero(ctx->context,
+						   sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
  */
 static void
-pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_send_begin(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
-					 send_replication_origin);
+			 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
 }
@@ -472,10 +509,28 @@ pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
  * COMMIT callback
  */
 static void
-pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
-					XLogRecPtr commit_lsn)
+pgoutput_commit_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn,
+		    XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool		sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the COMMIT message
+	 * too.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+
+	pfree(txndata);
+	if (!sent_begin_txn) {
+		elog(DEBUG1, "Skipping replication of an empty transaction in commit");
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -484,9 +539,22 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
 /*
  * BEGIN PREPARE callback
+ *
+ * We don't postpone sending BEGIN PREPARE till the first change in order to
+ * skip empty prepared transactions like we do for regular transactions. This
+ * is because there could be a restart of walsender after a prepared
+ * transaction is sent out. After the restart, when the corresponding COMMIT
+ * PREPARED is decoded, we will not know if the prepared transaction was
+ * skipped because it was empty. This is because we would have lost the
+ * in-memory txndata information that was present prior to the restart. This
+ * might result in sending a spurious COMMIT PREPARED without a correspodning
+ * prepared transaction at the subscriber.
+ *
+ * This could be solved with a protocol change that allows the apply worker
+ * to detect spurious COMMIT PREPARED commands and drop them.
  */
 static void
-pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_prepare_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
 
@@ -494,7 +562,7 @@ pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 	logicalrep_write_begin_prepare(ctx->out, txn);
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
-					 send_replication_origin);
+			 send_replication_origin);
 
 	OutputPluginWrite(ctx, true);
 }
@@ -506,7 +574,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +588,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +604,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -1141,6 +1209,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1216,6 +1285,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 &action))
 				break;
 
+		/*
+		 * Send BEGIN if we haven't yet.
+		 *
+		 * The reason this is delayed till after the row_filter check
+		 * is to make sure we don't send a begin if the change is not
+		 * send out due to the row_filter check failing.
+		 */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
@@ -1266,6 +1345,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1324,6 +1407,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1354,6 +1441,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1366,6 +1454,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 		xid = change->txn->xid;
 
 	old = MemoryContextSwitchTo(data->context);
+	txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
 	relids = palloc0(nrelations * sizeof(Oid));
 	nrelids = 0;
@@ -1392,11 +1481,21 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			continue;
 
 		relids[nrelids++] = relid;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
 		maybe_send_schema(ctx, change, relation, relentry);
 	}
 
 	if (nrelids > 0)
 	{
+		txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		OutputPluginPrepareWrite(ctx, true);
 		logicalrep_write_truncate(ctx->out,
 								  xid,
@@ -1429,6 +1528,18 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * messages.
+	 */
+	if (transactional) {
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1506,6 +1617,11 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
 
 /*
  * START STREAM callback
+ *
+ * We don't try and skip empty streams because streams might
+ * contain prepared transactions. Read the comments in the header of
+ * pgoutput_begin_prepare_txn() for details on why empty prepared
+ * transactions are not skipped.
  */
 static void
 pgoutput_stream_start(struct LogicalDecodingContext *ctx,
@@ -1598,7 +1714,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1619,7 +1735,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c
index ce163b9..11f7358 100644
--- a/src/backend/replication/syncrep.c
+++ b/src/backend/replication/syncrep.c
@@ -171,8 +171,7 @@ SyncRepWaitForLSN(XLogRecPtr lsn, bool commit)
 	 * described in SyncRepUpdateSyncStandbysDefined(). On the other hand, if
 	 * it's false, the lock is not necessary because we don't touch the queue.
 	 */
-	if (!SyncRepRequested() ||
-		!((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	if (!SyncRepEnabled())
 		return;
 
 	/* Cap the level for anything other than commit to remote flush only. */
@@ -539,6 +538,15 @@ SyncRepReleaseWaiters(void)
 }
 
 /*
+ * Check if synchronous replication is enabled
+ */
+bool
+SyncRepEnabled(void)
+{
+	return SyncRepRequested() && ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined;
+}
+
+/*
  * Calculate the synced Write, Flush and Apply positions among sync standbys.
  *
  * Return false if the number of sync standbys is less than
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2d0292a..21e7ac3 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool send_keepalive);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,12 +1450,20 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool send_keepalive)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we need
+	 * to send a keepalive to keep the MyWalSnd locations updated.
+	 */
+	if (send_keepalive && SyncRepEnabled())
+		WalSndKeepalive(true, ctx->write_location);
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1550,7 +1559,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2077,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3083,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3563,18 +3572,20 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * If writePtr is set, mark that as the LSN processed, else use sentPtr.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3613,7 +3624,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..9f59855 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool send_keepalive
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..eb91d17 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool send_keepalive);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/include/replication/syncrep.h b/src/include/replication/syncrep.h
index 27be230..7086532 100644
--- a/src/include/replication/syncrep.h
+++ b/src/include/replication/syncrep.h
@@ -90,6 +90,7 @@ extern void SyncRepCleanupAtProcExit(void);
 /* called by wal sender */
 extern void SyncRepInitConfig(void);
 extern void SyncRepReleaseWaiters(void);
+extern bool SyncRepEnabled(void);
 
 /* called by wal sender and user backend */
 extern int	SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys);
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index eca1c63..40c510b 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -473,6 +473,16 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab_full VALUES(0)");
 
 $node_publisher->wait_for_catchup('tap_sub');
 
+#create a new table which is not published, any transaction on this table 
+#should be skipped.
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_empty (a int)");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_empty SELECT generate_series(1,50)");
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+$node_publisher->wait_for_catchup('tap_sub');
 # note that data are different on provider and subscriber
 $result = $node_subscriber->safe_psql('postgres',
 	"SELECT count(*), min(a), max(a) FROM tab_ins");
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index eaf3e7a..4c575f3 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1608,6 +1608,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
1.8.3.1

#92Amit Kapila
amit.kapila16@gmail.com
In reply to: Ajin Cherian (#91)
1 attachment(s)
Re: logical replication empty transactions

On Sat, Mar 19, 2022 at 9:10 AM Ajin Cherian <itsajin@gmail.com> wrote:

On Thu, Mar 17, 2022 at 10:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

3. Can we add a simple test for it in one of the existing test
files(say in 001_rep_changes.pl)?

added a simple test.

This doesn't verify if the transaction is skipped. I think we should
extend this test to check for a DEBUG message in the Logs (you need to
probably set log_min_messages to DEBUG1 for this test). As an example,
you can check the patch [1]/messages/by-id/CAA4eK1JbLRj6pSUENfDFsqj0+adNob_=RPXpnUnWFBskVi5JhA@mail.gmail.com. Also, it seems by mistake you have added
wait_for_catchup() twice.

Few other comments:
=================
1. Let's keep the parameter name as skipped_empty_xact in
OutputPluginUpdateProgress so as to not confuse with the other patch's
[2]: /messages/by-id/CAA4eK1LGnaPuWs2M4sDfpd6JQZjoh4DGAsgUvNW=Or8i9z6K8w@mail.gmail.com
keep_alive message so as to not make the syncrep wait whereas in the
other patch we only need to send it periodically based on
wal_sender_timeout parameter.
2. The new function SyncRepEnabled() seems confusing to me as the
comments in SyncRepWaitForLSN() clearly state why we need to first
read the parameter 'sync_standbys_defined' without any lock then read
it again with a lock if the parameter is true. So, I just put that
check back and also added a similar check in WalSndUpdateProgress.
3.
@@ -1392,11 +1481,21 @@ pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
continue;

  relids[nrelids++] = relid;
+
+ /* Send BEGIN if we haven't yet */
+ if (txndata && !txndata->sent_begin_txn)
+ pgoutput_send_begin(ctx, txn);
  maybe_send_schema(ctx, change, relation, relentry);
  }
  if (nrelids > 0)
  {
+ txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+ /* Send BEGIN if we haven't yet */
+ if (txndata && !txndata->sent_begin_txn)
+ pgoutput_send_begin(ctx, txn);
+

Why do we need to try sending the begin in the second check? I think
it should be sufficient to do it in the above loop.

I have made these and a number of other changes in the attached patch.
Do let me know what you think of the attached?

[1]: /messages/by-id/CAA4eK1JbLRj6pSUENfDFsqj0+adNob_=RPXpnUnWFBskVi5JhA@mail.gmail.com
[2]: /messages/by-id/CAA4eK1LGnaPuWs2M4sDfpd6JQZjoh4DGAsgUvNW=Or8i9z6K8w@mail.gmail.com

--
With Regards,
Amit Kapila.

Attachments:

v27-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v27-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 505a36596a36c9b1c61876e6ef20c73ea33e11d8 Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Mon, 21 Mar 2022 10:29:01 +0530
Subject: [PATCH v27] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message, if there was
no other change for that transaction, do not send the COMMIT message.
This means that pgoutput will skip BEGIN/COMMIT messages for transactions
that are empty.

The patch also makes sure that in synchronous replication mode,
when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "streaming"
or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   6 +-
 src/backend/replication/pgoutput/pgoutput.c | 133 ++++++++++++++++++--
 src/backend/replication/walsender.c         |  37 ++++--
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/test/subscription/t/001_rep_changes.pl  |  10 ++
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 8 files changed, 171 insertions(+), 26 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13f2d..1c68ba4d76 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,14 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx,
+						   bool skipped_empty_xact)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid,
+						 skipped_empty_xact);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5fddab3a3d..1336227e5c 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -166,6 +166,36 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the transaction
+ * has sent BEGIN. BEGIN is only sent when the first change in a transaction
+ * is processed. This makes it possible to skip sending a pair of BEGIN/COMMIT
+ * messages for empty transactions which saves network bandwidth.
+ *
+ * This optimization is not used for prepared transactions because if the
+ * WALSender restarts after prepare of a transaction and before commit prepared
+ * of the same transaction then we won't be able to figure out if we have
+ * skipped sending BEGIN/PREPARE of a transaction as it was empty. This is
+ * because we would have lost the in-memory txndata information that was
+ * present prior to the restart. This will result in sending a spurious
+ * COMMIT PREPARED without a correspodning prepared transaction at the
+ * downstream which would lead to an error when it tries to process it.
+ *
+ * XXX We could achieve this optimization by changing protocol to send
+ * additional information so that downstream can detect that the corresponding
+ * prepare has not been sent. However, adding such a check for every
+ * transaction in the downstream could be costly so we might want to do it
+ * optionally.
+ *
+ * We also don't have this optimization for streamed transactions because
+ * they can contain prepared transactions.
+ */
+typedef struct PGOutputTxnData
+{
+	bool		sent_begin_txn;	/* flag indicating whether BEGIN has
+								 * been sent */
+}		PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,15 +482,42 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set of
+ * tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
-pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
+{
+	PGOutputTxnData	*txndata = MemoryContextAllocZero(ctx->context,
+													  sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -475,7 +532,26 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool		sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the COMMIT message
+	 * too.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+	pfree(txndata);
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipped replication of an empty transaction with XID: %u", txn->xid);
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -486,7 +562,7 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  * BEGIN PREPARE callback
  */
 static void
-pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_prepare_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
 
@@ -506,7 +582,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +596,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +612,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -1141,6 +1217,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1216,6 +1293,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 &action))
 				break;
 
+			/*
+			 * Send BEGIN if we haven't yet.
+			 *
+			 * We send the BEGIN message after ensuring that we will actually
+			 * send the change. This avoids sending a pair of BEGIN/COMMIT
+			 * messages for empty transactions.
+			 */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
@@ -1266,6 +1353,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1324,6 +1415,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1354,6 +1449,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1366,6 +1462,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 		xid = change->txn->xid;
 
 	old = MemoryContextSwitchTo(data->context);
+	txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
 	relids = palloc0(nrelations * sizeof(Oid));
 	nrelids = 0;
@@ -1392,6 +1489,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			continue;
 
 		relids[nrelids++] = relid;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		maybe_send_schema(ctx, change, relation, relentry);
 	}
 
@@ -1429,6 +1531,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * messages.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1598,7 +1713,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1619,7 +1734,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2d0292a092..7b4e650212 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool skipped_empty_xact);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,11 +1450,25 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool skipped_empty_xact)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
+	/*
+	 * When skipping empty transactions in synchronous replication, we send a
+	 * keepalive message to avoid delaying such transactions.
+	 *
+	 * It is okay to check sync_standbys_defined flag without lock here as
+	 * in the worst case we will just send an extra keepalive message when it
+	 * is really not required.
+	 */
+	if (skipped_empty_xact &&
+		SyncRepRequested() &&
+		((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+		WalSndKeepalive(true, ctx->write_location);
+
 	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
@@ -1550,7 +1565,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2083,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3089,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3563,18 +3578,20 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * If writePtr is set, mark that as the LSN processed, else use sentPtr.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3613,7 +3630,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9799..c6e4be95e1 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool skipped_empty_xact
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf76c..492fdb3d18 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool skipped_empty_xact);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index eca1c63335..33c529fff0 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -473,6 +473,16 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab_full VALUES(0)");
 
 $node_publisher->wait_for_catchup('tap_sub');
 
+# create a new table which is not published, any transaction on this table
+# should be skipped.
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_empty (a int)");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_empty SELECT generate_series(1,50)");
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+$node_publisher->wait_for_catchup('tap_sub');
 # note that data are different on provider and subscriber
 $result = $node_subscriber->safe_psql('postgres',
 	"SELECT count(*), min(a), max(a) FROM tab_ins");
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff3c4..d21d929c2d 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 93d5190508..139e51e9e5 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1610,6 +1610,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
2.28.0.windows.1

#93houzj.fnst@fujitsu.com
houzj.fnst@fujitsu.com
In reply to: Amit Kapila (#92)
1 attachment(s)
RE: logical replication empty transactions

On Monday, March 21, 2022 6:01 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Sat, Mar 19, 2022 at 9:10 AM Ajin Cherian <itsajin@gmail.com> wrote:

On Thu, Mar 17, 2022 at 10:43 PM Amit Kapila <amit.kapila16@gmail.com>

wrote:

3. Can we add a simple test for it in one of the existing test
files(say in 001_rep_changes.pl)?

added a simple test.

This doesn't verify if the transaction is skipped. I think we should
extend this test to check for a DEBUG message in the Logs (you need to
probably set log_min_messages to DEBUG1 for this test). As an example,
you can check the patch [1]. Also, it seems by mistake you have added
wait_for_catchup() twice.

I added a testcase to check the DEBUG message.

Few other comments:
=================
1. Let's keep the parameter name as skipped_empty_xact in
OutputPluginUpdateProgress so as to not confuse with the other patch's
[2] keep_alive parameter. I think in this case we must send the
keep_alive message so as to not make the syncrep wait whereas in the
other patch we only need to send it periodically based on
wal_sender_timeout parameter.
2. The new function SyncRepEnabled() seems confusing to me as the
comments in SyncRepWaitForLSN() clearly state why we need to first
read the parameter 'sync_standbys_defined' without any lock then read
it again with a lock if the parameter is true. So, I just put that
check back and also added a similar check in WalSndUpdateProgress.
3.
@@ -1392,11 +1481,21 @@ pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
continue;

relids[nrelids++] = relid;
+
+ /* Send BEGIN if we haven't yet */
+ if (txndata && !txndata->sent_begin_txn)
+ pgoutput_send_begin(ctx, txn);
maybe_send_schema(ctx, change, relation, relentry);
}
if (nrelids > 0)
{
+ txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+ /* Send BEGIN if we haven't yet */
+ if (txndata && !txndata->sent_begin_txn)
+ pgoutput_send_begin(ctx, txn);
+

Why do we need to try sending the begin in the second check? I think
it should be sufficient to do it in the above loop.

I have made these and a number of other changes in the attached patch.
Do let me know what you think of the attached?

The changes look good to me.
And I did some basic tests for the patch and didn’t find some other problems.

Attach the new version patch.

Best regards,
Hou zj

Attachments:

v28-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v28-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 6710a2b16e81ad3c340dd5ea24dab2b6bdaf7c19 Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Mon, 21 Mar 2022 10:29:01 +0530
Subject: [PATCH] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message, if there was
no other change for that transaction, do not send the COMMIT message.
This means that pgoutput will skip BEGIN/COMMIT messages for transactions
that are empty.

The patch also makes sure that in synchronous replication mode,
when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "streaming"
or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   6 +-
 src/backend/replication/pgoutput/pgoutput.c | 133 ++++++++++++++++++++++++++--
 src/backend/replication/walsender.c         |  37 +++++---
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/test/subscription/t/001_rep_changes.pl  |  24 +++++
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 8 files changed, 185 insertions(+), 26 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..1c68ba4 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,14 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx,
+						   bool skipped_empty_xact)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid,
+						 skipped_empty_xact);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5fddab3..1336227 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -166,6 +166,36 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the transaction
+ * has sent BEGIN. BEGIN is only sent when the first change in a transaction
+ * is processed. This makes it possible to skip sending a pair of BEGIN/COMMIT
+ * messages for empty transactions which saves network bandwidth.
+ *
+ * This optimization is not used for prepared transactions because if the
+ * WALSender restarts after prepare of a transaction and before commit prepared
+ * of the same transaction then we won't be able to figure out if we have
+ * skipped sending BEGIN/PREPARE of a transaction as it was empty. This is
+ * because we would have lost the in-memory txndata information that was
+ * present prior to the restart. This will result in sending a spurious
+ * COMMIT PREPARED without a correspodning prepared transaction at the
+ * downstream which would lead to an error when it tries to process it.
+ *
+ * XXX We could achieve this optimization by changing protocol to send
+ * additional information so that downstream can detect that the corresponding
+ * prepare has not been sent. However, adding such a check for every
+ * transaction in the downstream could be costly so we might want to do it
+ * optionally.
+ *
+ * We also don't have this optimization for streamed transactions because
+ * they can contain prepared transactions.
+ */
+typedef struct PGOutputTxnData
+{
+	bool		sent_begin_txn;	/* flag indicating whether BEGIN has
+								 * been sent */
+}		PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,15 +482,42 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set of
+ * tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
-pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
+{
+	PGOutputTxnData	*txndata = MemoryContextAllocZero(ctx->context,
+													  sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -475,7 +532,26 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool		sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the COMMIT message
+	 * too.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+	pfree(txndata);
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipped replication of an empty transaction with XID: %u", txn->xid);
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -486,7 +562,7 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  * BEGIN PREPARE callback
  */
 static void
-pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_prepare_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
 
@@ -506,7 +582,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +596,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +612,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -1141,6 +1217,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1217,6 +1294,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				break;
 
 			/*
+			 * Send BEGIN if we haven't yet.
+			 *
+			 * We send the BEGIN message after ensuring that we will actually
+			 * send the change. This avoids sending a pair of BEGIN/COMMIT
+			 * messages for empty transactions.
+			 */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
+			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
 			 */
@@ -1266,6 +1353,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1324,6 +1415,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1354,6 +1449,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1366,6 +1462,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 		xid = change->txn->xid;
 
 	old = MemoryContextSwitchTo(data->context);
+	txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
 	relids = palloc0(nrelations * sizeof(Oid));
 	nrelids = 0;
@@ -1392,6 +1489,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			continue;
 
 		relids[nrelids++] = relid;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		maybe_send_schema(ctx, change, relation, relentry);
 	}
 
@@ -1429,6 +1531,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * messages.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1598,7 +1713,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1619,7 +1734,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2d0292a..7b4e650 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool skipped_empty_xact);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,12 +1450,26 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool skipped_empty_xact)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we send a
+	 * keepalive message to avoid delaying such transactions.
+	 *
+	 * It is okay to check sync_standbys_defined flag without lock here as
+	 * in the worst case we will just send an extra keepalive message when it
+	 * is really not required.
+	 */
+	if (skipped_empty_xact &&
+		SyncRepRequested() &&
+		((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+		WalSndKeepalive(true, ctx->write_location);
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1550,7 +1565,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2083,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3089,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3563,18 +3578,20 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * If writePtr is set, mark that as the LSN processed, else use sentPtr.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3613,7 +3630,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..c6e4be9 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool skipped_empty_xact
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..492fdb3 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool skipped_empty_xact);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index eca1c63..ee9cc70 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -473,6 +473,30 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab_full VALUES(0)");
 
 $node_publisher->wait_for_catchup('tap_sub');
 
+# Check that we don't send BEGIN and COMMIT because of empty transaction
+# optimization.  We have to look for the DEBUG1 log messages about that, so
+# temporarily bump up the log verbosity.
+$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
+$node_subscriber->reload;
+
+# Note that the current location of the log file is not grabbed immediately
+# after reloading the configuration, but after sending one SQL command to
+# the node so as we are sure that the reloading has taken effect.
+my $log_location = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_notrep VALUES (11)");
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+my $logfile = slurp_file($node_subscriber->logfile, $log_location);
+ok( $logfile =~
+	  qr/Skipped replication of an empty transaction with XID/,
+	'empty transaction is skipped');
+
+$node_subscriber->append_conf('postgresql.conf',
+	"log_min_messages = warning");
+$node_subscriber->reload;
+
 # note that data are different on provider and subscriber
 $result = $node_subscriber->safe_psql('postgres',
 	"SELECT count(*), min(a), max(a) FROM tab_ins");
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 93d5190..139e51e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1610,6 +1610,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
2.7.2.windows.1

#94houzj.fnst@fujitsu.com
houzj.fnst@fujitsu.com
In reply to: houzj.fnst@fujitsu.com (#93)
1 attachment(s)
RE: logical replication empty transactions

On Monday, March 21, 2022 6:01 PM Amit Kapila <amit.kapila16@gmail.com>
wrote:

On Sat, Mar 19, 2022 at 9:10 AM Ajin Cherian <itsajin@gmail.com> wrote:

On Thu, Mar 17, 2022 at 10:43 PM Amit Kapila
<amit.kapila16@gmail.com>

wrote:

3. Can we add a simple test for it in one of the existing test
files(say in 001_rep_changes.pl)?

added a simple test.

This doesn't verify if the transaction is skipped. I think we should
extend this test to check for a DEBUG message in the Logs (you need to
probably set log_min_messages to DEBUG1 for this test). As an example,
you can check the patch [1]. Also, it seems by mistake you have added
wait_for_catchup() twice.

I added a testcase to check the DEBUG message.

Few other comments:
=================
1. Let's keep the parameter name as skipped_empty_xact in
OutputPluginUpdateProgress so as to not confuse with the other patch's
[2] keep_alive parameter. I think in this case we must send the
keep_alive message so as to not make the syncrep wait whereas in the
other patch we only need to send it periodically based on
wal_sender_timeout parameter.
2. The new function SyncRepEnabled() seems confusing to me as the
comments in SyncRepWaitForLSN() clearly state why we need to first
read the parameter 'sync_standbys_defined' without any lock then read
it again with a lock if the parameter is true. So, I just put that
check back and also added a similar check in WalSndUpdateProgress.
3.
@@ -1392,11 +1481,21 @@ pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
continue;

relids[nrelids++] = relid;
+
+ /* Send BEGIN if we haven't yet */
+ if (txndata && !txndata->sent_begin_txn) pgoutput_send_begin(ctx,
+ txn);
maybe_send_schema(ctx, change, relation, relentry);
}
if (nrelids > 0)
{
+ txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+ /* Send BEGIN if we haven't yet */
+ if (txndata && !txndata->sent_begin_txn) pgoutput_send_begin(ctx,
+ txn);
+

Why do we need to try sending the begin in the second check? I think
it should be sufficient to do it in the above loop.

I have made these and a number of other changes in the attached patch.
Do let me know what you think of the attached?

The changes look good to me.
And I did some basic tests for the patch and didn’t find some other problems.

Attach the new version patch.

Oh, sorry, I posted the wrong patch, here is the correct one.

Best regards,
Hou zj

Attachments:

v28-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v28-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 1c180de3ecbaa5a37d84946220906b52bcede521 Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Mon, 21 Mar 2022 10:29:01 +0530
Subject: [PATCH] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message, if there was
no other change for that transaction, do not send the COMMIT message.
This means that pgoutput will skip BEGIN/COMMIT messages for transactions
that are empty.

The patch also makes sure that in synchronous replication mode,
when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "streaming"
or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   6 +-
 src/backend/replication/pgoutput/pgoutput.c | 133 ++++++++++++++++++--
 src/backend/replication/walsender.c         |  37 ++++--
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/test/subscription/t/001_rep_changes.pl  |  24 ++++
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 8 files changed, 185 insertions(+), 26 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13f2d..1c68ba4d76 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,14 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx,
+						   bool skipped_empty_xact)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid,
+						 skipped_empty_xact);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5fddab3a3d..1336227e5c 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -166,6 +166,36 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the transaction
+ * has sent BEGIN. BEGIN is only sent when the first change in a transaction
+ * is processed. This makes it possible to skip sending a pair of BEGIN/COMMIT
+ * messages for empty transactions which saves network bandwidth.
+ *
+ * This optimization is not used for prepared transactions because if the
+ * WALSender restarts after prepare of a transaction and before commit prepared
+ * of the same transaction then we won't be able to figure out if we have
+ * skipped sending BEGIN/PREPARE of a transaction as it was empty. This is
+ * because we would have lost the in-memory txndata information that was
+ * present prior to the restart. This will result in sending a spurious
+ * COMMIT PREPARED without a correspodning prepared transaction at the
+ * downstream which would lead to an error when it tries to process it.
+ *
+ * XXX We could achieve this optimization by changing protocol to send
+ * additional information so that downstream can detect that the corresponding
+ * prepare has not been sent. However, adding such a check for every
+ * transaction in the downstream could be costly so we might want to do it
+ * optionally.
+ *
+ * We also don't have this optimization for streamed transactions because
+ * they can contain prepared transactions.
+ */
+typedef struct PGOutputTxnData
+{
+	bool		sent_begin_txn;	/* flag indicating whether BEGIN has
+								 * been sent */
+}		PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,15 +482,42 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set of
+ * tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
-pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
+{
+	PGOutputTxnData	*txndata = MemoryContextAllocZero(ctx->context,
+													  sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -475,7 +532,26 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool		sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the COMMIT message
+	 * too.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+	pfree(txndata);
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipped replication of an empty transaction with XID: %u", txn->xid);
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -486,7 +562,7 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  * BEGIN PREPARE callback
  */
 static void
-pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_prepare_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
 
@@ -506,7 +582,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +596,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +612,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -1141,6 +1217,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1216,6 +1293,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 &action))
 				break;
 
+			/*
+			 * Send BEGIN if we haven't yet.
+			 *
+			 * We send the BEGIN message after ensuring that we will actually
+			 * send the change. This avoids sending a pair of BEGIN/COMMIT
+			 * messages for empty transactions.
+			 */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
@@ -1266,6 +1353,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1324,6 +1415,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1354,6 +1449,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1366,6 +1462,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 		xid = change->txn->xid;
 
 	old = MemoryContextSwitchTo(data->context);
+	txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
 	relids = palloc0(nrelations * sizeof(Oid));
 	nrelids = 0;
@@ -1392,6 +1489,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			continue;
 
 		relids[nrelids++] = relid;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		maybe_send_schema(ctx, change, relation, relentry);
 	}
 
@@ -1429,6 +1531,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * messages.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1598,7 +1713,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1619,7 +1734,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2d0292a092..7b4e650212 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool skipped_empty_xact);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,11 +1450,25 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool skipped_empty_xact)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
+	/*
+	 * When skipping empty transactions in synchronous replication, we send a
+	 * keepalive message to avoid delaying such transactions.
+	 *
+	 * It is okay to check sync_standbys_defined flag without lock here as
+	 * in the worst case we will just send an extra keepalive message when it
+	 * is really not required.
+	 */
+	if (skipped_empty_xact &&
+		SyncRepRequested() &&
+		((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+		WalSndKeepalive(true, ctx->write_location);
+
 	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
@@ -1550,7 +1565,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2083,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3089,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3563,18 +3578,20 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * If writePtr is set, mark that as the LSN processed, else use sentPtr.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3613,7 +3630,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9799..c6e4be95e1 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool skipped_empty_xact
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf76c..492fdb3d18 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool skipped_empty_xact);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index eca1c63335..2904846348 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -473,6 +473,30 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab_full VALUES(0)");
 
 $node_publisher->wait_for_catchup('tap_sub');
 
+# Check that we don't send BEGIN and COMMIT because of empty transaction
+# optimization.  We have to look for the DEBUG1 log messages about that, so
+# temporarily bump up the log verbosity.
+$node_publisher->append_conf('postgresql.conf', "log_min_messages = debug1");
+$node_publisher->reload;
+
+# Note that the current location of the log file is not grabbed immediately
+# after reloading the configuration, but after sending one SQL command to
+# the node so as we are sure that the reloading has taken effect.
+$log_location = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_notrep VALUES (11)");
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+$logfile = slurp_file($node_publisher->logfile, $log_location);
+ok( $logfile =~
+	  qr/Skipped replication of an empty transaction with XID/,
+	'empty transaction is skipped');
+
+$node_publisher->append_conf('postgresql.conf',
+	"log_min_messages = warning");
+$node_publisher->reload;
+
 # note that data are different on provider and subscriber
 $result = $node_subscriber->safe_psql('postgres',
 	"SELECT count(*), min(a), max(a) FROM tab_ins");
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff3c4..d21d929c2d 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 93d5190508..139e51e9e5 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1610,6 +1610,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
2.18.4

#95Amit Kapila
amit.kapila16@gmail.com
In reply to: houzj.fnst@fujitsu.com (#94)
2 attachment(s)
Re: logical replication empty transactions

On Tue, Mar 22, 2022 at 7:25 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

On Monday, March 21, 2022 6:01 PM Amit Kapila <amit.kapila16@gmail.com>
wrote:

Oh, sorry, I posted the wrong patch, here is the correct one.

The test change looks good to me. I think additionally we can verify
that the record is not reflected in the subscriber table. Apart from
that, I had made minor changes mostly in the comments in the attached
patch. If those look okay to you, please include those in the next
version.

--
With Regards,
Amit Kapila.

Attachments:

v28-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v28-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 1c180de3ecbaa5a37d84946220906b52bcede521 Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Mon, 21 Mar 2022 10:29:01 +0530
Subject: [PATCH] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message, if there was
no other change for that transaction, do not send the COMMIT message.
This means that pgoutput will skip BEGIN/COMMIT messages for transactions
that are empty.

The patch also makes sure that in synchronous replication mode,
when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "streaming"
or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   6 +-
 src/backend/replication/pgoutput/pgoutput.c | 133 ++++++++++++++++++--
 src/backend/replication/walsender.c         |  37 ++++--
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/test/subscription/t/001_rep_changes.pl  |  24 ++++
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 8 files changed, 185 insertions(+), 26 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13f2d..1c68ba4d76 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,14 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx,
+						   bool skipped_empty_xact)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid,
+						 skipped_empty_xact);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5fddab3a3d..1336227e5c 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -166,6 +166,36 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the transaction
+ * has sent BEGIN. BEGIN is only sent when the first change in a transaction
+ * is processed. This makes it possible to skip sending a pair of BEGIN/COMMIT
+ * messages for empty transactions which saves network bandwidth.
+ *
+ * This optimization is not used for prepared transactions because if the
+ * WALSender restarts after prepare of a transaction and before commit prepared
+ * of the same transaction then we won't be able to figure out if we have
+ * skipped sending BEGIN/PREPARE of a transaction as it was empty. This is
+ * because we would have lost the in-memory txndata information that was
+ * present prior to the restart. This will result in sending a spurious
+ * COMMIT PREPARED without a correspodning prepared transaction at the
+ * downstream which would lead to an error when it tries to process it.
+ *
+ * XXX We could achieve this optimization by changing protocol to send
+ * additional information so that downstream can detect that the corresponding
+ * prepare has not been sent. However, adding such a check for every
+ * transaction in the downstream could be costly so we might want to do it
+ * optionally.
+ *
+ * We also don't have this optimization for streamed transactions because
+ * they can contain prepared transactions.
+ */
+typedef struct PGOutputTxnData
+{
+	bool		sent_begin_txn;	/* flag indicating whether BEGIN has
+								 * been sent */
+}		PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,15 +482,42 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send BEGIN message here. Instead, postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set of
+ * tables (instead of all tables) and transactions whose changes were on
+ * table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
-pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
+{
+	PGOutputTxnData	*txndata = MemoryContextAllocZero(ctx->context,
+													  sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -475,7 +532,26 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool		sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * If a BEGIN message was not yet sent, then it means there were no
+	 * relevant changes encountered, so we can skip the COMMIT message
+	 * too.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	txn->output_plugin_private = NULL;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+	pfree(txndata);
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipped replication of an empty transaction with XID: %u", txn->xid);
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -486,7 +562,7 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  * BEGIN PREPARE callback
  */
 static void
-pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_prepare_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
 
@@ -506,7 +582,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +596,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +612,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -1141,6 +1217,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1216,6 +1293,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 &action))
 				break;
 
+			/*
+			 * Send BEGIN if we haven't yet.
+			 *
+			 * We send the BEGIN message after ensuring that we will actually
+			 * send the change. This avoids sending a pair of BEGIN/COMMIT
+			 * messages for empty transactions.
+			 */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
@@ -1266,6 +1353,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1324,6 +1415,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1354,6 +1449,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1366,6 +1462,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 		xid = change->txn->xid;
 
 	old = MemoryContextSwitchTo(data->context);
+	txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
 	relids = palloc0(nrelations * sizeof(Oid));
 	nrelids = 0;
@@ -1392,6 +1489,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			continue;
 
 		relids[nrelids++] = relid;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		maybe_send_schema(ctx, change, relation, relentry);
 	}
 
@@ -1429,6 +1531,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * messages.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1598,7 +1713,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1619,7 +1734,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2d0292a092..7b4e650212 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool skipped_empty_xact);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,11 +1450,25 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool skipped_empty_xact)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
+	/*
+	 * When skipping empty transactions in synchronous replication, we send a
+	 * keepalive message to avoid delaying such transactions.
+	 *
+	 * It is okay to check sync_standbys_defined flag without lock here as
+	 * in the worst case we will just send an extra keepalive message when it
+	 * is really not required.
+	 */
+	if (skipped_empty_xact &&
+		SyncRepRequested() &&
+		((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+		WalSndKeepalive(true, ctx->write_location);
+
 	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
@@ -1550,7 +1565,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2083,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3089,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3563,18 +3578,20 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * If writePtr is set, mark that as the LSN processed, else use sentPtr.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3613,7 +3630,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9799..c6e4be95e1 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool skipped_empty_xact
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf76c..492fdb3d18 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool skipped_empty_xact);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index eca1c63335..2904846348 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -473,6 +473,30 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab_full VALUES(0)");
 
 $node_publisher->wait_for_catchup('tap_sub');
 
+# Check that we don't send BEGIN and COMMIT because of empty transaction
+# optimization.  We have to look for the DEBUG1 log messages about that, so
+# temporarily bump up the log verbosity.
+$node_publisher->append_conf('postgresql.conf', "log_min_messages = debug1");
+$node_publisher->reload;
+
+# Note that the current location of the log file is not grabbed immediately
+# after reloading the configuration, but after sending one SQL command to
+# the node so as we are sure that the reloading has taken effect.
+$log_location = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_notrep VALUES (11)");
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+$logfile = slurp_file($node_publisher->logfile, $log_location);
+ok( $logfile =~
+	  qr/Skipped replication of an empty transaction with XID/,
+	'empty transaction is skipped');
+
+$node_publisher->append_conf('postgresql.conf',
+	"log_min_messages = warning");
+$node_publisher->reload;
+
 # note that data are different on provider and subscriber
 $result = $node_subscriber->safe_psql('postgres',
 	"SELECT count(*), min(a), max(a) FROM tab_ins");
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff3c4..d21d929c2d 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 93d5190508..139e51e9e5 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1610,6 +1610,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
2.18.4

v28_diff_amit.1.patchapplication/octet-stream; name=v28_diff_amit.1.patchDownload
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 1336227e5c..5c15aa71c5 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -484,10 +484,10 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 /*
  * BEGIN callback.
  *
- * Don't send BEGIN message here. Instead, postpone it until the first
+ * Don't send the BEGIN message here instead postpone it until the first
  * change. In logical replication, a common scenario is to replicate a set of
  * tables (instead of all tables) and transactions whose changes were on
- * table(s) that are not published will produce empty transactions. These
+ * the table(s) that are not published will produce empty transactions. These
  * empty transactions will send BEGIN and COMMIT messages to subscribers,
  * using bandwidth on something with little/no use for logical replication.
  */
@@ -538,14 +538,13 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	Assert(txndata);
 
 	/*
-	 * If a BEGIN message was not yet sent, then it means there were no
-	 * relevant changes encountered, so we can skip the COMMIT message
-	 * too.
+	 * We don't need to send the commit message unless some relevant change
+	 * from this transaction has been sent to the downstream.
 	 */
 	sent_begin_txn = txndata->sent_begin_txn;
-	txn->output_plugin_private = NULL;
 	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
 	pfree(txndata);
+	txn->output_plugin_private = NULL;
 
 	if (!sent_begin_txn)
 	{
@@ -562,7 +561,7 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
  * BEGIN PREPARE callback
  */
 static void
-pgoutput_begin_prepare_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
+pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
 
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 7b4e650212..bba56746c4 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -3581,7 +3581,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
  *
- * If writePtr is set, mark that as the LSN processed, else use sentPtr.
+ * writePtr is the location up to which the WAL is sent. It is essentially
+ * the same as sentPtr but in some cases, we need to send keep alive before
+ * sentPtr is updated like when skipping empty transactions.
  */
 static void
 WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
#96houzj.fnst@fujitsu.com
houzj.fnst@fujitsu.com
In reply to: Amit Kapila (#95)
1 attachment(s)
RE: logical replication empty transactions

On Tuesday, March 22, 2022 7:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Tue, Mar 22, 2022 at 7:25 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

On Monday, March 21, 2022 6:01 PM Amit Kapila
<amit.kapila16@gmail.com>
wrote:

Oh, sorry, I posted the wrong patch, here is the correct one.

The test change looks good to me. I think additionally we can verify that the
record is not reflected in the subscriber table. Apart from that, I had made
minor changes mostly in the comments in the attached patch. If those look
okay to you, please include those in the next version.

Thanks, the changes look good to me, I merged the diff patch.

Attach the new version patch which include the following changes:

- Fix a typo
- Change the requestreply flag of the newly added WalSndKeepalive to false,
because the subscriber can judge whether it's necessary to post a reply based
on the received LSN.
- Add a testcase to make sure there is no data in subscriber side when the
transaction is skipped.
- Change the name of flag skipped_empty_xact to skipped_xact which seems more
understandable.
- Merge Amit's suggested changes.

Best regards,
Hou zj

Attachments:

v29-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v29-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 7fcc00e2046c659357e1daac886dee093c9ac62a Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Mon, 21 Mar 2022 10:29:01 +0530
Subject: [PATCH] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message, if there was
no other change for that transaction, do not send the COMMIT message.
This means that pgoutput will skip BEGIN/COMMIT messages for transactions
that are empty.

The patch also makes sure that in synchronous replication mode,
when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "streaming"
or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   6 +-
 src/backend/replication/pgoutput/pgoutput.c | 130 ++++++++++++++++++++++++++--
 src/backend/replication/walsender.c         |  39 ++++++---
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/test/subscription/t/001_rep_changes.pl  |  28 ++++++
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 8 files changed, 189 insertions(+), 25 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..1c68ba4 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,14 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx,
+						   bool skipped_xact)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid,
+						 skipped_xact);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5fddab3..f5638c6 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -166,6 +166,36 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the transaction
+ * has sent BEGIN. BEGIN is only sent when the first change in a transaction
+ * is processed. This makes it possible to skip sending a pair of BEGIN/COMMIT
+ * messages for empty transactions which saves network bandwidth.
+ *
+ * This optimization is not used for prepared transactions because if the
+ * WALSender restarts after prepare of a transaction and before commit prepared
+ * of the same transaction then we won't be able to figure out if we have
+ * skipped sending BEGIN/PREPARE of a transaction as it was empty. This is
+ * because we would have lost the in-memory txndata information that was
+ * present prior to the restart. This will result in sending a spurious
+ * COMMIT PREPARED without a corresponding prepared transaction at the
+ * downstream which would lead to an error when it tries to process it.
+ *
+ * XXX We could achieve this optimization by changing protocol to send
+ * additional information so that downstream can detect that the corresponding
+ * prepare has not been sent. However, adding such a check for every
+ * transaction in the downstream could be costly so we might want to do it
+ * optionally.
+ *
+ * We also don't have this optimization for streamed transactions because
+ * they can contain prepared transactions.
+ */
+typedef struct PGOutputTxnData
+{
+	bool		sent_begin_txn;	/* flag indicating whether BEGIN has
+								 * been sent */
+}		PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -452,15 +482,42 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send the BEGIN message here instead postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set of
+ * tables (instead of all tables) and transactions whose changes were on
+ * the table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
-pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
+{
+	PGOutputTxnData	*txndata = MemoryContextAllocZero(ctx->context,
+													  sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -475,7 +532,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool		sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * We don't need to send the commit message unless some relevant change
+	 * from this transaction has been sent to the downstream.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipped replication of an empty transaction with XID: %u", txn->xid);
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -506,7 +581,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -520,7 +595,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -536,7 +611,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -1141,6 +1216,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1217,6 +1293,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				break;
 
 			/*
+			 * Send BEGIN if we haven't yet.
+			 *
+			 * We send the BEGIN message after ensuring that we will actually
+			 * send the change. This avoids sending a pair of BEGIN/COMMIT
+			 * messages for empty transactions.
+			 */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
+			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
 			 */
@@ -1266,6 +1352,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1324,6 +1414,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1354,6 +1448,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1366,6 +1461,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 		xid = change->txn->xid;
 
 	old = MemoryContextSwitchTo(data->context);
+	txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
 	relids = palloc0(nrelations * sizeof(Oid));
 	nrelids = 0;
@@ -1392,6 +1488,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			continue;
 
 		relids[nrelids++] = relid;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		maybe_send_schema(ctx, change, relation, relentry);
 	}
 
@@ -1429,6 +1530,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * messages.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1598,7 +1712,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1619,7 +1733,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2d0292a..bba5674 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool skipped_xact);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,12 +1450,26 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool skipped_xact)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we send a
+	 * keepalive message to avoid delaying such transactions.
+	 *
+	 * It is okay to check sync_standbys_defined flag without lock here as
+	 * in the worst case we will just send an extra keepalive message when it
+	 * is really not required.
+	 */
+	if (skipped_xact &&
+		SyncRepRequested() &&
+		((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+		WalSndKeepalive(false, ctx->write_location);
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1550,7 +1565,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2083,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3089,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3563,18 +3578,22 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * writePtr is the location up to which the WAL is sent. It is essentially
+ * the same as sentPtr but in some cases, we need to send keep alive before
+ * sentPtr is updated like when skipping empty transactions.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3613,7 +3632,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..c6e4be9 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool skipped_xact
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..492fdb3 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool skipped_xact);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index eca1c63..1b0ed96 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -473,6 +473,34 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab_full VALUES(0)");
 
 $node_publisher->wait_for_catchup('tap_sub');
 
+# Check that we don't send BEGIN and COMMIT because of empty transaction
+# optimization.  We have to look for the DEBUG1 log messages about that, so
+# temporarily bump up the log verbosity.
+$node_publisher->append_conf('postgresql.conf', "log_min_messages = debug1");
+$node_publisher->reload;
+
+# Note that the current location of the log file is not grabbed immediately
+# after reloading the configuration, but after sending one SQL command to
+# the node so as we are sure that the reloading has taken effect.
+$log_location = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_notrep VALUES (11)");
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+$logfile = slurp_file($node_publisher->logfile, $log_location);
+ok( $logfile =~
+	  qr/Skipped replication of an empty transaction with XID/,
+	'empty transaction is skipped');
+
+$result = $node_subscriber->safe_psql('postgres',
+	"SELECT count(*) FROM tab_notrep");
+is($result, qq(0), 'check non-replicated table is empty on subscriber');
+
+$node_publisher->append_conf('postgresql.conf',
+	"log_min_messages = warning");
+$node_publisher->reload;
+
 # note that data are different on provider and subscriber
 $result = $node_subscriber->safe_psql('postgres',
 	"SELECT count(*), min(a), max(a) FROM tab_ins");
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 93d5190..139e51e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1610,6 +1610,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
2.7.2.windows.1

#97shiy.fnst@fujitsu.com
shiy.fnst@fujitsu.com
In reply to: houzj.fnst@fujitsu.com (#96)
2 attachment(s)
RE: logical replication empty transactions

On Thursday, March 24, 2022 11:19 AM Hou, Zhijie/侯 志杰 <houzj.fnst@fujitsu.com> wrote:

Attach the new version patch which include the following changes:

- Fix a typo
- Change the requestreply flag of the newly added WalSndKeepalive to false,
because the subscriber can judge whether it's necessary to post a reply
based
on the received LSN.
- Add a testcase to make sure there is no data in subscriber side when the
transaction is skipped.
- Change the name of flag skipped_empty_xact to skipped_xact which seems
more
understandable.
- Merge Amit's suggested changes.

Hi,

This patch skips sending BEGIN/COMMIT messages for empty transactions and saves
network bandwidth. So I tried to do a test to see how does it affect bandwidth.

This test refers to the previous test by Peter[1]/messages/by-id/CAHut+PuyqcDJO0X2BxY+9ycF+ew3x77FiCbTJQGnLDbNmMASZQ@mail.gmail.com. I temporarily modified the
code in worker.c to log the length of the data received by the subscriber (after
calling walrcv_receive()). At the conclusion of the test run, the logs are
processed to extract the numbers.

[1]: /messages/by-id/CAHut+PuyqcDJO0X2BxY+9ycF+ew3x77FiCbTJQGnLDbNmMASZQ@mail.gmail.com

The number of transactions is fixed (1000), and I tested different mixes of
empty and not-empty transactions sent - 0%, 25%, 50%, 100%. The patch will send
keepalive message when skipping empty transaction in synchronous replication
mode, so I tested both synchronous replication and asynchronous replication.

The results are as follows, and attach the bar chart.

Sync replication - size of sending data
--------------------------------------------------------------------
0% 25% 50% 75% 100%
HEAD 335211 281655 223661 170271 115108
patched 335217 256617 173878 98095 18108

Async replication - size of sending data
--------------------------------------------------------------------
0% 25% 50% 75% 100%
HEAD 339379 285835 236343 184227 115000
patched 335077 260953 180022 113333 18126

The details of the test is also attached.

Summary of result:
In both synchronous replication mode and asynchronous replication mode, as more
empty transactions, the improvement is more obvious. Even if when there is no
empty transaction, I can't see any overhead.

Regards,
Shi yu

Attachments:

performance_test.PNGimage/png; name=performance_test.PNGDownload
details.txttext/plain; name=details.txtDownload
#98houzj.fnst@fujitsu.com
houzj.fnst@fujitsu.com
In reply to: houzj.fnst@fujitsu.com (#96)
RE: logical replication empty transactions

On Thursday, March 24, 2022 11:19 AM houzj.fnst@fujitsu.com wrote:

On Tuesday, March 22, 2022 7:50 PM Amit Kapila <amit.kapila16@gmail.com>
wrote:

On Tue, Mar 22, 2022 at 7:25 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

On Monday, March 21, 2022 6:01 PM Amit Kapila
<amit.kapila16@gmail.com>
wrote:

Oh, sorry, I posted the wrong patch, here is the correct one.

The test change looks good to me. I think additionally we can verify
that the record is not reflected in the subscriber table. Apart from
that, I had made minor changes mostly in the comments in the attached
patch. If those look okay to you, please include those in the next version.

Thanks, the changes look good to me, I merged the diff patch.

Attach the new version patch which include the following changes:

- Fix a typo
- Change the requestreply flag of the newly added WalSndKeepalive to false,
because the subscriber can judge whether it's necessary to post a reply
based
on the received LSN.
- Add a testcase to make sure there is no data in subscriber side when the
transaction is skipped.
- Change the name of flag skipped_empty_xact to skipped_xact which seems
more
understandable.
- Merge Amit's suggested changes.

I did some more review for the newly added keepalive message and confirmed that
it's necessary to send this in sync mode.

+	if (skipped_xact &&
+		SyncRepRequested() &&
+		((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+		WalSndKeepalive(false, ctx->write_location);

Because in sync replication, the publisher need to get the reply from
subscirber to release the waiter. After applying the patch, we don't send empty
transaction to subscriber, so we won't get a reply without this keepalive
message. Although the walsender usually invoke WalSndWaitForWal() which will
also send a keepalive message to subscriber, and we could get a reply and
release the wait. But WalSndWaitForWal() is not always invoked for each record.
When reading the page, we won't invoke WalSndWaitForWal() if we already have
the record in our buffer[1]ReadPageInternal( ... /* check whether we have all the requested data already */ if (targetSegNo == state->seg.ws_segno && targetPageOff == state->segoff && reqLen <= state->readLen) return state->readLen; ....

[1]: ReadPageInternal( ... /* check whether we have all the requested data already */ if (targetSegNo == state->seg.ws_segno && targetPageOff == state->segoff && reqLen <= state->readLen) return state->readLen; ...
...
/* check whether we have all the requested data already */
if (targetSegNo == state->seg.ws_segno &&
targetPageOff == state->segoff && reqLen <= state->readLen)
return state->readLen;
...

Based on above, if we don't have the newly added keepalive message in the
patch, the transaction could wait for a bit more time to finish.

For example, I did some experiments to confirm:
1. Set LOG_SNAPSHOT_INTERVAL_MS and checkpoint_timeout to a bigger value to
make sure it doesn't generate extra WAL which could affect the test.
2. Use debugger to attach the walsender and let it stop in the WalSndWaitForWal()
3. Start two clients and modify un-published table
postgres1 # INSERT INTO not_rep VALUES(1);
---- waiting
postgres2 # INSERT INTO not_rep VALUES(1);
---- waiting
4. Release the walsender, and we can see it won't send a keepalive to
subscriber until it has handled all the above two transactions, which means
the two transaction will wait until all of them has been decoded. This
behavior doesn't looks good and is inconsistent with the current
behavior(the transaction will finish after decoding it or after sending it
to sub if necessary).

So, I think the newly add keepalive message makes sense.

Best regards,
Hou zj

#99houzj.fnst@fujitsu.com
houzj.fnst@fujitsu.com
In reply to: houzj.fnst@fujitsu.com (#98)
1 attachment(s)
RE: logical replication empty transactions

On Friday, March 25, 2022 8:31 AM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:

On Thursday, March 24, 2022 11:19 AM houzj.fnst@fujitsu.com wrote:

On Tuesday, March 22, 2022 7:50 PM Amit Kapila

<amit.kapila16@gmail.com>

wrote:

On Tue, Mar 22, 2022 at 7:25 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

On Monday, March 21, 2022 6:01 PM Amit Kapila
<amit.kapila16@gmail.com>
wrote:

Oh, sorry, I posted the wrong patch, here is the correct one.

The test change looks good to me. I think additionally we can verify
that the record is not reflected in the subscriber table. Apart from
that, I had made minor changes mostly in the comments in the attached
patch. If those look okay to you, please include those in the next version.

Thanks, the changes look good to me, I merged the diff patch.

Attach the new version patch which include the following changes:

- Fix a typo
- Change the requestreply flag of the newly added WalSndKeepalive to false,
because the subscriber can judge whether it's necessary to post a reply
based
on the received LSN.
- Add a testcase to make sure there is no data in subscriber side when the
transaction is skipped.
- Change the name of flag skipped_empty_xact to skipped_xact which seems
more
understandable.
- Merge Amit's suggested changes.

I did some more review for the newly added keepalive message and confirmed
that it's necessary to send this in sync mode.

Since commit 75b1521 added decoding of sequence to logical
replication, this patch needs to have send begin message in
pgoutput_sequence if necessary.

Attach the new version patch with this change.

Best regards,
Hou zj

Attachments:

v30-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v30-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From c474ba9252f07205d113e6038807e0a50e8c54a5 Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Mon, 21 Mar 2022 10:29:01 +0530
Subject: [PATCH] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message, if there was
no other change for that transaction, do not send the COMMIT message.
This means that pgoutput will skip BEGIN/COMMIT messages for transactions
that are empty.

The patch also makes sure that in synchronous replication mode,
when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "streaming"
or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   6 +-
 src/backend/replication/pgoutput/pgoutput.c | 143 ++++++++++++++++++++++++++--
 src/backend/replication/walsender.c         |  39 ++++++--
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/test/subscription/t/001_rep_changes.pl  |  28 ++++++
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 8 files changed, 202 insertions(+), 25 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..e1f14ae 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,14 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx,
+						   bool skipped_xact)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid,
+						 skipped_xact);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 4cdc698..edfdfa1 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -171,6 +171,36 @@ typedef struct RelationSyncEntry
 	AttrMap    *attrmap;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the transaction
+ * has sent BEGIN. BEGIN is only sent when the first change in a transaction
+ * is processed. This makes it possible to skip sending a pair of BEGIN/COMMIT
+ * messages for empty transactions which saves network bandwidth.
+ *
+ * This optimization is not used for prepared transactions because if the
+ * WALSender restarts after prepare of a transaction and before commit prepared
+ * of the same transaction then we won't be able to figure out if we have
+ * skipped sending BEGIN/PREPARE of a transaction as it was empty. This is
+ * because we would have lost the in-memory txndata information that was
+ * present prior to the restart. This will result in sending a spurious
+ * COMMIT PREPARED without a corresponding prepared transaction at the
+ * downstream which would lead to an error when it tries to process it.
+ *
+ * XXX We could achieve this optimization by changing protocol to send
+ * additional information so that downstream can detect that the corresponding
+ * prepare has not been sent. However, adding such a check for every
+ * transaction in the downstream could be costly so we might want to do it
+ * optionally.
+ *
+ * We also don't have this optimization for streamed transactions because
+ * they can contain prepared transactions.
+ */
+typedef struct PGOutputTxnData
+{
+	bool		sent_begin_txn;	/* flag indicating whether BEGIN has
+								 * been sent */
+}		PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -471,15 +501,42 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send the BEGIN message here instead postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set of
+ * tables (instead of all tables) and transactions whose changes were on
+ * the table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
+ */
+static void
+pgoutput_begin_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
+{
+	PGOutputTxnData	*txndata = MemoryContextAllocZero(ctx->context,
+													  sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
  */
 static void
-pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -494,7 +551,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool		sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * We don't need to send the commit message unless some relevant change
+	 * from this transaction has been sent to the downstream.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipped replication of an empty transaction with XID: %u", txn->xid);
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -525,7 +600,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -539,7 +614,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -555,7 +630,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -1160,6 +1235,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1236,6 +1312,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				break;
 
 			/*
+			 * Send BEGIN if we haven't yet.
+			 *
+			 * We send the BEGIN message after ensuring that we will actually
+			 * send the change. This avoids sending a pair of BEGIN/COMMIT
+			 * messages for empty transactions.
+			 */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
+			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
 			 */
@@ -1285,6 +1371,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1343,6 +1433,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1373,6 +1467,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1385,6 +1480,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 		xid = change->txn->xid;
 
 	old = MemoryContextSwitchTo(data->context);
+	txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
 	relids = palloc0(nrelations * sizeof(Oid));
 	nrelids = 0;
@@ -1411,6 +1507,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			continue;
 
 		relids[nrelids++] = relid;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		maybe_send_schema(ctx, change, relation, relentry);
 	}
 
@@ -1448,6 +1549,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * messages.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1492,6 +1606,19 @@ pgoutput_sequence(LogicalDecodingContext *ctx,
 	if (!relentry->pubactions.pubsequence)
 		return;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * sequence changes.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_sequence(ctx->out,
 							  relation,
@@ -1662,7 +1789,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1683,7 +1810,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2d0292a..e1dabbd 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,15 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool skipped_xact);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1449,12 +1450,26 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * Write the current position to the lag tracker (see XLogSendPhysical).
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool skipped_xact)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
 
 	/*
+	 * When skipping empty transactions in synchronous replication, we send a
+	 * keepalive message to avoid delaying such transactions.
+	 *
+	 * It is okay to check sync_standbys_defined flag without lock here as
+	 * in the worst case we will just send an extra keepalive message when it
+	 * is really not required.
+	 */
+	if (skipped_xact &&
+		SyncRepRequested() &&
+		((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+		WalSndKeepalive(false, ctx->write_location);
+
+	/*
 	 * Track lag no more than once per WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS to
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
@@ -1550,7 +1565,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2083,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3089,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3563,18 +3578,22 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * writePtr is the location up to which the WAL is sent. It is essentially
+ * the same as sentPtr but in some cases, we need to send keep alive before
+ * sentPtr is updated like when skipping empty transactions.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3613,7 +3632,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..a6ef16a 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool skipped_xact
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..fe85d49 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool skipped_xact);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index eca1c63..1b0ed96 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -473,6 +473,34 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab_full VALUES(0)");
 
 $node_publisher->wait_for_catchup('tap_sub');
 
+# Check that we don't send BEGIN and COMMIT because of empty transaction
+# optimization.  We have to look for the DEBUG1 log messages about that, so
+# temporarily bump up the log verbosity.
+$node_publisher->append_conf('postgresql.conf', "log_min_messages = debug1");
+$node_publisher->reload;
+
+# Note that the current location of the log file is not grabbed immediately
+# after reloading the configuration, but after sending one SQL command to
+# the node so as we are sure that the reloading has taken effect.
+$log_location = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_notrep VALUES (11)");
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+$logfile = slurp_file($node_publisher->logfile, $log_location);
+ok( $logfile =~
+	  qr/Skipped replication of an empty transaction with XID/,
+	'empty transaction is skipped');
+
+$result = $node_subscriber->safe_psql('postgres',
+	"SELECT count(*) FROM tab_notrep");
+is($result, qq(0), 'check non-replicated table is empty on subscriber');
+
+$node_publisher->append_conf('postgresql.conf',
+	"log_min_messages = warning");
+$node_publisher->reload;
+
 # note that data are different on provider and subscriber
 $result = $node_subscriber->safe_psql('postgres',
 	"SELECT count(*), min(a), max(a) FROM tab_ins");
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 4968803..d0a02b7 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1610,6 +1610,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
2.7.2.windows.1

#100Amit Kapila
amit.kapila16@gmail.com
In reply to: houzj.fnst@fujitsu.com (#99)
Re: logical replication empty transactions

On Fri, Mar 25, 2022 at 12:50 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

Attach the new version patch with this change.

Few comments:
=================
1. I think we can move the keep_alive check after the tracklag record
check to keep it consistent with another patch [1]/messages/by-id/OS3PR01MB6275C64F264662E84D2FB7AE9E1D9@OS3PR01MB6275.jpnprd01.prod.outlook.com.
2. Add the comment about the new parameter skipped_xact atop
WalSndUpdateProgress.
3. I think we need to call pq_flush_if_writable after sending a
keepalive message to avoid delaying sync transactions.

[1]: /messages/by-id/OS3PR01MB6275C64F264662E84D2FB7AE9E1D9@OS3PR01MB6275.jpnprd01.prod.outlook.com

--
With Regards,
Amit Kapila.

#101houzj.fnst@fujitsu.com
houzj.fnst@fujitsu.com
In reply to: Amit Kapila (#100)
1 attachment(s)
RE: logical replication empty transactions

On Monday, March 28, 2022 3:08 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Fri, Mar 25, 2022 at 12:50 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

Attach the new version patch with this change.

Few comments:

Thanks for the comments.

=================
1. I think we can move the keep_alive check after the tracklag record
check to keep it consistent with another patch [1].

Changed.

2. Add the comment about the new parameter skipped_xact atop
WalSndUpdateProgress.

Added.

3. I think we need to call pq_flush_if_writable after sending a
keepalive message to avoid delaying sync transactions.

Agreed.
If we don’t flush the data, we might flush the keepalive later than before. And
we could get the reply later as well and then the release of syncwait could be
delayed.

Attach the new version patch which addressed the above comments.
The patch also adds a loop after the newly added keepalive message
to make sure the message is actually flushed to the client like what
did in WalSndWriteData.

Best regards,
Hou zj

Attachments:

v32-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v32-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From 3aa75c0a90b1a47d7add2d111992500d8eadaa21 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@cn.fujitsu.com>
Date: Sun, 27 Mar 2022 10:09:16 +0800
Subject: [PATCH] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty (because it does not
contain changes from the selected publications). It is a waste of CPU
cycles and network bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change. While processing a COMMIT message, if there was
no other change for that transaction, do not send the COMMIT message.
This means that pgoutput will skip BEGIN/COMMIT messages for transactions
that are empty.

The patch also makes sure that in synchronous replication mode,
when skipping empty transactions, keepalive messages
are sent to keep the LSN locations updated on the standby.

This patch does not skip empty transactions that are "streaming"
or "two-phase".

Discussion:
https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   6 +-
 src/backend/replication/pgoutput/pgoutput.c | 143 ++++++++++++++++++++++++++--
 src/backend/replication/walsender.c         |  72 +++++++++++---
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/test/subscription/t/001_rep_changes.pl  |  28 ++++++
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 8 files changed, 230 insertions(+), 30 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..e1f14ae 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,14 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx,
+						   bool skipped_xact)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid,
+						 skipped_xact);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 893833e..aafea01 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -183,6 +183,36 @@ typedef struct RelationSyncEntry
 	MemoryContext entry_cxt;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the transaction
+ * has sent BEGIN. BEGIN is only sent when the first change in a transaction
+ * is processed. This makes it possible to skip sending a pair of BEGIN/COMMIT
+ * messages for empty transactions which saves network bandwidth.
+ *
+ * This optimization is not used for prepared transactions because if the
+ * WALSender restarts after prepare of a transaction and before commit prepared
+ * of the same transaction then we won't be able to figure out if we have
+ * skipped sending BEGIN/PREPARE of a transaction as it was empty. This is
+ * because we would have lost the in-memory txndata information that was
+ * present prior to the restart. This will result in sending a spurious
+ * COMMIT PREPARED without a corresponding prepared transaction at the
+ * downstream which would lead to an error when it tries to process it.
+ *
+ * XXX We could achieve this optimization by changing protocol to send
+ * additional information so that downstream can detect that the corresponding
+ * prepare has not been sent. However, adding such a check for every
+ * transaction in the downstream could be costly so we might want to do it
+ * optionally.
+ *
+ * We also don't have this optimization for streamed transactions because
+ * they can contain prepared transactions.
+ */
+typedef struct PGOutputTxnData
+{
+	bool		sent_begin_txn;	/* flag indicating whether BEGIN has
+								 * been sent */
+}		PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -488,15 +518,42 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send the BEGIN message here instead postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set of
+ * tables (instead of all tables) and transactions whose changes were on
+ * the table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
-pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
+{
+	PGOutputTxnData	*txndata = MemoryContextAllocZero(ctx->context,
+													  sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is where the BEGIN is actually sent. This is called while processing
+ * the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -511,7 +568,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool		sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * We don't need to send the commit message unless some relevant change
+	 * from this transaction has been sent to the downstream.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "Skipped replication of an empty transaction with XID: %u", txn->xid);
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -542,7 +617,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -556,7 +631,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -572,7 +647,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -1295,6 +1370,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1371,6 +1447,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				break;
 
 			/*
+			 * Send BEGIN if we haven't yet.
+			 *
+			 * We send the BEGIN message after ensuring that we will actually
+			 * send the change. This avoids sending a pair of BEGIN/COMMIT
+			 * messages for empty transactions.
+			 */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
+			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
 			 */
@@ -1420,6 +1506,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1480,6 +1570,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1510,6 +1604,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1522,6 +1617,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 		xid = change->txn->xid;
 
 	old = MemoryContextSwitchTo(data->context);
+	txndata = (PGOutputTxnData *) txn->output_plugin_private;
 
 	relids = palloc0(nrelations * sizeof(Oid));
 	nrelids = 0;
@@ -1548,6 +1644,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			continue;
 
 		relids[nrelids++] = relid;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		maybe_send_schema(ctx, change, relation, relentry);
 	}
 
@@ -1585,6 +1686,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * messages.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1629,6 +1743,19 @@ pgoutput_sequence(LogicalDecodingContext *ctx,
 	if (!relentry->pubactions.pubsequence)
 		return;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * sequence changes.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_sequence(ctx->out,
 							  relation,
@@ -1799,7 +1926,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1820,7 +1947,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2d0292a..f5b0302 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,16 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void ProcessPendingWritesAndTimeOut(void);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool skipped_xact);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1399,6 +1401,16 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
 	}
 
 	/* If we have pending write here, go to slow path */
+	ProcessPendingWritesAndTimeOut();
+}
+
+/*
+ * Wait until there is no pending write. Also process replies from the other
+ * side and check timeouts during that.
+ */
+static void
+ProcessPendingWritesAndTimeOut(void)
+{
 	for (;;)
 	{
 		long		sleeptime;
@@ -1447,9 +1459,12 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * LogicalDecodingContext 'update_progress' callback.
  *
  * Write the current position to the lag tracker (see XLogSendPhysical).
+ *
+ * When skipping empty transactions, send a keepalive message if necessary.
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool skipped_xact)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
@@ -1459,12 +1474,35 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
 #define WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS	1000
-	if (!TimestampDifferenceExceeds(sendTime, now,
-									WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS))
-		return;
+	if (TimestampDifferenceExceeds(sendTime, now,
+								   WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS))
+	{
+		LagTrackerWrite(lsn, now);
+		sendTime = now;
+	}
 
-	LagTrackerWrite(lsn, now);
-	sendTime = now;
+	/*
+	 * When skipping empty transactions in synchronous replication, we send a
+	 * keepalive message to avoid delaying such transactions.
+	 *
+	 * It is okay to check sync_standbys_defined flag without lock here as
+	 * in the worst case we will just send an extra keepalive message when it
+	 * is really not required.
+	 */
+	if (skipped_xact &&
+		SyncRepRequested() &&
+		((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	{
+		WalSndKeepalive(false, ctx->write_location);
+
+		/* Try to flush pending output to the client */
+		if (pq_flush_if_writable() != 0)
+			WalSndShutdown();
+
+		/* If we have pending write here, make sure it's actually flushed */
+		if (pq_is_send_pending())
+			ProcessPendingWritesAndTimeOut();
+	}
 }
 
 /*
@@ -1550,7 +1588,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2106,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3112,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3563,18 +3601,22 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * writePtr is the location up to which the WAL is sent. It is essentially
+ * the same as sentPtr but in some cases, we need to send keep alive before
+ * sentPtr is updated like when skipping empty transactions.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3613,7 +3655,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..a6ef16a 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool skipped_xact
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..fe85d49 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool skipped_xact);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index eca1c63..1b0ed96 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -473,6 +473,34 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab_full VALUES(0)");
 
 $node_publisher->wait_for_catchup('tap_sub');
 
+# Check that we don't send BEGIN and COMMIT because of empty transaction
+# optimization.  We have to look for the DEBUG1 log messages about that, so
+# temporarily bump up the log verbosity.
+$node_publisher->append_conf('postgresql.conf', "log_min_messages = debug1");
+$node_publisher->reload;
+
+# Note that the current location of the log file is not grabbed immediately
+# after reloading the configuration, but after sending one SQL command to
+# the node so as we are sure that the reloading has taken effect.
+$log_location = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_notrep VALUES (11)");
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+$logfile = slurp_file($node_publisher->logfile, $log_location);
+ok( $logfile =~
+	  qr/Skipped replication of an empty transaction with XID/,
+	'empty transaction is skipped');
+
+$result = $node_subscriber->safe_psql('postgres',
+	"SELECT count(*) FROM tab_notrep");
+is($result, qq(0), 'check non-replicated table is empty on subscriber');
+
+$node_publisher->append_conf('postgresql.conf',
+	"log_min_messages = warning");
+$node_publisher->reload;
+
 # note that data are different on provider and subscriber
 $result = $node_subscriber->safe_psql('postgres',
 	"SELECT count(*), min(a), max(a) FROM tab_ins");
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 85c808a..e01fb17 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1610,6 +1610,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
2.7.2.windows.1

#102Masahiko Sawada
sawada.mshk@gmail.com
In reply to: houzj.fnst@fujitsu.com (#101)
Re: logical replication empty transactions

On Mon, Mar 28, 2022 at 9:22 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

On Monday, March 28, 2022 3:08 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Fri, Mar 25, 2022 at 12:50 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

Attach the new version patch with this change.

Few comments:

Thanks for the comments.

=================
1. I think we can move the keep_alive check after the tracklag record
check to keep it consistent with another patch [1].

Changed.

2. Add the comment about the new parameter skipped_xact atop
WalSndUpdateProgress.

Added.

3. I think we need to call pq_flush_if_writable after sending a
keepalive message to avoid delaying sync transactions.

Agreed.
If we don’t flush the data, we might flush the keepalive later than before. And
we could get the reply later as well and then the release of syncwait could be
delayed.

Attach the new version patch which addressed the above comments.
The patch also adds a loop after the newly added keepalive message
to make sure the message is actually flushed to the client like what
did in WalSndWriteData.

Thank you for updating the patch!

Some comments:

+       if (skipped_xact &&
+               SyncRepRequested() &&
+               ((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+       {
+               WalSndKeepalive(false, ctx->write_location);

I think we can use 'lsn' since it is actually ctx->write_location.

---
+       if (!sent_begin_txn)
+       {
+               elog(DEBUG1, "Skipped replication of an empty
transaction with XID: %u", txn->xid);
+               return;
+       }

The log message should start with lowercase.

---
+# Note that the current location of the log file is not grabbed immediately
+# after reloading the configuration, but after sending one SQL command to
+# the node so as we are sure that the reloading has taken effect.
+$log_location = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_notrep VALUES (11)");
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+$logfile = slurp_file($node_publisher->logfile, $log_location);

I think we should get the log location of the publisher node, not
subscriber node.

Regards,

--
Masahiko Sawada
EDB: https://www.enterprisedb.com/

#103houzj.fnst@fujitsu.com
houzj.fnst@fujitsu.com
In reply to: Masahiko Sawada (#102)
1 attachment(s)
RE: logical replication empty transactions

On Tuesday, March 29, 2022 3:20 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

Some comments:

Thanks for the comments!

+       if (skipped_xact &&
+               SyncRepRequested() &&
+               ((volatile WalSndCtlData *)
WalSndCtl)->sync_standbys_defined)
+       {
+               WalSndKeepalive(false, ctx->write_location);

I think we can use 'lsn' since it is actually ctx->write_location.

Agreed, and changed.

---
+       if (!sent_begin_txn)
+       {
+               elog(DEBUG1, "Skipped replication of an empty
transaction with XID: %u", txn->xid);
+               return;
+       }

The log message should start with lowercase.

Changed.

---
+# Note that the current location of the log file is not grabbed
+immediately # after reloading the configuration, but after sending one
+SQL command to # the node so as we are sure that the reloading has taken
effect.
+$log_location = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_notrep VALUES
+(11)");
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+$logfile = slurp_file($node_publisher->logfile, $log_location);

I think we should get the log location of the publisher node, not subscriber
node.

Changed.

Attach the new version patch which addressed the
above comments and slightly adjusted some code comments.

Best regards,
Hou zj

Attachments:

v33-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v33-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From fe48eca27591a64bc6fc40864aabc2b7118f3a54 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@cn.fujitsu.com>
Date: Sun, 27 Mar 2022 10:09:16 +0800
Subject: [PATCH] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
subscriber even if the transaction is empty. This can happen because
transaction doesn't contain changes from the selected publications or all
the changes got filtered. It is a waste of CPU cycles and network
bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change is sent. While processing a COMMIT message, if
there was no other change for that transaction, do not send the COMMIT
message. This allows us to skip sending BEGIN/COMMIT messages for empty
transactions.

When skipping empty transactions in synchronous replication mode, we send
a keepalive message to avoid delaying such transactions.

Author: Ajin Cherian, Hou Zhijie, Euler Taveira
Reviewed-by: Peter Smith, Takamichi Osumi, Shi Yu, Masahiko Sawada, Greg Nancarrow, Vignesh C, Amit Kapila
Discussion: https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   6 +-
 src/backend/replication/pgoutput/pgoutput.c | 141 ++++++++++++++++++++++++++--
 src/backend/replication/walsender.c         |  72 +++++++++++---
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/test/subscription/t/001_rep_changes.pl  |  28 ++++++
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 8 files changed, 228 insertions(+), 30 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..e1f14ae 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,14 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx,
+						   bool skipped_xact)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid,
+						 skipped_xact);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 893833e..20d0b1e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -183,6 +183,36 @@ typedef struct RelationSyncEntry
 	MemoryContext entry_cxt;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the transaction
+ * has sent BEGIN. BEGIN is only sent when the first change in a transaction
+ * is processed. This makes it possible to skip sending a pair of BEGIN/COMMIT
+ * messages for empty transactions which saves network bandwidth.
+ *
+ * This optimization is not used for prepared transactions because if the
+ * WALSender restarts after prepare of a transaction and before commit prepared
+ * of the same transaction then we won't be able to figure out if we have
+ * skipped sending BEGIN/PREPARE of a transaction as it was empty. This is
+ * because we would have lost the in-memory txndata information that was
+ * present prior to the restart. This will result in sending a spurious
+ * COMMIT PREPARED without a corresponding prepared transaction at the
+ * downstream which would lead to an error when it tries to process it.
+ *
+ * XXX We could achieve this optimization by changing protocol to send
+ * additional information so that downstream can detect that the corresponding
+ * prepare has not been sent. However, adding such a check for every
+ * transaction in the downstream could be costly so we might want to do it
+ * optionally.
+ *
+ * We also don't have this optimization for streamed transactions because
+ * they can contain prepared transactions.
+ */
+typedef struct PGOutputTxnData
+{
+	bool		sent_begin_txn;	/* flag indicating whether BEGIN has
+								 * been sent */
+}		PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -488,15 +518,41 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send the BEGIN message here instead postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set of
+ * tables (instead of all tables) and transactions whose changes were on
+ * the table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
-pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
+{
+	PGOutputTxnData	*txndata = MemoryContextAllocZero(ctx->context,
+													  sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is called while processing the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -511,7 +567,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool		sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * We don't need to send the commit message unless some relevant change
+	 * from this transaction has been sent to the downstream.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "skipped replication of an empty transaction with XID: %u", txn->xid);
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -542,7 +616,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -556,7 +630,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -572,7 +646,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -1295,6 +1369,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1371,6 +1446,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				break;
 
 			/*
+			 * Send BEGIN if we haven't yet.
+			 *
+			 * We send the BEGIN message after ensuring that we will actually
+			 * send the change. This avoids sending a pair of BEGIN/COMMIT
+			 * messages for empty transactions.
+			 */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
+			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
 			 */
@@ -1420,6 +1505,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1480,6 +1569,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1510,6 +1603,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1548,6 +1642,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			continue;
 
 		relids[nrelids++] = relid;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		maybe_send_schema(ctx, change, relation, relentry);
 	}
 
@@ -1585,6 +1684,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * messages.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1629,6 +1741,19 @@ pgoutput_sequence(LogicalDecodingContext *ctx,
 	if (!relentry->pubactions.pubsequence)
 		return;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * sequence changes.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_sequence(ctx->out,
 							  relation,
@@ -1799,7 +1924,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1820,7 +1945,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2d0292a..6a3e96f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,16 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void ProcessPendingWritesAndTimeOut(void);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool skipped_xact);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1399,6 +1401,16 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
 	}
 
 	/* If we have pending write here, go to slow path */
+	ProcessPendingWritesAndTimeOut();
+}
+
+/*
+ * Wait until there is no pending write. Also process replies from the other
+ * side and check timeouts during that.
+ */
+static void
+ProcessPendingWritesAndTimeOut(void)
+{
 	for (;;)
 	{
 		long		sleeptime;
@@ -1447,9 +1459,12 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * LogicalDecodingContext 'update_progress' callback.
  *
  * Write the current position to the lag tracker (see XLogSendPhysical).
+ *
+ * When skipping empty transactions, send a keepalive message if necessary.
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool skipped_xact)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
@@ -1459,12 +1474,35 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
 #define WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS	1000
-	if (!TimestampDifferenceExceeds(sendTime, now,
-									WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS))
-		return;
+	if (TimestampDifferenceExceeds(sendTime, now,
+								   WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS))
+	{
+		LagTrackerWrite(lsn, now);
+		sendTime = now;
+	}
 
-	LagTrackerWrite(lsn, now);
-	sendTime = now;
+	/*
+	 * When skipping empty transactions in synchronous replication, we send a
+	 * keepalive message to avoid delaying such transactions.
+	 *
+	 * It is okay to check sync_standbys_defined flag without lock here as
+	 * in the worst case we will just send an extra keepalive message when it
+	 * is really not required.
+	 */
+	if (skipped_xact &&
+		SyncRepRequested() &&
+		((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	{
+		WalSndKeepalive(false, lsn);
+
+		/* Try to flush pending output to the client */
+		if (pq_flush_if_writable() != 0)
+			WalSndShutdown();
+
+		/* If we have pending write here, make sure it's actually flushed */
+		if (pq_is_send_pending())
+			ProcessPendingWritesAndTimeOut();
+	}
 }
 
 /*
@@ -1550,7 +1588,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2106,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3112,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3563,18 +3601,22 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * writePtr is the location up to which the WAL is sent. It is essentially
+ * the same as sentPtr but in some cases, we need to send keep alive before
+ * sentPtr is updated like when skipping empty transactions.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3613,7 +3655,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..a6ef16a 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool skipped_xact
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..fe85d49 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool skipped_xact);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index eca1c63..c35d83f 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -473,6 +473,34 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab_full VALUES(0)");
 
 $node_publisher->wait_for_catchup('tap_sub');
 
+# Check that we don't send BEGIN and COMMIT because of empty transaction
+# optimization.  We have to look for the DEBUG1 log messages about that, so
+# temporarily bump up the log verbosity.
+$node_publisher->append_conf('postgresql.conf', "log_min_messages = debug1");
+$node_publisher->reload;
+
+# Note that the current location of the log file is not grabbed immediately
+# after reloading the configuration, but after sending one SQL command to
+# the node so as we are sure that the reloading has taken effect.
+$log_location = -s $node_publisher->logfile;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_notrep VALUES (11)");
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+$logfile = slurp_file($node_publisher->logfile, $log_location);
+ok( $logfile =~
+	  qr/skipped replication of an empty transaction with XID/,
+	'empty transaction is skipped');
+
+$result = $node_subscriber->safe_psql('postgres',
+	"SELECT count(*) FROM tab_notrep");
+is($result, qq(0), 'check non-replicated table is empty on subscriber');
+
+$node_publisher->append_conf('postgresql.conf',
+	"log_min_messages = warning");
+$node_publisher->reload;
+
 # note that data are different on provider and subscriber
 $result = $node_subscriber->safe_psql('postgres',
 	"SELECT count(*), min(a), max(a) FROM tab_ins");
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 85c808a..e01fb17 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1610,6 +1610,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
2.7.2.windows.1

#104Amit Kapila
amit.kapila16@gmail.com
In reply to: houzj.fnst@fujitsu.com (#103)
Re: logical replication empty transactions

On Tue, Mar 29, 2022 at 2:05 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

Attach the new version patch which addressed the
above comments and slightly adjusted some code comments.

The patch looks good to me. One minor suggestion is to change the
function name ProcessPendingWritesAndTimeOut() to
ProcessPendingWrites().

--
With Regards,
Amit Kapila.

#105houzj.fnst@fujitsu.com
houzj.fnst@fujitsu.com
In reply to: Amit Kapila (#104)
1 attachment(s)
RE: logical replication empty transactions

On Tuesday, March 29, 2022 5:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Tue, Mar 29, 2022 at 2:05 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

Attach the new version patch which addressed the above comments and
slightly adjusted some code comments.

The patch looks good to me. One minor suggestion is to change the function
name ProcessPendingWritesAndTimeOut() to ProcessPendingWrites().

Thanks for the comment.
Attach the new version patch with this change.

Best regards,
Hou zj

Attachments:

v34-0001-Skip-empty-transactions-for-logical-replication.patchapplication/octet-stream; name=v34-0001-Skip-empty-transactions-for-logical-replication.patchDownload
From fe48eca27591a64bc6fc40864aabc2b7118f3a54 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@cn.fujitsu.com>
Date: Sun, 27 Mar 2022 10:09:16 +0800
Subject: [PATCH] Skip empty transactions for logical replication.

The current logical replication behavior is to send every transaction to
the subscriber even if the transaction is empty. This can happen because
transaction doesn't contain changes from the selected publications or all
the changes got filtered. It is a waste of CPU cycles and network
bandwidth to build/transmit these empty transactions.

This patch addresses the above problem by postponing the BEGIN message
until the first change is sent. While processing a COMMIT message, if
there was no other change for that transaction, do not send the COMMIT
message. This allows us to skip sending BEGIN/COMMIT messages for empty
transactions.

When skipping empty transactions in synchronous replication mode, we send
a keepalive message to avoid delaying such transactions.

Author: Ajin Cherian, Hou Zhijie, Euler Taveira
Reviewed-by: Peter Smith, Takamichi Osumi, Shi Yu, Masahiko Sawada, Greg Nancarrow, Vignesh C, Amit Kapila
Discussion: https://postgr.es/m/CAMkU=1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q@mail.gmail.com
---
 src/backend/replication/logical/logical.c   |   6 +-
 src/backend/replication/pgoutput/pgoutput.c | 141 ++++++++++++++++++++++++++--
 src/backend/replication/walsender.c         |  72 +++++++++++---
 src/include/replication/logical.h           |   3 +-
 src/include/replication/output_plugin.h     |   2 +-
 src/test/subscription/t/001_rep_changes.pl  |  28 ++++++
 src/test/subscription/t/020_messages.pl     |   5 +-
 src/tools/pgindent/typedefs.list            |   1 +
 8 files changed, 228 insertions(+), 30 deletions(-)

diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13..e1f14ae 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -683,12 +683,14 @@ OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write)
  * Update progress tracking (if supported).
  */
 void
-OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx)
+OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx,
+						   bool skipped_xact)
 {
 	if (!ctx->update_progress)
 		return;
 
-	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid);
+	ctx->update_progress(ctx, ctx->write_location, ctx->write_xid,
+						 skipped_xact);
 }
 
 /*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 893833e..20d0b1e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -183,6 +183,36 @@ typedef struct RelationSyncEntry
 	MemoryContext entry_cxt;
 } RelationSyncEntry;
 
+/*
+ * Maintain a per-transaction level variable to track whether the transaction
+ * has sent BEGIN. BEGIN is only sent when the first change in a transaction
+ * is processed. This makes it possible to skip sending a pair of BEGIN/COMMIT
+ * messages for empty transactions which saves network bandwidth.
+ *
+ * This optimization is not used for prepared transactions because if the
+ * WALSender restarts after prepare of a transaction and before commit prepared
+ * of the same transaction then we won't be able to figure out if we have
+ * skipped sending BEGIN/PREPARE of a transaction as it was empty. This is
+ * because we would have lost the in-memory txndata information that was
+ * present prior to the restart. This will result in sending a spurious
+ * COMMIT PREPARED without a corresponding prepared transaction at the
+ * downstream which would lead to an error when it tries to process it.
+ *
+ * XXX We could achieve this optimization by changing protocol to send
+ * additional information so that downstream can detect that the corresponding
+ * prepare has not been sent. However, adding such a check for every
+ * transaction in the downstream could be costly so we might want to do it
+ * optionally.
+ *
+ * We also don't have this optimization for streamed transactions because
+ * they can contain prepared transactions.
+ */
+typedef struct PGOutputTxnData
+{
+	bool		sent_begin_txn;	/* flag indicating whether BEGIN has
+								 * been sent */
+}		PGOutputTxnData;
+
 /* Map used to remember which relation schemas we sent. */
 static HTAB *RelationSyncCache = NULL;
 
@@ -488,15 +518,41 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
 }
 
 /*
- * BEGIN callback
+ * BEGIN callback.
+ *
+ * Don't send the BEGIN message here instead postpone it until the first
+ * change. In logical replication, a common scenario is to replicate a set of
+ * tables (instead of all tables) and transactions whose changes were on
+ * the table(s) that are not published will produce empty transactions. These
+ * empty transactions will send BEGIN and COMMIT messages to subscribers,
+ * using bandwidth on something with little/no use for logical replication.
  */
 static void
-pgoutput_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
+pgoutput_begin_txn(LogicalDecodingContext * ctx, ReorderBufferTXN * txn)
+{
+	PGOutputTxnData	*txndata = MemoryContextAllocZero(ctx->context,
+													  sizeof(PGOutputTxnData));
+
+	txn->output_plugin_private = txndata;
+}
+
+/*
+ * Send BEGIN.
+ *
+ * This is called while processing the first change of the transaction.
+ */
+static void
+pgoutput_send_begin(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
 {
 	bool		send_replication_origin = txn->origin_id != InvalidRepOriginId;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+	Assert(txndata);
+	Assert(!txndata->sent_begin_txn);
 
 	OutputPluginPrepareWrite(ctx, !send_replication_origin);
 	logicalrep_write_begin(ctx->out, txn);
+	txndata->sent_begin_txn = true;
 
 	send_repl_origin(ctx, txn->origin_id, txn->origin_lsn,
 					 send_replication_origin);
@@ -511,7 +567,25 @@ static void
 pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+	bool		sent_begin_txn;
+
+	Assert(txndata);
+
+	/*
+	 * We don't need to send the commit message unless some relevant change
+	 * from this transaction has been sent to the downstream.
+	 */
+	sent_begin_txn = txndata->sent_begin_txn;
+	OutputPluginUpdateProgress(ctx, !sent_begin_txn);
+	pfree(txndata);
+	txn->output_plugin_private = NULL;
+
+	if (!sent_begin_txn)
+	{
+		elog(DEBUG1, "skipped replication of an empty transaction with XID: %u", txn->xid);
+		return;
+	}
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit(ctx->out, txn, commit_lsn);
@@ -542,7 +616,7 @@ static void
 pgoutput_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					 XLogRecPtr prepare_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_prepare(ctx->out, txn, prepare_lsn);
@@ -556,7 +630,7 @@ static void
 pgoutput_commit_prepared_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 							 XLogRecPtr commit_lsn)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_commit_prepared(ctx->out, txn, commit_lsn);
@@ -572,7 +646,7 @@ pgoutput_rollback_prepared_txn(LogicalDecodingContext *ctx,
 							   XLogRecPtr prepare_end_lsn,
 							   TimestampTz prepare_time)
 {
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_rollback_prepared(ctx->out, txn, prepare_end_lsn,
@@ -1295,6 +1369,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				Relation relation, ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	TransactionId xid = InvalidTransactionId;
@@ -1371,6 +1446,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				break;
 
 			/*
+			 * Send BEGIN if we haven't yet.
+			 *
+			 * We send the BEGIN message after ensuring that we will actually
+			 * send the change. This avoids sending a pair of BEGIN/COMMIT
+			 * messages for empty transactions.
+			 */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
+			/*
 			 * Schema should be sent using the original relation because it
 			 * also sends the ancestor's relation.
 			 */
@@ -1420,6 +1505,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 									 relentry, &action))
 				break;
 
+			/* Send BEGIN if we haven't yet */
+			if (txndata && !txndata->sent_begin_txn)
+				pgoutput_send_begin(ctx, txn);
+
 			maybe_send_schema(ctx, change, relation, relentry);
 
 			OutputPluginPrepareWrite(ctx, true);
@@ -1480,6 +1569,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 										 relentry, &action))
 					break;
 
+				/* Send BEGIN if we haven't yet */
+				if (txndata && !txndata->sent_begin_txn)
+					pgoutput_send_begin(ctx, txn);
+
 				maybe_send_schema(ctx, change, relation, relentry);
 
 				OutputPluginPrepareWrite(ctx, true);
@@ -1510,6 +1603,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				  int nrelations, Relation relations[], ReorderBufferChange *change)
 {
 	PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+	PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 	int			i;
@@ -1548,6 +1642,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			continue;
 
 		relids[nrelids++] = relid;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+
 		maybe_send_schema(ctx, change, relation, relentry);
 	}
 
@@ -1585,6 +1684,19 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = txn->xid;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * messages.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_message(ctx->out,
 							 xid,
@@ -1629,6 +1741,19 @@ pgoutput_sequence(LogicalDecodingContext *ctx,
 	if (!relentry->pubactions.pubsequence)
 		return;
 
+	/*
+	 * Output BEGIN if we haven't yet. Avoid for non-transactional
+	 * sequence changes.
+	 */
+	if (transactional)
+	{
+		PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+		/* Send BEGIN if we haven't yet */
+		if (txndata && !txndata->sent_begin_txn)
+			pgoutput_send_begin(ctx, txn);
+	}
+
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_sequence(ctx->out,
 							  relation,
@@ -1799,7 +1924,7 @@ pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 	Assert(!in_streaming);
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
@@ -1820,7 +1945,7 @@ pgoutput_stream_prepare_txn(LogicalDecodingContext *ctx,
 {
 	Assert(rbtxn_is_streamed(txn));
 
-	OutputPluginUpdateProgress(ctx);
+	OutputPluginUpdateProgress(ctx, false);
 	OutputPluginPrepareWrite(ctx, true);
 	logicalrep_write_stream_prepare(ctx->out, txn, prepare_lsn);
 	OutputPluginWrite(ctx, true);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2d0292a..6a3e96f 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -242,14 +242,16 @@ static void ProcessStandbyMessage(void);
 static void ProcessStandbyReplyMessage(void);
 static void ProcessStandbyHSFeedbackMessage(void);
 static void ProcessRepliesIfAny(void);
-static void WalSndKeepalive(bool requestReply);
+static void ProcessPendingWrites(void);
+static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
 static void WalSndKeepaliveIfNecessary(void);
 static void WalSndCheckTimeOut(void);
 static long WalSndComputeSleeptime(TimestampTz now);
 static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
 static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
 static void WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
-static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid);
+static void WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+								 bool skipped_xact);
 static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc);
 static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
 static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
@@ -1399,6 +1401,16 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
 	}
 
 	/* If we have pending write here, go to slow path */
+	ProcessPendingWrites();
+}
+
+/*
+ * Wait until there is no pending write. Also process replies from the other
+ * side and check timeouts during that.
+ */
+static void
+ProcessPendingWrites(void)
+{
 	for (;;)
 	{
 		long		sleeptime;
@@ -1447,9 +1459,12 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
  * LogicalDecodingContext 'update_progress' callback.
  *
  * Write the current position to the lag tracker (see XLogSendPhysical).
+ *
+ * When skipping empty transactions, send a keepalive message if necessary.
  */
 static void
-WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
+WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
+					 bool skipped_xact)
 {
 	static TimestampTz sendTime = 0;
 	TimestampTz now = GetCurrentTimestamp();
@@ -1459,12 +1474,35 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
 	 * avoid flooding the lag tracker when we commit frequently.
 	 */
 #define WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS	1000
-	if (!TimestampDifferenceExceeds(sendTime, now,
-									WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS))
-		return;
+	if (TimestampDifferenceExceeds(sendTime, now,
+								   WALSND_LOGICAL_LAG_TRACK_INTERVAL_MS))
+	{
+		LagTrackerWrite(lsn, now);
+		sendTime = now;
+	}
 
-	LagTrackerWrite(lsn, now);
-	sendTime = now;
+	/*
+	 * When skipping empty transactions in synchronous replication, we send a
+	 * keepalive message to avoid delaying such transactions.
+	 *
+	 * It is okay to check sync_standbys_defined flag without lock here as
+	 * in the worst case we will just send an extra keepalive message when it
+	 * is really not required.
+	 */
+	if (skipped_xact &&
+		SyncRepRequested() &&
+		((volatile WalSndCtlData *) WalSndCtl)->sync_standbys_defined)
+	{
+		WalSndKeepalive(false, lsn);
+
+		/* Try to flush pending output to the client */
+		if (pq_flush_if_writable() != 0)
+			WalSndShutdown();
+
+		/* If we have pending write here, make sure it's actually flushed */
+		if (pq_is_send_pending())
+			ProcessPendingWrites();
+	}
 }
 
 /*
@@ -1550,7 +1588,7 @@ WalSndWaitForWal(XLogRecPtr loc)
 		if (MyWalSnd->flush < sentPtr &&
 			MyWalSnd->write < sentPtr &&
 			!waiting_for_ping_response)
-			WalSndKeepalive(false);
+			WalSndKeepalive(false, InvalidXLogRecPtr);
 
 		/* check whether we're done */
 		if (loc <= RecentFlushPtr)
@@ -2068,7 +2106,7 @@ ProcessStandbyReplyMessage(void)
 
 	/* Send a reply if the standby requested one. */
 	if (replyRequested)
-		WalSndKeepalive(false);
+		WalSndKeepalive(false, InvalidXLogRecPtr);
 
 	/*
 	 * Update shared state for this WalSender process based on reply data from
@@ -3074,7 +3112,7 @@ WalSndDone(WalSndSendDataCallback send_data)
 		proc_exit(0);
 	}
 	if (!waiting_for_ping_response)
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 }
 
 /*
@@ -3563,18 +3601,22 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
  *
  * If requestReply is set, the message requests the other party to send
  * a message back to us, for heartbeat purposes.  We also set a flag to
- * let nearby code that we're waiting for that response, to avoid
+ * let nearby code know that we're waiting for that response, to avoid
  * repeated requests.
+ *
+ * writePtr is the location up to which the WAL is sent. It is essentially
+ * the same as sentPtr but in some cases, we need to send keep alive before
+ * sentPtr is updated like when skipping empty transactions.
  */
 static void
-WalSndKeepalive(bool requestReply)
+WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
 {
 	elog(DEBUG2, "sending replication keepalive");
 
 	/* construct the message... */
 	resetStringInfo(&output_message);
 	pq_sendbyte(&output_message, 'k');
-	pq_sendint64(&output_message, sentPtr);
+	pq_sendint64(&output_message, XLogRecPtrIsInvalid(writePtr) ? sentPtr : writePtr);
 	pq_sendint64(&output_message, GetCurrentTimestamp());
 	pq_sendbyte(&output_message, requestReply ? 1 : 0);
 
@@ -3613,7 +3655,7 @@ WalSndKeepaliveIfNecessary(void)
 											wal_sender_timeout / 2);
 	if (last_processing >= ping_time)
 	{
-		WalSndKeepalive(true);
+		WalSndKeepalive(true, InvalidXLogRecPtr);
 
 		/* Try to flush pending output to the client */
 		if (pq_flush_if_writable() != 0)
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9..a6ef16a 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -26,7 +26,8 @@ typedef LogicalOutputPluginWriterWrite LogicalOutputPluginWriterPrepareWrite;
 
 typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct LogicalDecodingContext *lr,
 														 XLogRecPtr Ptr,
-														 TransactionId xid
+														 TransactionId xid,
+														 bool skipped_xact
 );
 
 typedef struct LogicalDecodingContext
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf..fe85d49 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -270,6 +270,6 @@ typedef struct OutputPluginCallbacks
 /* Functions in replication/logical/logical.c */
 extern void OutputPluginPrepareWrite(struct LogicalDecodingContext *ctx, bool last_write);
 extern void OutputPluginWrite(struct LogicalDecodingContext *ctx, bool last_write);
-extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx);
+extern void OutputPluginUpdateProgress(struct LogicalDecodingContext *ctx, bool skipped_xact);
 
 #endif							/* OUTPUT_PLUGIN_H */
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index eca1c63..c35d83f 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -473,6 +473,34 @@ $node_publisher->safe_psql('postgres', "INSERT INTO tab_full VALUES(0)");
 
 $node_publisher->wait_for_catchup('tap_sub');
 
+# Check that we don't send BEGIN and COMMIT because of empty transaction
+# optimization.  We have to look for the DEBUG1 log messages about that, so
+# temporarily bump up the log verbosity.
+$node_publisher->append_conf('postgresql.conf', "log_min_messages = debug1");
+$node_publisher->reload;
+
+# Note that the current location of the log file is not grabbed immediately
+# after reloading the configuration, but after sending one SQL command to
+# the node so as we are sure that the reloading has taken effect.
+$log_location = -s $node_publisher->logfile;
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab_notrep VALUES (11)");
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+$logfile = slurp_file($node_publisher->logfile, $log_location);
+ok( $logfile =~
+	  qr/skipped replication of an empty transaction with XID/,
+	'empty transaction is skipped');
+
+$result = $node_subscriber->safe_psql('postgres',
+	"SELECT count(*) FROM tab_notrep");
+is($result, qq(0), 'check non-replicated table is empty on subscriber');
+
+$node_publisher->append_conf('postgresql.conf',
+	"log_min_messages = warning");
+$node_publisher->reload;
+
 # note that data are different on provider and subscriber
 $result = $node_subscriber->safe_psql('postgres',
 	"SELECT count(*), min(a), max(a) FROM tab_ins");
diff --git a/src/test/subscription/t/020_messages.pl b/src/test/subscription/t/020_messages.pl
index b5045ff..d21d929 100644
--- a/src/test/subscription/t/020_messages.pl
+++ b/src/test/subscription/t/020_messages.pl
@@ -87,9 +87,8 @@ $result = $node_publisher->safe_psql(
 			'publication_names', 'tap_pub')
 ));
 
-# 66 67 == B C == BEGIN COMMIT
-is( $result, qq(66
-67),
+# no message and no BEGIN and COMMIT because of empty transaction optimization
+is($result, qq(),
 	'option messages defaults to false so message (M) is not available on slot'
 );
 
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 85c808a..e01fb17 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1610,6 +1610,7 @@ PGMessageField
 PGModuleMagicFunction
 PGNoticeHooks
 PGOutputData
+PGOutputTxnData
 PGPROC
 PGP_CFB
 PGP_Context
-- 
2.7.2.windows.1

#106shiy.fnst@fujitsu.com
shiy.fnst@fujitsu.com
In reply to: houzj.fnst@fujitsu.com (#105)
3 attachment(s)
RE: logical replication empty transactions

On Tue, Mar 29, 2022 5:15 PM Hou, Zhijie/侯 志杰 <houzj.fnst@fujitsu.com> wrote:

Thanks for the comment.
Attach the new version patch with this change.

Hi,

I did a performance test for this patch to see if it affects performance when
publishing empty transactions, based on the v32 patch.

In this test, I use synchronous logical replication, and publish a table with no
operations on it. The test uses pgbench, each run takes 15 minutes, and I take
median of 3 runs. Drop and recreate db after each run.

The results are as follows, and attach the bar chart. The details of the test is
also attached.

TPS - publishing empty transactions (scale factor 1)
--------------------------------------------------------------------
4 threads 8 threads 16 threads
HEAD 4818.2837 4353.6243 3888.5995
patched 5111.2936 4555.1629 4024.4286

TPS - publishing empty transactions (scale factor 100)
--------------------------------------------------------------------
4 threads 8 threads 16 threads
HEAD 9066.6465 16118.0453 21485.1207
patched 9357.3361 16638.6409 24503.6829

There is an improvement of more than 3% after applying this patch, and in the
best case, it improves by 14%, which looks good to me.

Regards,
Shi yu

Attachments:

TPS_empty_transactions_scale_factor_1.PNGimage/png; name=TPS_empty_transactions_scale_factor_1.PNGDownload
TPS_empty_transactions_scale_factor_100.PNGimage/png; name=TPS_empty_transactions_scale_factor_100.PNGDownload
�PNG


IHDR��I��sRGB���gAMA���a	pHYs���o�d{�IDATx^����9��]W�=f]�8
��e����
��]B�*c�q�~�9�+���>��_��`&��dDV�2��F�^����&�~8u�������iL����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i�k�����W��:�^���{x��u������:F�����	�:C�c���u������:�^���{x��u��=����u����_	���{x��u������:�^���{�O�'X���]�����:�^���{x��u����1�����O����~%x��u������:�^���{x�ct��?��`�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i��!�B!�B!t�xc�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i�k�����W��:�^���{x��u������:F�����	�:C�c���u������:�^���{x��u��=����u����_	���{x��u������:�^���{�O�'X���]�����:�^���{x��u����1�����O����~%x��u������:�^���{x�ct��?��`�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i�k�����W��:�^���{x��u������:F�����	�:C�c���u������:�^���{x��u��=����u����_	���{x��u������:�^���{�O�'X���]�����:�^���{x��u����1�����O����~%x��u������:�^���{x�ct��?��`�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i������B���~#�~���7B!����Go�u��<��EV����� ��^'����
��+�{{�������?k�u�`��A%��	�u"X�A���`
Z2��/���5��^'���`M���5h����kb�T�{��^'�5k:��%����!X#`P	�u�{��t�� X�����K�&�`����>������O�q���N������;��o��
�c��}���t�� X�����K�&�`�U��R��8����p�����[<_��_�qx�^�_�v|���'��� �4�� X�����K�&���8hMV����Y��05�+6��;�������o��g���������{��(�m�>�����`~��5k����	��X�M�����9����*4�Y(a���k��{���%������?������.�D~�l�%p

��j��>�>d�M�:r��z������*�=���`
�MvMo�nM~������$�x�����`�
k:��%�����&/�F���5�o��X�������9o��5	��r�*\����I��s.�G~�5���_�e���kP�����z�#�~w������m�Y����0��E-�u����t�@KF��%XcM^4�&��k~_��Q������E!�,�{!F��I�������zYh��o�P��z�M�`����
�Jo^k
s�z��`M��d��_�51���A�h���������dMzr�B'@�B��������|�s���]vMo���� ��m���]��~�vk�w��\�h X�A�-���`M�5yq�4��~��}�BS�������O�p
R�� 
����]�
�uJ��Fv=o�k��f�B��8��{'X[��)X;�:�5@���5h����kb������d�{�������VA�kB�pN�M�9c�c������!��s/!�Fx����y�RB��a��m��%���`�
k:��%�����&/�F���5���z�vi�MSc��P���X!dH�}�`�@4�K�p�y�%!D�����o�]���ze�=�����}�b�b�y�����j�\�h X�A�������w��/_�->_�~���������w�y;����j|��q�^��i��3��/��k���i4Y�^����u�{��t�� X�{"?5����|���?���O����`
������`M��Jx���D���`M��3���0o�5��@�Be|��yw���k��7X�R�G0��u�g���1����!X#`P	�u�{��t�� X�{��`-Gj��kz��A��a��_�51k*��Nx����5kp����M�cj��mo��1������/��M�Z���y��|�����;���b����yV�m�c�c��s��,�^��6�p\X��?(&����������K��{ �'(�3�?��c���a��k#_�QZ[#��]?&��c�y���y���q��w���O��!��'�w1B�������:�9����>4��k��8��p���7�s���B`q�������k���!G��7g;'6�1�q!H��
�G ��������1����c�����c�g'��`��xM�PW�5'����������vL�-�#����j��k��3N���k�y��w��v�m�k��l[ \��i��y�6^|<�7���}/�*�X��5yv�������N��5����:���a��{%#yo�p,,$���@�/4�q�b�����B@�F>/;&!BH�0���<�(��>��9!���������{��C�������_�u����ul�������������^����9&������������/��wL�|��{��s��������s|������y��������Zg�T�{��^'�5k:������L�>k�c��y�]'wK�P���yPa�C�����V�=�����K�!���b������|���+>.���m�)����B�|�F>����k�������bJc����J�������_L>�]+�4�t��>�j��(]/���ks��e��_�51k�*��Nx����5kp�������5�&��*� 6vh���a�����P_^��{�����C��si�@�Ia�yh��m�s-�=�fi�j
k�U�Q�vn��+P�f��a�����[���z�5k���x��V�^�7�*l��0��/���5���^'���`M���5�g�����@m_������=��y�a�A��U

BHR
�x�-�6���y�q�a�����Z��v\��������&�����Z�{�f�^����9�1K��u���c�k�k���3�]��+QZ���p��V������������c��_�51k�*��Nx����5kp��
xNm_�Z�6�w�|^�9*B�B����aE����P"&�_������?����P'�%?f�����@>o#�k��{�{���a�>���	���(���3F86�����#�w�8[�3�x;/&_��s �����^�y�}3��/���5���^'���`M���5�g�  P�g��=n�C�������k��}h����������s5�s���#W�y!�������n��cO�%���\7��0�'���b�9l[���������>��������:�u���%P�?�n<�@^W ^��|��3N~�a����x��������g���=S�*�-xe�������f��_�51i����o�<�~������>���_N��v�wt���G�����y��\����_�G6���v���#?����#����������K�����U<��/���/O����?���y����EO�L��m���
������{X;o�/YO���6������oZ��O/�� �n�t.�����{�����r������f������}
k:�����9o��}�88���`c�B�d�sB�`��"��
�B0��yp����C��|S\o-�����qq�c�1y@��|	?b�s�����m�s�?�\����(�5��
ux�y���sJ�����a.���@���&��5���GL�?����b��p�)�����0��U8.(�����	��X�w
��k� ������~��|��d��1��u�G>n�M���45�Q�3}��?OO�9^����������tl���\�����?==�D��?������y<��4'7@qk����/Z�)�I�������^�V���=_3�g
�T����e�zuo��2������}wUe�]���Y����{P@���`
�uX@@s%:9����@�V��:#2��/��k��A�;k�����������?���s���^�����(8������?�� ����:6�9�o1�p������}>����*���'^�����?����������
��������������yo�Y$�^�5k:�^�ZJ��-�����1������K�&�E�f
t��w�B�6�������C��n�-(��u(��5X����g�cRx3��/������������^�g�������+��i�O�y����hkZ;wc}�~���s��Y�������E�{��{T����^_b��'����f��o��5k���eM�b�=z�`-�u��uE�����kb���Ms�umJ#U����M��-�_N��{���r��	�R�AA�����|���O�~L!���Lm_t�I�c�?� ��4@�a���,P��}6����v���i�~���[�>�����e�UP�
�J58�w������[��m���f��8 ������5k����	��X�w
�����7�VAZ�����7���$
v��l[T<v�����@v�[�pB���;��UC�.o_Z��D�]���nx�����pIq�OA�������������Uuo�yW&/���f�� X�A�-���`M��k�>�f��5������Jq�S~��`z�'�}S�_�w�������B�|-'Y`��1�$X�UC�.g���eRq}�@h���	k<k��:�1�U�L�?��_n����E���y�~&/������t�@KF��%XcM�5hzc�6�c��i���/��3��V�z����u������!��AO��?�@`�B��}��"X;6Vv\�����|/���}>g�[�w^e�t�(x�������W�k�+������O�kK�y8�������o��������{{�<���t�� X�����K�&���������h���$������w�:��%?o|�k	���!���>���f����u�P��E�y��J�Ma�u�P���s;�AH2��.uk��W���8����C��������k���s3�����X����]��=�\���_��������]�1��
k:��%�����&/�F�����_�^x���D���`M��d��_�51k*��Nx����5k����	���0���:��Nk:�t��>��~��|j�������|��lyv��o��-[���1�l
�����kb�T�{��^'�5k:���\���=��O�>�>~��|:6VG-�����?�>�oc�!����p4F��]k���5y!����]��9D�G�w����Z����B��k6�o����m����8\��������Q�k��&�n�Q55������	z?��Nx��yF��W2���lpTB��������u���a����9��1�v�������l�	^��m����������Zg�T�{��^'�5k:��Ax{)�3��=hk�������`-KA��,��J�>/D��,?&�X���1ZAy�a���/�R�����/yj�������G`��_�51k�*��Nx����5k����<
�M���`&�Zqd���'��������}�
�cl[����x���m���6��#����e5�1�8�������1������9�6�����`M���Jx���D���`M��1�j� '�������y��>���������'�y�������U���\���|1!X��m�58����!X#\P	�u�{��t�� X8��Pi��������(���������������)��j������#�w��o8�����k�>������?�}����~���oo_E!`�`�Vx���D���`M��1��5#5A����=b�s��9y���jc��h��s���T�����y�=�#��+X��jcz��mv<�����`M�{k���������R�W��5�����9���_���}?��+�������?O��a���:���Uy����~���&��l{���js�SC�/��������6��������/�<��^�s*������[*�-���u}�{�T���.���W��Z�.�A���5�c`����XB������O~N)X+]���-#��m��Q��`����#���j�|(��{cu�k���{g��_�51���A�[D��JM�x���@������~,�N�%`�s
���Szn���}K����|�����Ns���~���R�a���g�zN!�:`	��?���������jZ>��P�~���{�=�m��rU_����^]����oa
k5���� X8{��Xxc���B��<���[��B�e�5m[���_?����%�bB}�������_��{�h���=�*��3��/���4X�����o�����y��{x��Q���N�\��pl��6�/��{��_��������?.��J�{>���k��>k�#����?X�wb���|_y�W�s��,S����j�S������8Ly�a���������~������P_TOr��R{~l�^�z�F5���� X8^e�K���8�	��wN�^8?;��8���i�o�~vN��*����-�l�x,������_������R�f���1b�������7�#2��/��k��AS��!���%����k^M����kspv	�.�����������s:�|~r��8�~c-o���`���V��B����;W�cT�-��c<���������K��]����M~OkZ���SP3}N��p��;�''��������/�D�v���6S�^>k���k���{P@���`
��B �+'aAqH��c�����f����D�u������}�vS>~i<�p��8�3�q�������[�p����K�&���k��$��'����Mo�]�$[k��3�7�&9���s�n>f4�l���7hnr�M�Js47��f|j��}n�3k
��>h���T��o��ySP���g1��������K����j��uY��Sg�����3��:\������Y�=���2�+�������������=k{
����=( X�A�p,|��!�y	�#�����`M�5y����]��o����F���]{��x��%X���k��AX9<3��yADM�1�}S0���x�f����K��]��J=K�a�8��c���u-���kZ�[��������������[�Z�-���������0��S�{P@���`
�8��A*�,���K�&���k����Zh5}c�;o�O��Z�U�iz������������:�O'������`�V���E��z����n��Jv�B(���s?�u�ck@����6�T�>�R������:�Y��P������`M������A�k���pDF��%XcM�5h������,(���<XK���m����V�����re�
o����8\xk���������^������\��\e���<l�B�<������{o.�*�j��a����g����(�����=L�1/���E��_��
�����v����.����fM5�A���5h����kb�`-��u�o��a��_~��|\�?�+���kR�����1a�I�����9���kK��L�?�&��?
��?��j����0�cT��������<���Y�\���:M�1�b
�}u_� (:�Zw��Y��/��K�29/?��z�;z���Aq�������R��u�����j��=( X�A�-���`M�5y���	�~r�Mn���>�{��^'�5k:��%����!X#`P	�u�{��t�� X�����K�&�`��A%��	�u"X�A���`
Z2��/���4XO:��Nx����5k����W��m@�dMB�~�~#�~�������Q�k��_���{x��u��=�@xc-}�"X����:�^���{x�c$�	��C��>k�AWXx��u����1��k�!XK����+������:�^���{�5�����Zk��_���{x��u��=�@��g-��C�����pzzY>��������z�tJo����������q������G�u��=�@��g-}�����85I�������<�>�x~<=$)�-��|�c�mN���7�W��L�:o_�����o������y����-�O����rO��x�������u���M�;"~W��Od����9�O�N�?x�c$�	��C��>ki�5k��7�f	�&6��k���o�<=�s1�1��]�;�����(�B���.�!�%�1�������=��
���>c���u���|O^0O����O��,������#��c��:F�~����.������t����e��o�����=�vL��ZA��>k	����*�Bs�u}s!~���Y#6_��?>'����P������s��>[�g��<j���j��&2>6������������R��������t	���>S�|>~�(��c]���V�����Z:���8���?�s��
�o�z?�\��~���O/����
����x(�^�H��y�����%|�?��y�g�oa��^�cj���%k���.X�h�BsRSs{9g���|m���:��6���^�7T�sC���7&='"������1n���u���-�Z�o����
���{��]��w=��[|������7�0�������k�����u���3�]���W�������3��}����������)��,�
�g{�������)��u;@/��g-]�Vlb�����d��U���5`��2����:9w���[�^|���|���B���{o��l�Z�Po9���F���t�}Vk�)�Y�6���t�-^&U��ci���=2��<n~<��p<�y�1�w�J~g�������?~:v�]0�}��������`����O��-�@��Y`�^�cj���%k����k���Y�{���3�&+�\c}���$�������-k��A~Pwm�����Sc��W�`-o6�kf�Zr����3S�q���o#6Z�}U���u�a
/���y���x_���L�:�z���q�F��]�N����5�����1���>���jD�h�	�j����_�5��Bhl���k��G&���cm
o�F6n��Fni��
����zy�������_�|��wnm>6�];��}k��u��n�WMq���!��^I���z�S�w"����<��>[���{�������w��{��v��a���;������<���k�Tq ���S;������ XK������K�H����Y��U�T	�VT��-���K��5zS�}�<7�i�7_�;<�F�����:6���y��������nc]�47��������t������d��^�;������_~^�/����kh��r����o�������^gfl����L�?�>����}9���/;�>������������`oot�~�:Q��{k�������)��u;@/��g-i�f�
��q
J���������t�x����%�u��z|~���	��4����U	H��GcN�[_w:.-l
��y����O�r���������N�X��\�=��_�q��������r�4����E����<�^��x��������{����u{���^K���������������`oaZ�7���j�x{���{l�����(��u;@��g-y�������:�������:�^���������Zk�-x��u����1��k�!XK����+4Z:�^���{x�c$�	��C��>k
��=����{x��u�����'X��Z���`B!�:���c����������P?o0��{x��u�����'X�o���Zk��~-x��u����1��k�!XK���:C�4Z*�^���{x�c$�	��C��>k�u��i�T������:�^�H����`-}�"X���h��{x��u�����'X��Z��E�����R��:�^���{#yO�����Y�`�3�O���u������:F��`
�?k���Zg��FK���{x��u��=�@��g-���P?��
����:�^���{�5�����Zk��~-x��u����1��k�!XK���:C�4Z*�^���{x�c$�	��C��>k�u��i�T������:�^�H����`-}�"X���h��{x��u�������`�������>m���K��>>~�x�����e���cj���k���Zg��FK���{x��u����{�B�d������q��m�?��l�Y�e��=�vL��ZB��>k�u��i�T������:�^�H���`oA�V�fA��������cj������k���Zg��FK���{x��u�������W���_���JeTYg�n�S;��x-!XK���:C�4Z*�^���{x�c$�oy������_/�Z]�cZ�����Y�`�3�O���u������:F�~���Q�VZ�����8�
A���cj�����}/����B��e������Y�`�3�O���u������:F�~����N���'{c���2���z�q�[����)��u;@/��g���B!�����������Y��o������-���pa��(��u;@����^U�����
x��u����1����?��~�������k���Zg��FK���{x��u��=��^��ZG�`-}�"X���h��{x��u�����'X����Z�����Zk��~-x��u����1��k�^x�Wk���Y�`�3�O���u������:F��`
�/�j��B��>k�u��i�T������:�^�H���{�_�uT��g-���P?��
����:�^���{�5x/�����
�Z��E�����R��:�^���{#yO���|��Q!XK���:C�4Z*�^���{x�c$�	�������:*k���Zg��FK���{x��u��=��^��ZG�`-}�"X���h��{x��u�����'X����Z�����Zk��~-x��u����1��k�^x�Wk���Y�`�3�O���u������:F��`
�/�j��B��>k�u��i�T������:�^�H���{�_�uT��g-���P?��
����:�^���{�5x/�����
�Z��E�����R��:�^���{#yO���|��Q!XK���:C�4Z*�^���{x�c$�	�������v�^��ZG�`-}�"X���h��{x��u�����'X8&^��Z[x�Wk���Y�`�3�O���u������:F�~����.������t����e��o�����=�vL���/�j�-�����
�Z���`B!�:��<����>��?}�t������?�~����S;��x��|��^��ZG�����Q�k��~�`P��:�^���{#y��cnAT�,����lo������1�}�n8^��Z[x�Wk�XK���:C�4Z*�^���{x�c$�o}��7���/#�,p��������i1���_���|��Q!XK���:C�4Z*�^���{x�c$�oy��T���=AW����;^��Z[x�Wk���Y�`�3�O���u������:F�~���Q���X@R!�z��1�cJ�n������k�
!����������Z��G/���k��G&�"X���h��{x��u�������`o���jy�d-��������1�}�n8^��Z[x�Wk���Y�`�3�O���u������:F�~����UI�b����f��v���l��wL�����G��Zk/�j��B��>k�u��i�T������:�^�H���`pT���������:*k���Zg��FK���{x��u��=��1�������Z�����Zk��~-x��u����1��k���Zk/�j��B��>k�u��i�T������:�^�H��/�j�-�����
�Z��E�����R��:�^���{#yO�pL���������:*k���Zg��FK���{x��u��=��1�������Z�����Zk��~-x��u����1��k���Zk/�j��B��>k�u��i�T������:�^�H��/�j�-�����
�Z��E�����R��:�^���{#yO�pL���������:*k���Zg��FK���{x��u��=��1�������Z�����Zk��~-x��u����1��k���Zk/�j��B��>k�u��i�T������:�^�H��/�j�-�����
�Z��E�����R��:�^���{#yO�pL���������:*k����m@!�B�h"X8&^��Z[x�Wk�����Go�u��y�A���{x��u��=��^��Z��_����N�uTxc-}�"X���h��{x��u�����'X���<Zkt����������
�Z��E�����R��:�^���{#yO���x���x�Wkm��Sk���Y�`�3�O���u������:F��`
��x�Gk����o�Zko�Z�����Zk��~-x��u����1��{���~�������/����g������e��o����o���b<�{�=Z��x���xk�Z[x��ZG�`-}�"X���h��{x��u�������`o��Q���Z�9`�>�<�9�k��=�vL���/�h�^��Z���Qkm��Sk���Y�`�3�O���u������:F���{�L��s��*����x�u{L����[�	/�h�^��Z���Qkm��Sk���Y�`�3�O���u������:F����Z����_����O�>M�n�S;��x��z�V
/�h������������
�Z��E�����R��:�^���{#y��`-����S`e��j����_�5���R^��Z�<������y�$o�Z��G,o�Z��G/���k��G&�"X���h��{x��u�������������a�UT� ���1�cJ�n�p$����jx�Gk���F����N�uT��g-���P?��
����:�^�����
���5{����0���u{L����[�	/�h�^��Z���Qkm��Sk���Y�`�3�O���u������:F�~����f����L� -����o����B���1���[�/�h�^��Z���Qkm��Sk���Y�`�3�O���u������:F�~�{�����U�<Zkt�5j�-�uj��B��>k�u��i�T������:�^�H��/�h�^��Z���Qkm��Sk���Y�`�3�O���u������:F��`
��x�Gk����o�Zko�Z�����Zk��~-x��u����1��k��=Z��x���xk�Z[x��ZG�`-}�"X���h��{x��u�����'X8&^��Z5����F�[����[��:*k���Zg��FK���{x��u��=��1�B�����5:����:��Q!XK���:C�4Z*�^���{x�c$�	���z�V
/�h������������
�Z���`B!�:����z�V
/�h�������������}�z����7�:C�����u������:F��`
��x�Gk����o�Zko�Z����Z��E�����R��:�^���{#yO�pL����jx�Gk���F����N�uT��g-���P?��
����:�^���{�5�c���U�<Zkt�5j�-�uj��B��>k�u��i�T������:�^�H��/�h�^��Z���Qkm��Sk���Y�`�3�O���u������:F��`
��x�Gk����o�Zko�Z�����Zk��~-x��u����1��k��=Z��x���xk�Z[x��ZG�`-}�"X���h��{x��u�����'X8&^��Z5����F�[����[��:*k���Zg��FK���{x��u��=��1�B�����5:����:��Q!XK���:C�4Z*�^���{x�c$�	���z�V
/�h������������
�Z��E�����R��:�^���{#yO�pL����jx�Gk���F����N�uT��g-���P?��
����:�^��������?}�������e�������o��-[�o{L����3^��Z5����F�[����[��:*k���Zg��FK���{x��u����{�,��P-����?���1�k{L����;^��Z5����F�[����[��:*k���Zg��FK���{x��u���-�H�HY0���f�{m��S�w�v�#���U�<Zkt�5j�-�uj��B��>k�u��i�T������:�^�H��5X��O�>M��{m���b<�{�=Z��x���xk�Z[x��ZG�`-}�"X���h��{x��u�����'X�����_M�!��z��7� /�h-o#�[�����[�����K�}�Z�������:C�4Z*�^���{x�c$�oy����8�
��{m��S�w�v�#���U�<Zkt�5j�-�uj��B��>k�u��i�T������:�^�H��%X����x��1�cJ�n�p$����jx�Gk���F����N�uT��g-���P?��
����:�^����=�_�~=}��!Q �����-������	�����(x�Gk����o�Zko�Z�����Zk��~-x��u����1���?����5��5:^��Z5�5j������������
�Z���`B!�:��������^��Z���Sk�����F�[����[��:*���}/�*�X����
����:�^���{����5���������x��Z5�5j������������
o���Zk��~-x��u����1��k��kv[k��m������jxk�Z���Qkm��Sk���Y�`�3�O���u������:F��`�xx�nkm�5��5:�:�V
o�Zkt�5j�-�uj��B��>k�u��i�T������:�^�H����m�-�f��F�[�����Qk���F����N�uT��g-���P?��
����:�^���{����5���������x��Z5�5j������������
�Z��E�������NO/�G/O�����L�����s��x��;W	���{����5���������x��Z5�5j������������
�Z��E�����yz8=<<���������!Ine�����y�����'_��}�O���.��w���1��k��kv[k��m������jxk�Z���Qkm��Sk���Y�`�3���?5�O��G����������������k��^�|�������.��w���1��k��kv[k��m������jxk�Z���Qkm��Sk���Y�`�3�������������N8&�7��7{������O���[��za��0]���E�o�v"LAG�����3��S-q�Y�6�����s?�����:��.�0��\%x�c$�	�������^��Z���Sk�����F�[����[��:*k���Zg�?�����2\)��7��_'���4$H�Y�{	������s>���a�*��?r���~ks����?���S-�����E��WX�}��;W	���{����5���������x��Z5�5j������������
�Z��E�����Oo'd�(�������M~z�Z�C�q�7i��y!A<��?�����9\���;���s�[]�'��X�}��;W	���{����5���������x��Z5�5j������������
�Z��E�����!�2D�_��I��!�4N�o	���"��Ku(I�����n?L������;�����������*�{#yO�v<�f�������o�Z���F�5:����:��Q!XK���:C�����O�Eqc��~!B��AA|�wn"\p���{o�3�{��S��Z���uy+���U��:F��`�xx�nkm�5��5:�:�V
o�Zkt�5j�-�uj��B��>k�u��K����"l����_����]��B?t�v$����?g�;�}�y�m�|��I��l2��\%x�c$�	�������^��Z���Sk�����F�[����[��:*k���Zg��T�F���K#�����+�!w�7��z|~>_c>�����s8w
 ���o���q�&�oG���O2��V��b9��.F����u��=���������kv[kt�uj�����xk�Z[x��ZG�`-}�"X���h��{x��u����������_O>|�����������O��v\���1�9�-��+^��Z[x�nk���N�U�[��o�Zko�Z�����Zk��~-x��u����1��o
��0-`�-�2��>�W���c^s�[��g�f�������o�Z���F�5:����:��Q!XK���:C�4Z*�^���{x�c$�[kZ���}������c^s�k�9^��Z[x�nk���N�U�[��o�Zko�Z�����Zk��~-x��u����1��o
��m��V�A��o����1�9�-��3^��Z[x�nk���N�U�[��o�Zko�Z�����Z�!�B�����-�
���@kO���s�2�=�5���������x��Z5�5j�������������}�z����7�:C�����u������:F��=���?N�f�WqP��[������������k�M%��m-o��f���y�$o�Z��G��F���c$yk�Z�<by��Z�<z��^��?2���Zg��FK���{x��u���{=�[0e������)�2��8���c^s�[��g�f�������o�Z���F�5:����:��Q!XK���:C�4Z*�^���{x�c$���`o��B� ������`�l���BXg�zM�5��;^��Z[x�nk���N�U�[��o�Zko�Z�����Zk��~-x��u����1���?���m�-�f��F�[�����Qk���F����N�uT��g-���P?��
����:�^���{����5���������x��Z5�5j������������
�Z��E�����R��:�^���{#yO�v<�f�������o�Z���F�5:����:��Q!XK���:C�wR������a���������|z��{8=�,���x~��{:E�&�����>���9���l���#������5/���)������ygq�r��Yxy��%s���u���o���������������I����6�+��,��;W	���{����5���������x��Z5�5j������������
�Z��E�������F<�BH�rz:7���>9��������L������L!�*�����Z�s[s����7�i�������k�Y=�9s�g�2��Y����k�����kv�~�O������?_�������*���y�w���1��k��kv[k��m������jxk�Z���Qkm��Sk���Y�`�3���
��`n���|������u���4��B����r��#������{^T��������lE|��pn6��^X]Sx�;s	5&��������.��*���y�w���1��k��kv[k��m������jxk�Z���Qkm��Sk���Y�`�3����k���}��{��1���^:>b	&����j[en9�s���\��w�Q���#����C��t�i>�kl�{a������K����w���������w~��U}�����*�{#yO�v<�f�������o�Z���F�5:����:��Q!XK���:C��Q�"\uz�_���Fj���V���J�5�p`
��a@in)~�v97�������F�3[��7��H6�4���s��;�����K>���^���_������]�<i��2��\%x�c$�	�������^��Z���Sk�����F�[����[��:*k���Zg����k�N��lj���z��q���%��kW�u� %���5�I�3�s�9����F>����/����u�������s��31��7��\%x�c$�	�������^��Z���Sk�����F�[����[��:*k���Zg������B��6��_kL3������9����`O���{��]���p��[�la}�����O�������d���6o�{���������D8?/���U��:F��`�xx�nkm�5��5:�:�V
o�Zkt�5j�-�uj��B��>k�u�����9P�6�Yh7���G���VOzy�����WG�c�1�� ���;n(L�Cp����E����nXTY�p^�}9t
x�,��}�~���/��xu}��yJ����s�w������;W	���{����5���������x��Z5�5j������������
�Z��E�������� ���7�����_�w���_������.���V�)p������gJ���������[k����/�I����q�n�W�R_�W��Z��p�������y�w���1��k��kv[k��m������jxk�Z���Qkm��Sk���Y�`�3�O���u������:F��`�xx�nkm�5��5:�:�V
o�Zkt�5j�-�uj��B��>k�u��i�T������:�^�H����m�-�f��F�[�����Qk���F����N�uT��g-���P?��
����:�^���{����5���������x��Z5�5j������������
�Z��E�����R��:�^���{#yO�v<�f�������o�Z���F�5:����:��Q!XK��l�����@!�z��������^��Z���Sk�����F�[����[��:*���}/�*�X���]�����:�^���{����5���������x��Z5�5j������������
o���Zk��~-x��u����1����`������&}��y��R:�����m�-�f��F�[�����Qk���F����N�uT��g-���P?��
����:�^����{x�����r>~�x�����i�t�{m?^��Z[x�nk���N�U�[��o�Zko�Z�����Zk��~-x��u����1����`oo��o��bJ����#�5���������x��Z5�5j������������
�Z��E�����R��:�^���{#y/�Zn}��5	���1���Hx�nkm�5��5:�:�V
o�Zkt�5j�-�uj��B��>k�u����8=?>��,�g~<?���{�O�1��s^���O/������n�FK���{x��u��=���m�5��5:�:�V
o�Zkt�5j�-�uj��B��>k�u������A��������t>ou�3�����������R���SR�����e_��k�+����i-�I�=�W1%�������\��63*�1������u��wY��kf��~k.��\q�-����9����.�K6���?��(�{#y/�Zn���Q:�����_�5��Bhl���k��G&�"X�����`�R;~jX���cnb�a�-���OuD�JR��Q����=����tz������������)�&��/�����}m.�{3��<��l�x�������f�\]��n�o�?���o����L���k���|
3���s��������{��l�J���v�����Zk�y{���Y�9_���?Fhp�5�>����&
����z���|����I�����Q�^����|��t)y�u��W�����������{�����(�[�=�h7��#�y���uO����{y����,�2�7��������1�k;@/��g-��������t��S�����4k>�aC������&:�.���_��z�����'#�����c��+_����\�m��2
A���1��lE�6�Kx��E��-{<�����w���{�\����{j�l�gD���g+[��;G	���������Zk�y{�~S�6��1�la�����(5�?I�vf
�a��2~�2�x�y	��!�����)pX�w�|����j��|^L��A�r��<�����5���w��Ys���w���'�k��D�Y�4�@������i��%x�c$�	��C��>k�u����Mi�|�������,\u�f��u�~���\���y�����)_
2n1����}��@����p��or=+�W7�w
��f�~�vVi��}�����?���}�������'������q3��(�{#yO�����Y�`�3o�OS�7���=s��	g��-�R3���\�8�)�4e/o=o�`3�B��iM���������\�kX���{V<�2n`��	wjk��������~O���&�{2�|�;�]?[�@o��%x�c$�	��C��>k�u����M���M��35�~J5���[��>�o������b����r����{����^(b�Et�4
�������\^���(
R��'�����?o�W������a~LR�����B?�����k���_���\{��_�������03�w���1��k�!XK���:C�wP���cB�?
^{^1`0��:�E���u.��1��!9��x�����%�/R2�]�r���y���^C���������t��:��{df�����������r�-]�p�u3���U��:F��`
�?k���Zg��FK���{x��u��=�@��g-���P?��
����:�^���{�5�����Zk��~-x��u����1��kc������O��Ou���|������������o����-�^�s�[���������'h�ylk����e������a����5Z`s�9��Z��E�����R��:�^���{#yO���c
��X�`-Q��n
�Z?,��k�{�A��[?�7_[���3n���I~�V��E���`P_���B!��kD��scMjI�@���7W�~���^a{���p�]74����a�)n���d��!X��0RS������M[�����s�{ �o�k�S ^S�s��7l���`-�C�.�y(f��<��M������p�x����}�o����c��m��B��!\�g�����Q�k�����W��:�^���{#yO��sc��5��8��	���
��5����=n�C(b�1^�n�!P	��v����}������~a�
;.k��l��9����^b�������g/�C�������~0yk��\��a����x

�\����~�1��v���������Y�`�3�O���u������:F��`�����4��rL�������oMp�p��o���������ym�5���@v�=aAGH�{%V8.���?��z!�����&!
����5������];^�pmS�-���%�n8�g�`-}�"X���h��{x��u�����'X�y�.�^x�v����w���c�s����!���i����#�H�=a��5��+1�Z���s���07/����������kj�?��5�,��]���a�����p��}��Cw���Y�`�3�O���u������:F��`m�F9&on��#��P$O�8��pN8>�5�y�o'_'[�8�
�;7��!���km��������\��
��|m������2`k����a�f�����������9�c��?������{?#k���Zg��FK���{x��u��=����rLhh����������g����M�5���v��?s�$�������8���x��g
�J<�����
�m�m_~
�o'��4������p������-�n���?����������������+?#k���Zg��FK���{x��u��=�@��g-���P�����tzxxX�xz��l7~<�/�NO/������sk�r��}y��=f;��~����E��{{������a����d=�s/l������oZ��)�dO�������G�}�]������W���������*�{#yO�����Y�`�3��O�x��O�C�rz����)\x|>e��Dm_���6�����1
���G)�0�y<|���4n
����E�94��e�����5���������n�[��\&�W���.
��W�^�.�q��4��w���1��k�!XK���:C����!A���88��[Q���uls��k�Q;o;���}>c�O�yg���kV���F����W��d��2:N�j�}W$��+�����0��\%x�c$�	��C��>k�u�����Uo��������_������_=�m_������i�P?o>��=���g�7v���w��7�A[���6|e}�S���s����^������\��fZ�w%�7�^�.������s������'X��Z��E����������
�_7���S#?5�Qp0K_��S;v�s
�a�SY����l�����.��]�4[�������X�;����,���/��lox����[�����w���1��k�!XK���:C��X��7���!
v��);��W�P ��-cD5�{��oW
���}NHA����~����.)��)prC����s���#_���2��\%x�c$�	��C��>k�u�����88(�S�>�96Q�_��S<�y
�o�D���Mc(�w����J]��r(��_\�9�)z�}����x_�������x��q���Z��u����u��=�@��g-���P���?7�y�pi��F=�}!D�����T��1�q��^����`������=�{��}�5�0e
w���_���oL:N�_�rA��|cm���5�|�y�}��%����9���uI��gF����u��=�@��g-���P���/���U����B���$3(�s��u.8!B��)��n���~��G2��.uk��W���8����C�@�}��e��5�y��3h�c].��}�x_�&^�.������;W	���{�5�����Zk��~-x��u����1��k�!XK���:C�4Z*�^���{x�c$�	��C��>k�u��i�T������:�^�H����`-}�"X���h��{x��u�����'X��Z���`B!�:���c����������P?o0��{x��u�����'X�o���Zk��~-x��u����1��k�!XK���:C�4Z*�^���{x�c$�	��C��>k�u��i�T������:�^�H����������?/�f>~�x��������}��i9b���cc�s������-�G��}[>��Z��E�����R��:�^���{#yO����`�2�qy�����o*���������Y�`�3�O���u������:F��`
�t�_���b^���/_�\(]�������KQ1a�u�=~k�j1�����	��Za�a[\K�=�o�kxA#�H���Zk���;������������c�~���c�����l?�9=?��������b��_)�Q�g����?�z{_��_C�����z�h��?�����y_|n:�z�zz���u
�������x���=������'X�-XaQ�����0���-�L�����-�@-(&�/�x�|���0.�i����=�ul[|};�`
F�`-}�"X���A�Ss5����������m����g�Tx�������[B�h���yS��2����y_�K�����}?�O�����e�����c}�^y|:�Q|n}��n��������W�2�����
��1��k��X���--��l���jA1�Xr�����pL�y���j�m�5Ma{|
�X��!XK���:C�������-��6��;����k����|_y�W�9�a���{o.�j����������s�x�)��W^�j��^����%#>�6�Q�w�]yx/d$�	��k�=��q���6y�ZPL>V�l[��c�����v�5���Y�`�3�������k���y�p�kg�����ZN>FL�o	I�9��(��w�R����~������Tc
m��z��������j�W���Y:S=�'��^�H���
���(��s,����=x�ZP��e�^ xq�e���5�����?��k� ���l>k0*k���Zg��>���������L���Mf�f�
p*�B�6Fi��=
��*�<������f�����������1�y��Ju��������UC�-�W�2�{|���{#yO��~�Z�VZ��Y�j��xa�^���������e���5l8�B�����?�cM�p
�o�������Zk���{������y�9�_G��;����8/)&��/� [5j��o
nvy_7�����U?�[O�}�zq[
��~��/g�u��=������'X���#^���`��`-}�"X���a���'bz[&�ki�}�{�X��������\v���������Qh&.c�kz}�*�7�3�����x{
j�j?K9�~���=\�{#yO�0k��Z��E����������Xc�b�0/Oi��9��gs�s�kj�1����\�:y�Q��{���\C������^����������Lvnu�f�y_�"&�����d?Ka.+������{��{#yO�����Y�`�3��[3�zI�c	.���
��}��?y�P���L6�k��������s)�P���}A��s�+s�\��|���kv���U/b�^�����0v����{X��:F��`
�?k���Zg��FK���{x��u��=�@��g-���P?��
����:�^���{�5�����Zk��~-x��u����1��k�!XK���:C�4Z*�^���{x�c$�	��C��>k=��W#���B!��������Q�k�����W��:�^���{#yO���XK���:C�4Z*�^���{x�c$�	��C��>k�u��i�T������:�^�H����`-}�"X���h��{x��u�����'X��Z��E��������������||^�N���+������:�^���{#yO�����Y�`�3�����xz��������{x��u��=�@��g-������`m'6���E��gg�/O���x����������L�T���t���f�y�Z���9/���6����>�]�����:��N:��S������>���3��o�����U_��[Y�jMw�=������'X��Z��E������������Tj�&w�_���O/�qV�������s��
�����j�>;A�����\������|������-�i
��N�]��-��^�c�P"����\����M���P�w����tG�
�:���yM�TX�k����{k��o�����[������u��tO��
��1���`�B����Y�`�3���0��,��q������9il���r�z��)�?�c��'�=�S
E�������s���������U���s�
I�S;�:^�|]�����K��-�B��uk�"&��5����z�Y{�}��[F���}�S��s�^�T�ws]"����{�{!x��u������:F�����	�:�]����oH�F3>�y�c���
M��!7�y��3���b�W�X]W/��������o\�J|�������7/����A����c�{���V_�v����Q��3�5H��������o�����ln������{�z�'d��l�om]����{�{!x��u������:F�����	�:�]��T��a�,N�nhL'y���5�n���;5��AK��jh���r/�?7���������u�
�:`����Y%&_��lY�:�}��A~����;s��W�WY�������s�:������Z{�}�z��{k��������v�%a=�������:�^���{x�ct��?��`�3����c��Z�V��fyi��N5=��3����+�����s�Cp�{�6���a��_Y�9�1��5W�u����{���V_q��o���2���fk���~�n�N<�Y����O����t�}_��zo��e�X�����a�u������:�^���{�O�'X��v�s�7�Ic�5������&xn4�M����?o����/��k�/���O�M����������/���B0�xX��
	&�@���c�����w�R������.��}����!Qm��v�_p�����{k��^���w�}_�����u�n������p�u������:�^���{�O�'X��v��i���Q���Ys �����oiA�m���i��)u��0��xr����F�{���	ys?{�����\|���|v���������)
,��!�����{s)�W�S�q�o�|�>_��^�D�u�������=k?������������U����d�gc&�[]��5�������:�^���{x�ct��?��`�3���Md���T���/M{�F�J��QQ�f0L!�������Z�{!?/	�P!�V<�~�o��T�Q�w�Y�����3q0m��:�����L7����Y}�{�LRc�U|	���+�[\�ZM��=��{x��u������:F�����	�:�]����������_	���{x��u������:�^���{�O�'X��v�k�����:�^���{x��u����1�����O����~%x��u������:�^���{x�ct��?��`�3�?v�J�^���{x��u������:�^���SZ��m@}52�!�B!�BGo�u����_	���{x��u������:�^���{�O�'X���]�����:�^���{x��u����1�����O����~%x��u������:�^���{x�ct��?��`�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i�k���;������������c�~���c�����l�yy��?�'�	�����c-ce�/��8/����)�xo�k^�s����n����Y������?�O��s2+��������}`x�U���w������qz~�n��xG�C
���{x��u����1�����O�������F<j���!�y9==>�[��)���	;���|\����;7���/S�[\���Ds���rPq9����=i���������������6�j�OT�w��C��i���;����yu7��z�9X�'��{X��:�^���{x��u��=����u����������X����Zi{�8D������C�������z��R�u����1
��
l�q�\&�c����}P�/'>����5[�{���9w�=������:�^���{�{O�i�k���;��X3�5���8|����R0�X8�o�M��������po�8�_������_���KX��^���(��:������d�}6���w!;����5s�;���r�88�g���������:�^���{�O�'X���G�sm�-�4�S����;����R0�X��C����0'�?o��C��yL�F����RGm�QZ��m�������:X�������}�����5�}��[kV��12�����������:�^���{x�ct��?��`�3����C�k?�cJ�D-�X�l����G����������{��)���"�1�}i��������WZ�������1�}z���������7�Y�Mc�X�{�>������:�^���{x�ct��?��`�3��������
k�����fL���K�:�����R�f��������s|Lm_Nt�M�Wq����$7�q�������b}�(����{��-�<F�j��;��������:�^���{�{O�i�k���{�.
�5�QZ2�9��n����>o����{�<�B'|[�i#,b�{_���m�7�$�}�u�z�{_?&�{f�c^����~�/�V������y;���\��y�/O����c�����:�^���{x��u��=����u������io������_�{���
KS=o��??0��`u�@��)�H�������g2�[����R����6��|}�$�E�W���n���f��6����[k6�v���8T�'�a���{x��u����1�����O����~%x��u������:�^���{x�ct��?��`�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i��!�B!�B!t�xc�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i�k�����W��:�^���{x��u������:F�����	�:C�c���u������:�^���{x��u��=����u����_	���{x��u������:�^���{�O�'X���]�����:�^���{x��u����1�����O����~%x��u������:�^���{x�ct��?��`�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i�k�����W��:�^���{x��u������:F�����	�:C�c���u������:�^���{x��u��=����u����_	���{x��u������:�^���{�O�'X���]�����:�^���{x��u����1�����O����~%x��u������:�^���{x�ct��?��`�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i��!�B!�B!t�xc�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i�k�����W��:�^���{x��u������:F�����	�:C�c���u������:�^���{x��u��=����u����_	���{x��u������:�^���{�O�'X���]�����:�^���{x��u����1�����O����~%x��u������:�^���{x�ct��?��`�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i�k�����W��:�^���{x��u������:F�����	�:C�c���u������:�^���{x��u��=����u����_	���{x��u������:�^���{�O�'X���]�����:�^���{x��u����1�����O����~%x��u������:�^���{x�ct��?��`�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i��!�B!�B!t�xc�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i�k�����W��:�^���{x��u������:F�����	�:C�c���u������:�^���{x��u��=����u����_	���{x��u������:�^���{�O�'X���]�����:�^���{x��u����1�����O����~%x��u������:�^���{x�ct��?��`�3�?v�J�^���{x��u������:�^���SZ?�Zg����������:�^���{x��u��������~���P���+�{x��u������:�^���{�{O�i�k�����W��:�^���{x��u������:F�����	�:C�c���u������:�^���{x��u��=����u����_	���{x��u������:�^���{�O�'X���]�����:�^���{x��u����1�����O����~%x��u������:�^���{x�ct��?��t�� �B����IEND�B`�
details.txttext/plain; name=details.txtDownload
#107Masahiko Sawada
sawada.mshk@gmail.com
In reply to: houzj.fnst@fujitsu.com (#105)
Re: logical replication empty transactions

On Tue, Mar 29, 2022 at 6:15 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

On Tuesday, March 29, 2022 5:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Tue, Mar 29, 2022 at 2:05 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

Attach the new version patch which addressed the above comments and
slightly adjusted some code comments.

The patch looks good to me. One minor suggestion is to change the function
name ProcessPendingWritesAndTimeOut() to ProcessPendingWrites().

Thanks for the comment.
Attach the new version patch with this change.

Thank you for updating the patch. Looks good to me.

Regards,

--
Masahiko Sawada
EDB: https://www.enterprisedb.com/

#108Amit Kapila
amit.kapila16@gmail.com
In reply to: Masahiko Sawada (#107)
Re: logical replication empty transactions

On Wed, Mar 30, 2022 at 7:15 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Tue, Mar 29, 2022 at 6:15 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:

Thanks for the comment.
Attach the new version patch with this change.

Thank you for updating the patch. Looks good to me.

Pushed.

--
With Regards,
Amit Kapila.