Proposal: Filter irrelevant change before reassemble transactions during logical decoding
Hi,
During logical decoding, if there is a large write transaction, some
spill files will be written to disk,
depending on the setting of max_changes_in_memory.
This behavior can effectively avoid OOM, but if the transaction
generates a lot of change before commit,
a large number of files may fill the disk. For example, you can update
a TB-level table.
Of course, this is also inevitable.
But I found an inelegant phenomenon. If the updated large table is not
published, its changes will also
be written with a large number of spill files. Look at an example below:
publisher:
```
create table tbl_pub(id int, val1 text, val2 text,val3 text);
create table tbl_t1(id int, val1 text, val2 text,val3 text);
CREATE PUBLICATION mypub FOR TABLE public.tbl_pub;
```
subscriber:
```
create table tbl_pub(id int, val1 text, val2 text,val3 text);
create table tbl_t1(id int, val1 text, val2 text,val3 text);
CREATE SUBSCRIPTION mysub CONNECTION 'host=127.0.0.1 port=5432
user=postgres dbname=postgres' PUBLICATION mypub;
```
publisher:
```
begin;
insert into tbl_t1 select i,repeat('xyzzy', i),repeat('abcba',
i),repeat('dfds', i) from generate_series(0,999999) i;
```
Later you will see a large number of spill files in the
"/$PGDATA/pg_replslot/mysub/" directory.
```
$ll -sh
total 4.5G
4.0K -rw------- 1 postgres postgres 200 Nov 30 09:24 state
17M -rw------- 1 postgres postgres 17M Nov 30 08:22 xid-750-lsn-0-10000000.spill
12M -rw------- 1 postgres postgres 12M Nov 30 08:20 xid-750-lsn-0-1000000.spill
17M -rw------- 1 postgres postgres 17M Nov 30 08:23 xid-750-lsn-0-11000000.spill
......
```
We can see that table tbl_t1 is not published in mypub. It is also not
sent downstream because it is subscribed.
After the transaction is reorganized, the pgoutput decoding plug-in
filters out these change of unpublished relation
when sending logical changes. see function pgoutput_change.
Above all, if after constructing a change and before queuing a change
into a transaction, we filter out unpublished
relation-related changes, will it make logical decoding less laborious
and avoid disk growth as much as possible?
This is just an immature idea. I haven't started to implement it yet.
Maybe it was designed this way because there
are key factors that I didn't consider. So I want to hear everyone's
opinions, especially the designers of logic decoding.
This is just an immature idea. I haven't started to implement it yet.
Maybe it was designed this way because there
are key factors that I didn't consider. So I want to hear everyone's
opinions, especially the designers of logic decoding.
Attached is the patch I used to implement this optimization.
The main designs are as follows:
1. Added a callback LogicalDecodeFilterByRelCB for the output plugin.
2. Added this callback function pgoutput_table_filter for the pgoutput plugin.
Its main implementation is based on the table filter in the
pgoutput_change function.
3. After constructing a change and before Queue a change into a transaction,
use RelidByRelfilenumber to obtain the relation associated with the change,
just like obtaining the relation in the ReorderBufferProcessTXN function.
4. Relation may be a toast, and there is no good way to get its real
table relation
based on toast relation. Here, I get the real table oid through
toast relname, and
then get the real table relation.
5. This filtering takes into account INSERT/UPDATE/INSERT. Other
changes have not
been considered yet and can be expanded in the future.
6. The table filter in pgoutput_change and the get relation in
ReorderBufferProcessTXN
can be deleted. This has not been done yet. This is the next step.
Sincerely look forward to your feedback.
Regards, lijie
Attachments:
v1-Filter-irrelevant-change-before-reassemble-transacti.patchapplication/octet-stream; name=v1-Filter-irrelevant-change-before-reassemble-transacti.patchDownload
From b4ac04d5fd7e87ee199f996dfda96305a6dac967 Mon Sep 17 00:00:00 2001
From: "adger.lj" <adger.lj@alibaba-inc.com>
Date: Fri, 1 Dec 2023 13:58:33 +0800
Subject: [PATCH] Filter irrelevant change before reassemble transactions
during logical decoding
In order to reduce unnecessary logical decoding, irrelevant relationship
changes can be filtered out before reorganizing transaction fragments.
This can effectively reduce useless changes and prevent disk space from
being filled up with irrelevant data.
By design, Added a callback LogicalDecodeFilterByRelCB for the output plugin.
We implemented a function pgoutput_table_filter for pgoutput. And RelationSyncCache
is reused to determine whether a relationship-related change should be filtered out.
Referring to the implementation of the function pgoutput_change, currently only
insert/update/delete is can filtered, and other types of changes are not considered.
Perhaps more detailed analysis can be done and more filters can be filtered.
The filtering in pgoutput_change and ReorderBufferProcessTXN is no longer needed,
but is retained for now. Here's what to do next.
---
src/backend/replication/logical/decode.c | 156 +++++++++++++++++++-
src/backend/replication/logical/logical.c | 31 ++++
src/backend/replication/pgoutput/pgoutput.c | 55 +++++++
src/include/replication/logical.h | 2 +
src/include/replication/output_plugin.h | 8 +
src/test/subscription/t/034_table_filter.pl | 91 ++++++++++++
6 files changed, 339 insertions(+), 4 deletions(-)
create mode 100644 src/test/subscription/t/034_table_filter.pl
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 1237118e84..ae28d2145b 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -35,6 +35,7 @@
#include "access/xlogrecord.h"
#include "access/xlogutils.h"
#include "catalog/pg_control.h"
+#include "common/string.h"
#include "replication/decode.h"
#include "replication/logical.h"
#include "replication/message.h"
@@ -42,6 +43,7 @@
#include "replication/reorderbuffer.h"
#include "replication/snapbuild.h"
#include "storage/standby.h"
+#include "utils/relfilenumbermap.h"
/* individual record(group)'s handlers */
static void DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
@@ -155,7 +157,7 @@ xlog_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_PARAMETER_CHANGE:
{
xl_parameter_change *xlrec =
- (xl_parameter_change *) XLogRecGetData(buf->record);
+ (xl_parameter_change *) XLogRecGetData(buf->record);
/*
* If wal_level on the primary is reduced to less than
@@ -581,6 +583,137 @@ FilterByOrigin(LogicalDecodingContext *ctx, RepOriginId origin_id)
return filter_by_origin_cb_wrapper(ctx, origin_id);
}
+static bool
+FilterByTable(LogicalDecodingContext *ctx, ReorderBufferChange *change)
+{
+ ReorderBuffer *rb = ctx->reorder;
+ Relation relation = NULL;
+ Oid reloid;
+ bool result = false;
+ bool using_subtxn;
+
+ if (ctx->callbacks.filter_by_origin_cb == NULL)
+ return false;
+
+ switch (change->action)
+ {
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ break;
+ default:
+ return false;
+ }
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("filter change by table");
+ else
+ StartTransactionCommand();
+
+ reloid = RelidByRelfilenumber(change->data.tp.rlocator.spcOid,
+ change->data.tp.rlocator.relNumber);
+ if (reloid == InvalidOid)
+ goto filter_done;
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (!RelationIsValid(relation))
+ elog(ERROR, "could not open relation with OID %u (for filenumber \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.rlocator,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ {
+ result = true;
+ goto filter_done;
+ }
+
+ /*
+ * Ignore temporary heaps created during DDL unless the plugin has asked
+ * for them.
+ */
+ if (relation->rd_rel->relrewrite && !rb->output_rewrites)
+ {
+ result = true;
+ goto filter_done;
+ }
+
+ /*
+ * For now ignore sequence changes entirely. Most of the time they don't
+ * log changes using records we understand, so it doesn't make sense to
+ * handle the few cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ {
+ result = true;
+ goto filter_done;
+ }
+
+ if (IsToastRelation(relation))
+ {
+ Oid real_reloid = InvalidOid;
+
+ /* pg_toast_ len is 9 */
+ char *toast_name = RelationGetRelationName(relation);
+ char *start_ch = &toast_name[9];
+
+ real_reloid = strtoint(start_ch, NULL, 10);
+
+ if (real_reloid == InvalidOid)
+ goto filter_done;
+
+ RelationClose(relation);
+
+ relation = RelationIdGetRelation(real_reloid);
+
+ if (!RelationIsValid(relation))
+ elog(ERROR, "could not open relation with OID %u (for filenumber \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.rlocator,
+ MAIN_FORKNUM));
+ }
+
+ result = filter_by_table_cb_wrapper(ctx, relation, change);
+
+filter_done:
+
+ if (result && RelationIsValid(relation))
+ elog(LOG, "logical filter change by table %s", RelationGetRelationName(relation));
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ if (RelationIsValid(relation))
+ {
+ RelationClose(relation);
+ }
+
+ AbortCurrentTransaction();
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ if (result)
+ ReorderBufferReturnChange(rb, change, false);
+
+ return result;
+}
+
/*
* Handle rmgr LOGICALMSG_ID records for LogicalDecodingProcessRecord().
*/
@@ -940,6 +1073,9 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
+ if (FilterByTable(ctx, change))
+ return;
+
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
change,
xlrec->flags & XLH_INSERT_ON_TOAST_RELATION);
@@ -1009,6 +1145,9 @@ DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
+ if (FilterByTable(ctx, change))
+ return;
+
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
change, false);
}
@@ -1065,6 +1204,9 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
+ if (FilterByTable(ctx, change))
+ return;
+
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
change, false);
}
@@ -1201,11 +1343,14 @@ DecodeMultiInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
else
change->data.tp.clear_toast_afterwards = false;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change, false);
-
/* move to the next xl_multi_insert_tuple entry */
data += datalen;
+
+ if (FilterByTable(ctx, change))
+ continue;;
+
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, change, false);
}
Assert(data == tupledata + tuplelen);
}
@@ -1240,6 +1385,9 @@ DecodeSpecConfirm(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
+ if (FilterByTable(ctx, change))
+ return;
+
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
change, false);
}
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8288da5277..f5da5199dc 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1225,6 +1225,37 @@ filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, RepOriginId origin_id)
return ret;
}
+bool
+filter_by_table_cb_wrapper(LogicalDecodingContext *ctx, Relation relation, ReorderBufferChange *change)
+{
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+ bool ret;
+
+ Assert(!ctx->fast_forward);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "filter_by_table";
+ state.report_location = InvalidXLogRecPtr;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = false;
+ ctx->end_xact = false;
+
+ /* do the actual work: call callback */
+ ret = ctx->callbacks.filter_by_table_cb(ctx, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
+ return ret;
+}
+
static void
message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index f9ed1083df..54a04b54e7 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -57,6 +57,9 @@ static void pgoutput_message(LogicalDecodingContext *ctx,
Size sz, const char *message);
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static bool pgoutput_table_filter(struct LogicalDecodingContext *ctx,
+ Relation relation,
+ ReorderBufferChange *change);
static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn);
static void pgoutput_prepare_txn(LogicalDecodingContext *ctx,
@@ -259,6 +262,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_prepared_cb = pgoutput_commit_prepared_txn;
cb->rollback_prepared_cb = pgoutput_rollback_prepared_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
+ cb->filter_by_table_cb = pgoutput_table_filter;
cb->shutdown_cb = pgoutput_shutdown;
/* transaction streaming */
@@ -1679,6 +1683,57 @@ pgoutput_origin_filter(LogicalDecodingContext *ctx,
return false;
}
+/*
+ * Return true if the relation has not been published, false otherwise.
+ */
+static bool
+pgoutput_table_filter(struct LogicalDecodingContext *ctx,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ RelationSyncEntry *relentry;
+ ReorderBufferChangeType action = change->action;
+
+ if (!is_publishable_relation(relation))
+ return true;
+
+ relentry = get_rel_sync_entry(data, relation);
+
+ /* First check the table filter */
+ switch (action)
+ {
+ case REORDER_BUFFER_CHANGE_INSERT:
+ if (!relentry->pubactions.pubinsert)
+ return true;
+ break;
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ if (!relentry->pubactions.pubupdate)
+ return true;
+ break;
+ case REORDER_BUFFER_CHANGE_DELETE:
+ if (!relentry->pubactions.pubdelete)
+ return true;
+
+ /*
+ * This is only possible if deletes are allowed even when replica
+ * identity is not defined for a table. Since the DELETE action
+ * can't be published, we simply return.
+ */
+ if (!change->data.tp.oldtuple)
+ {
+ elog(DEBUG1, "didn't send DELETE change because of missing oldtuple");
+ return true;
+ }
+ break;
+ default:
+ Assert(false);
+ }
+
+ return false;
+}
+
+
/*
* Shutdown the output plugin.
*
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index dffc0d1564..452425ba5c 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -145,6 +145,8 @@ extern void LogicalConfirmReceivedLocation(XLogRecPtr lsn);
extern bool filter_prepare_cb_wrapper(LogicalDecodingContext *ctx,
TransactionId xid, const char *gid);
extern bool filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, RepOriginId origin_id);
+extern bool filter_by_table_cb_wrapper(LogicalDecodingContext *ctx, Relation relation,
+ ReorderBufferChange *change);
extern void ResetLogicalStreamingState(void);
extern void UpdateDecodingStats(LogicalDecodingContext *ctx);
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 2ffcf17505..4c885639d7 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -96,6 +96,13 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ctx,
RepOriginId origin_id);
+/*
+ * Filter changes by table.
+ */
+typedef bool (*LogicalDecodeFilterByRelCB) (struct LogicalDecodingContext *ctx,
+ Relation relation,
+ ReorderBufferChange *change);
+
/*
* Called to shutdown an output plugin.
*/
@@ -222,6 +229,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeCommitCB commit_cb;
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
+ LogicalDecodeFilterByRelCB filter_by_table_cb;
LogicalDecodeShutdownCB shutdown_cb;
/* streaming of changes at prepare time */
diff --git a/src/test/subscription/t/034_table_filter.pl b/src/test/subscription/t/034_table_filter.pl
new file mode 100644
index 0000000000..8d4f962adc
--- /dev/null
+++ b/src/test/subscription/t/034_table_filter.pl
@@ -0,0 +1,91 @@
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+# Basic logical replication test
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "create table tbl_pub(id int, val1 text, val2 text,size int);");
+$node_publisher->safe_psql('postgres',
+ "create table tbl_t1(id int, val1 text, val2 text,size int);");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION mypub FOR TABLE public.tbl_pub;");
+$node_publisher->safe_psql('postgres',
+qq(
+CREATE OR REPLACE FUNCTION check_replication_status() RETURNS VOID AS \$\$
+DECLARE
+ replication_record pg_stat_replication;
+BEGIN
+ LOOP
+ SELECT *
+ INTO replication_record
+ FROM pg_stat_replication
+ WHERE application_name = 'mysub';
+
+ IF replication_record.replay_lsn = replication_record.write_lsn THEN
+ EXIT;
+ END IF;
+
+ PERFORM pg_sleep(1);
+ END LOOP;
+END;
+\$\$ LANGUAGE plpgsql;));
+
+# Create some preexisting content on subscriber
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_subscriber->safe_psql('postgres',
+ "create table tbl_pub(id int, val1 text, val2 text,size text);");
+$node_subscriber->safe_psql('postgres',
+ "create table tbl_t1(id int, val1 text, val2 text,size text);");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub"
+);
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, 'mysub');
+
+# test filter
+$node_publisher->safe_psql('postgres',
+qq(BEGIN;
+insert into tbl_t1 select 1, 'xyzzy', 'abcba', sum(size) from pg_ls_replslotdir('mysub');
+insert into tbl_t1 select i,repeat('xyzzy', i),repeat('abcba',i),(select sum(size) from pg_ls_replslotdir('mysub')) from generate_series(2,9999) i;
+update tbl_t1 set val2 = repeat('xyzzy',id) where id > 1 and id < 10001;
+select check_replication_status();
+insert into tbl_t1 select 10001, 'xyzzy', 'abcba', sum(size) from pg_ls_replslotdir('mysub');
+COMMIT;)
+);
+
+my $minsize = $node_publisher->safe_psql('postgres',
+ "select size from tbl_t1 order by size asc limit 1;");
+my $maxsize = $node_publisher->safe_psql('postgres',
+ "select size from tbl_t1 order by size desc limit 1;");
+is($minsize, $maxsize, 'check decode filter table between maxsize and minsize');
+
+
+my $fristrow = $node_publisher->safe_psql('postgres',
+ "select size from tbl_t1 where id = 1;");
+my $lastrow = $node_publisher->safe_psql('postgres',
+ "select size from tbl_t1 where id = 10001;");
+is($minsize, $maxsize, 'check decode filter table between fristrow and lastrow');
+
+is($minsize, $lastrow, 'check decode filter table between minsize and lastrow');
+
+print "minsize: " . $minsize . "maxsize: " . $maxsize ."fristrow: " . $fristrow ."lastrow: " . $lastrow . "\n";
+
+done_testing();
\ No newline at end of file
--
2.39.3
On Fri, Dec 1, 2023 at 1:55 PM li jie <ggysxcq@gmail.com> wrote:
This is just an immature idea. I haven't started to implement it yet.
Maybe it was designed this way because there
are key factors that I didn't consider. So I want to hear everyone's
opinions, especially the designers of logic decoding.Attached is the patch I used to implement this optimization.
The main designs are as follows:
1. Added a callback LogicalDecodeFilterByRelCB for the output plugin.2. Added this callback function pgoutput_table_filter for the pgoutput plugin.
Its main implementation is based on the table filter in the
pgoutput_change function.3. After constructing a change and before Queue a change into a transaction,
use RelidByRelfilenumber to obtain the relation associated with the change,
just like obtaining the relation in the ReorderBufferProcessTXN function.4. Relation may be a toast, and there is no good way to get its real
table relation
based on toast relation. Here, I get the real table oid through
toast relname, and
then get the real table relation.
This may be helpful for the case you have mentioned but how about
cases where there is nothing to filter by relation? It will add
overhead related to the transaction start/end and others for each
change. Currently, we do that just once for all the changes that need
to be processed. I wonder why the spilling can't be avoided with GUC
logical_decoding_work_mem?
--
With Regards,
Amit Kapila.
This may be helpful for the case you have mentioned but how about
cases where there is nothing to filter by relation?
You can see the complete antecedent in the email [1]. Relation that has
not been published will also generate changes and put them into the entire
transaction group, which will increase invalid memory or disk space.
It will add
overhead related to the transaction start/end and others for each
change. Currently, we do that just once for all the changes that need
to be processed.
Yes, it will only be processed once at present. It is done when applying
each change when the transaction is committed. This patch hopes to
advance it to the time when constructing the change, and determines the
change queue into a based on whether the relation is published.
I wonder why the spilling can't be avoided with GUC
logical_decoding_work_mem?
Of course you can, but this will only convert disk space into memory space.
For details, please see the case in Email [1].
Regards, lijie
This may be helpful for the case you have mentioned but how about
cases where there is nothing to filter by relation?
You can see the complete antecedent in the email [1]/messages/by-id/CAGfChW51P944nM5h0HTV9HistvVfwBxNaMt_s-OZ9t=uXz+Zbg@mail.gmail.com. Relation that has
not been published will also generate changes and put them into the entire
transaction group, which will increase invalid memory or disk space.
It will add
overhead related to the transaction start/end and others for each
change. Currently, we do that just once for all the changes that need
to be processed.
Yes, it will only be processed once at present. It is done when applying
each change when the transaction is committed. This patch hopes to
advance it to the time when constructing the change, and determines the
change queue into a based on whether the relation is published.
I wonder why the spilling can't be avoided with GUC
logical_decoding_work_mem?
Of course you can, but this will only convert disk space into memory space.
For details, please see the case in Email [1]/messages/by-id/CAGfChW51P944nM5h0HTV9HistvVfwBxNaMt_s-OZ9t=uXz+Zbg@mail.gmail.com.
[1]: /messages/by-id/CAGfChW51P944nM5h0HTV9HistvVfwBxNaMt_s-OZ9t=uXz+Zbg@mail.gmail.com
Regards, lijie
Amit Kapila <amit.kapila16@gmail.com> 于2023年12月2日周六 12:11写道:
Show quoted text
On Fri, Dec 1, 2023 at 1:55 PM li jie <ggysxcq@gmail.com> wrote:
This is just an immature idea. I haven't started to implement it yet.
Maybe it was designed this way because there
are key factors that I didn't consider. So I want to hear everyone's
opinions, especially the designers of logic decoding.Attached is the patch I used to implement this optimization.
The main designs are as follows:
1. Added a callback LogicalDecodeFilterByRelCB for the output plugin.2. Added this callback function pgoutput_table_filter for the pgoutput plugin.
Its main implementation is based on the table filter in the
pgoutput_change function.3. After constructing a change and before Queue a change into a transaction,
use RelidByRelfilenumber to obtain the relation associated with the change,
just like obtaining the relation in the ReorderBufferProcessTXN function.4. Relation may be a toast, and there is no good way to get its real
table relation
based on toast relation. Here, I get the real table oid through
toast relname, and
then get the real table relation.This may be helpful for the case you have mentioned but how about
cases where there is nothing to filter by relation? It will add
overhead related to the transaction start/end and others for each
change. Currently, we do that just once for all the changes that need
to be processed. I wonder why the spilling can't be avoided with GUC
logical_decoding_work_mem?--
With Regards,
Amit Kapila.
Of course you can, but this will only convert disk space into memory space.
For details, please see the case in Email [1].[1]
/messages/by-id/CAGfChW51P944nM5h0HTV9HistvVfwBxNaMt_s-OZ9t=uXz+Zbg@mail.gmail.comRegards, lijie
Hi lijie,
Overall, I think the patch is a good improvement. Some comments from first
run through of patch:
1. The patch no longer applies cleanly, please rebase.
2. While testing the patch, I saw something strange. If I try to truncate a
table that is published. I still see the message:
2024-03-18 22:25:51.243 EDT [29385] LOG: logical filter change by table
pg_class
This gives the impression that the truncate operation on the published
table has been filtered but it hasn't. Also the log message needs to be
reworded. Maybe, "Logical filtering change by non-published table
<relation_name>"
3. Below code:
@@ -1201,11 +1343,14 @@ DecodeMultiInsert(LogicalDecodingContext *ctx,
XLogRecordBuffer *buf)
+
+ if (FilterByTable(ctx, change))
+ continue;;
extra semi-colon after continue.
4. I am not sure if this is possible, but is there a way to avoid the
overhead in the patch if the publication publishes "ALL TABLES"?
5. In function: pgoutput_table_filter() - this code appears to be filtering
out not just unpublished tables but also applying row based filters on
published tables as well. Is this really within the scope of the feature?
regards,
Ajin Cherian
Fujitsu Australia
Dear Li,
Thanks for proposing and sharing the PoC. Here are my high-level comments.
1.
What if ALTER PUBLICATION ... ADD is executed in parallel?
Should we publish added tables if the altering is done before the transaction is
committed? The current patch seems unable to do so because changes for added
tables have not been queued at COMMIT.
If we should not publish such tables, why?
2.
This patch could not apply as-is. Please rebase.
3. FilterByTable()
```
+ if (ctx->callbacks.filter_by_origin_cb == NULL)
+ return false;
```
filter_by_table_cb should be checked instead of filter_by_origin_cb.
Current patch crashes if the filter_by_table_cb() is not implemented.
4. DecodeSpecConfirm()
```
+ if (FilterByTable(ctx, change))
+ return;
+
```
I'm not sure it is needed. Can you explain the reason why you added?
5. FilterByTable
```
+ switch (change->action)
+ {
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ break;
+ default:
+ return false;
+ }
```
IIUC, REORDER_BUFFER_CHANGE_TRUNCATE also targes the user table, so I think
it should be accepted. Thought?
6.
I got strange errors when I tested the feature. I thought this implied there were
bugs in your patch.
1. implemented no-op filter atop test_decoding like attached
2. ran `make check` for test_decoding modle
3. some tests failed. Note that "filter" was a test added by me.
regression.diffs was also attached.
```
not ok 1 - ddl 970 ms
ok 2 - xact 36 ms
not ok 3 - rewrite 525 ms
not ok 4 - toast 736 ms
ok 5 - permissions 50 ms
ok 6 - decoding_in_xact 39 ms
not ok 7 - decoding_into_rel 57 ms
ok 8 - binary 21 ms
not ok 9 - prepared 33 ms
ok 10 - replorigin 93 ms
ok 11 - time 25 ms
ok 12 - messages 47 ms
ok 13 - spill 8063 ms
ok 14 - slot 124 ms
ok 15 - truncate 37 ms
not ok 16 - stream 60 ms
ok 17 - stats 157 ms
ok 18 - twophase 122 ms
not ok 19 - twophase_stream 57 ms
not ok 20 - filter 20 ms
```
Currently I'm not 100% sure the reason, but I think it may read the latest system
catalog even if ALTER SUBSCRIPTION is executed after changes.
In below example, an attribute is altered text->somenum, after inserting data.
But get_changes() outputs as somenum.
```
BEGIN
- table public.replication_example: INSERT: id[integer]:1 somedata[integer]:1 text[character varying]:'1'
- table public.replication_example: INSERT: id[integer]:2 somedata[integer]:1 text[character varying]:'2'
+ table public.replication_example: INSERT: id[integer]:1 somedata[integer]:1 somenum[character varying]:'1'
+ table public.replication_example: INSERT: id[integer]:2 somedata[integer]:1 somenum[character varying]:'2'
COMMIT
```
Also, if the relfilenuber of the relation is changed, an ERROR is raised.
```
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-----------------------------------------------------------------------------
- BEGIN
- table public.tr_pkey: INSERT: id2[integer]:2 data[integer]:1 id[integer]:2
- COMMIT
- BEGIN
- table public.tr_pkey: DELETE: id[integer]:1
- table public.tr_pkey: DELETE: id[integer]:2
- COMMIT
-(7 rows)
-
+ERROR: could not map filenumber "base/16384/16397" to relation OID
```
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/
Attachments:
0001-WIP-implement-no-op-filter.txttext/plain; name=0001-WIP-implement-no-op-filter.txtDownload
From 9bfb34dad67745f1155238a4b11bbe1d28284b50 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Mon, 20 May 2024 03:19:56 +0000
Subject: [PATCH] WIP: implement no-op filter
---
contrib/test_decoding/Makefile | 2 +-
contrib/test_decoding/expected/filter.out | 0
contrib/test_decoding/meson.build | 1 +
contrib/test_decoding/sql/filter.sql | 13 +++++++++++++
contrib/test_decoding/test_decoding.c | 18 ++++++++++++++++++
src/backend/replication/logical/decode.c | 2 +-
6 files changed, 34 insertions(+), 2 deletions(-)
create mode 100644 contrib/test_decoding/expected/filter.out
create mode 100644 contrib/test_decoding/sql/filter.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..b9b6034b9f 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,7 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream filter
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot
diff --git a/contrib/test_decoding/expected/filter.out b/contrib/test_decoding/expected/filter.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..539a148708 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'filter',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/filter.sql b/contrib/test_decoding/sql/filter.sql
new file mode 100644
index 0000000000..4cd4f1d824
--- /dev/null
+++ b/contrib/test_decoding/sql/filter.sql
@@ -0,0 +1,13 @@
+-- predictability
+SET synchronous_commit = on;
+
+-- define table which will be filtered
+CREATE TABLE test_tobeskipped (a int);
+
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+INSERT INTO test_tobeskipped VALUES (generate_series(1, 10));
+
+-- check that changes to test_tobeskipped are not output
+
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 7c50d13969..6f0d076305 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -68,6 +68,9 @@ static void pg_decode_truncate(LogicalDecodingContext *ctx,
ReorderBufferChange *change);
static bool pg_decode_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static bool pg_decode_filter_table(struct LogicalDecodingContext *ctx,
+ Relation relation,
+ ReorderBufferChange *change);
static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr lsn,
bool transactional, const char *prefix,
@@ -133,6 +136,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->truncate_cb = pg_decode_truncate;
cb->commit_cb = pg_decode_commit_txn;
cb->filter_by_origin_cb = pg_decode_filter;
+ cb->filter_by_table_cb = pg_decode_filter_table;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
cb->filter_prepare_cb = pg_decode_filter_prepare;
@@ -467,6 +471,20 @@ pg_decode_filter(LogicalDecodingContext *ctx,
return false;
}
+/*
+ * Filter out by the table.
+ *
+ * It is implemented for demonstating and testing purpose. If the table
+ * contains the "_tobeskipped" subsctring, then we filter out.
+ */
+static bool
+pg_decode_filter_table(struct LogicalDecodingContext *ctx,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ return false;
+}
+
/*
* Print literal `outputstr' already represented as string of type `typid'
* into stringbuf `s'.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index f2b2e33521..4266a9b353 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -589,7 +589,7 @@ FilterByTable(LogicalDecodingContext *ctx, ReorderBufferChange *change)
bool result = false;
bool using_subtxn;
- if (ctx->callbacks.filter_by_origin_cb == NULL)
+ if (ctx->callbacks.filter_by_table_cb == NULL)
return false;
switch (change->action)
--
2.43.0
regression.diffsapplication/octet-stream; name=regression.diffsDownload
diff -U3 /home/hayato/patchTest/postgres/contrib/test_decoding/expected/ddl.out /home/hayato/patchTest/postgres/contrib/test_decoding/results/ddl.out
--- /home/hayato/patchTest/postgres/contrib/test_decoding/expected/ddl.out 2024-05-14 00:12:11.528895957 +0000
+++ /home/hayato/patchTest/postgres/contrib/test_decoding/results/ddl.out 2024-05-20 04:49:00.215181926 +0000
@@ -90,26 +90,26 @@
INSERT INTO replication_example(somedata, somenum) VALUES (4, 1);
-- collect all changes
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
----------------------------------------------------------------------------------------------------------------------------
+ data
+-------------------------------------------------------------------------------------------------------------
BEGIN
- table public.replication_example: INSERT: id[integer]:1 somedata[integer]:1 text[character varying]:'1'
- table public.replication_example: INSERT: id[integer]:2 somedata[integer]:1 text[character varying]:'2'
+ table public.replication_example: INSERT: id[integer]:1 somedata[integer]:1 somenum[character varying]:'1'
+ table public.replication_example: INSERT: id[integer]:2 somedata[integer]:1 somenum[character varying]:'2'
COMMIT
BEGIN
- table public.replication_example: INSERT: id[integer]:3 somedata[integer]:2 text[character varying]:'1' bar[integer]:4
+ table public.replication_example: INSERT: id[integer]:3 somedata[integer]:2 somenum[character varying]:'1'
COMMIT
BEGIN
- table public.replication_example: INSERT: id[integer]:4 somedata[integer]:2 text[character varying]:'2' bar[integer]:4
- table public.replication_example: INSERT: id[integer]:5 somedata[integer]:2 text[character varying]:'3' bar[integer]:4
- table public.replication_example: INSERT: id[integer]:6 somedata[integer]:2 text[character varying]:'4' bar[integer]:null
+ table public.replication_example: INSERT: id[integer]:4 somedata[integer]:2 somenum[character varying]:'2'
+ table public.replication_example: INSERT: id[integer]:5 somedata[integer]:2 somenum[character varying]:'3'
+ table public.replication_example: INSERT: id[integer]:6 somedata[integer]:2 somenum[character varying]:'4'
COMMIT
BEGIN
- table public.replication_example: INSERT: id[integer]:7 somedata[integer]:3 text[character varying]:'1'
+ table public.replication_example: INSERT: id[integer]:7 somedata[integer]:3 somenum[character varying]:'1'
COMMIT
BEGIN
- table public.replication_example: INSERT: id[integer]:8 somedata[integer]:3 text[character varying]:'2'
- table public.replication_example: INSERT: id[integer]:9 somedata[integer]:3 text[character varying]:'3'
+ table public.replication_example: INSERT: id[integer]:8 somedata[integer]:3 somenum[character varying]:'2'
+ table public.replication_example: INSERT: id[integer]:9 somedata[integer]:3 somenum[character varying]:'3'
COMMIT
BEGIN
table public.replication_example: INSERT: id[integer]:10 somedata[integer]:4 somenum[character varying]:'1'
@@ -135,13 +135,13 @@
COMMIT;
-- show changes
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-------------------------------------------------------------------------------------------------------------------------------------------
+ data
+---------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
- table public.replication_example: INSERT: id[integer]:11 somedata[integer]:5 somenum[integer]:1
+ table public.replication_example: INSERT: id[integer]:11 somedata[integer]:5 somenum[integer]:1 zaphod1[integer]:null zaphod2[integer]:null
COMMIT
BEGIN
- table public.replication_example: INSERT: id[integer]:12 somedata[integer]:6 somenum[integer]:1
+ table public.replication_example: INSERT: id[integer]:12 somedata[integer]:6 somenum[integer]:1 zaphod1[integer]:null zaphod2[integer]:null
table public.replication_example: INSERT: id[integer]:13 somedata[integer]:6 somenum[integer]:2 zaphod1[integer]:1
table public.replication_example: INSERT: id[integer]:14 somedata[integer]:6 somenum[integer]:3 zaphod1[integer]:null zaphod2[integer]:1
table public.replication_example: INSERT: id[integer]:15 somedata[integer]:6 somenum[integer]:4 zaphod1[integer]:2 zaphod2[integer]:null
@@ -243,32 +243,13 @@
ALTER TABLE tr_unique RENAME TO tr_pkey;
ALTER TABLE tr_pkey ADD COLUMN id serial primary key;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-rewrites', '1');
- data
------------------------------------------------------------------------------
- BEGIN
- table public.tr_unique: INSERT: id2[integer]:1 data[integer]:10
- COMMIT
- BEGIN
- table public.tr_pkey: INSERT: id2[integer]:1 data[integer]:10 id[integer]:1
- COMMIT
-(6 rows)
-
+ERROR: could not map filenumber "base/16384/16397" to relation OID
INSERT INTO tr_pkey(data) VALUES(1);
--show deletion with primary key
DELETE FROM tr_pkey;
/* display results */
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-----------------------------------------------------------------------------
- BEGIN
- table public.tr_pkey: INSERT: id2[integer]:2 data[integer]:1 id[integer]:2
- COMMIT
- BEGIN
- table public.tr_pkey: DELETE: id[integer]:1
- table public.tr_pkey: DELETE: id[integer]:2
- COMMIT
-(7 rows)
-
+ERROR: could not map filenumber "base/16384/16397" to relation OID
/*
* check that disk spooling works (also for logical messages)
*/
@@ -291,15 +272,7 @@
FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')
GROUP BY substring(data, 1, 24)
ORDER BY 1,2;
- count | min | max
--------+-----------------------------------------------------------------------+------------------------------------------------------------------------
- 1 | BEGIN | BEGIN
- 1 | COMMIT | COMMIT
- 1 | message: transactional: 1 prefix: test, sz: 14 content:tx logical msg | message: transactional: 1 prefix: test, sz: 14 content:tx logical msg
- 1 | table public.tr_oddlength: INSERT: id[text]:'ab' data[text]:'foo' | table public.tr_oddlength: INSERT: id[text]:'ab' data[text]:'foo'
- 20467 | table public.tr_etoomuch: DELETE: id[integer]:1 | table public.tr_etoomuch: UPDATE: id[integer]:9999 data[integer]:-9999
-(5 rows)
-
+ERROR: could not map filenumber "base/16384/16397" to relation OID
-- check updates of primary keys work correctly
BEGIN;
CREATE TABLE spoolme AS SELECT g.i FROM generate_series(1, 5000) g(i);
@@ -312,13 +285,7 @@
SELECT data
FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')
WHERE data ~ 'UPDATE';
- data
--------------------------------------------------------------------------------------------------------------
- table public.tr_etoomuch: UPDATE: old-key: id[integer]:5000 new-tuple: id[integer]:-5000 data[integer]:5000
- table public.tr_oddlength: UPDATE: old-key: id[text]:'ab' new-tuple: id[text]:'x' data[text]:'quux'
- table public.tr_oddlength: UPDATE: old-key: id[text]:'x' new-tuple: id[text]:'yy' data[text]:'a'
-(3 rows)
-
+ERROR: could not map filenumber "base/16384/16397" to relation OID
-- check that a large, spooled, upsert works
INSERT INTO tr_etoomuch (id, data)
SELECT g.i, -g.i FROM generate_series(8000, 12000) g(i)
@@ -327,14 +294,7 @@
FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1') WITH ORDINALITY
GROUP BY 1
ORDER BY min(ordinality);
- substring | count
--------------------------------+-------
- BEGIN | 1
- table public.tr_etoomuch: UPD | 2235
- table public.tr_etoomuch: INS | 1766
- COMMIT | 1
-(4 rows)
-
+ERROR: could not map filenumber "base/16384/16397" to relation OID
/*
* check whether we decode subtransactions correctly in relation with each
* other
@@ -356,18 +316,7 @@
RELEASE SAVEPOINT b;
COMMIT;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-----------------------------------------------------------------------
- BEGIN
- table public.tr_sub: INSERT: id[integer]:1 path[text]:'1-top-#1'
- table public.tr_sub: INSERT: id[integer]:2 path[text]:'1-top-1-#1'
- table public.tr_sub: INSERT: id[integer]:3 path[text]:'1-top-1-#2'
- table public.tr_sub: INSERT: id[integer]:4 path[text]:'1-top-2-1-#1'
- table public.tr_sub: INSERT: id[integer]:5 path[text]:'1-top-2-1-#2'
- table public.tr_sub: INSERT: id[integer]:6 path[text]:'1-top-2-#1'
- COMMIT
-(8 rows)
-
+ERROR: could not map filenumber "base/16384/16397" to relation OID
-- check that we handle xlog assignments correctly
BEGIN;
-- nest 80 subtxns
@@ -395,16 +344,7 @@
INSERT INTO tr_sub(path) VALUES ('2-top-#1');
COMMIT;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-------------------------------------------------------------------------
- BEGIN
- table public.tr_sub: INSERT: id[integer]:7 path[text]:'2-top-1...--#1'
- table public.tr_sub: INSERT: id[integer]:8 path[text]:'2-top-1...--#2'
- table public.tr_sub: INSERT: id[integer]:9 path[text]:'2-top-1...--#3'
- table public.tr_sub: INSERT: id[integer]:10 path[text]:'2-top-#1'
- COMMIT
-(6 rows)
-
+ERROR: could not map filenumber "base/16384/16397" to relation OID
-- make sure rollbacked subtransactions aren't decoded
BEGIN;
INSERT INTO tr_sub(path) VALUES ('3-top-2-#1');
@@ -416,15 +356,7 @@
INSERT INTO tr_sub(path) VALUES ('3-top-2-#2');
COMMIT;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
------------------------------------------------------------------------
- BEGIN
- table public.tr_sub: INSERT: id[integer]:11 path[text]:'3-top-2-#1'
- table public.tr_sub: INSERT: id[integer]:12 path[text]:'3-top-2-1-#1'
- table public.tr_sub: INSERT: id[integer]:14 path[text]:'3-top-2-#2'
- COMMIT
-(5 rows)
-
+ERROR: could not map filenumber "base/16384/16397" to relation OID
-- test whether a known, but not yet logged toplevel xact, followed by a
-- subxact commit is handled correctly
BEGIN;
@@ -445,16 +377,7 @@
INSERT INTO tr_sub(path) VALUES ('5-top-1-#1');
COMMIT;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
----------------------------------------------------------------------
- BEGIN
- table public.tr_sub: INSERT: id[integer]:15 path[text]:'4-top-1-#1'
- COMMIT
- BEGIN
- table public.tr_sub: INSERT: id[integer]:16 path[text]:'5-top-1-#1'
- COMMIT
-(6 rows)
-
+ERROR: could not map filenumber "base/16384/16397" to relation OID
-- check that DDL in aborted subtransactions handled correctly
CREATE TABLE tr_sub_ddl(data int);
BEGIN;
@@ -466,13 +389,7 @@
INSERT INTO tr_sub_ddl VALUES(43);
COMMIT;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
---------------------------------------------------
- BEGIN
- table public.tr_sub_ddl: INSERT: data[bigint]:43
- COMMIT
-(3 rows)
-
+ERROR: could not map filenumber "base/16384/16397" to relation OID
/*
* Check whether treating a table as a catalog table works somewhat
*/
@@ -543,22 +460,7 @@
INSERT INTO replication_metadata(relation, options)
VALUES ('zaphod', NULL);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-------------------------------------------------------------------------------------------------------------------------------------
- BEGIN
- table public.replication_metadata: INSERT: id[integer]:1 relation[name]:'foo' options[text[]]:'{a,b}'
- COMMIT
- BEGIN
- table public.replication_metadata: INSERT: id[integer]:2 relation[name]:'bar' options[text[]]:'{a,b}'
- COMMIT
- BEGIN
- table public.replication_metadata: INSERT: id[integer]:3 relation[name]:'blub' options[text[]]:null
- COMMIT
- BEGIN
- table public.replication_metadata: INSERT: id[integer]:4 relation[name]:'zaphod' options[text[]]:null rewritemeornot[integer]:null
- COMMIT
-(12 rows)
-
+ERROR: could not map filenumber "base/16384/16397" to relation OID
/*
* check whether we handle updates/deletes correct with & without a pkey
*/
@@ -663,151 +565,7 @@
-- produce 200kB of useless dashes. Turn that off temporarily to avoid it.
\pset format unaligned
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-data
-BEGIN
-table public.table_without_key: INSERT: id[integer]:1 data[integer]:1
-table public.table_without_key: INSERT: id[integer]:2 data[integer]:2
-COMMIT
-BEGIN
-table public.table_without_key: DELETE: (no-tuple-data)
-COMMIT
-BEGIN
-table public.table_without_key: UPDATE: id[integer]:2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_without_key: UPDATE: id[integer]:-2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_without_key: UPDATE: id[integer]:2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_without_key: UPDATE: old-key: id[integer]:2 data[integer]:3 new-tuple: id[integer]:-2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_without_key: UPDATE: old-key: id[integer]:-2 data[integer]:3 new-tuple: id[integer]:2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_without_key: UPDATE: old-key: id[integer]:2 data[integer]:3 new-tuple: id[integer]:-2 data[integer]:3 new_column[text]:null
-COMMIT
-BEGIN
-table public.table_without_key: UPDATE: old-key: id[integer]:-2 data[integer]:3 new-tuple: id[integer]:2 data[integer]:3 new_column[text]:'someval'
-COMMIT
-BEGIN
-table public.table_without_key: DELETE: id[integer]:2 data[integer]:3 new_column[text]:'someval'
-COMMIT
-BEGIN
-table public.table_with_pkey: INSERT: id[integer]:1 data[integer]:1
-table public.table_with_pkey: INSERT: id[integer]:2 data[integer]:2
-COMMIT
-BEGIN
-table public.table_with_pkey: DELETE: id[integer]:1
-COMMIT
-BEGIN
-table public.table_with_pkey: UPDATE: id[integer]:2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_with_pkey: UPDATE: old-key: id[integer]:2 new-tuple: id[integer]:-2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_with_pkey: UPDATE: old-key: id[integer]:-2 new-tuple: id[integer]:2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_with_pkey: UPDATE: old-key: id[integer]:2 new-tuple: id[integer]:-2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_with_pkey: UPDATE: old-key: id[integer]:-2 new-tuple: id[integer]:2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_with_pkey: DELETE: id[integer]:2
-COMMIT
-BEGIN
-table public.table_with_unique_not_null: INSERT: id[integer]:1 data[integer]:1
-table public.table_with_unique_not_null: INSERT: id[integer]:2 data[integer]:2
-COMMIT
-BEGIN
-table public.table_with_unique_not_null: DELETE: (no-tuple-data)
-COMMIT
-BEGIN
-table public.table_with_unique_not_null: UPDATE: id[integer]:2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_with_unique_not_null: UPDATE: id[integer]:-2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_with_unique_not_null: UPDATE: id[integer]:2 data[integer]:3
-COMMIT
-BEGIN
-table public.table_with_unique_not_null: DELETE: (no-tuple-data)
-COMMIT
-BEGIN
-table public.table_with_unique_not_null: INSERT: id[integer]:3 data[integer]:1
-table public.table_with_unique_not_null: INSERT: id[integer]:4 data[integer]:2
-COMMIT
-BEGIN
-table public.table_with_unique_not_null: DELETE: id[integer]:3
-COMMIT
-BEGIN
-table public.table_with_unique_not_null: UPDATE: id[integer]:4 data[integer]:3
-COMMIT
-BEGIN
-table public.table_with_unique_not_null: UPDATE: old-key: id[integer]:4 new-tuple: id[integer]:-4 data[integer]:3
-COMMIT
-BEGIN
-table public.table_with_unique_not_null: UPDATE: old-key: id[integer]:-4 new-tuple: id[integer]:4 data[integer]:3
-COMMIT
-BEGIN
-table public.table_with_unique_not_null: DELETE: id[integer]:4
-COMMIT
-BEGIN
-table public.table_dropped_index_with_pk: INSERT: a[integer]:1 b[integer]:1 c[integer]:1
-table public.table_dropped_index_with_pk: INSERT: a[integer]:2 b[integer]:2 c[integer]:2
-table public.table_dropped_index_with_pk: INSERT: a[integer]:3 b[integer]:3 c[integer]:3
-COMMIT
-BEGIN
-table public.table_dropped_index_with_pk: UPDATE: a[integer]:4 b[integer]:1 c[integer]:1
-COMMIT
-BEGIN
-table public.table_dropped_index_with_pk: UPDATE: a[integer]:2 b[integer]:5 c[integer]:2
-COMMIT
-BEGIN
-table public.table_dropped_index_with_pk: UPDATE: a[integer]:3 b[integer]:6 c[integer]:7
-COMMIT
-BEGIN
-table public.table_dropped_index_with_pk: DELETE: (no-tuple-data)
-COMMIT
-BEGIN
-table public.table_dropped_index_with_pk: DELETE: (no-tuple-data)
-COMMIT
-BEGIN
-table public.table_dropped_index_no_pk: INSERT: a[integer]:1 b[integer]:1 c[integer]:1
-table public.table_dropped_index_no_pk: INSERT: a[integer]:2 b[integer]:2 c[integer]:2
-table public.table_dropped_index_no_pk: INSERT: a[integer]:3 b[integer]:3 c[integer]:3
-COMMIT
-BEGIN
-table public.table_dropped_index_no_pk: UPDATE: a[integer]:4 b[integer]:1 c[integer]:1
-COMMIT
-BEGIN
-table public.table_dropped_index_no_pk: UPDATE: a[integer]:2 b[integer]:5 c[integer]:2
-COMMIT
-BEGIN
-table public.table_dropped_index_no_pk: UPDATE: a[integer]:3 b[integer]:6 c[integer]:7
-COMMIT
-BEGIN
-table public.table_dropped_index_no_pk: DELETE: (no-tuple-data)
-COMMIT
-BEGIN
-table public.table_dropped_index_no_pk: DELETE: (no-tuple-data)
-COMMIT
-BEGIN
-table public.toasttable: INSERT: id[integer]:1 toasted_col1[text]:'12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403140414051406140714081409141014111412141314141415141614171418141914201421142214231424142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532153315341535153615371538153915401541154215431544154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613161416151616161716181619162016211622162316241625162616271628162916301631163216331634163516361637163816391640164116421643164416451646164716481649165016511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709171017111712171317141715171617171718171917201721172217231724172517261727172817291730173117321733173417351736173717381739174017411742174317441745174617471748174917501751175217531754175517561757175817591760176117621763176417651766176717681769177017711772177317741775177617771778177917801781178217831784178517861787178817891790179117921793179417951796179717981799180018011802180318041805180618071808180918101811181218131814181518161817181818191820182118221823182418251826182718281829183018311832183318341835183618371838183918401841184218431844184518461847184818491850185118521853185418551856185718581859186018611862186318641865186618671868186918701871187218731874187518761877187818791880188118821883188418851886188718881889189018911892189318941895189618971898189919001901190219031904190519061907190819091910191119121913191419151916191719181919192019211922192319241925192619271928192919301931193219331934193519361937193819391940194119421943194419451946194719481949195019511952195319541955195619571958195919601961196219631964196519661967196819691970197119721973197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000' rand1[double precision]:79 toasted_col2[text]:null rand2[double precision]:1578
-COMMIT
-BEGIN
-table public.toasttable: INSERT: id[integer]:2 toasted_col1[text]:null rand1[double precision]:3077 toasted_col2[text]:'0001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500' rand2[double precision]:4576
-COMMIT
-BEGIN
-table public.toasttable: UPDATE: id[integer]:1 toasted_col1[text]:'12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403140414051406140714081409141014111412141314141415141614171418141914201421142214231424142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532153315341535153615371538153915401541154215431544154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613161416151616161716181619162016211622162316241625162616271628162916301631163216331634163516361637163816391640164116421643164416451646164716481649165016511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709171017111712171317141715171617171718171917201721172217231724172517261727172817291730173117321733173417351736173717381739174017411742174317441745174617471748174917501751175217531754175517561757175817591760176117621763176417651766176717681769177017711772177317741775177617771778177917801781178217831784178517861787178817891790179117921793179417951796179717981799180018011802180318041805180618071808180918101811181218131814181518161817181818191820182118221823182418251826182718281829183018311832183318341835183618371838183918401841184218431844184518461847184818491850185118521853185418551856185718581859186018611862186318641865186618671868186918701871187218731874187518761877187818791880188118821883188418851886188718881889189018911892189318941895189618971898189919001901190219031904190519061907190819091910191119121913191419151916191719181919192019211922192319241925192619271928192919301931193219331934193519361937193819391940194119421943194419451946194719481949195019511952195319541955195619571958195919601961196219631964196519661967196819691970197119721973197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000' rand1[double precision]:79 toasted_col2[text]:null rand2[double precision]:1578
-COMMIT
-(143 rows)
+ERROR: could not map filenumber "base/16384/16397" to relation OID
\pset format aligned
INSERT INTO toasttable(toasted_col1) SELECT string_agg(g.i::text, '') FROM generate_series(1, 2000) g(i);
-- update of second column, first column unchanged
@@ -818,18 +576,10 @@
DROP TABLE toasttable;
\pset format unaligned
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-data
-BEGIN
-table public.toasttable: INSERT: id[integer]:3 toasted_col1[text]:'12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403140414051406140714081409141014111412141314141415141614171418141914201421142214231424142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532153315341535153615371538153915401541154215431544154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613161416151616161716181619162016211622162316241625162616271628162916301631163216331634163516361637163816391640164116421643164416451646164716481649165016511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709171017111712171317141715171617171718171917201721172217231724172517261727172817291730173117321733173417351736173717381739174017411742174317441745174617471748174917501751175217531754175517561757175817591760176117621763176417651766176717681769177017711772177317741775177617771778177917801781178217831784178517861787178817891790179117921793179417951796179717981799180018011802180318041805180618071808180918101811181218131814181518161817181818191820182118221823182418251826182718281829183018311832183318341835183618371838183918401841184218431844184518461847184818491850185118521853185418551856185718581859186018611862186318641865186618671868186918701871187218731874187518761877187818791880188118821883188418851886188718881889189018911892189318941895189618971898189919001901190219031904190519061907190819091910191119121913191419151916191719181919192019211922192319241925192619271928192919301931193219331934193519361937193819391940194119421943194419451946194719481949195019511952195319541955195619571958195919601961196219631964196519661967196819691970197119721973197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000' rand1[double precision]:6075 toasted_col2[text]:null rand2[double precision]:7574
-COMMIT
-BEGIN
-table public.toasttable: UPDATE: id[integer]:1 toasted_col1[text]:unchanged-toast-datum rand1[double precision]:79 toasted_col2[text]:'12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403140414051406140714081409141014111412141314141415141614171418141914201421142214231424142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532153315341535153615371538153915401541154215431544154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613161416151616161716181619162016211622162316241625162616271628162916301631163216331634163516361637163816391640164116421643164416451646164716481649165016511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709171017111712171317141715171617171718171917201721172217231724172517261727172817291730173117321733173417351736173717381739174017411742174317441745174617471748174917501751175217531754175517561757175817591760176117621763176417651766176717681769177017711772177317741775177617771778177917801781178217831784178517861787178817891790179117921793179417951796179717981799180018011802180318041805180618071808180918101811181218131814181518161817181818191820182118221823182418251826182718281829183018311832183318341835183618371838183918401841184218431844184518461847184818491850185118521853185418551856185718581859186018611862186318641865186618671868186918701871187218731874187518761877187818791880188118821883188418851886188718881889189018911892189318941895189618971898189919001901190219031904190519061907190819091910191119121913191419151916191719181919192019211922192319241925192619271928192919301931193219331934193519361937193819391940194119421943194419451946194719481949195019511952195319541955195619571958195919601961196219631964196519661967196819691970197119721973197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000' rand2[double precision]:1578
-COMMIT
-(6 rows)
+ERROR: could not map filenumber "base/16384/16397" to relation OID
-- done, free logical replication slot
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-data
-(0 rows)
+ERROR: could not map filenumber "base/16384/16397" to relation OID
\pset format aligned
SELECT pg_drop_replication_slot('regression_slot');
pg_drop_replication_slot
diff -U3 /home/hayato/patchTest/postgres/contrib/test_decoding/expected/rewrite.out /home/hayato/patchTest/postgres/contrib/test_decoding/results/rewrite.out
--- /home/hayato/patchTest/postgres/contrib/test_decoding/expected/rewrite.out 2023-12-05 09:40:08.864621359 +0000
+++ /home/hayato/patchTest/postgres/contrib/test_decoding/results/rewrite.out 2024-05-20 04:49:00.802181926 +0000
@@ -116,21 +116,21 @@
-- make old files go away
CHECKPOINT;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ data
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
- table public.replication_example: INSERT: id[integer]:2 somedata[integer]:2 text[character varying]:null
+ table public.replication_example: INSERT: id[integer]:2 somedata[integer]:2 text[character varying]:null testcolumn1[integer]:null testcolumn2[integer]:null testcolumn3[integer]:null
table public.replication_example: INSERT: id[integer]:3 somedata[integer]:3 text[character varying]:null testcolumn1[integer]:1
COMMIT
BEGIN
- table public.replication_example: INSERT: id[integer]:4 somedata[integer]:3 text[character varying]:null testcolumn1[integer]:null
+ table public.replication_example: INSERT: id[integer]:4 somedata[integer]:3 text[character varying]:null testcolumn1[integer]:null testcolumn2[integer]:null testcolumn3[integer]:null
table public.replication_example: INSERT: id[integer]:5 somedata[integer]:4 text[character varying]:null testcolumn1[integer]:2 testcolumn2[integer]:1
COMMIT
BEGIN
- table public.replication_example: INSERT: id[integer]:6 somedata[integer]:5 text[character varying]:null testcolumn1[integer]:3 testcolumn2[integer]:null
+ table public.replication_example: INSERT: id[integer]:6 somedata[integer]:5 text[character varying]:null testcolumn1[integer]:3 testcolumn2[integer]:null testcolumn3[integer]:null
COMMIT
BEGIN
- table public.replication_example: INSERT: id[integer]:7 somedata[integer]:6 text[character varying]:null testcolumn1[integer]:4 testcolumn2[integer]:null
+ table public.replication_example: INSERT: id[integer]:7 somedata[integer]:6 text[character varying]:null testcolumn1[integer]:4 testcolumn2[integer]:null testcolumn3[integer]:null
table public.replication_example: INSERT: id[integer]:8 somedata[integer]:7 text[character varying]:null testcolumn1[integer]:5 testcolumn2[integer]:null testcolumn3[integer]:1
COMMIT
(15 rows)
diff -U3 /home/hayato/patchTest/postgres/contrib/test_decoding/expected/toast.out /home/hayato/patchTest/postgres/contrib/test_decoding/results/toast.out
--- /home/hayato/patchTest/postgres/contrib/test_decoding/expected/toast.out 2024-01-23 12:00:13.548375422 +0000
+++ /home/hayato/patchTest/postgres/contrib/test_decoding/results/toast.out 2024-05-20 04:49:01.549181926 +0000
@@ -352,14 +352,7 @@
DROP TABLE toasted_several;
SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')
WHERE data NOT LIKE '%INSERT: %';
- regexp_replace
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
- BEGIN
- table public.toasted_several: UPDATE: old-key: id[integer]:1 toasted_key[text]:'98765432109876543210..7654321098765432109876543210987654321098765432109876543210' toasted_col2[text]:unchanged-toast-datum
- table public.toasted_several: DELETE: id[integer]:1 toasted_key[text]:'98765432109876543210987654321..876543210987654321098765432109876543210987654321098765432109876543210987654321098765432109876543210'
- COMMIT
-(4 rows)
-
+ERROR: could not map filenumber "base/16384/16709" to relation OID
/*
* Test decoding relation rewrite with toast. The insert into tbl2 within the
* same transaction is there to check that there is no remaining toast_hash not
@@ -374,14 +367,7 @@
INSERT INTO tbl2 VALUES(1);
commit;
SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- substr
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
- BEGIN
- table public.tbl1: INSERT: a[integer]:1 b[text]:'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
- table public.tbl2: INSERT: a[integer]:1
- COMMIT
-(4 rows)
-
+ERROR: could not map filenumber "base/16384/16709" to relation OID
SELECT pg_drop_replication_slot('regression_slot');
pg_drop_replication_slot
--------------------------
diff -U3 /home/hayato/patchTest/postgres/contrib/test_decoding/expected/decoding_into_rel.out /home/hayato/patchTest/postgres/contrib/test_decoding/results/decoding_into_rel.out
--- /home/hayato/patchTest/postgres/contrib/test_decoding/expected/decoding_into_rel.out 2023-10-18 07:11:15.274329904 +0000
+++ /home/hayato/patchTest/postgres/contrib/test_decoding/results/decoding_into_rel.out 2024-05-20 04:49:01.713181926 +0000
@@ -66,43 +66,11 @@
SELECT data FROM pg_logical_slot_peek_changes(slot_name, NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
END$$ LANGUAGE plpgsql;
SELECT * FROM slot_changes_wrapper('regression_slot');
- slot_changes_wrapper
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
- BEGIN
- table public.changeresult: INSERT: data[text]:'BEGIN'
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''BEGIN'''
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''table public.somechange: INSERT: id[integer]:1'''
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''COMMIT'''
- table public.changeresult: INSERT: data[text]:'COMMIT'
- table public.changeresult: INSERT: data[text]:'BEGIN'
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''BEGIN'''
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''table public.changeresult: INSERT: data[text]:''''BEGIN'''''''
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''table public.changeresult: INSERT: data[text]:''''table public.somechange: INSERT: id[integer]:1'''''''
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''table public.changeresult: INSERT: data[text]:''''COMMIT'''''''
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''COMMIT'''
- table public.changeresult: INSERT: data[text]:'COMMIT'
- COMMIT
-(14 rows)
-
+ERROR: could not map filenumber "base/16384/16767" to relation OID
+CONTEXT: SQL statement "SELECT data FROM pg_logical_slot_peek_changes(slot_name, NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')"
+PL/pgSQL function slot_changes_wrapper(name) line 3 at RETURN QUERY
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
- BEGIN
- table public.changeresult: INSERT: data[text]:'BEGIN'
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''BEGIN'''
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''table public.somechange: INSERT: id[integer]:1'''
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''COMMIT'''
- table public.changeresult: INSERT: data[text]:'COMMIT'
- table public.changeresult: INSERT: data[text]:'BEGIN'
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''BEGIN'''
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''table public.changeresult: INSERT: data[text]:''''BEGIN'''''''
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''table public.changeresult: INSERT: data[text]:''''table public.somechange: INSERT: id[integer]:1'''''''
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''table public.changeresult: INSERT: data[text]:''''COMMIT'''''''
- table public.changeresult: INSERT: data[text]:'table public.changeresult: INSERT: data[text]:''COMMIT'''
- table public.changeresult: INSERT: data[text]:'COMMIT'
- COMMIT
-(14 rows)
-
+ERROR: could not map filenumber "base/16384/16767" to relation OID
SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
?column?
----------
diff -U3 /home/hayato/patchTest/postgres/contrib/test_decoding/expected/prepared.out /home/hayato/patchTest/postgres/contrib/test_decoding/results/prepared.out
--- /home/hayato/patchTest/postgres/contrib/test_decoding/expected/prepared.out 2023-10-18 07:11:15.275329904 +0000
+++ /home/hayato/patchTest/postgres/contrib/test_decoding/results/prepared.out 2024-05-20 04:49:01.780181926 +0000
@@ -40,32 +40,7 @@
DROP TABLE test_prepared2;
-- show results
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
--------------------------------------------------------------------------
- BEGIN
- table public.test_prepared1: INSERT: id[integer]:1
- COMMIT
- BEGIN
- table public.test_prepared1: INSERT: id[integer]:2
- COMMIT
- BEGIN
- table public.test_prepared1: INSERT: id[integer]:4
- COMMIT
- BEGIN
- table public.test_prepared2: INSERT: id[integer]:7
- COMMIT
- BEGIN
- table public.test_prepared1: INSERT: id[integer]:5
- table public.test_prepared1: INSERT: id[integer]:6 data[text]:'frakbar'
- COMMIT
- BEGIN
- table public.test_prepared1: INSERT: id[integer]:8 data[text]:null
- COMMIT
- BEGIN
- table public.test_prepared2: INSERT: id[integer]:9
- COMMIT
-(22 rows)
-
+ERROR: could not map filenumber "base/16384/16773" to relation OID
SELECT pg_drop_replication_slot('regression_slot');
pg_drop_replication_slot
--------------------------
diff -U3 /home/hayato/patchTest/postgres/contrib/test_decoding/expected/stream.out /home/hayato/patchTest/postgres/contrib/test_decoding/results/stream.out
--- /home/hayato/patchTest/postgres/contrib/test_decoding/expected/stream.out 2024-04-23 00:42:15.088956778 +0000
+++ /home/hayato/patchTest/postgres/contrib/test_decoding/results/stream.out 2024-05-20 04:49:10.264181926 +0000
@@ -31,6 +31,37 @@
----------------------------------------------------------
opening a streamed block for transaction
streaming message: transactional: 1 prefix: test, sz: 50
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
closing a streamed block for transaction
aborting streamed (sub)transaction
opening a streamed block for transaction
@@ -56,7 +87,7 @@
streaming change for transaction
closing a streamed block for transaction
committing streamed transaction
-(27 rows)
+(58 rows)
-- streaming test for toast changes
ALTER TABLE stream_test ALTER COLUMN data set storage external;
diff -U3 /home/hayato/patchTest/postgres/contrib/test_decoding/expected/twophase_stream.out /home/hayato/patchTest/postgres/contrib/test_decoding/results/twophase_stream.out
--- /home/hayato/patchTest/postgres/contrib/test_decoding/expected/twophase_stream.out 2024-04-23 00:42:15.088956778 +0000
+++ /home/hayato/patchTest/postgres/contrib/test_decoding/results/twophase_stream.out 2024-05-20 04:49:10.619181926 +0000
@@ -33,6 +33,37 @@
----------------------------------------------------------
opening a streamed block for transaction
streaming message: transactional: 1 prefix: test, sz: 50
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
closing a streamed block for transaction
aborting streamed (sub)transaction
opening a streamed block for transaction
@@ -58,7 +89,7 @@
streaming change for transaction
closing a streamed block for transaction
preparing streamed transaction 'test1'
-(27 rows)
+(58 rows)
COMMIT PREPARED 'test1';
--should show the COMMIT PREPARED and the other changes in the transaction
@@ -89,9 +120,40 @@
----------------------------------------------------------
opening a streamed block for transaction
streaming message: transactional: 1 prefix: test, sz: 50
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
+ streaming change for transaction
closing a streamed block for transaction
aborting streamed (sub)transaction
-(4 rows)
+(35 rows)
COMMIT PREPARED 'test1_nodecode';
-- should show the inserts but not show a COMMIT PREPARED but a COMMIT
diff -U3 /home/hayato/patchTest/postgres/contrib/test_decoding/expected/filter.out /home/hayato/patchTest/postgres/contrib/test_decoding/results/filter.out
--- /home/hayato/patchTest/postgres/contrib/test_decoding/expected/filter.out 2024-05-20 03:13:35.881181926 +0000
+++ /home/hayato/patchTest/postgres/contrib/test_decoding/results/filter.out 2024-05-20 04:49:10.648181926 +0000
@@ -0,0 +1,29 @@
+-- predictability
+SET synchronous_commit = on;
+-- define table which will be filtered
+CREATE TABLE test_tobeskipped (a int);
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+INSERT INTO test_tobeskipped VALUES (generate_series(1, 10));
+-- check that changes to test_tobeskipped are not output
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+------------------------------------------------------
+ BEGIN
+ table public.test_tobeskipped: INSERT: a[integer]:1
+ table public.test_tobeskipped: INSERT: a[integer]:2
+ table public.test_tobeskipped: INSERT: a[integer]:3
+ table public.test_tobeskipped: INSERT: a[integer]:4
+ table public.test_tobeskipped: INSERT: a[integer]:5
+ table public.test_tobeskipped: INSERT: a[integer]:6
+ table public.test_tobeskipped: INSERT: a[integer]:7
+ table public.test_tobeskipped: INSERT: a[integer]:8
+ table public.test_tobeskipped: INSERT: a[integer]:9
+ table public.test_tobeskipped: INSERT: a[integer]:10
+ COMMIT
+(12 rows)
+
On Tue, Mar 19, 2024 at 1:49 PM Ajin Cherian <itsajin@gmail.com> wrote:
Of course you can, but this will only convert disk space into memory
space.
For details, please see the case in Email [1].[1]
/messages/by-id/CAGfChW51P944nM5h0HTV9HistvVfwBxNaMt_s-OZ9t=uXz+Zbg@mail.gmail.comRegards, lijie
In some testing, I see a crash:
(gdb) bt
#0 0x00007fa5bcbfd277 in raise () from /lib64/libc.so.6
#1 0x00007fa5bcbfe968 in abort () from /lib64/libc.so.6
#2 0x00000000009e0940 in ExceptionalCondition (
conditionName=conditionName@entry=0x7fa5ab8b9842 "RelationSyncCache !=
NULL",
fileName=fileName@entry=0x7fa5ab8b9820 "pgoutput.c",
lineNumber=lineNumber@entry=1991)
at assert.c:66
#3 0x00007fa5ab8b7804 in get_rel_sync_entry (data=data@entry=0x2492288,
relation=relation@entry=0x7fa5be30a768) at pgoutput.c:1991
#4 0x00007fa5ab8b7cda in pgoutput_table_filter (ctx=<optimized out>,
relation=0x7fa5be30a768,
change=0x24c5c20) at pgoutput.c:1671
#5 0x0000000000813761 in filter_by_table_cb_wrapper (ctx=ctx@entry=0x2491fd0,
relation=relation@entry=0x7fa5be30a768, change=change@entry=0x24c5c20)
at logical.c:1268
#6 0x000000000080e20f in FilterByTable (ctx=ctx@entry=0x2491fd0,
change=change@entry=0x24c5c20)
at decode.c:690
#7 0x000000000080e8e3 in DecodeInsert (ctx=ctx@entry=0x2491fd0,
buf=buf@entry=0x7fff0db92550)
at decode.c:1070
#8 0x000000000080f43d in heap_decode (ctx=ctx@entry=0x2491fd0,
buf=buf@entry=0x7fff0db92550)
at decode.c:485
#9 0x000000000080eca6 in LogicalDecodingProcessRecord
(ctx=ctx@entry=0x2491fd0,
record=0x2492368)
at decode.c:118
#10 0x000000000081338f in DecodingContextFindStartpoint
(ctx=ctx@entry=0x2491fd0)
at logical.c:672
#11 0x000000000083c650 in CreateReplicationSlot (cmd=cmd@entry=0x2490970)
at walsender.c:1323
#12 0x000000000083fd48 in exec_replication_command (
cmd_string=cmd_string@entry=0x239c880 "CREATE_REPLICATION_SLOT
\"pg_16387_sync_16384_7371301304766135621\" LOGICAL pgoutput (SNAPSHOT
'use')") at walsender.c:2116
The reason for the crash is that the RelationSyncCache was NULL prior to
reaching a consistent point.
Hi li jie, I see that you created a new thread with an updated version of
this patch [1]/messages/by-id/CAGfChW7+ZMN4_NHPgz24MM42HVO83ecr9TLfpihJ=M0s1GkXFw@mail.gmail.com. I used that patch and addressed the crash seen above,
rebased the patch and addressed a few other comments.
I'm happy to help you with this patch and address comments if you are not
available.
regards,
Ajin Cherian
Fujitsu Australia
[1]: /messages/by-id/CAGfChW7+ZMN4_NHPgz24MM42HVO83ecr9TLfpihJ=M0s1GkXFw@mail.gmail.com
/messages/by-id/CAGfChW7+ZMN4_NHPgz24MM42HVO83ecr9TLfpihJ=M0s1GkXFw@mail.gmail.com
Attachments:
v2-0001-Reduce-useless-changes-before-reassemble-during-l.patchapplication/octet-stream; name=v2-0001-Reduce-useless-changes-before-reassemble-during-l.patchDownload
From e5484c1e46457103f8e3d3a5e126ae4825c80fc6 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <ajinc@fast.au.fujitsu.com>
Date: Tue, 21 May 2024 23:57:55 -0400
Subject: [PATCH v2] Reduce useless changes before reassemble during logical
replication.
In order to reduce unnecessary logical replication, irrelevant relationship
changes can be filtered out before reorganizing transaction fragments.
This can effectively reduce useless changes and prevent storage space from
being filled up with irrelevant data.
By design, Added a callback LogicalDecodeFilterByRelCB for the output plugin.
We implemented a function pgoutput_table_filter for pgoutput. And RelationSyncCache
is reused to determine whether a relationship-related change should be filtered out.
referring to the implementation of the function pgoutput_change, currently only
insert/update/delete is can filtered, and other types of changes are not considered.
Perhaps more detailed analysis can be done and more filters can be filtered.
Most of the code in the FilterByTable function is transplanted from the ReorderBufferProcessTXN
function, which can be called before the ReorderBufferQueueChange function.It is
also the encapsulation of the callback function filter_by_table_cb in logical.c.
In general, this patch concentrates the filtering of changes in the ReorderBufferProcessTXN
function and the pgoutput_change function into the FilterByTable function, and calls
it before the ReorderBufferQueueChange function.
Author: Lijie
---
src/backend/replication/logical/decode.c | 157 +++++++++++++++++++++++++++-
src/backend/replication/logical/logical.c | 31 ++++++
src/backend/replication/pgoutput/pgoutput.c | 89 ++++++++++------
src/include/replication/logical.h | 2 +
src/include/replication/output_plugin.h | 7 ++
src/test/subscription/t/034_table_filter.pl | 91 ++++++++++++++++
6 files changed, 339 insertions(+), 38 deletions(-)
create mode 100644 src/test/subscription/t/034_table_filter.pl
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 7a86f84..01b43cc 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -33,12 +33,14 @@
#include "access/xlogreader.h"
#include "access/xlogrecord.h"
#include "catalog/pg_control.h"
+#include "common/string.h"
#include "replication/decode.h"
#include "replication/logical.h"
#include "replication/message.h"
#include "replication/reorderbuffer.h"
#include "replication/snapbuild.h"
#include "storage/standbydefs.h"
+#include "utils/relfilenumbermap.h"
/* individual record(group)'s handlers */
static void DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
@@ -152,7 +154,7 @@ xlog_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_PARAMETER_CHANGE:
{
xl_parameter_change *xlrec =
- (xl_parameter_change *) XLogRecGetData(buf->record);
+ (xl_parameter_change *) XLogRecGetData(buf->record);
/*
* If wal_level on the primary is reduced to less than
@@ -578,6 +580,138 @@ FilterByOrigin(LogicalDecodingContext *ctx, RepOriginId origin_id)
return filter_by_origin_cb_wrapper(ctx, origin_id);
}
+static bool
+FilterByTable(LogicalDecodingContext *ctx, ReorderBufferChange *change)
+{
+ ReorderBuffer *rb = ctx->reorder;
+ Relation relation = NULL;
+ SnapBuild *builder = ctx->snapshot_builder;
+ Oid reloid;
+ bool result = false;
+ bool using_subtxn;
+
+ if (ctx->callbacks.filter_by_table_cb == NULL)
+ return false;
+
+ if(SnapBuildCurrentState(builder) < SNAPBUILD_CONSISTENT)
+ return false;
+
+ switch (change->action)
+ {
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ break;
+ default:
+ return false;
+ }
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("filter change by table");
+ else
+ StartTransactionCommand();
+
+ reloid = RelidByRelfilenumber(change->data.tp.rlocator.spcOid,
+ change->data.tp.rlocator.relNumber);
+ if (reloid == InvalidOid)
+ {
+ result = true;
+ goto filter_done;
+ }
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (!RelationIsValid(relation))
+ elog(ERROR, "could not open relation with OID %u (for filenumber \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.rlocator,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ {
+ result = true;
+ goto filter_done;
+ }
+
+ /*
+ * Ignore temporary heaps created during DDL unless the plugin has asked
+ * for them.
+ */
+ if (relation->rd_rel->relrewrite && !rb->output_rewrites)
+ {
+ result = true;
+ goto filter_done;
+ }
+
+ /*
+ * For now ignore sequence changes entirely. Most of the time they don't
+ * log changes using records we understand, so it doesn't make sense to
+ * handle the few cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ {
+ result = true;
+ goto filter_done;
+ }
+
+ if (IsToastRelation(relation))
+ {
+ Oid real_reloid = InvalidOid;
+
+ /* pg_toast_ len is 9 */
+ char *toast_name = RelationGetRelationName(relation);
+ char *start_ch = &toast_name[9];
+
+ real_reloid = strtoint(start_ch, NULL, 10);
+
+ if (real_reloid == InvalidOid)
+ elog(ERROR, "cannot get the real table oid for toast table %s, error: %m", toast_name);
+
+ RelationClose(relation);
+
+ relation = RelationIdGetRelation(real_reloid);
+
+ if (!RelationIsValid(relation))
+ elog(ERROR, "could not open real relation with OID %u (for toast table filenumber \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.rlocator,
+ MAIN_FORKNUM));
+ }
+
+ result = filter_by_table_cb_wrapper(ctx, relation, change);
+
+filter_done:
+
+ if (result && RelationIsValid(relation))
+ elog(DEBUG1, "logical filter change by table %s", RelationGetRelationName(relation));
+
+ if (RelationIsValid(relation))
+ RelationClose(relation);
+
+ AbortCurrentTransaction();
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ if (result)
+ ReorderBufferReturnChange(rb, change, false);
+
+ return result;
+}
+
/*
* Handle rmgr LOGICALMSG_ID records for LogicalDecodingProcessRecord().
*/
@@ -937,6 +1071,9 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
+ if (FilterByTable(ctx, change))
+ return;
+
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
change,
xlrec->flags & XLH_INSERT_ON_TOAST_RELATION);
@@ -1006,6 +1143,9 @@ DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
+ if (FilterByTable(ctx, change))
+ return;
+
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
change, false);
}
@@ -1062,6 +1202,9 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
+ if (FilterByTable(ctx, change))
+ return;
+
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
change, false);
}
@@ -1198,11 +1341,14 @@ DecodeMultiInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
else
change->data.tp.clear_toast_afterwards = false;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change, false);
-
/* move to the next xl_multi_insert_tuple entry */
data += datalen;
+
+ if (FilterByTable(ctx, change))
+ continue;
+
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, change, false);
}
Assert(data == tupledata + tuplelen);
}
@@ -1237,6 +1383,9 @@ DecodeSpecConfirm(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
+ if (FilterByTable(ctx, change))
+ return;
+
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
change, false);
}
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 97a4d99..49ac184 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1242,6 +1242,37 @@ filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, RepOriginId origin_id)
return ret;
}
+bool
+filter_by_table_cb_wrapper(LogicalDecodingContext *ctx, Relation relation, ReorderBufferChange *change)
+{
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+ bool ret;
+
+ Assert(!ctx->fast_forward);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "filter_by_table";
+ state.report_location = InvalidXLogRecPtr;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = false;
+ ctx->end_xact = false;
+
+ /* do the actual work: call callback */
+ ret = ctx->callbacks.filter_by_table_cb(ctx, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
+ return ret;
+}
+
static void
message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index d2b35cf..3d1de74 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -56,6 +56,9 @@ static void pgoutput_message(LogicalDecodingContext *ctx,
Size sz, const char *message);
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static bool pgoutput_table_filter(struct LogicalDecodingContext *ctx,
+ Relation relation,
+ ReorderBufferChange *change);
static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn);
static void pgoutput_prepare_txn(LogicalDecodingContext *ctx,
@@ -258,6 +261,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_prepared_cb = pgoutput_commit_prepared_txn;
cb->rollback_prepared_cb = pgoutput_rollback_prepared_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
+ cb->filter_by_table_cb = pgoutput_table_filter;
cb->shutdown_cb = pgoutput_shutdown;
/* transaction streaming */
@@ -1414,9 +1418,6 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
TupleTableSlot *old_slot = NULL;
TupleTableSlot *new_slot = NULL;
- if (!is_publishable_relation(relation))
- return;
-
/*
* Remember the xid for the change in streaming mode. We need to send xid
* with each change in the streaming mode so that subscriber can make
@@ -1427,37 +1428,6 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
xid = change->txn->xid;
relentry = get_rel_sync_entry(data, relation);
-
- /* First check the table filter */
- switch (action)
- {
- case REORDER_BUFFER_CHANGE_INSERT:
- if (!relentry->pubactions.pubinsert)
- return;
- break;
- case REORDER_BUFFER_CHANGE_UPDATE:
- if (!relentry->pubactions.pubupdate)
- return;
- break;
- case REORDER_BUFFER_CHANGE_DELETE:
- if (!relentry->pubactions.pubdelete)
- return;
-
- /*
- * This is only possible if deletes are allowed even when replica
- * identity is not defined for a table. Since the DELETE action
- * can't be published, we simply return.
- */
- if (!change->data.tp.oldtuple)
- {
- elog(DEBUG1, "didn't send DELETE change because of missing oldtuple");
- return;
- }
- break;
- default:
- Assert(false);
- }
-
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
@@ -1684,6 +1654,57 @@ pgoutput_origin_filter(LogicalDecodingContext *ctx,
}
/*
+ * Return true if the relation has not been published, false otherwise.
+ */
+static bool
+pgoutput_table_filter(struct LogicalDecodingContext *ctx,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ RelationSyncEntry *relentry;
+ ReorderBufferChangeType action = change->action;
+
+ if (!is_publishable_relation(relation))
+ return true;
+
+ relentry = get_rel_sync_entry(data, relation);
+
+ /* First check the table filter */
+ switch (action)
+ {
+ case REORDER_BUFFER_CHANGE_INSERT:
+ if (!relentry->pubactions.pubinsert)
+ return true;
+ break;
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ if (!relentry->pubactions.pubupdate)
+ return true;
+ break;
+ case REORDER_BUFFER_CHANGE_DELETE:
+ if (!relentry->pubactions.pubdelete)
+ return true;
+
+ /*
+ * This is only possible if deletes are allowed even when replica
+ * identity is not defined for a table. Since the DELETE action
+ * can't be published, we simply return.
+ */
+ if (!change->data.tp.oldtuple)
+ {
+ elog(DEBUG1, "didn't send DELETE change because of missing oldtuple");
+ return true;
+ }
+ break;
+ default:
+ Assert(false);
+ }
+
+ return false;
+}
+
+
+/*
* Shutdown the output plugin.
*
* Note, we don't need to clean the data->context and data->cachectx as
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index aff38e8..dae1411 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -145,6 +145,8 @@ extern void LogicalConfirmReceivedLocation(XLogRecPtr lsn);
extern bool filter_prepare_cb_wrapper(LogicalDecodingContext *ctx,
TransactionId xid, const char *gid);
extern bool filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, RepOriginId origin_id);
+extern bool filter_by_table_cb_wrapper(LogicalDecodingContext *ctx, Relation relation,
+ ReorderBufferChange *change);
extern void ResetLogicalStreamingState(void);
extern void UpdateDecodingStats(LogicalDecodingContext *ctx);
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 44988eb..030eb5a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -97,6 +97,12 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
RepOriginId origin_id);
/*
+ * Filter changes by table.
+ */
+typedef bool (*LogicalDecodeFilterByRelCB) (struct LogicalDecodingContext *ctx,
+ Relation relation, ReorderBufferChange *change);
+
+/*
* Called to shutdown an output plugin.
*/
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
@@ -222,6 +228,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeCommitCB commit_cb;
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
+ LogicalDecodeFilterByRelCB filter_by_table_cb;
LogicalDecodeShutdownCB shutdown_cb;
/* streaming of changes at prepare time */
diff --git a/src/test/subscription/t/034_table_filter.pl b/src/test/subscription/t/034_table_filter.pl
new file mode 100644
index 0000000..8d4f962
--- /dev/null
+++ b/src/test/subscription/t/034_table_filter.pl
@@ -0,0 +1,91 @@
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+# Basic logical replication test
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "create table tbl_pub(id int, val1 text, val2 text,size int);");
+$node_publisher->safe_psql('postgres',
+ "create table tbl_t1(id int, val1 text, val2 text,size int);");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION mypub FOR TABLE public.tbl_pub;");
+$node_publisher->safe_psql('postgres',
+qq(
+CREATE OR REPLACE FUNCTION check_replication_status() RETURNS VOID AS \$\$
+DECLARE
+ replication_record pg_stat_replication;
+BEGIN
+ LOOP
+ SELECT *
+ INTO replication_record
+ FROM pg_stat_replication
+ WHERE application_name = 'mysub';
+
+ IF replication_record.replay_lsn = replication_record.write_lsn THEN
+ EXIT;
+ END IF;
+
+ PERFORM pg_sleep(1);
+ END LOOP;
+END;
+\$\$ LANGUAGE plpgsql;));
+
+# Create some preexisting content on subscriber
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_subscriber->safe_psql('postgres',
+ "create table tbl_pub(id int, val1 text, val2 text,size text);");
+$node_subscriber->safe_psql('postgres',
+ "create table tbl_t1(id int, val1 text, val2 text,size text);");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub"
+);
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, 'mysub');
+
+# test filter
+$node_publisher->safe_psql('postgres',
+qq(BEGIN;
+insert into tbl_t1 select 1, 'xyzzy', 'abcba', sum(size) from pg_ls_replslotdir('mysub');
+insert into tbl_t1 select i,repeat('xyzzy', i),repeat('abcba',i),(select sum(size) from pg_ls_replslotdir('mysub')) from generate_series(2,9999) i;
+update tbl_t1 set val2 = repeat('xyzzy',id) where id > 1 and id < 10001;
+select check_replication_status();
+insert into tbl_t1 select 10001, 'xyzzy', 'abcba', sum(size) from pg_ls_replslotdir('mysub');
+COMMIT;)
+);
+
+my $minsize = $node_publisher->safe_psql('postgres',
+ "select size from tbl_t1 order by size asc limit 1;");
+my $maxsize = $node_publisher->safe_psql('postgres',
+ "select size from tbl_t1 order by size desc limit 1;");
+is($minsize, $maxsize, 'check decode filter table between maxsize and minsize');
+
+
+my $fristrow = $node_publisher->safe_psql('postgres',
+ "select size from tbl_t1 where id = 1;");
+my $lastrow = $node_publisher->safe_psql('postgres',
+ "select size from tbl_t1 where id = 10001;");
+is($minsize, $maxsize, 'check decode filter table between fristrow and lastrow');
+
+is($minsize, $lastrow, 'check decode filter table between minsize and lastrow');
+
+print "minsize: " . $minsize . "maxsize: " . $maxsize ."fristrow: " . $fristrow ."lastrow: " . $lastrow . "\n";
+
+done_testing();
\ No newline at end of file
--
1.8.3.1
On Wed, May 22, 2024 at 2:17 PM Ajin Cherian <itsajin@gmail.com> wrote:
The reason for the crash is that the RelationSyncCache was NULL prior to
reaching a consistent point.
Hi li jie, I see that you created a new thread with an updated version of
this patch [1]. I used that patch and addressed the crash seen above,
rebased the patch and addressed a few other comments.
I'm happy to help you with this patch and address comments if you are not
available.regards,
Ajin Cherian
Fujitsu Australia
[1] -
/messages/by-id/CAGfChW7+ZMN4_NHPgz24MM42HVO83ecr9TLfpihJ=M0s1GkXFw@mail.gmail.com
I was discussing this with Kuroda-san who made a patch to add a table
filter with test_decoding plugin. The filter does nothing, just returns
false( doesn't filter anything) and I see that 8 test_decoding tests fail.
In my analysis, I could see that some of the failures were because the new
filter logic was accessing the relation cache using the latest snapshot for
relids which was getting incorrect relation information while decoding
attribute values.
for eg:
CREATE TABLE replication_example(id SERIAL PRIMARY KEY, somedata int, text
varchar(120));
BEGIN;
INSERT INTO replication_example(somedata, text) VALUES (1, 1);
INSERT INTO replication_example(somedata, text) VALUES (1, 2);
COMMIT;
ALTER TABLE replication_example RENAME COLUMN text TO somenum;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL,
'include-xids', '0', 'skip-empty-xacts', '1');
Here, after decoding, the changes for the INSERT, were reflecting the new
column name (somenum) which was altered in a later transaction. This is
because the new Filterby Table() logic was getting relcache with latest
snapshot and does not use a historic snapshot like logical decoding should
be doing. This is because the changes are at the decode.c level and not the
reorderbuffer level and does not have access to the txn from the
reorderbuffer. This problem could be fixed by invalidating the cache in the
FilterByTable() logic, but this does not solve other problems like the
table name itself is changed in a later transaction. I think the patch has
a fundamental problem that the table filter logic does not respect the
snapshot of the transaction being decoded. The historic snapshot is
currently only set up when the actual changes are committed or streamed at
ReorderBufferProcessTXN().
If the purpose of the patch is to filter out unnecessary changes prior to
actual decode, then it will use an invalid snapshot and have lots of
problems. Otherwise this logic has to be moved to the reorderbuffer level
and there will be a big overhead of extracting reorderbuffer while each
change is queued in memory/disk.
regards,
Ajin Cherian
Fujitsu Australia
Hi,
I think the proposed feature is interesting, and after reviewing the
original discussion thread, I'd like to help move it forward. To that end, I've
summarized the key points of the discussion so far, including the challenges
highlighted in the thread and some issues I identified off-list, along with
potential solutions. My colleague will share a new POC patch set that has been
developed based on these lines.
(Note that the previous email was blocked, this is the second attempt)
The original proposal in this thread suggests filtering out changes that won't
be published before decoding. This approach could prevent the serialization of
numerous unnecessary records to disk when using non-streaming mode for large
transactions. I find this to be a valid proposal. Although a recent commit has
enabled streaming mode for built-in logical replication by default, this
strategy could also save memory and CPU cycles in streaming mode, especially
when most changes can be filtered out.
While the proposal is interesting, several issues have been identified both in
this thread and off-list:
- Performance
Determining whether to filter a change requires information about the relation
or publication from the catalog, requiring a transaction to be started. When
most changes in a transaction are unfilterable, the overhead of starting a
transaction for each record is significant.
This is not a big issue because we can cache whether a change corresponding to
a table is filterable using a hash table.
However, a single hash search may not be sufficiently cheap if performed for
every record before decoding. In my tests, this caching approach produced
noticeable overhead (around 4%), mainly due to the cost of computing the hash
key and function calls in a hot path
--4.86%--hash_search
|--3.55%--tag_hash
| --2.96%--hash_bytes
--0.37%--hash_search_with_hash_value
Such overhead may not be acceptable for all users, given valid use cases like
publishing most tables on an instance (e.g., during an upgrade using logical
replication), where this filtering wouldn't provide a benefit and would only
incur overhead.
A tiny approach to minimize overhead is to suspend filtering for a certain
period when an unfilterable change is encountered. In other words, continue
filtering changes (using hash cache) if the last record was successfully
filtered out. If an unfilterable change is found, skip filtering the next 100
(an arbitrary number) changes. This approach aims to reduce frequent hash
searches when most changes can't be filtered out, lowering the overhead to less
than 1% on my machine. (This is a simple idea, and better algorithms could
exist.)
- Snapshot construction
The main challenge discussed in this thread is constructing a correct historic
snapshot for filtering purposes. Creating a historic snapshot while decoding an
in-progress transaction is feasible, as it's already done in streaming mode.
However, there could be pending records (INTERNAL_SNAPSHOT, COMMAND_ID, or
INVALIDATION) that will update the built-in snapshot. We shouldn't use a
current snapshot without processing these records.
But thinking from another perspective, why not perform filtering only when
there are no such pending records. By tracking whether the decoder has any of
these records, we could skip filtering if they are present. These special
records are generally generated only during DDL execution, which shouldn't be
frequent; thus, the filter could still benefit many scenarios.
- new output plugin callback
To perform filtering, a new callback (e.g., change_filter) would be needed to
invoke before decoding the record. Using pgoutput as an example, we would call
get_rel_sync_entry() in the callback to determine if a change corresponding to
a table can be filtered.
get_rel_sync_entry() requires catalog access if it doesn't find the cache entry
in the hash table. But as mentioned in the "Performance" section, we need to
minimize catalog access and transaction start/stop. So, ideally, the callback
should return whether it needs catalog access, allowing the caller
(reorderbuffer) to start a transaction if necessary.
The callback interface could be:
(Note: The function or parameter names can be refined; the example just
provided what's the input and output information.)
typedef bool (LogicalDecodeFilterChangeCB) (struct LogicalDecodingContext ctx,
Oid relid,
ReorderBufferChangeType type,
bool in_txn, bool *cache_valid);
To signal that decoding should be skipped for the given change type, it returns
true; false otherwise. The in_txn parameter indicates whether the callback is
invoked within a transaction block. When in_txn is false, and if making a
decision to filter a change requires being inside a transaction block, such as
needing access to the catalog, set *cache_valid to false. The caller should
invoke this for each record, and invoke this without starting a transaction, if
the returned cache_valid is false, then the caller should should reinvoke the
callback after starting a transaction.
Another alternative approach in Li jie patch[1]/messages/by-id/CAGfChW7XpZGkxOpHZnv69+H_AyKzbffcrumyNu4Fz0u+1ADPxA@mail.gmail.com is having the output plugin
callback always provide all record types that can be published for a relation
and cache them at the reorderbuffer level. However, I think this isn't feasible
because the reorderbuffer isn't (and needn't to be) aware of when to invalidate
such a cache. Although we could register the cache invalidation callback like
pgoutput does in init_rel_sync_cache(), many third-party output plugins could
have their own caching mechanisms, making it impossible for reorderbuffer to
simulate cache management for all. Therefore, what the reorderbuffer should do,
in my opinion, would be to directly ask the output plugin to see if a result
can be obtained without entering a transaction, as per the interface I
mentioned earlier.
[1]: /messages/by-id/CAGfChW7XpZGkxOpHZnv69+H_AyKzbffcrumyNu4Fz0u+1ADPxA@mail.gmail.com
Best Regards,
Hou zj
On Fri, Jan 24, 2025 at 5:28 PM Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>
wrote:
Hi,
I think the proposed feature is interesting, and after reviewing the
original discussion thread, I'd like to help move it forward. To that end,
I've
summarized the key points of the discussion so far, including the
challenges
highlighted in the thread and some issues I identified off-list, along with
potential solutions. My colleague will share a new POC patch set that has
been
developed based on these lines.
(Note that the previous email was blocked, this is the second attempt)
Thanks Hou-san,
Here's a patch-set created based on the design changes proposed by Hou-san.
P.S: It has reached v12 after internal review/rework iterations
Regards,
Ajin Cherian
Fujitsu Australia
Attachments:
v12-0001-Track-transactions-with-internal-snapshot-change.patchapplication/octet-stream; name=v12-0001-Track-transactions-with-internal-snapshot-change.patchDownload
From 269184738e6b5b5e888bb4dd44d7f3ebec6151cf Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Fri, 24 Jan 2025 00:11:29 -0500
Subject: [PATCH v12 1/3] Track transactions with internal snapshot changes
Track transactions which have snapshot changes with a new flag RBTXN_HAS_SNAPSHOT_CHANGES
---
src/backend/replication/logical/reorderbuffer.c | 11 +++++++++++
src/include/replication/reorderbuffer.h | 7 +++++++
2 files changed, 18 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 79b60df..121a1c2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -823,6 +823,14 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
toptxn->txn_flags |= RBTXN_HAS_STREAMABLE_CHANGE;
}
+ else if (change->action == REORDER_BUFFER_CHANGE_INVALIDATION ||
+ change->action == REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT ||
+ change->action == REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID)
+ {
+ ReorderBufferTXN *toptxn = rbtxn_get_toptxn(txn);
+
+ toptxn->txn_flags |= RBTXN_HAS_SNAPSHOT_CHANGES;
+ }
change->lsn = lsn;
change->txn = txn;
@@ -1747,6 +1755,9 @@ ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn, bool txn_prep
txn->txn_flags |= RBTXN_IS_SERIALIZED_CLEAR;
}
+ /* All snapshot changes up to this point have been processed. */
+ txn->txn_flags &= ~RBTXN_HAS_SNAPSHOT_CHANGES;
+
/* also reset the number of entries in the transaction */
txn->nentries_mem = 0;
txn->nentries = 0;
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index a669658..0329a69 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_PREPARE 0x0040
#define RBTXN_SKIPPED_PREPARE 0x0080
#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_HAS_SNAPSHOT_CHANGES 0x0200
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -210,6 +211,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_HAS_STREAMABLE_CHANGE) != 0 \
)
+/* Does this transaction have snapshot changes? */
+#define rbtxn_has_snapshot_changes(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_HAS_SNAPSHOT_CHANGES) != 0 \
+)
+
/*
* Has this transaction been streamed to downstream?
*
--
1.8.3.1
v12-0002-Filter-transactions-that-need-not-be-published.patchapplication/octet-stream; name=v12-0002-Filter-transactions-that-need-not-be-published.patchDownload
From e6827ae016721819e682596e5370b98e595dfaa1 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Fri, 24 Jan 2025 00:30:55 -0500
Subject: [PATCH v12 2/3] Filter transactions that need not be published
This adds logic to filter transactions early (at decode time, rather than at streaming time) so
they can be skipped if they do not contain tables that are part of the publication list of the
logical replication walsender.
Determining whether to filter a change requires information about the relation
and the publication from the catalog, requiring a transaction to be started.
When most changes in a transaction are unfilterable, the overhead of starting a
transaction for each record is significant. To reduce this overhead a hash cache of relation file
locators is created. Even then a hash search for every record before recording has considerable
overhead especially for use cases where most tables in an instance are published. To further reduce
this overhead a simple approach is used to suspend filtering for a certain number of changes
(100) when an unfilterable change is encountered. In other words, continue filtering changes if
the last record was filtered out. If an unfilterable change is found, skip filtering the next 100
changes.
---
src/backend/replication/logical/decode.c | 5 +
src/backend/replication/logical/reorderbuffer.c | 423 ++++++++++++++++++++----
src/include/replication/reorderbuffer.h | 12 +
src/tools/pgindent/typedefs.list | 2 +
4 files changed, 374 insertions(+), 68 deletions(-)
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 0bff0f1..c5f1083 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -915,6 +915,11 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ if (ctx->reorder->can_filter_change &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, &target_locator, true))
+ return;
+
change = ReorderBufferGetChange(ctx->reorder);
if (!(xlrec->flags & XLH_INSERT_IS_SPECULATIVE))
change->action = REORDER_BUFFER_CHANGE_INSERT;
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 121a1c2..5974c77 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -108,6 +108,7 @@
#include "storage/fd.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
@@ -226,8 +227,10 @@ static ReorderBufferTXN *ReorderBufferTXNByXid(ReorderBuffer *rb,
XLogRecPtr lsn, bool create_as_top);
static void ReorderBufferTransferSnapToParent(ReorderBufferTXN *txn,
ReorderBufferTXN *subtxn);
-
static void AssertTXNLsnOrder(ReorderBuffer *rb);
+static Relation ReorderBufferGetRelation(ReorderBuffer *rb,
+ RelFileLocator *rlocator,
+ bool has_tuple);
/* ---------------------------------------
* support functions for lsn-order iterating over the ->changes of a
@@ -276,6 +279,8 @@ static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
*/
static inline bool ReorderBufferCanStream(ReorderBuffer *rb);
static inline bool ReorderBufferCanStartStreaming(ReorderBuffer *rb);
+static Snapshot ReorderBufferStreamTXNSnapshot(ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
@@ -302,6 +307,48 @@ static void ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition, Size sz);
/*
+ * ---------------------------------------
+ * RelFileLocator filtering
+ * ---------------------------------------
+ * This hash table serves as a lookup table for determining if a relation can
+ * be filtered before being decoded and queued into the buffer.
+ *
+ * The hash table shares the same lifespan as the reorder buffer. This is
+ * crucial because each reorderbuffer may have different configurations or be
+ * associated with different output plugins, affecting the types of tables to
+ * be processed.
+ */
+
+static HTAB *RelFileLocatorFilterCache = NULL;
+
+static bool relation_callbacks_registered = false;
+
+typedef struct ReorderBufferRelFileLocatorKey
+{
+ Oid reltablespace;
+ RelFileNumber relfilenumber;
+} ReorderBufferRelFileLocatorKey;
+
+/* entry for hash table we use to track if the relation can be filtered */
+typedef struct ReorderBufferRelFileLocatorEnt
+{
+ ReorderBufferRelFileLocatorKey key;
+ Oid relid;
+ bool filterable;
+} ReorderBufferRelFileLocatorEnt;
+
+static void RelFileLocatorCacheInvalidateCallback(Datum arg, Oid relid);
+static void ReorderBufferMemoryResetcallback(void *arg);
+
+/*
+ * After encountering a change that cannot be filtered out, filtering is
+ * temporarily suspended. Filtering resumes after processing every 100 changes.
+ * This strategy helps to minimize the overhead of performing a hash table
+ * search for each record, especially when most changes are not filterable.
+ */
+#define CHANGES_THRESHOLD_FOR_FILTER 100
+
+/*
* Allocate a new ReorderBuffer and clean out any old serialized state from
* prior ReorderBuffer instances for the same slot.
*/
@@ -359,6 +406,34 @@ ReorderBufferAllocate(void)
buffer->by_txn = hash_create("ReorderBufferByXid", 1000, &hash_ctl,
HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ /*
+ * To support early filtering of changes, this hash table acts
+ * as a lookup table to determine if the corresponding relation is
+ * required to be decoded and queued into the buffer. The hash table
+ * shares the same lifespan as the reorder buffer.
+ *
+ * Also setup the callback to invalidate cache when relations are updated.
+ */
+ hash_ctl.keysize = sizeof(ReorderBufferRelFileLocatorKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferRelFileLocatorEnt);
+ hash_ctl.hcxt = buffer->context;
+
+ RelFileLocatorFilterCache =
+ hash_create("RelFileLocatorFilterCache", 1000, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ buffer->relfile_callback.arg = NULL;
+ buffer->relfile_callback.func = ReorderBufferMemoryResetcallback;
+ MemoryContextRegisterResetCallback(buffer->context, &buffer->relfile_callback);
+
+ if (!relation_callbacks_registered)
+ {
+ /* Watch for invalidation events. */
+ CacheRegisterRelcacheCallback(RelFileLocatorCacheInvalidateCallback,
+ (Datum) 0);
+ relation_callbacks_registered = true;
+ }
+
buffer->by_txn_last_xid = InvalidTransactionId;
buffer->by_txn_last_txn = NULL;
@@ -369,6 +444,8 @@ ReorderBufferAllocate(void)
/* txn_heap is ordered by transaction size */
buffer->txn_heap = pairingheap_allocate(ReorderBufferTXNSizeCompare, NULL);
+ buffer->can_filter_change = true;
+
buffer->spillTxns = 0;
buffer->spillCount = 0;
buffer->spillBytes = 0;
@@ -840,6 +917,16 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn->nentries++;
txn->nentries_mem++;
+ /*
+ * If we're not filtering and we've crossed the change threshold,
+ * attempt to filter again
+ */
+ if (!rb->can_filter_change && ++rb->processed_changes >= CHANGES_THRESHOLD_FOR_FILTER)
+ {
+ rb->can_filter_change = true;
+ rb->processed_changes = 0;
+ }
+
/* update memory accounting information */
ReorderBufferChangeMemoryUpdate(rb, change, NULL, true,
ReorderBufferChangeSize(change));
@@ -2105,6 +2192,81 @@ ReorderBufferResetTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
+ * Return the relation corresponding to the given RelFileLocator 'rlocator' if
+ * relevant to decoding. Otherwise, return NULL.
+ *
+ * We don't decode catalog files, relations that are not logically logged,
+ * temporary heaps and sequences.
+ */
+static Relation
+ReorderBufferGetRelation(ReorderBuffer *rb, RelFileLocator *rlocator,
+ bool has_tuple)
+{
+ bool filterable = false;
+ Relation relation;
+ Oid reloid;
+
+ reloid = RelidByRelfilenumber(rlocator->spcOid, rlocator->relNumber);
+
+ if (!OidIsValid(reloid))
+ {
+ if (!has_tuple)
+ {
+ /*
+ * Mapped catalog tuple without data, emitted while catalog table was in
+ * the process of being rewritten. We can fail to look up the
+ * relfilenumber, because the relmapper has no "historic" view, in
+ * contrast to the normal catalog during decoding. Thus repeated rewrites
+ * can cause a lookup failure. That's OK because we do not decode catalog
+ * changes anyway. Normally such tuples would be skipped over below, but
+ * we can't identify whether the table should be logically logged without
+ * mapping the relfilenumber to the oid.
+ */
+ return NULL;
+ }
+
+ elog(ERROR, "could not map filenumber \"%s\" to relation OID",
+ relpathperm(*rlocator, MAIN_FORKNUM));
+ }
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (!RelationIsValid(relation))
+ elog(ERROR, "could not open relation with OID %u (for filenumber \"%s\")",
+ reloid, relpathperm(*rlocator, MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ filterable = true;
+
+ else if (relation->rd_rel->relrewrite && !rb->output_rewrites)
+ {
+ /*
+ * Ignore temporary heaps created during DDL unless the plugin has asked
+ * for them.
+ */
+ filterable = true;
+ }
+
+ else if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ {
+ /*
+ * For now ignore sequence changes entirely. Most of the time they don't
+ * log changes using records we understand, so it doesn't make sense to
+ * handle the few cases we do.
+ */
+ filterable = true;
+ }
+
+ if (filterable)
+ {
+ RelationClose(relation);
+ return NULL;
+ }
+
+ return relation;
+}
+
+/*
* Helper function for ReorderBufferReplay and ReorderBufferStreamTXN.
*
* Send data of a transaction (and its subtransactions) to the
@@ -2176,7 +2338,6 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
{
Relation relation = NULL;
- Oid reloid;
CHECK_FOR_INTERRUPTS();
@@ -2235,55 +2396,11 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_DELETE:
Assert(snapshot_now);
- reloid = RelidByRelfilenumber(change->data.tp.rlocator.spcOid,
- change->data.tp.rlocator.relNumber);
-
- /*
- * Mapped catalog tuple without data, emitted while
- * catalog table was in the process of being rewritten. We
- * can fail to look up the relfilenumber, because the
- * relmapper has no "historic" view, in contrast to the
- * normal catalog during decoding. Thus repeated rewrites
- * can cause a lookup failure. That's OK because we do not
- * decode catalog changes anyway. Normally such tuples
- * would be skipped over below, but we can't identify
- * whether the table should be logically logged without
- * mapping the relfilenumber to the oid.
- */
- if (reloid == InvalidOid &&
- change->data.tp.newtuple == NULL &&
- change->data.tp.oldtuple == NULL)
- goto change_done;
- else if (reloid == InvalidOid)
- elog(ERROR, "could not map filenumber \"%s\" to relation OID",
- relpathperm(change->data.tp.rlocator,
- MAIN_FORKNUM));
-
- relation = RelationIdGetRelation(reloid);
+ relation = ReorderBufferGetRelation(rb, &change->data.tp.rlocator,
+ (change->data.tp.newtuple != NULL ||
+ change->data.tp.oldtuple != NULL));
if (!RelationIsValid(relation))
- elog(ERROR, "could not open relation with OID %u (for filenumber \"%s\")",
- reloid,
- relpathperm(change->data.tp.rlocator,
- MAIN_FORKNUM));
-
- if (!RelationIsLogicallyLogged(relation))
- goto change_done;
-
- /*
- * Ignore temporary heaps created during DDL unless the
- * plugin has asked for them.
- */
- if (relation->rd_rel->relrewrite && !rb->output_rewrites)
- goto change_done;
-
- /*
- * For now ignore sequence changes entirely. Most of the
- * time they don't log changes using records we
- * understand, so it doesn't make sense to handle the few
- * cases we do.
- */
- if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
goto change_done;
/* user-triggered change */
@@ -4059,19 +4176,13 @@ ReorderBufferCanStartStreaming(ReorderBuffer *rb)
}
/*
- * Send data of a large transaction (and its subtransactions) to the
- * output plugin, but using the stream API.
+ * This function generates or retrieves a consistent snapshot of a transaction
+ * that is currently in progress for logical replication.
*/
-static void
-ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+static Snapshot
+ReorderBufferStreamTXNSnapshot(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
Snapshot snapshot_now;
- CommandId command_id;
- Size stream_bytes;
- bool txn_is_streamed;
-
- /* We can never reach here for a subtransaction. */
- Assert(rbtxn_is_toptxn(txn));
/*
* We can't make any assumptions about base snapshot here, similar to what
@@ -4114,12 +4225,11 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (txn->base_snapshot == NULL)
{
Assert(txn->ninvalidations == 0);
- return;
+ return NULL;
}
- command_id = FirstCommandId;
snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
- txn, command_id);
+ txn, FirstCommandId);
}
else
{
@@ -4132,17 +4242,40 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* the LSN condition in the previous branch (so no need to walk
* through subxacts again). In fact, we must not do that as we may be
* using snapshot half-way through the subxact.
- */
- command_id = txn->command_id;
-
- /*
+ *
* We can't use txn->snapshot_now directly because after the last
* streaming run, we might have got some new sub-transactions. So we
* need to add them to the snapshot.
*/
snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
- txn, command_id);
+ txn, txn->command_id);
+ }
+
+ return snapshot_now;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Snapshot snapshot_now;
+ Size stream_bytes;
+ bool txn_is_streamed;
+
+ /* We can never reach here for a subtransaction. */
+ Assert(rbtxn_is_toptxn(txn));
+
+ snapshot_now = ReorderBufferStreamTXNSnapshot(rb, txn);
+
+ /* This transaction didn't make any changes to the database till now. */
+ if (snapshot_now == NULL)
+ return;
+ if (txn->snapshot_now != NULL)
+ {
/* Free the previously copied snapshot. */
Assert(txn->snapshot_now->copied);
ReorderBufferFreeSnap(rb, txn->snapshot_now);
@@ -4160,7 +4293,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* Process and send the changes to output plugin. */
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
- command_id, true);
+ snapshot_now->curcid, true);
rb->streamCount += 1;
rb->streamBytes += stream_bytes;
@@ -5349,3 +5482,157 @@ restart:
*cmax = ent->cmax;
return true;
}
+
+/*
+ * Flush mapping entries when the corresponding relations are updated in pg_class.
+ */
+static void
+RelFileLocatorCacheInvalidateCallback(Datum arg, Oid relid)
+{
+ HASH_SEQ_STATUS status;
+ ReorderBufferRelFileLocatorEnt *entry;
+
+ if (!RelFileLocatorFilterCache)
+ return;
+
+ hash_seq_init(&status, RelFileLocatorFilterCache);
+
+ /* slightly inefficient algorithm but not performance critical path */
+ while ((entry = (ReorderBufferRelFileLocatorEnt *) hash_seq_search(&status)) != NULL)
+ {
+ /*
+ * If relid is InvalidOid, signaling a complete reset, we must remove
+ * all entries, otherwise just remove the specific relation's entry.
+ * Always remove negative cache entries.
+ */
+ if (relid == InvalidOid || /* complete reset */
+ entry->relid == InvalidOid || /* negative cache entry */
+ entry->relid == relid) /* individual flushed relation */
+ {
+ if (hash_search(RelFileLocatorFilterCache,
+ &entry->key,
+ HASH_REMOVE,
+ NULL) == NULL)
+ elog(ERROR, "hash table corrupted");
+ }
+ }
+}
+
+/*
+ * Context reset/delete callback for RelFileLocatorFilterCache.
+ */
+static void
+ReorderBufferMemoryResetcallback(void *arg)
+{
+ RelFileLocatorFilterCache = NULL;
+}
+
+/*
+ * Determine whether the record corresponding to the relation identified by
+ * 'rlocator' needs to be filtered and not decoded and queued in the buffer.
+ *
+ * Determining the necessity of decoding requires accessing relation
+ * information from the system catalog, which requires a historical snapshot.
+ * We cannot directly use the current snapshot of the transaction due to the
+ * presence of INTERNAL_SNAPSHOT, COMMAND_ID, or INVALIDATION records in the
+ * buffer that could modify the snapshot. A proper snapshot can only be
+ * constructed after these records are processed in ReorderBufferProcessTXN, or
+ * if we are decoding a transaction without these records. See comment on top
+ * of ReorderBufferGetRelation() to see list of relations that are not
+ * decoded by the reorderbuffer.
+ *
+ * To reduce the overhead from the system catalog access and transaction
+ * management, the results are cached in the 'RelFileLocatorFilterCache' hash
+ * table.
+ *
+ * Returns true if the relation can be filtered; otherwise, false.
+ */
+bool
+ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, RelFileLocator *rlocator,
+ bool has_tuple)
+{
+ bool found;
+ Relation relation;
+ bool using_subtxn;
+ Snapshot snapshot_now;
+ ReorderBufferTXN *txn,
+ *toptxn;
+ ReorderBufferRelFileLocatorEnt *entry;
+ ReorderBufferRelFileLocatorKey key;
+
+ Assert(RelFileLocatorFilterCache);
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+ Assert(txn);
+ toptxn = rbtxn_get_toptxn(txn);
+ rb->can_filter_change = false;
+
+ /*
+ * We cannot construct an accurate historical snapshot until all pending
+ * records in this transaction that might update the snapshot are
+ * processed.
+ */
+ if (rbtxn_has_snapshot_changes(toptxn))
+ return false;
+
+ key.reltablespace = rlocator->spcOid;
+ key.relfilenumber = rlocator->relNumber;
+ entry = hash_search(RelFileLocatorFilterCache, &key, HASH_ENTER, &found);
+
+ if (found)
+ {
+ rb->can_filter_change = entry->filterable;
+ return entry->filterable;
+ }
+
+ /* constructs a temporary historical snapshot */
+ snapshot_now = ReorderBufferStreamTXNSnapshot(rb, toptxn);
+ Assert(snapshot_now);
+
+ /* build data to be able to lookup the CommandIds of catalog tuples */
+ ReorderBufferBuildTupleCidHash(rb, toptxn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, toptxn->tuplecid_hash);
+
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("filter change by RelFileLocator");
+ else
+ StartTransactionCommand();
+
+ relation = ReorderBufferGetRelation(rb, rlocator, has_tuple);
+
+ if (RelationIsValid(relation))
+ {
+ entry->relid = RelationGetRelid(relation);
+ entry->filterable = false;
+ RelationClose(relation);
+ }
+ else
+ {
+ entry->relid = InvalidOid;
+ entry->filterable = true;
+ }
+
+ rb->can_filter_change = entry->filterable;
+
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access have
+ * persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ return entry->filterable;
+}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0329a69..b1267e9 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -622,6 +622,7 @@ struct ReorderBuffer
* Private memory context.
*/
MemoryContext context;
+ MemoryContextCallback relfile_callback;
/*
* Memory contexts for specific types objects
@@ -642,6 +643,12 @@ struct ReorderBuffer
/* Max-heap for sizes of all top-level and sub transactions */
pairingheap *txn_heap;
+ /* should we try to filter the change? */
+ bool can_filter_change;
+
+ /* number of changes after a failed attempt at filtering */
+ int8 processed_changes;
+
/*
* Statistics about transactions spilled to disk.
*
@@ -742,4 +749,9 @@ extern void ReorderBufferSetRestartPoint(ReorderBuffer *rb, XLogRecPtr ptr);
extern void StartupReorderBuffer(void);
+extern bool ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, RelFileLocator *rlocator,
+ bool has_tuple);
+extern bool ReorderBufferCanFilterChanges(ReorderBuffer *rb);
+
#endif
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index a2644a2..0feca52 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2433,6 +2433,8 @@ ReorderBufferIterTXNEntry
ReorderBufferIterTXNState
ReorderBufferMessageCB
ReorderBufferPrepareCB
+ReorderBufferRelFileLocatorEnt
+ReorderBufferRelFileLocatorKey
ReorderBufferRollbackPreparedCB
ReorderBufferStreamAbortCB
ReorderBufferStreamChangeCB
--
1.8.3.1
v12-0003-Introduce-a-output-plugin-callback-to-filter-cha.patchapplication/octet-stream; name=v12-0003-Introduce-a-output-plugin-callback-to-filter-cha.patchDownload
From 4f3f09618e91d9a0e36bfa3163a64384df6ee4d2 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Fri, 24 Jan 2025 00:40:53 -0500
Subject: [PATCH v12 3/3] Introduce a output plugin callback to filter changes
This new output plugin callback provides an option to logical decoding plugins to filter out
changes early. The primary purpose of the callback is to conserve memory and processing cycles
by excluding changes that are not required by output plugins.
---
doc/src/sgml/logicaldecoding.sgml | 41 ++++++++++++++++++-
src/backend/replication/logical/decode.c | 22 +++++++++-
src/backend/replication/logical/logical.c | 42 +++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 45 +++++++++++++++++----
src/backend/replication/pgoutput/pgoutput.c | 54 +++++++++++++++++++++++++
src/include/replication/output_plugin.h | 20 +++++++++
src/include/replication/reorderbuffer.h | 8 ++++
src/test/subscription/t/013_partition.pl | 7 ++++
8 files changed, 228 insertions(+), 11 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 1c4ae38..a603c47 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -560,6 +560,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeCommitCB commit_cb;
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
+ LogicalDecodeFilterChangeCB filter_change_cb;
LogicalDecodeShutdownCB shutdown_cb;
LogicalDecodeFilterPrepareCB filter_prepare_cb;
LogicalDecodeBeginPrepareCB begin_prepare_cb;
@@ -582,8 +583,8 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
and <function>commit_cb</function> callbacks are required,
while <function>startup_cb</function>, <function>truncate_cb</function>,
<function>message_cb</function>, <function>filter_by_origin_cb</function>,
- and <function>shutdown_cb</function> are optional.
- If <function>truncate_cb</function> is not set but a
+ <function>shutdown_cb</function>, and <function>filter_change_cb</function>
+ are optional. If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
@@ -871,6 +872,42 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-filter-change">
+ <title>Change Filter Callback</title>
+
+ <para>
+ The optional <function>filter_change_cb</function> is called before a
+ change record is decoded to determine whether the change can be filtered
+ out.
+<programlisting>
+typedef bool (*LogicalDecodeFilterChangeCB) (struct LogicalDecodingContext *ctx,
+ Oid relid,
+ ReorderBufferChangeType change_type,
+ bool in_txn, bool *cache_valid);
+</programlisting>
+ To indicate that decoding can be skipped for the given change
+ <parameter>change_type</parameter>, return <literal>true</literal>;
+ <literal>false</literal> otherwise.
+ The <parameter>in_txn</parameter> parameter indicates whether the
+ callback is invoked within a transaction block.
+ When <parameter>in_txn</parameter> is false, and if making a decision to filter a change requires being inside a
+ transaction block, such as needing access to the catalog, set
+ <parameter>*cache_valid</parameter> to <literal>false</literal>.
+ This ensures that the callback will be reinvoked once a transaction block
+ starts. If a decision can be made immediately, set
+ <parameter>*cache_valid</parameter> to <literal>true</literal>.
+ </para>
+ <para>
+ The primary purpose of this callback function is to optimize memory usage
+ and processing efficiency by filtering out changes that are unnecessary for
+ output plugins. It enables output plugins to selectively process relevant
+ changes. Caching filtering decisions locally is recommended, as it enables
+ the callback to provide cached results without repeatedly initiating
+ transactions or querying the catalog. This approach minimizes overhead
+ and improves efficiency during the decoding process.
+ </para>
+ </sect3>
+
<sect3 id="logicaldecoding-output-plugin-message">
<title>Generic Message Callback</title>
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index c5f1083..7c01e7c 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -576,6 +576,17 @@ FilterByOrigin(LogicalDecodingContext *ctx, RepOriginId origin_id)
}
/*
+ * Check if filtering changes before decoding is supported and we're not suppressing filter
+ * changes currently.
+ */
+static inline bool
+FilterChangeIsEnabled(LogicalDecodingContext *ctx)
+{
+ return (ctx->callbacks.filter_change_cb != NULL &&
+ ctx->reorder->can_filter_change);
+}
+
+/*
* Handle rmgr LOGICALMSG_ID records for LogicalDecodingProcessRecord().
*/
void
@@ -915,9 +926,16 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
- if (ctx->reorder->can_filter_change &&
+ /*
+ * When filtering changes, determine if the relation associated with the change
+ * can be skipped. This could be because the relation is unlogged or because
+ * the plugin has opted to exclude this relation from decoding.
+ */
+ if (FilterChangeIsEnabled(ctx) &&
ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, &target_locator, true))
+ buf->origptr, &target_locator,
+ REORDER_BUFFER_CHANGE_INSERT,
+ true))
return;
change = ReorderBufferGetChange(ctx->reorder);
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 0b25efa..adc5383 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -74,6 +74,9 @@ static void truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+static bool filter_change_cb_wrapper(ReorderBuffer *cache, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn,
+ bool *cache_valid);
/* streaming callbacks */
static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
@@ -220,6 +223,7 @@ StartupDecodingContext(List *output_plugin_options,
/* wrap output plugin callbacks, so we can add error context information */
ctx->reorder->begin = begin_cb_wrapper;
ctx->reorder->apply_change = change_cb_wrapper;
+ ctx->reorder->filter_change = filter_change_cb_wrapper;
ctx->reorder->apply_truncate = truncate_cb_wrapper;
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
@@ -1243,6 +1247,44 @@ filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, RepOriginId origin_id)
return ret;
}
+static bool
+filter_change_cb_wrapper(ReorderBuffer *cache, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn,
+ bool *cache_valid)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+ bool ret;
+
+ Assert(!ctx->fast_forward);
+
+ /* check if the filter change callback is supported */
+ if (ctx->callbacks.filter_change_cb == NULL)
+ return false;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "filter_change";
+ state.report_location = InvalidXLogRecPtr;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = false;
+ ctx->end_xact = false;
+
+ /* do the actual work: call callback */
+ ret = ctx->callbacks.filter_change_cb(ctx, relid, change_type, in_txn, cache_valid);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
+ return ret;
+}
+
static void
message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 5974c77..052d91f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5550,9 +5550,11 @@ ReorderBufferMemoryResetcallback(void *arg)
bool
ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, RelFileLocator *rlocator,
+ ReorderBufferChangeType change_type,
bool has_tuple)
{
bool found;
+ bool cache_valid;
Relation relation;
bool using_subtxn;
Snapshot snapshot_now;
@@ -5566,7 +5568,6 @@ ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
Assert(txn);
toptxn = rbtxn_get_toptxn(txn);
- rb->can_filter_change = false;
/*
* We cannot construct an accurate historical snapshot until all pending
@@ -5583,7 +5584,25 @@ ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
if (found)
{
rb->can_filter_change = entry->filterable;
- return entry->filterable;
+
+ /*
+ * Quick return if we already know that the relation is not to be decoded.
+ * These are for special relations that are unlogged and for sequences
+ * and catalogs.
+ */
+ if (entry->filterable)
+ return true;
+
+ /* Allow the output plugin to filter relations */
+ rb->can_filter_change = rb->filter_change(rb, entry->relid, change_type,
+ false, &cache_valid);
+
+ /*
+ * If plugin had the relation ready in cache, the response is valid, else
+ * we'll need to call the plugin a second time within a transaction.
+ */
+ if (cache_valid)
+ return rb->can_filter_change;
}
/* constructs a temporary historical snapshot */
@@ -5607,18 +5626,30 @@ ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
if (RelationIsValid(relation))
{
- entry->relid = RelationGetRelid(relation);
+ if (IsToastRelation(relation))
+ {
+ Oid real_reloid = InvalidOid;
+ char *toast_name = RelationGetRelationName(relation);
+ /* pg_toast_ len is 9 */
+ char *start_ch = &toast_name[9];
+
+ real_reloid = pg_strtoint32(start_ch);
+ entry->relid = real_reloid;
+ }
+ else
+ entry->relid = RelationGetRelid(relation);
+
entry->filterable = false;
+ rb->can_filter_change = rb->filter_change(rb, entry->relid, change_type,
+ true, &cache_valid);
RelationClose(relation);
}
else
{
entry->relid = InvalidOid;
- entry->filterable = true;
+ rb->can_filter_change = entry->filterable = true;
}
- rb->can_filter_change = entry->filterable;
-
ReorderBufferFreeSnap(rb, snapshot_now);
TeardownHistoricSnapshot(false);
@@ -5634,5 +5665,5 @@ ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- return entry->filterable;
+ return rb->can_filter_change;
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index a363c88..c15b053 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -56,6 +56,9 @@ static void pgoutput_message(LogicalDecodingContext *ctx,
Size sz, const char *message);
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static bool pgoutput_filter_change(LogicalDecodingContext *ctx, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn,
+ bool *cache_valid);
static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn);
static void pgoutput_prepare_txn(LogicalDecodingContext *ctx,
@@ -267,6 +270,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_prepared_cb = pgoutput_commit_prepared_txn;
cb->rollback_prepared_cb = pgoutput_rollback_prepared_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
+ cb->filter_change_cb = pgoutput_filter_change;
cb->shutdown_cb = pgoutput_shutdown;
/* transaction streaming */
@@ -1742,6 +1746,56 @@ pgoutput_origin_filter(LogicalDecodingContext *ctx,
}
/*
+ * Determine whether a change to the specified relation should be published.
+ *
+ * See the comments atop of LogicalDecodeFilterChangeCB for details.
+ */
+static bool
+pgoutput_filter_change(LogicalDecodingContext *ctx, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn, bool *cache_valid)
+{
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ if (in_txn)
+ {
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+ entry = get_rel_sync_entry(data, relation);
+ *cache_valid = true;
+ }
+ else
+ {
+ entry = (RelationSyncEntry *) hash_search(RelationSyncCache,
+ &relid,
+ HASH_FIND, cache_valid);
+ if (!*cache_valid)
+ return false;
+ }
+
+ /*
+ * If the pubaction is not supported by this publication then return true to say
+ * the change for this entry can be skipped.
+ */
+ switch (change_type)
+ {
+ case REORDER_BUFFER_CHANGE_INSERT:
+ return !entry->pubactions.pubinsert;
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ return !entry->pubactions.pubupdate;
+ case REORDER_BUFFER_CHANGE_DELETE:
+ return !entry->pubactions.pubdelete;
+ default:
+ /* allow any other changes that are not explicitly filtered */
+ return false;
+ }
+
+}
+
+/*
* Shutdown the output plugin.
*
* Note, we don't need to clean the data->context, data->cachectx, and
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 8d4d5b71..f82bc1d 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -97,6 +97,25 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
RepOriginId origin_id);
/*
+ * This callback is called before a change record is decoded to determine
+ * whether the change can be filtered out before entering the reorder buffer.
+ *
+ * Typically, the output plugin needs to refer to the system catalog to make
+ * this decision. To enhance efficiency, the plugin should cache the lookup
+ * result for each relation, minimizing catalog access on subsequent calls.
+ *
+ * If the callback is called with 'in_txn' set as false (the reorder
+ * buffer has not started a transaction), the output plugin should set '*cache_valid'
+ * to false to indicate that the result is not available in its internal cache.
+ * If 'in_txn' is true, the plugin can create a cache entry after querying the
+ * catalog.
+ */
+typedef bool (*LogicalDecodeFilterChangeCB) (struct LogicalDecodingContext *ctx,
+ Oid relid,
+ ReorderBufferChangeType change_type,
+ bool in_txn, bool *cache_valid);
+
+/*
* Called to shutdown an output plugin.
*/
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
@@ -222,6 +241,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeCommitCB commit_cb;
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
+ LogicalDecodeFilterChangeCB filter_change_cb;
LogicalDecodeShutdownCB shutdown_cb;
/* streaming of changes at prepare time */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index b1267e9..b138365 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -468,6 +468,12 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* filter change callback signature */
+typedef bool (*ReorderBufferFilterChangeCB) (ReorderBuffer *rb,
+ Oid relid,
+ ReorderBufferChangeType change_type,
+ bool in_txn, bool *cache_valid);
+
/* begin prepare callback signature */
typedef void (*ReorderBufferBeginPrepareCB) (ReorderBuffer *rb,
ReorderBufferTXN *txn);
@@ -578,6 +584,7 @@ struct ReorderBuffer
*/
ReorderBufferBeginCB begin;
ReorderBufferApplyChangeCB apply_change;
+ ReorderBufferFilterChangeCB filter_change;
ReorderBufferApplyTruncateCB apply_truncate;
ReorderBufferCommitCB commit;
ReorderBufferMessageCB message;
@@ -751,6 +758,7 @@ extern void StartupReorderBuffer(void);
extern bool ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, RelFileLocator *rlocator,
+ ReorderBufferChangeType change_type,
bool has_tuple);
extern bool ReorderBufferCanFilterChanges(ReorderBuffer *rb);
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 14a3bea..180eaa0 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -468,6 +468,13 @@ $node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab4 (a int PRIMARY KEY)");
$node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab4_1 (a int PRIMARY KEY)");
+
+# Ensure that the subscription 'sub2' catches up with the latest changes. This
+# is necessary for the walsender to update its historic snapshot. Otherwise,
+# the walsender might retain an outdated snapshot, potentially preventing it
+# from accessing the newly created publication.
+$node_publisher->wait_for_catchup('sub2');
+
# Since we specified publish_via_partition_root in pub_all and
# pub_lower_level, all partition tables use their root tables' identity and
# schema. We set the list of publications so that the FOR ALL TABLES
--
1.8.3.1
Hi Ajin,
Some review comments for patch v12-0001.
======
Commit message
1.
Track transactions which have snapshot changes with a new flag
RBTXN_HAS_SNAPSHOT_CHANGES
~
The commit message only says *what* it does, but not *why* this patch
even exists. TBH, I don't understand why this patch needs to be
separated from your patch 0002, because 0001 makes no independent use
of the flag, nor is it separately tested.
Anyway, if it is going to remain separated then IMO at least the the
message should explain the intended purpose e.g. why the subsequent
patches require this flagged info and how they will use it.
======
src/include/replication/reorderbuffer.h
2.
+/* Does this transaction have snapshot changes? */
+#define rbtxn_has_snapshot_changes(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_HAS_SNAPSHOT_CHANGES) != 0 \
+)
+
Is the below wording maybe a more plain way to say that:
/* Does this transaction make changes to the current snapshot? */
======
Kind Regards,
Peter Smith.
Fujitsu Australia
Dear Ajin, Hou,
- Snapshot construction
I understand the approach that we do not try to filter for all the workloads; do
just best-effort.
I played with your PoC and here are my comments.
1.
Can you add tests for better understanding? I've tried for tes_decoding like attached,
but it couldn't pass the regression test. Cases "stream.sql" and "twophase_stream.sql"
were failed.
2.
I think we can extend the skip mechanism to UPDATE/DELETE/MultiInsert/SpecConfirm.
Regarding the TRUNCATE, I'm not sure we can handle hte TRUNCATE case because the we
can't track RelFileLocator anymore.
3.
Both ReorderBuffer and RelFileLocatorFilterCache have the same lifetime but
RelFileLocatorFilterCache is provided as the global variable; it is quite strange.
Can you somehow avoid this? I considered idea but could not because we do not have
APIs to Unregister the relcacheCallback.
4.
For output plugins which does not cache the catalog, it may not able to do filtering
without the transaction. For them, the early filter may degrade the performance
because it requires to open transactions for every changes.
Is my understanding correct?
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
v20250131-0001-Try-to-add-tests.txttext/plain; name=v20250131-0001-Try-to-add-tests.txtDownload
From 1fc88208de13fae8d287d456b899d4ebe01558be Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Fri, 31 Jan 2025 21:45:31 +0900
Subject: [PATCH v20250131] Try to add tests
---
contrib/test_decoding/Makefile | 2 +-
contrib/test_decoding/expected/filter.out | 26 +++++++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
contrib/test_decoding/sql/filter.sql | 16 ++++++++++++++
contrib/test_decoding/test_decoding.c | 21 ++++++++++++++++++
5 files changed, 65 insertions(+), 1 deletion(-)
create mode 100644 contrib/test_decoding/expected/filter.out
create mode 100644 contrib/test_decoding/sql/filter.sql
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index a4ba1a509a..a66ab2cbf8 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,7 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream filter
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error catalog_change_snapshot \
diff --git a/contrib/test_decoding/expected/filter.out b/contrib/test_decoding/expected/filter.out
new file mode 100644
index 0000000000..a3b43f2772
--- /dev/null
+++ b/contrib/test_decoding/expected/filter.out
@@ -0,0 +1,26 @@
+-- predictability
+SET synchronous_commit = on;
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+-- Create two tables
+CREATE TABLE test(id int);
+CREATE TABLE test_skipped(id int);
+-- Changes for XXX_skipped are skipped
+BEGIN;
+INSERT INTO test_skipped VALUES (1);
+COMMIT;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+------
+(0 rows)
+
+SELECT pg_drop_replication_slot('regression_slot');
+ pg_drop_replication_slot
+--------------------------
+
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index 54d65d3f30..9f1f98e6e2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -41,6 +41,7 @@ tests += {
'stats',
'twophase',
'twophase_stream',
+ 'filter',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/sql/filter.sql b/contrib/test_decoding/sql/filter.sql
new file mode 100644
index 0000000000..08ec80eb03
--- /dev/null
+++ b/contrib/test_decoding/sql/filter.sql
@@ -0,0 +1,16 @@
+-- predictability
+SET synchronous_commit = on;
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+-- Create two tables
+CREATE TABLE test(id int);
+CREATE TABLE test_skipped(id int);
+
+-- Changes for XXX_skipped are skipped
+BEGIN;
+INSERT INTO test_skipped VALUES (1);
+COMMIT;
+
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+SELECT pg_drop_replication_slot('regression_slot');
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 0113b19636..789a4bc2aa 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -68,6 +68,9 @@ static void pg_decode_truncate(LogicalDecodingContext *ctx,
ReorderBufferChange *change);
static bool pg_decode_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static bool pg_decode_filter_change(LogicalDecodingContext *ctx, Oid relid,
+ ReorderBufferChangeType change_type,
+ bool in_txn, bool *cache_valid);
static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr lsn,
bool transactional, const char *prefix,
@@ -133,6 +136,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->truncate_cb = pg_decode_truncate;
cb->commit_cb = pg_decode_commit_txn;
cb->filter_by_origin_cb = pg_decode_filter;
+ cb->filter_change_cb = pg_decode_filter_change;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
cb->filter_prepare_cb = pg_decode_filter_prepare;
@@ -467,6 +471,23 @@ pg_decode_filter(LogicalDecodingContext *ctx,
return false;
}
+static bool
+pg_decode_filter_change(LogicalDecodingContext *ctx, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn,
+ bool *cache_valid)
+{
+ if (in_txn)
+ {
+ Relation relation = RelationIdGetRelation(relid);
+ char *relname = NameStr(relation->rd_rel->relname);
+
+ if (strstr(relname, "_skipped") != NULL)
+ return true;
+ }
+
+ return false;
+}
+
/*
* Print literal `outputstr' already represented as string of type `typid'
* into stringbuf `s'.
--
2.43.5
On Tue, 28 Jan 2025 at 08:40, Ajin Cherian <itsajin@gmail.com> wrote:
Here's a patch-set created based on the design changes proposed by Hou-san.
Few comments:
1) Shouldn't we do the same thing for other DecodeXXX functions?
@@ -915,6 +915,11 @@ DecodeInsert(LogicalDecodingContext *ctx,
XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ if (ctx->reorder->can_filter_change &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder,
XLogRecGetXid(r),
+
buf->origptr, &target_locator, true))
+ return;
+
2) Let's add some debug logs so that it will be easy to verify the
changes that are getting filtered, or else we will have to debug and
verify them:
@@ -915,6 +915,11 @@ DecodeInsert(LogicalDecodingContext *ctx,
XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ if (ctx->reorder->can_filter_change &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder,
XLogRecGetXid(r),
+
buf->origptr, &target_locator, true))
+ return;
3) Also there are no tests currently, probably if we add the above
mentioned debug logs we could add few tests and verify them based on
the logs.
4) Can you elaborate a bit in the commit message why we need to
capture if a transaction has snapshot changes:
Subject: [PATCH v12 1/3] Track transactions with internal snapshot changes
Track transactions which have snapshot changes with a new flag
RBTXN_HAS_SNAPSHOT_CHANGES
Regards,
Vignesh
On Friday, January 31, 2025 9:43 PM Kuroda, Hayato/黒田 隼人 <kuroda.hayato@fujitsu.com>
Dear Ajin, Hou,
- Snapshot construction
I understand the approach that we do not try to filter for all the workloads; do
just best-effort.3.
Both ReorderBuffer and RelFileLocatorFilterCache have the same lifetime but
RelFileLocatorFilterCache is provided as the global variable; it is quite strange.
Can you somehow avoid this? I considered idea but could not because we do
not have APIs to Unregister the relcacheCallback.
I think we already have precedent of this (e.g., RelationSyncCache in pgoutput.c),
so, I am OK for it unless we come up with better ideas.
4.
For output plugins which does not cache the catalog, it may not able to do
filtering without the transaction. For them, the early filter may degrade the
performance because it requires to open transactions for every changes.
Is my understanding correct?
Partially correct. To be precise, I think we should recommend that output
plugin that cannot filter without transaction can avoid implementing this
callback in the first place, so that it would not cause degradation.
Best Regards,
Hou zj
On Wed, Jan 29, 2025 at 9:31 AM Peter Smith <smithpb2250@gmail.com> wrote:
Hi Ajin,
Some review comments for patch v12-0001.
======
Commit message
1.
Track transactions which have snapshot changes with a new flag
RBTXN_HAS_SNAPSHOT_CHANGES
~
The commit message only says *what* it does, but not *why* this patch
even exists. TBH, I don't understand why this patch needs to be
separated from your patch 0002, because 0001 makes no independent use
of the flag, nor is it separately tested.
Anyway, if it is going to remain separated then IMO at least the the
message should explain the intended purpose e.g. why the subsequent
patches require this flagged info and how they will use it.
Fixed.
======
src/include/replication/reorderbuffer.h
2.
+/* Does this transaction have snapshot changes? */
+#define rbtxn_has_snapshot_changes(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_HAS_SNAPSHOT_CHANGES) != 0 \
+)
+
Is the below wording maybe a more plain way to say that:
/* Does this transaction make changes to the current snapshot? */
Fixed as suggested.
On Sat, Feb 1, 2025 at 12:43 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
Dear Ajin, Hou,
- Snapshot construction
I understand the approach that we do not try to filter for all the workloads; do
just best-effort.
I played with your PoC and here are my comments.
1.
Can you add tests for better understanding? I've tried for
tes_decoding like attached,
but it couldn't pass the regression test. Cases "stream.sql" and
"twophase_stream.sql"
were failed.
I've added tests. I've analysed your patch and the regression test
failure. The tests fail to detect concurrent aborts.I think the reason
for that is because of the filter check logic access of the catalog,
the changes are cached.
As a result, when the actual decoding happens, the catalog is not
accessed as the relation detailsare in cache. Without catalog access,
concurrent aborts cannot be detected as concurrent aborts are
detectedonly when the catalog is accessed. There is a new thread by
Sawada-san on a more efficient detection of concurrent aborts,I don't
know if that will solve this issue, otherwise I don't know how to fix
this in a meaningful way. Caching improvesperformance, and at the same
time it prevents detection of concurrent aborts.
2.
I think we can extend the skip mechanism to
UPDATE/DELETE/MultiInsert/SpecConfirm.
Regarding the TRUNCATE, I'm not sure we can handle hte TRUNCATE case
because the we
can't track RelFileLocator anymore.
Updated.
On Tue, Feb 4, 2025 at 2:19 PM vignesh C <vignesh21@gmail.com> wrote:
On Tue, 28 Jan 2025 at 08:40, Ajin Cherian <itsajin@gmail.com> wrote:
Here's a patch-set created based on the design changes proposed
by Hou-san.
Few comments:
1) Shouldn't we do the same thing for other DecodeXXX functions?
@@ -915,6 +915,11 @@ DecodeInsert(LogicalDecodingContext *ctx,
XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ if (ctx->reorder->can_filter_change &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder,
XLogRecGetXid(r),
+
buf->origptr, &target_locator, true))
+ return;
+
Updated.
2) Let's add some debug logs so that it will be easy to verify the
changes that are getting filtered, or else we will have to debug and
verify them:
@@ -915,6 +915,11 @@ DecodeInsert(LogicalDecodingContext *ctx,
XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ if (ctx->reorder->can_filter_change &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder,
XLogRecGetXid(r),
+
buf->origptr, &target_locator, true))
+ return;
3) Also there are no tests currently, probably if we add the above
mentioned debug logs we could add few tests and verify them based on
the logs.
Updated.
4) Can you elaborate a bit in the commit message why we need to
capture if a transaction has snapshot changes:
Subject: [PATCH v12 1/3] Track transactions with internal snapshot changes
Track transactions which have snapshot changes with a new flag
RBTXN_HAS_SNAPSHOT_CHANGES
Updated.
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
v13-0003-Introduce-a-output-plugin-callback-to-filter-cha.patchapplication/octet-stream; name=v13-0003-Introduce-a-output-plugin-callback-to-filter-cha.patchDownload
From 944993771427e94d8e74411303c876b61d757bc9 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Tue, 11 Feb 2025 23:49:50 -0500
Subject: [PATCH v13 3/3] Introduce a output plugin callback to filter changes
This new output plugin callback provides an option to logical decoding plugins to filter out
changes early. The primary purpose of the callback is to conserve memory and processing cycles
by excluding changes that are not required by output plugins.
---
doc/src/sgml/logicaldecoding.sgml | 41 ++++++++++++++-
src/backend/replication/logical/decode.c | 70 ++++++++++++++++++++++++-
src/backend/replication/logical/logical.c | 42 +++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 45 +++++++++++++---
src/backend/replication/pgoutput/pgoutput.c | 70 +++++++++++++++++++++++++
src/include/replication/output_plugin.h | 20 +++++++
src/include/replication/reorderbuffer.h | 8 +++
src/test/subscription/t/001_rep_changes.pl | 18 ++++++-
src/test/subscription/t/013_partition.pl | 7 +++
9 files changed, 308 insertions(+), 13 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 1c4ae38..a603c47 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -560,6 +560,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeCommitCB commit_cb;
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
+ LogicalDecodeFilterChangeCB filter_change_cb;
LogicalDecodeShutdownCB shutdown_cb;
LogicalDecodeFilterPrepareCB filter_prepare_cb;
LogicalDecodeBeginPrepareCB begin_prepare_cb;
@@ -582,8 +583,8 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
and <function>commit_cb</function> callbacks are required,
while <function>startup_cb</function>, <function>truncate_cb</function>,
<function>message_cb</function>, <function>filter_by_origin_cb</function>,
- and <function>shutdown_cb</function> are optional.
- If <function>truncate_cb</function> is not set but a
+ <function>shutdown_cb</function>, and <function>filter_change_cb</function>
+ are optional. If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
@@ -871,6 +872,42 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-filter-change">
+ <title>Change Filter Callback</title>
+
+ <para>
+ The optional <function>filter_change_cb</function> is called before a
+ change record is decoded to determine whether the change can be filtered
+ out.
+<programlisting>
+typedef bool (*LogicalDecodeFilterChangeCB) (struct LogicalDecodingContext *ctx,
+ Oid relid,
+ ReorderBufferChangeType change_type,
+ bool in_txn, bool *cache_valid);
+</programlisting>
+ To indicate that decoding can be skipped for the given change
+ <parameter>change_type</parameter>, return <literal>true</literal>;
+ <literal>false</literal> otherwise.
+ The <parameter>in_txn</parameter> parameter indicates whether the
+ callback is invoked within a transaction block.
+ When <parameter>in_txn</parameter> is false, and if making a decision to filter a change requires being inside a
+ transaction block, such as needing access to the catalog, set
+ <parameter>*cache_valid</parameter> to <literal>false</literal>.
+ This ensures that the callback will be reinvoked once a transaction block
+ starts. If a decision can be made immediately, set
+ <parameter>*cache_valid</parameter> to <literal>true</literal>.
+ </para>
+ <para>
+ The primary purpose of this callback function is to optimize memory usage
+ and processing efficiency by filtering out changes that are unnecessary for
+ output plugins. It enables output plugins to selectively process relevant
+ changes. Caching filtering decisions locally is recommended, as it enables
+ the callback to provide cached results without repeatedly initiating
+ transactions or querying the catalog. This approach minimizes overhead
+ and improves efficiency during the decoding process.
+ </para>
+ </sect3>
+
<sect3 id="logicaldecoding-output-plugin-message">
<title>Generic Message Callback</title>
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index c5f1083..fd6a177 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -576,6 +576,17 @@ FilterByOrigin(LogicalDecodingContext *ctx, RepOriginId origin_id)
}
/*
+ * Check if filtering changes before decoding is supported and we're not suppressing filter
+ * changes currently.
+ */
+static inline bool
+FilterChangeIsEnabled(LogicalDecodingContext *ctx)
+{
+ return (ctx->callbacks.filter_change_cb != NULL &&
+ ctx->reorder->can_filter_change);
+}
+
+/*
* Handle rmgr LOGICALMSG_ID records for LogicalDecodingProcessRecord().
*/
void
@@ -915,9 +926,16 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
- if (ctx->reorder->can_filter_change &&
+ /*
+ * When filtering changes, determine if the relation associated with the change
+ * can be skipped. This could be because the relation is unlogged or because
+ * the plugin has opted to exclude this relation from decoding.
+ */
+ if (FilterChangeIsEnabled(ctx) &&
ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, &target_locator, true))
+ buf->origptr, &target_locator,
+ REORDER_BUFFER_CHANGE_INSERT,
+ true))
return;
change = ReorderBufferGetChange(ctx->reorder);
@@ -970,6 +988,18 @@ DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ /*
+ * When filtering changes, determine if the relation associated with the change
+ * can be skipped. This could be because the relation is unlogged or because
+ * the plugin has opted to exclude this relation from decoding.
+ */
+ if (FilterChangeIsEnabled(ctx) &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, &target_locator,
+ REORDER_BUFFER_CHANGE_UPDATE,
+ true))
+ return;
+
change = ReorderBufferGetChange(ctx->reorder);
change->action = REORDER_BUFFER_CHANGE_UPDATE;
change->origin_id = XLogRecGetOrigin(r);
@@ -1036,6 +1066,18 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ /*
+ * When filtering changes, determine if the relation associated with the change
+ * can be skipped. This could be because the relation is unlogged or because
+ * the plugin has opted to exclude this relation from decoding.
+ */
+ if (FilterChangeIsEnabled(ctx) &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, &target_locator,
+ REORDER_BUFFER_CHANGE_DELETE,
+ true))
+ return;
+
change = ReorderBufferGetChange(ctx->reorder);
if (xlrec->flags & XLH_DELETE_IS_SUPER)
@@ -1139,6 +1181,18 @@ DecodeMultiInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
return;
/*
+ * When filtering changes, determine if the relation associated with the change
+ * can be skipped. This could be because the relation is unlogged or because
+ * the plugin has opted to exclude this relation from decoding.
+ */
+ if (FilterChangeIsEnabled(ctx) &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, &rlocator,
+ REORDER_BUFFER_CHANGE_INSERT,
+ true))
+ return;
+
+ /*
* We know that this multi_insert isn't for a catalog, so the block should
* always have data even if a full-page write of it is taken.
*/
@@ -1231,6 +1285,18 @@ DecodeSpecConfirm(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ /*
+ * When filtering changes, determine if the relation associated with the change
+ * can be skipped. This could be because the relation is unlogged or because
+ * the plugin has opted to exclude this relation from decoding.
+ */
+ if (FilterChangeIsEnabled(ctx) &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, &target_locator,
+ REORDER_BUFFER_CHANGE_INSERT,
+ true))
+ return;
+
change = ReorderBufferGetChange(ctx->reorder);
change->action = REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM;
change->origin_id = XLogRecGetOrigin(r);
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8ea846b..635e1de 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -74,6 +74,9 @@ static void truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+static bool filter_change_cb_wrapper(ReorderBuffer *cache, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn,
+ bool *cache_valid);
/* streaming callbacks */
static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
@@ -220,6 +223,7 @@ StartupDecodingContext(List *output_plugin_options,
/* wrap output plugin callbacks, so we can add error context information */
ctx->reorder->begin = begin_cb_wrapper;
ctx->reorder->apply_change = change_cb_wrapper;
+ ctx->reorder->filter_change = filter_change_cb_wrapper;
ctx->reorder->apply_truncate = truncate_cb_wrapper;
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
@@ -1224,6 +1228,44 @@ filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, RepOriginId origin_id)
return ret;
}
+static bool
+filter_change_cb_wrapper(ReorderBuffer *cache, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn,
+ bool *cache_valid)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+ bool ret;
+
+ Assert(!ctx->fast_forward);
+
+ /* check if the filter change callback is supported */
+ if (ctx->callbacks.filter_change_cb == NULL)
+ return false;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "filter_change";
+ state.report_location = InvalidXLogRecPtr;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = false;
+ ctx->end_xact = false;
+
+ /* do the actual work: call callback */
+ ret = ctx->callbacks.filter_change_cb(ctx, relid, change_type, in_txn, cache_valid);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
+ return ret;
+}
+
static void
message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 7be0033..9a1dfb6 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -5549,9 +5549,11 @@ ReorderBufferMemoryResetcallback(void *arg)
bool
ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, RelFileLocator *rlocator,
+ ReorderBufferChangeType change_type,
bool has_tuple)
{
bool found;
+ bool cache_valid;
Relation relation;
bool using_subtxn;
Snapshot snapshot_now;
@@ -5565,7 +5567,6 @@ ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
Assert(txn);
toptxn = rbtxn_get_toptxn(txn);
- rb->can_filter_change = false;
/*
* We cannot construct an accurate historical snapshot until all pending
@@ -5582,7 +5583,25 @@ ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
if (found)
{
rb->can_filter_change = entry->filterable;
- return entry->filterable;
+
+ /*
+ * Quick return if we already know that the relation is not to be decoded.
+ * These are for special relations that are unlogged and for sequences
+ * and catalogs.
+ */
+ if (entry->filterable)
+ return true;
+
+ /* Allow the output plugin to filter relations */
+ rb->can_filter_change = rb->filter_change(rb, entry->relid, change_type,
+ false, &cache_valid);
+
+ /*
+ * If plugin had the relation ready in cache, the response is valid, else
+ * we'll need to call the plugin a second time within a transaction.
+ */
+ if (cache_valid)
+ return rb->can_filter_change;
}
/* constructs a temporary historical snapshot */
@@ -5606,18 +5625,30 @@ ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
if (RelationIsValid(relation))
{
- entry->relid = RelationGetRelid(relation);
+ if (IsToastRelation(relation))
+ {
+ Oid real_reloid = InvalidOid;
+ char *toast_name = RelationGetRelationName(relation);
+ /* pg_toast_ len is 9 */
+ char *start_ch = &toast_name[9];
+
+ real_reloid = pg_strtoint32(start_ch);
+ entry->relid = real_reloid;
+ }
+ else
+ entry->relid = RelationGetRelid(relation);
+
entry->filterable = false;
+ rb->can_filter_change = rb->filter_change(rb, entry->relid, change_type,
+ true, &cache_valid);
RelationClose(relation);
}
else
{
entry->relid = InvalidOid;
- entry->filterable = true;
+ rb->can_filter_change = entry->filterable = true;
}
- rb->can_filter_change = entry->filterable;
-
ReorderBufferFreeSnap(rb, snapshot_now);
TeardownHistoricSnapshot(false);
@@ -5633,5 +5664,5 @@ ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- return entry->filterable;
+ return rb->can_filter_change;
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 7d464f6..93eb579 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -57,6 +57,9 @@ static void pgoutput_message(LogicalDecodingContext *ctx,
Size sz, const char *message);
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static bool pgoutput_filter_change(LogicalDecodingContext *ctx, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn,
+ bool *cache_valid);
static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn);
static void pgoutput_prepare_txn(LogicalDecodingContext *ctx,
@@ -268,6 +271,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_prepared_cb = pgoutput_commit_prepared_txn;
cb->rollback_prepared_cb = pgoutput_rollback_prepared_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
+ cb->filter_change_cb = pgoutput_filter_change;
cb->shutdown_cb = pgoutput_shutdown;
/* transaction streaming */
@@ -1744,6 +1748,72 @@ pgoutput_origin_filter(LogicalDecodingContext *ctx,
}
/*
+ * Determine whether a change to the specified relation should be published.
+ *
+ * See the comments atop of LogicalDecodeFilterChangeCB for details.
+ */
+static bool
+pgoutput_filter_change(LogicalDecodingContext *ctx, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn, bool *cache_valid)
+{
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ if (in_txn)
+ {
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+ entry = get_rel_sync_entry(data, relation);
+ *cache_valid = true;
+ }
+ else
+ {
+ entry = (RelationSyncEntry *) hash_search(RelationSyncCache,
+ &relid,
+ HASH_FIND, cache_valid);
+ if (!*cache_valid)
+ return false;
+ }
+
+ /*
+ * If the pubaction is not supported by this publication then return true to say
+ * the change for this entry can be skipped.
+ */
+ switch (change_type)
+ {
+ case REORDER_BUFFER_CHANGE_INSERT:
+ if (!entry->pubactions.pubinsert)
+ {
+ elog(DEBUG1, "Filtering INSERT");
+ return true;
+ }
+ break;
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ if (!entry->pubactions.pubupdate)
+ {
+ elog(DEBUG1, "Filtering UPDATE");
+ return true;
+ }
+ break;
+ case REORDER_BUFFER_CHANGE_DELETE:
+ if (!entry->pubactions.pubdelete)
+ {
+ elog(DEBUG1, "Filtering DELETE");
+ return true;
+ }
+ break;
+ default:
+ /* allow any other changes that are not explicitly filtered */
+ return false;
+ }
+
+ return false;
+}
+
+/*
* Shutdown the output plugin.
*
* Note, we don't need to clean the data->context, data->cachectx, and
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 8d4d5b71..f82bc1d 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -97,6 +97,25 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
RepOriginId origin_id);
/*
+ * This callback is called before a change record is decoded to determine
+ * whether the change can be filtered out before entering the reorder buffer.
+ *
+ * Typically, the output plugin needs to refer to the system catalog to make
+ * this decision. To enhance efficiency, the plugin should cache the lookup
+ * result for each relation, minimizing catalog access on subsequent calls.
+ *
+ * If the callback is called with 'in_txn' set as false (the reorder
+ * buffer has not started a transaction), the output plugin should set '*cache_valid'
+ * to false to indicate that the result is not available in its internal cache.
+ * If 'in_txn' is true, the plugin can create a cache entry after querying the
+ * catalog.
+ */
+typedef bool (*LogicalDecodeFilterChangeCB) (struct LogicalDecodingContext *ctx,
+ Oid relid,
+ ReorderBufferChangeType change_type,
+ bool in_txn, bool *cache_valid);
+
+/*
* Called to shutdown an output plugin.
*/
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
@@ -222,6 +241,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeCommitCB commit_cb;
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
+ LogicalDecodeFilterChangeCB filter_change_cb;
LogicalDecodeShutdownCB shutdown_cb;
/* streaming of changes at prepare time */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7d805d4..0b83ef4 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -468,6 +468,12 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* filter change callback signature */
+typedef bool (*ReorderBufferFilterChangeCB) (ReorderBuffer *rb,
+ Oid relid,
+ ReorderBufferChangeType change_type,
+ bool in_txn, bool *cache_valid);
+
/* begin prepare callback signature */
typedef void (*ReorderBufferBeginPrepareCB) (ReorderBuffer *rb,
ReorderBufferTXN *txn);
@@ -578,6 +584,7 @@ struct ReorderBuffer
*/
ReorderBufferBeginCB begin;
ReorderBufferApplyChangeCB apply_change;
+ ReorderBufferFilterChangeCB filter_change;
ReorderBufferApplyTruncateCB apply_truncate;
ReorderBufferCommitCB commit;
ReorderBufferMessageCB message;
@@ -751,6 +758,7 @@ extern void StartupReorderBuffer(void);
extern bool ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, RelFileLocator *rlocator,
+ ReorderBufferChangeType change_type,
bool has_tuple);
extern bool ReorderBufferCanFilterChanges(ReorderBuffer *rb);
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 8726fe0..bced62c 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -485,9 +485,10 @@ $node_publisher->wait_for_catchup('tap_sub');
# Check that we don't send BEGIN and COMMIT because of empty transaction
# optimization. We have to look for the DEBUG1 log messages about that, so
-# temporarily bump up the log verbosity.
+# temporarily bump up the log verbosity. Also confirm that unpublished
+# changes are filtered out after a restart.
$node_publisher->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_publisher->reload;
+$node_publisher->restart;
# Note that the current location of the log file is not grabbed immediately
# after reloading the configuration, but after sending one SQL command to
@@ -495,6 +496,8 @@ $node_publisher->reload;
$log_location = -s $node_publisher->logfile;
$node_publisher->safe_psql('postgres', "INSERT INTO tab_notrep VALUES (11)");
+$node_publisher->safe_psql('postgres', "UPDATE tab_notrep SET a = 2 WHERE a = 1");
+$node_publisher->safe_psql('postgres', "DELETE FROM tab_notrep WHERE a = 2");
$node_publisher->wait_for_catchup('tap_sub');
@@ -502,6 +505,17 @@ $logfile = slurp_file($node_publisher->logfile, $log_location);
ok($logfile =~ qr/skipped replication of an empty transaction with XID/,
'empty transaction is skipped');
+# Check that an unpublished change is filtered out.
+$logfile = slurp_file($node_publisher->logfile, $log_location);
+ok($logfile =~ qr/Filtering INSERT/,
+ 'unpublished INSERT is filtered');
+
+ok($logfile =~ qr/Filtering UPDATE/,
+ 'unpublished UPDATE is filtered');
+
+ok($logfile =~ qr/Filtering DELETE/,
+ 'unpublished DELETE is filtered');
+
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM tab_notrep");
is($result, qq(0), 'check non-replicated table is empty on subscriber');
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 14a3bea..180eaa0 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -468,6 +468,13 @@ $node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab4 (a int PRIMARY KEY)");
$node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab4_1 (a int PRIMARY KEY)");
+
+# Ensure that the subscription 'sub2' catches up with the latest changes. This
+# is necessary for the walsender to update its historic snapshot. Otherwise,
+# the walsender might retain an outdated snapshot, potentially preventing it
+# from accessing the newly created publication.
+$node_publisher->wait_for_catchup('sub2');
+
# Since we specified publish_via_partition_root in pub_all and
# pub_lower_level, all partition tables use their root tables' identity and
# schema. We set the list of publications so that the FOR ALL TABLES
--
1.8.3.1
v13-0001-Track-transactions-with-internal-snapshot-change.patchapplication/octet-stream; name=v13-0001-Track-transactions-with-internal-snapshot-change.patchDownload
From fec9567d30f333fb1b4b94bca4b7289146beadbc Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Tue, 11 Feb 2025 03:25:34 -0500
Subject: [PATCH v13 1/3] Track transactions with internal snapshot changes
Track transactions that make changes to the current snapshot with a new flag
RBTXN_HAS_SNAPSHOT_CHANGES. This will allow logical decoding to accumulate changes
which modify the snapshot before creating a historic snaphot for a transaction.
---
src/backend/replication/logical/reorderbuffer.c | 11 +++++++++++
src/include/replication/reorderbuffer.h | 7 +++++++
2 files changed, 18 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 10a3766..3d22aa9 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -823,6 +823,14 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
toptxn->txn_flags |= RBTXN_HAS_STREAMABLE_CHANGE;
}
+ else if (change->action == REORDER_BUFFER_CHANGE_INVALIDATION ||
+ change->action == REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT ||
+ change->action == REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID)
+ {
+ ReorderBufferTXN *toptxn = rbtxn_get_toptxn(txn);
+
+ toptxn->txn_flags |= RBTXN_HAS_SNAPSHOT_CHANGES;
+ }
change->lsn = lsn;
change->txn = txn;
@@ -1747,6 +1755,9 @@ ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn, bool txn_prep
txn->txn_flags |= RBTXN_IS_SERIALIZED_CLEAR;
}
+ /* All snapshot changes up to this point have been processed. */
+ txn->txn_flags &= ~RBTXN_HAS_SNAPSHOT_CHANGES;
+
/* also reset the number of entries in the transaction */
txn->nentries_mem = 0;
txn->nentries = 0;
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index a669658..b0f26c7 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_PREPARE 0x0040
#define RBTXN_SKIPPED_PREPARE 0x0080
#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_HAS_SNAPSHOT_CHANGES 0x0200
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -210,6 +211,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_HAS_STREAMABLE_CHANGE) != 0 \
)
+/* Does this transaction make changes to the current snapshot? */
+#define rbtxn_has_snapshot_changes(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_HAS_SNAPSHOT_CHANGES) != 0 \
+)
+
/*
* Has this transaction been streamed to downstream?
*
--
1.8.3.1
v13-0002-Filter-transactions-that-need-not-be-published.patchapplication/octet-stream; name=v13-0002-Filter-transactions-that-need-not-be-published.patchDownload
From cb1e99a1bea946f4da3f8ae6b0068d2464adfafa Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Tue, 11 Feb 2025 04:07:21 -0500
Subject: [PATCH v13 2/3] Filter transactions that need not be published
This adds logic to filter transactions early (at decode time, rather than at streaming time) so
they can be skipped if they do not contain tables that are part of the publication list of the
logical replication walsender.
Determining whether to filter a change requires information about the relation
and the publication from the catalog, requiring a transaction to be started.
When most changes in a transaction are unfilterable, the overhead of starting a
transaction for each record is significant. To reduce this overhead a hash cache of relation file
locators is created. Even then a hash search for every record before recording has considerable
overhead especially for use cases where most tables in an instance are published. To further reduce
this overhead a simple approach is used to suspend filtering for a certain number of changes
(100) when an unfilterable change is encountered. In other words, continue filtering changes if
the last record was filtered out. If an unfilterable change is found, skip filtering the next 100
changes.
---
src/backend/replication/logical/decode.c | 5 +
src/backend/replication/logical/reorderbuffer.c | 423 ++++++++++++++++++++----
src/include/replication/reorderbuffer.h | 12 +
src/tools/pgindent/typedefs.list | 2 +
4 files changed, 374 insertions(+), 68 deletions(-)
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 0bff0f1..c5f1083 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -915,6 +915,11 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ if (ctx->reorder->can_filter_change &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, &target_locator, true))
+ return;
+
change = ReorderBufferGetChange(ctx->reorder);
if (!(xlrec->flags & XLH_INSERT_IS_SPECULATIVE))
change->action = REORDER_BUFFER_CHANGE_INSERT;
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3d22aa9..7be0033 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -108,6 +108,7 @@
#include "storage/fd.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
@@ -226,8 +227,10 @@ static ReorderBufferTXN *ReorderBufferTXNByXid(ReorderBuffer *rb,
XLogRecPtr lsn, bool create_as_top);
static void ReorderBufferTransferSnapToParent(ReorderBufferTXN *txn,
ReorderBufferTXN *subtxn);
-
static void AssertTXNLsnOrder(ReorderBuffer *rb);
+static Relation ReorderBufferGetRelation(ReorderBuffer *rb,
+ RelFileLocator *rlocator,
+ bool has_tuple);
/* ---------------------------------------
* support functions for lsn-order iterating over the ->changes of a
@@ -276,6 +279,8 @@ static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
*/
static inline bool ReorderBufferCanStream(ReorderBuffer *rb);
static inline bool ReorderBufferCanStartStreaming(ReorderBuffer *rb);
+static Snapshot ReorderBufferStreamTXNSnapshot(ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
@@ -302,6 +307,48 @@ static void ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition, Size sz);
/*
+ * ---------------------------------------
+ * RelFileLocator filtering
+ * ---------------------------------------
+ * This hash table serves as a lookup table for determining if a relation can
+ * be filtered before being decoded and queued into the buffer.
+ *
+ * The hash table shares the same lifespan as the reorder buffer. This is
+ * crucial because each reorderbuffer may have different configurations or be
+ * associated with different output plugins, affecting the types of tables to
+ * be processed.
+ */
+
+static HTAB *RelFileLocatorFilterCache = NULL;
+
+static bool relation_callbacks_registered = false;
+
+typedef struct ReorderBufferRelFileLocatorKey
+{
+ Oid reltablespace;
+ RelFileNumber relfilenumber;
+} ReorderBufferRelFileLocatorKey;
+
+/* entry for hash table we use to track if the relation can be filtered */
+typedef struct ReorderBufferRelFileLocatorEnt
+{
+ ReorderBufferRelFileLocatorKey key;
+ Oid relid;
+ bool filterable;
+} ReorderBufferRelFileLocatorEnt;
+
+static void RelFileLocatorCacheInvalidateCallback(Datum arg, Oid relid);
+static void ReorderBufferMemoryResetcallback(void *arg);
+
+/*
+ * After encountering a change that cannot be filtered out, filtering is
+ * temporarily suspended. Filtering resumes after processing every 100 changes.
+ * This strategy helps to minimize the overhead of performing a hash table
+ * search for each record, especially when most changes are not filterable.
+ */
+#define CHANGES_THRESHOLD_FOR_FILTER 100
+
+/*
* Allocate a new ReorderBuffer and clean out any old serialized state from
* prior ReorderBuffer instances for the same slot.
*/
@@ -359,6 +406,34 @@ ReorderBufferAllocate(void)
buffer->by_txn = hash_create("ReorderBufferByXid", 1000, &hash_ctl,
HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ /*
+ * To support early filtering of changes, this hash table acts
+ * as a lookup table to determine if the corresponding relation is
+ * required to be decoded and queued into the buffer. The hash table
+ * shares the same lifespan as the reorder buffer.
+ *
+ * Also setup the callback to invalidate cache when relations are updated.
+ */
+ hash_ctl.keysize = sizeof(ReorderBufferRelFileLocatorKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferRelFileLocatorEnt);
+ hash_ctl.hcxt = buffer->context;
+
+ RelFileLocatorFilterCache =
+ hash_create("RelFileLocatorFilterCache", 1000, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ buffer->relfile_callback.arg = NULL;
+ buffer->relfile_callback.func = ReorderBufferMemoryResetcallback;
+ MemoryContextRegisterResetCallback(buffer->context, &buffer->relfile_callback);
+
+ if (!relation_callbacks_registered)
+ {
+ /* Watch for invalidation events. */
+ CacheRegisterRelcacheCallback(RelFileLocatorCacheInvalidateCallback,
+ (Datum) 0);
+ relation_callbacks_registered = true;
+ }
+
buffer->by_txn_last_xid = InvalidTransactionId;
buffer->by_txn_last_txn = NULL;
@@ -369,6 +444,8 @@ ReorderBufferAllocate(void)
/* txn_heap is ordered by transaction size */
buffer->txn_heap = pairingheap_allocate(ReorderBufferTXNSizeCompare, NULL);
+ buffer->can_filter_change = true;
+
buffer->spillTxns = 0;
buffer->spillCount = 0;
buffer->spillBytes = 0;
@@ -840,6 +917,16 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn->nentries++;
txn->nentries_mem++;
+ /*
+ * If we're not filtering and we've crossed the change threshold,
+ * attempt to filter again
+ */
+ if (!rb->can_filter_change && ++rb->processed_changes >= CHANGES_THRESHOLD_FOR_FILTER)
+ {
+ rb->can_filter_change = true;
+ rb->processed_changes = 0;
+ }
+
/* update memory accounting information */
ReorderBufferChangeMemoryUpdate(rb, change, NULL, true,
ReorderBufferChangeSize(change));
@@ -2105,6 +2192,81 @@ ReorderBufferResetTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
+ * Return the relation corresponding to the given RelFileLocator 'rlocator' if
+ * relevant to decoding. Otherwise, return NULL.
+ *
+ * We don't decode catalog files, relations that are not logically logged,
+ * temporary heaps and sequences.
+ */
+static Relation
+ReorderBufferGetRelation(ReorderBuffer *rb, RelFileLocator *rlocator,
+ bool has_tuple)
+{
+ bool filterable = false;
+ Relation relation;
+ Oid reloid;
+
+ reloid = RelidByRelfilenumber(rlocator->spcOid, rlocator->relNumber);
+
+ if (!OidIsValid(reloid))
+ {
+ if (!has_tuple)
+ {
+ /*
+ * Mapped catalog tuple without data, emitted while catalog table was in
+ * the process of being rewritten. We can fail to look up the
+ * relfilenumber, because the relmapper has no "historic" view, in
+ * contrast to the normal catalog during decoding. Thus repeated rewrites
+ * can cause a lookup failure. That's OK because we do not decode catalog
+ * changes anyway. Normally such tuples would be skipped over below, but
+ * we can't identify whether the table should be logically logged without
+ * mapping the relfilenumber to the oid.
+ */
+ return NULL;
+ }
+
+ elog(ERROR, "could not map filenumber \"%s\" to relation OID",
+ relpathperm(*rlocator, MAIN_FORKNUM));
+ }
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (!RelationIsValid(relation))
+ elog(ERROR, "could not open relation with OID %u (for filenumber \"%s\")",
+ reloid, relpathperm(*rlocator, MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ filterable = true;
+
+ else if (relation->rd_rel->relrewrite && !rb->output_rewrites)
+ {
+ /*
+ * Ignore temporary heaps created during DDL unless the plugin has asked
+ * for them.
+ */
+ filterable = true;
+ }
+
+ else if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ {
+ /*
+ * For now ignore sequence changes entirely. Most of the time they don't
+ * log changes using records we understand, so it doesn't make sense to
+ * handle the few cases we do.
+ */
+ filterable = true;
+ }
+
+ if (filterable)
+ {
+ RelationClose(relation);
+ return NULL;
+ }
+
+ return relation;
+}
+
+/*
* Helper function for ReorderBufferReplay and ReorderBufferStreamTXN.
*
* Send data of a transaction (and its subtransactions) to the
@@ -2176,7 +2338,6 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
{
Relation relation = NULL;
- Oid reloid;
CHECK_FOR_INTERRUPTS();
@@ -2235,55 +2396,11 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_DELETE:
Assert(snapshot_now);
- reloid = RelidByRelfilenumber(change->data.tp.rlocator.spcOid,
- change->data.tp.rlocator.relNumber);
-
- /*
- * Mapped catalog tuple without data, emitted while
- * catalog table was in the process of being rewritten. We
- * can fail to look up the relfilenumber, because the
- * relmapper has no "historic" view, in contrast to the
- * normal catalog during decoding. Thus repeated rewrites
- * can cause a lookup failure. That's OK because we do not
- * decode catalog changes anyway. Normally such tuples
- * would be skipped over below, but we can't identify
- * whether the table should be logically logged without
- * mapping the relfilenumber to the oid.
- */
- if (reloid == InvalidOid &&
- change->data.tp.newtuple == NULL &&
- change->data.tp.oldtuple == NULL)
- goto change_done;
- else if (reloid == InvalidOid)
- elog(ERROR, "could not map filenumber \"%s\" to relation OID",
- relpathperm(change->data.tp.rlocator,
- MAIN_FORKNUM));
-
- relation = RelationIdGetRelation(reloid);
+ relation = ReorderBufferGetRelation(rb, &change->data.tp.rlocator,
+ (change->data.tp.newtuple != NULL ||
+ change->data.tp.oldtuple != NULL));
if (!RelationIsValid(relation))
- elog(ERROR, "could not open relation with OID %u (for filenumber \"%s\")",
- reloid,
- relpathperm(change->data.tp.rlocator,
- MAIN_FORKNUM));
-
- if (!RelationIsLogicallyLogged(relation))
- goto change_done;
-
- /*
- * Ignore temporary heaps created during DDL unless the
- * plugin has asked for them.
- */
- if (relation->rd_rel->relrewrite && !rb->output_rewrites)
- goto change_done;
-
- /*
- * For now ignore sequence changes entirely. Most of the
- * time they don't log changes using records we
- * understand, so it doesn't make sense to handle the few
- * cases we do.
- */
- if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
goto change_done;
/* user-triggered change */
@@ -4058,19 +4175,13 @@ ReorderBufferCanStartStreaming(ReorderBuffer *rb)
}
/*
- * Send data of a large transaction (and its subtransactions) to the
- * output plugin, but using the stream API.
+ * This function generates or retrieves a consistent snapshot of a transaction
+ * that is currently in progress for logical replication.
*/
-static void
-ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+static Snapshot
+ReorderBufferStreamTXNSnapshot(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
Snapshot snapshot_now;
- CommandId command_id;
- Size stream_bytes;
- bool txn_is_streamed;
-
- /* We can never reach here for a subtransaction. */
- Assert(rbtxn_is_toptxn(txn));
/*
* We can't make any assumptions about base snapshot here, similar to what
@@ -4113,12 +4224,11 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (txn->base_snapshot == NULL)
{
Assert(txn->ninvalidations == 0);
- return;
+ return NULL;
}
- command_id = FirstCommandId;
snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
- txn, command_id);
+ txn, FirstCommandId);
}
else
{
@@ -4131,17 +4241,40 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* the LSN condition in the previous branch (so no need to walk
* through subxacts again). In fact, we must not do that as we may be
* using snapshot half-way through the subxact.
- */
- command_id = txn->command_id;
-
- /*
+ *
* We can't use txn->snapshot_now directly because after the last
* streaming run, we might have got some new sub-transactions. So we
* need to add them to the snapshot.
*/
snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
- txn, command_id);
+ txn, txn->command_id);
+ }
+
+ return snapshot_now;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Snapshot snapshot_now;
+ Size stream_bytes;
+ bool txn_is_streamed;
+
+ /* We can never reach here for a subtransaction. */
+ Assert(rbtxn_is_toptxn(txn));
+
+ snapshot_now = ReorderBufferStreamTXNSnapshot(rb, txn);
+
+ /* This transaction didn't make any changes to the database till now. */
+ if (snapshot_now == NULL)
+ return;
+ if (txn->snapshot_now != NULL)
+ {
/* Free the previously copied snapshot. */
Assert(txn->snapshot_now->copied);
ReorderBufferFreeSnap(rb, txn->snapshot_now);
@@ -4159,7 +4292,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* Process and send the changes to output plugin. */
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
- command_id, true);
+ snapshot_now->curcid, true);
rb->streamCount += 1;
rb->streamBytes += stream_bytes;
@@ -5348,3 +5481,157 @@ restart:
*cmax = ent->cmax;
return true;
}
+
+/*
+ * Flush mapping entries when the corresponding relations are updated in pg_class.
+ */
+static void
+RelFileLocatorCacheInvalidateCallback(Datum arg, Oid relid)
+{
+ HASH_SEQ_STATUS status;
+ ReorderBufferRelFileLocatorEnt *entry;
+
+ if (!RelFileLocatorFilterCache)
+ return;
+
+ hash_seq_init(&status, RelFileLocatorFilterCache);
+
+ /* slightly inefficient algorithm but not performance critical path */
+ while ((entry = (ReorderBufferRelFileLocatorEnt *) hash_seq_search(&status)) != NULL)
+ {
+ /*
+ * If relid is InvalidOid, signaling a complete reset, we must remove
+ * all entries, otherwise just remove the specific relation's entry.
+ * Always remove negative cache entries.
+ */
+ if (relid == InvalidOid || /* complete reset */
+ entry->relid == InvalidOid || /* negative cache entry */
+ entry->relid == relid) /* individual flushed relation */
+ {
+ if (hash_search(RelFileLocatorFilterCache,
+ &entry->key,
+ HASH_REMOVE,
+ NULL) == NULL)
+ elog(ERROR, "hash table corrupted");
+ }
+ }
+}
+
+/*
+ * Context reset/delete callback for RelFileLocatorFilterCache.
+ */
+static void
+ReorderBufferMemoryResetcallback(void *arg)
+{
+ RelFileLocatorFilterCache = NULL;
+}
+
+/*
+ * Determine whether the record corresponding to the relation identified by
+ * 'rlocator' needs to be filtered and not decoded and queued in the buffer.
+ *
+ * Determining the necessity of decoding requires accessing relation
+ * information from the system catalog, which requires a historical snapshot.
+ * We cannot directly use the current snapshot of the transaction due to the
+ * presence of INTERNAL_SNAPSHOT, COMMAND_ID, or INVALIDATION records in the
+ * buffer that could modify the snapshot. A proper snapshot can only be
+ * constructed after these records are processed in ReorderBufferProcessTXN, or
+ * if we are decoding a transaction without these records. See comment on top
+ * of ReorderBufferGetRelation() to see list of relations that are not
+ * decoded by the reorderbuffer.
+ *
+ * To reduce the overhead from the system catalog access and transaction
+ * management, the results are cached in the 'RelFileLocatorFilterCache' hash
+ * table.
+ *
+ * Returns true if the relation can be filtered; otherwise, false.
+ */
+bool
+ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, RelFileLocator *rlocator,
+ bool has_tuple)
+{
+ bool found;
+ Relation relation;
+ bool using_subtxn;
+ Snapshot snapshot_now;
+ ReorderBufferTXN *txn,
+ *toptxn;
+ ReorderBufferRelFileLocatorEnt *entry;
+ ReorderBufferRelFileLocatorKey key;
+
+ Assert(RelFileLocatorFilterCache);
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+ Assert(txn);
+ toptxn = rbtxn_get_toptxn(txn);
+ rb->can_filter_change = false;
+
+ /*
+ * We cannot construct an accurate historical snapshot until all pending
+ * records in this transaction that might update the snapshot are
+ * processed.
+ */
+ if (rbtxn_has_snapshot_changes(toptxn))
+ return false;
+
+ key.reltablespace = rlocator->spcOid;
+ key.relfilenumber = rlocator->relNumber;
+ entry = hash_search(RelFileLocatorFilterCache, &key, HASH_ENTER, &found);
+
+ if (found)
+ {
+ rb->can_filter_change = entry->filterable;
+ return entry->filterable;
+ }
+
+ /* constructs a temporary historical snapshot */
+ snapshot_now = ReorderBufferStreamTXNSnapshot(rb, toptxn);
+ Assert(snapshot_now);
+
+ /* build data to be able to lookup the CommandIds of catalog tuples */
+ ReorderBufferBuildTupleCidHash(rb, toptxn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, toptxn->tuplecid_hash);
+
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("filter change by RelFileLocator");
+ else
+ StartTransactionCommand();
+
+ relation = ReorderBufferGetRelation(rb, rlocator, has_tuple);
+
+ if (RelationIsValid(relation))
+ {
+ entry->relid = RelationGetRelid(relation);
+ entry->filterable = false;
+ RelationClose(relation);
+ }
+ else
+ {
+ entry->relid = InvalidOid;
+ entry->filterable = true;
+ }
+
+ rb->can_filter_change = entry->filterable;
+
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access have
+ * persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ return entry->filterable;
+}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index b0f26c7..7d805d4 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -622,6 +622,7 @@ struct ReorderBuffer
* Private memory context.
*/
MemoryContext context;
+ MemoryContextCallback relfile_callback;
/*
* Memory contexts for specific types objects
@@ -642,6 +643,12 @@ struct ReorderBuffer
/* Max-heap for sizes of all top-level and sub transactions */
pairingheap *txn_heap;
+ /* should we try to filter the change? */
+ bool can_filter_change;
+
+ /* number of changes after a failed attempt at filtering */
+ int8 processed_changes;
+
/*
* Statistics about transactions spilled to disk.
*
@@ -742,4 +749,9 @@ extern void ReorderBufferSetRestartPoint(ReorderBuffer *rb, XLogRecPtr ptr);
extern void StartupReorderBuffer(void);
+extern bool ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, RelFileLocator *rlocator,
+ bool has_tuple);
+extern bool ReorderBufferCanFilterChanges(ReorderBuffer *rb);
+
#endif
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 656ecd9..9b43262 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2433,6 +2433,8 @@ ReorderBufferIterTXNEntry
ReorderBufferIterTXNState
ReorderBufferMessageCB
ReorderBufferPrepareCB
+ReorderBufferRelFileLocatorEnt
+ReorderBufferRelFileLocatorKey
ReorderBufferRollbackPreparedCB
ReorderBufferStreamAbortCB
ReorderBufferStreamChangeCB
--
1.8.3.1
Hi Ajin,
Some review comments for the patch v13-0002.
======
src/backend/replication/logical/reorderbuffer.c
1. GENERAL
I felt that a summary/overview of how all this filter/cache logic
works should be given in the large file header comment at the top of
this file. There may be some overlap with comments that are already in
the "RelFileLocator filtering" section.
~~~
ReorderBufferRelFileLocatorEnt:
2.
+/* entry for hash table we use to track if the relation can be filtered */
+typedef struct ReorderBufferRelFileLocatorEnt
/* Hash table entry used to determine if the relation can be filtered. */
~~~
ReorderBufferQueueChange:
3.
+ /*
+ * If we're not filtering and we've crossed the change threshold,
+ * attempt to filter again
+ */
SUGGESTION
If filtering was suspended, and we've crossed the change threshold
then reenable filtering.
~~~
ReorderBufferGetRelation:
4.
+static Relation
+ReorderBufferGetRelation(ReorderBuffer *rb, RelFileLocator *rlocator,
+ bool has_tuple)
Would a better name be ReorderBufferGetRelationForDecoding(). Yeah,
it's a bit long but perhaps it explains the context/purpose better.
~~~
5.
+ if (filterable)
+ {
+ RelationClose(relation);
+ return NULL;
+ }
I wonder if some small descriptive comment would be helpful here just
to say we are returning NULL to indicate that this relation is not
needed and yada yada...
~~~
RelFileLocatorCacheInvalidateCallback:
6.
+ /* slightly inefficient algorithm but not performance critical path */
+ while ((entry = (ReorderBufferRelFileLocatorEnt *)
hash_seq_search(&status)) != NULL)
+ {
+ /*
+ * If relid is InvalidOid, signaling a complete reset, we must remove
+ * all entries, otherwise just remove the specific relation's entry.
+ * Always remove negative cache entries.
+ */
+ if (relid == InvalidOid || /* complete reset */
+ entry->relid == InvalidOid || /* negative cache entry */
+ entry->relid == relid) /* individual flushed relation */
6a.
Maybe uppercase that 1st comment.
~
6b.
It seems a bit unusual to be referring to "negative cache entries". I
thought it should be described in terms of InvalidOid since that is
what it is using in the condition.
~
6c.
If the relid parameter can take special values like "If relid is
InvalidOid, signaling a complete reset" that sounds like the kind of
thing that should be documented in the function comment.
~~~
ReorderBufferFilterByRelFileLocator
7.
Despite the extra indenting required, I wondered if the logic may be
easier to read (e.g. it shows the association of the
rb->can_filter_change and entry->filterable more clearly) if this is
refactored slightly by sharing a single common return like below:
BEFORE
...
+ key.reltablespace = rlocator->spcOid;
+ key.relfilenumber = rlocator->relNumber;
+ entry = hash_search(RelFileLocatorFilterCache, &key, HASH_ENTER, &found);
+
+ if (found)
+ {
+ rb->can_filter_change = entry->filterable;
+ return entry->filterable;
+ }
...
+ rb->can_filter_change = entry->filterable;
...
+ return entry->filterable;
+}
AFTER
...
+ key.reltablespace = rlocator->spcOid;
+ key.relfilenumber = rlocator->relNumber;
+ entry = hash_search(RelFileLocatorFilterCache, &key, HASH_ENTER, &found);
+
+ if (!found)
+ {
...
+ }
+
+ rb->can_filter_change = entry->filterable;
+ return entry->filterable;
+}
======
src/include/replication/reorderbuffer.h
8.
+ /* should we try to filter the change? */
+ bool can_filter_change;
+
I think most of my difficulty reading this patch was due to this field
name 'can_filter_change'.
'can_filter_change' somehow sounded to me like it is past tense. e.g.
like as if we already found some change and we yes, we CAN filter it.
But AFAICT the real meaning is simply that (when the flag is true) we
are ALLOWED to check to see if there is anything filterable. In fact,
the change may or may not be filterable.
Can this be renamed to something more "correct"? e.g.
- 'allow_change_filtering'
- 'enable_change_filtering'
- etc.
~~
9.
+ /* number of changes after a failed attempt at filtering */
+ int8 processed_changes;
Maybe 'unfiltered_changes_count' is a better name for this field?
~~~
10.
+extern bool ReorderBufferCanFilterChanges(ReorderBuffer *rb);
Should match the 'can_filter_change' field name, so if you change that
(see comment #8), then change this too.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
On Wed, Feb 12, 2025 at 10:41 AM Ajin Cherian <itsajin@gmail.com> wrote:
On Wed, Jan 29, 2025 at 9:31 AM Peter Smith <smithpb2250@gmail.com> wrote:
Hi Ajin,
Some review comments for patch v12-0001.
======
Commit message1.
Track transactions which have snapshot changes with a new flag
RBTXN_HAS_SNAPSHOT_CHANGES~
The commit message only says *what* it does, but not *why* this patch
even exists. TBH, I don't understand why this patch needs to be
separated from your patch 0002, because 0001 makes no independent use
of the flag, nor is it separately tested.Anyway, if it is going to remain separated then IMO at least the the
message should explain the intended purpose e.g. why the subsequent
patches require this flagged info and how they will use it.Fixed.
I still can't get from 0001's commit message the reason for tracking
the snapshot changes separately. Also, please find my comments for
0002's commit message.
When most changes in a transaction are unfilterable, the overhead of
starting a transaction for each record is significant.
Can you tell what is the exact overhead by testing it? IIRC, that was
the initial approach. It is better if you can mention in the commit
message what was overhead. It will be easier for reviewers.
To reduce this overhead a hash cache of relation file locators is
created. Even then a hash search for every record before recording has
considerable overhead especially for use cases where most tables in an
instance are published.
Again, can you share the link of performance data for this overhead
and if you have not published then please share it and also mention it
in commit message?
To further reduce this overhead a simple approach is used to suspend
filtering for a certain number of changes (100) when an unfilterable
change is encountered. In other words, continue filtering changes if
the last record was filtered out. If an unfilterable change is found,
skip filtering the next 100 changes.
Can we try different thresholds for this like 10, 50, 100, 200, etc.
to decide what is a good threshold value to skip filtering changes?
--
With Regards,
Amit Kapila.
Hi Ajin.
FYI - Patch set v13* no longer applies cleanly. Needs rebasing.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
On Mon, Feb 17, 2025 at 10:08 AM Peter Smith <smithpb2250@gmail.com> wrote:
Hi Ajin.
FYI - Patch set v13* no longer applies cleanly. Needs rebasing.
I've rebased the patch. I've also merged patch 1 into patch 2 as the
functionality of the changes in patch 1 are only n patch 2.
On Mon, Feb 17, 2025 at 10:08 AM Peter Smith <smithpb2250@gmail.com> wrote:
Hi Ajin.
FYI - Patch set v13* no longer applies cleanly. Needs rebasing.
I've rebased the patch. I've also merged patch 1 into patch 2 as the
functionality of the changes in patch 1 are only in patch 2. So only 2
patches in this version.
On Fri, Feb 14, 2025 at 6:18 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
Again, can you share the link of performance data for this overhead
and if you have not published then please share it and also mention it
in commit message?
I will run the performance tests and post an update with the results.
Can we try different thresholds for this like 10, 50, 100, 200, etc.
to decide what is a good threshold value to skip filtering changes?
Ok, will do this. For this patch, I reset the count to 0, else the
test case fails as filtering could be skipped due to throttling. I
think we need a way for the user to set this threshold via a GUC and
that can be used for testing.
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
v14-0001-Filter-transactions-that-need-not-be-published.patchapplication/octet-stream; name=v14-0001-Filter-transactions-that-need-not-be-published.patchDownload
From 56714f125f0ec94760a6964c7935129f498bb888 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Mon, 17 Feb 2025 04:53:15 -0500
Subject: [PATCH v14 1/2] Filter transactions that need not be published.
This adds logic to filter transactions early (at decode time, rather than at streaming time) so
they can be skipped if they do not contain tables that are part of the publication list of the
logical replication walsender.
Determining whether to filter a change requires information about the relation
and the publication from the catalog, requiring a transaction to be started.
When most changes in a transaction are unfilterable, the overhead of starting a
transaction for each record is significant. To reduce this overhead a hash cache of relation file
locators is created. Even then a hash search for every record before recording has considerable
overhead especially for use cases where most tables in an instance are published. To further reduce
this overhead a simple approach is used to suspend filtering for a certain number of changes
(100) when an unfilterable change is encountered. In other words, continue filtering changes if
the last record was filtered out. If an unfilterable change is found, skip filtering the next 100
changes.
---
src/backend/replication/logical/decode.c | 5 +
src/backend/replication/logical/reorderbuffer.c | 453 ++++++++++++++++++++----
src/include/replication/reorderbuffer.h | 19 +
src/tools/pgindent/typedefs.list | 2 +
4 files changed, 411 insertions(+), 68 deletions(-)
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 24d88f3..978e38a 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -915,6 +915,11 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ if (ctx->reorder->can_filter_change &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, &target_locator, true))
+ return;
+
change = ReorderBufferGetChange(ctx->reorder);
if (!(xlrec->flags & XLH_INSERT_IS_SPECULATIVE))
change->action = REORDER_BUFFER_CHANGE_INSERT;
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index b42f400..d2e06ca 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -82,6 +82,22 @@
* a bit more memory to the oldest subtransactions, because it's likely
* they are the source for the next sequence of changes.
*
+ * We also try and filter changes that are not relevant for logical decoding
+ * as well as give the option for plugins to filter changes in advance.
+ * Determining whether to filter a change requires information about the
+ * relation from the catalog, requring a transaction to be started.
+ * When most changes in a transaction are unfilterable, the overhead of
+ * starting a transaction for each record is significant. To reduce this
+ * overhead a hash cache of relation file locators is created. Even then a
+ * hash search for every record before recording has considerable overhead
+ * especially for use cases where most tables in an instance are not filtered.
+ * To further reduce this overhead a simple approach is used to suspend
+ * filtering for a certain number of changes CHANGES_THRESHOLD_FOR_FILTER
+ * when an unfilterable change is encountered. In other words, continue
+ * filtering changes if the last record was filtered out. If an unfilterable
+ * change is found, skip filtering the next CHANGES_THRESHOLD_FOR_FILTER
+ * changes.
+ *
* -------------------------------------------------------------------------
*/
#include "postgres.h"
@@ -109,6 +125,7 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
@@ -227,8 +244,10 @@ static ReorderBufferTXN *ReorderBufferTXNByXid(ReorderBuffer *rb,
XLogRecPtr lsn, bool create_as_top);
static void ReorderBufferTransferSnapToParent(ReorderBufferTXN *txn,
ReorderBufferTXN *subtxn);
-
static void AssertTXNLsnOrder(ReorderBuffer *rb);
+static Relation ReorderBufferGetRelation(ReorderBuffer *rb,
+ RelFileLocator *rlocator,
+ bool has_tuple);
/* ---------------------------------------
* support functions for lsn-order iterating over the ->changes of a
@@ -279,6 +298,8 @@ static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
*/
static inline bool ReorderBufferCanStream(ReorderBuffer *rb);
static inline bool ReorderBufferCanStartStreaming(ReorderBuffer *rb);
+static Snapshot ReorderBufferStreamTXNSnapshot(ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
@@ -305,6 +326,48 @@ static void ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition, Size sz);
/*
+ * ---------------------------------------
+ * RelFileLocator filtering
+ * ---------------------------------------
+ * This hash table serves as a lookup table for determining if a relation can
+ * be filtered before being decoded and queued into the buffer.
+ *
+ * The hash table shares the same lifespan as the reorder buffer. This is
+ * crucial because each reorderbuffer may have different configurations or be
+ * associated with different output plugins, affecting the types of tables to
+ * be processed.
+ */
+
+static HTAB *RelFileLocatorFilterCache = NULL;
+
+static bool relation_callbacks_registered = false;
+
+typedef struct ReorderBufferRelFileLocatorKey
+{
+ Oid reltablespace;
+ RelFileNumber relfilenumber;
+} ReorderBufferRelFileLocatorKey;
+
+/* Hash table entry used to determine if the relation can be filtered. */
+typedef struct ReorderBufferRelFileLocatorEnt
+{
+ ReorderBufferRelFileLocatorKey key;
+ Oid relid;
+ bool filterable;
+} ReorderBufferRelFileLocatorEnt;
+
+static void RelFileLocatorCacheInvalidateCallback(Datum arg, Oid relid);
+static void ReorderBufferMemoryResetcallback(void *arg);
+
+/*
+ * After encountering a change that cannot be filtered out, filtering is
+ * temporarily suspended. Filtering resumes after processing every 100 changes.
+ * This strategy helps to minimize the overhead of performing a hash table
+ * search for each record, especially when most changes are not filterable.
+ */
+#define CHANGES_THRESHOLD_FOR_FILTER 100
+
+/*
* Allocate a new ReorderBuffer and clean out any old serialized state from
* prior ReorderBuffer instances for the same slot.
*/
@@ -362,6 +425,34 @@ ReorderBufferAllocate(void)
buffer->by_txn = hash_create("ReorderBufferByXid", 1000, &hash_ctl,
HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ /*
+ * To support early filtering of changes, this hash table acts
+ * as a lookup table to determine if the corresponding relation is
+ * required to be decoded and queued into the buffer. The hash table
+ * shares the same lifespan as the reorder buffer.
+ *
+ * Also setup the callback to invalidate cache when relations are updated.
+ */
+ hash_ctl.keysize = sizeof(ReorderBufferRelFileLocatorKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferRelFileLocatorEnt);
+ hash_ctl.hcxt = buffer->context;
+
+ RelFileLocatorFilterCache =
+ hash_create("RelFileLocatorFilterCache", 1000, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ buffer->relfile_callback.arg = NULL;
+ buffer->relfile_callback.func = ReorderBufferMemoryResetcallback;
+ MemoryContextRegisterResetCallback(buffer->context, &buffer->relfile_callback);
+
+ if (!relation_callbacks_registered)
+ {
+ /* Watch for invalidation events. */
+ CacheRegisterRelcacheCallback(RelFileLocatorCacheInvalidateCallback,
+ (Datum) 0);
+ relation_callbacks_registered = true;
+ }
+
buffer->by_txn_last_xid = InvalidTransactionId;
buffer->by_txn_last_txn = NULL;
@@ -372,6 +463,8 @@ ReorderBufferAllocate(void)
/* txn_heap is ordered by transaction size */
buffer->txn_heap = pairingheap_allocate(ReorderBufferTXNSizeCompare, NULL);
+ buffer->can_filter_change = true;
+
buffer->spillTxns = 0;
buffer->spillCount = 0;
buffer->spillBytes = 0;
@@ -826,6 +919,14 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
toptxn->txn_flags |= RBTXN_HAS_STREAMABLE_CHANGE;
}
+ else if (change->action == REORDER_BUFFER_CHANGE_INVALIDATION ||
+ change->action == REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT ||
+ change->action == REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID)
+ {
+ ReorderBufferTXN *toptxn = rbtxn_get_toptxn(txn);
+
+ toptxn->txn_flags |= RBTXN_HAS_SNAPSHOT_CHANGES;
+ }
change->lsn = lsn;
change->txn = txn;
@@ -835,6 +936,17 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn->nentries++;
txn->nentries_mem++;
+ /*
+ * If filtering was suspended and we've crossed the change threshold,
+ * attempt to filter again
+ */
+ if (!rb->can_filter_change && (++rb->unfiltered_changes_count
+ >= CHANGES_THRESHOLD_FOR_FILTER))
+ {
+ rb->can_filter_change = true;
+ rb->unfiltered_changes_count = 0;
+ }
+
/* update memory accounting information */
ReorderBufferChangeMemoryUpdate(rb, change, NULL, true,
ReorderBufferChangeSize(change));
@@ -1734,6 +1846,9 @@ ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn, bool txn_prep
txn->txn_flags |= RBTXN_IS_SERIALIZED_CLEAR;
}
+ /* All snapshot changes up to this point have been processed. */
+ txn->txn_flags &= ~RBTXN_HAS_SNAPSHOT_CHANGES;
+
/* also reset the number of entries in the transaction */
txn->nentries_mem = 0;
txn->nentries = 0;
@@ -2177,6 +2292,82 @@ ReorderBufferResetTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
+ * Return the relation corresponding to the given RelFileLocator 'rlocator' if
+ * relevant to decoding. Otherwise, return NULL.
+ *
+ * We don't decode catalog files, relations that are not logically logged,
+ * temporary heaps and sequences.
+ */
+static Relation
+ReorderBufferGetRelation(ReorderBuffer *rb, RelFileLocator *rlocator,
+ bool has_tuple)
+{
+ bool filterable = false;
+ Relation relation;
+ Oid reloid;
+
+ reloid = RelidByRelfilenumber(rlocator->spcOid, rlocator->relNumber);
+
+ if (!OidIsValid(reloid))
+ {
+ if (!has_tuple)
+ {
+ /*
+ * Mapped catalog tuple without data, emitted while catalog table was in
+ * the process of being rewritten. We can fail to look up the
+ * relfilenumber, because the relmapper has no "historic" view, in
+ * contrast to the normal catalog during decoding. Thus repeated rewrites
+ * can cause a lookup failure. That's OK because we do not decode catalog
+ * changes anyway. Normally such tuples would be skipped over below, but
+ * we can't identify whether the table should be logically logged without
+ * mapping the relfilenumber to the oid.
+ */
+ return NULL;
+ }
+
+ elog(ERROR, "could not map filenumber \"%s\" to relation OID",
+ relpathperm(*rlocator, MAIN_FORKNUM));
+ }
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (!RelationIsValid(relation))
+ elog(ERROR, "could not open relation with OID %u (for filenumber \"%s\")",
+ reloid, relpathperm(*rlocator, MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ filterable = true;
+
+ else if (relation->rd_rel->relrewrite && !rb->output_rewrites)
+ {
+ /*
+ * Ignore temporary heaps created during DDL unless the plugin has asked
+ * for them.
+ */
+ filterable = true;
+ }
+
+ else if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ {
+ /*
+ * For now ignore sequence changes entirely. Most of the time they don't
+ * log changes using records we understand, so it doesn't make sense to
+ * handle the few cases we do.
+ */
+ filterable = true;
+ }
+
+ /* Return NULL to indicate that this relation need not be decoded. */
+ if (filterable)
+ {
+ RelationClose(relation);
+ return NULL;
+ }
+
+ return relation;
+}
+
+/*
* Helper function for ReorderBufferReplay and ReorderBufferStreamTXN.
*
* Send data of a transaction (and its subtransactions) to the
@@ -2248,7 +2439,6 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
{
Relation relation = NULL;
- Oid reloid;
CHECK_FOR_INTERRUPTS();
@@ -2307,55 +2497,11 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_DELETE:
Assert(snapshot_now);
- reloid = RelidByRelfilenumber(change->data.tp.rlocator.spcOid,
- change->data.tp.rlocator.relNumber);
-
- /*
- * Mapped catalog tuple without data, emitted while
- * catalog table was in the process of being rewritten. We
- * can fail to look up the relfilenumber, because the
- * relmapper has no "historic" view, in contrast to the
- * normal catalog during decoding. Thus repeated rewrites
- * can cause a lookup failure. That's OK because we do not
- * decode catalog changes anyway. Normally such tuples
- * would be skipped over below, but we can't identify
- * whether the table should be logically logged without
- * mapping the relfilenumber to the oid.
- */
- if (reloid == InvalidOid &&
- change->data.tp.newtuple == NULL &&
- change->data.tp.oldtuple == NULL)
- goto change_done;
- else if (reloid == InvalidOid)
- elog(ERROR, "could not map filenumber \"%s\" to relation OID",
- relpathperm(change->data.tp.rlocator,
- MAIN_FORKNUM));
-
- relation = RelationIdGetRelation(reloid);
+ relation = ReorderBufferGetRelation(rb, &change->data.tp.rlocator,
+ (change->data.tp.newtuple != NULL ||
+ change->data.tp.oldtuple != NULL));
if (!RelationIsValid(relation))
- elog(ERROR, "could not open relation with OID %u (for filenumber \"%s\")",
- reloid,
- relpathperm(change->data.tp.rlocator,
- MAIN_FORKNUM));
-
- if (!RelationIsLogicallyLogged(relation))
- goto change_done;
-
- /*
- * Ignore temporary heaps created during DDL unless the
- * plugin has asked for them.
- */
- if (relation->rd_rel->relrewrite && !rb->output_rewrites)
- goto change_done;
-
- /*
- * For now ignore sequence changes entirely. Most of the
- * time they don't log changes using records we
- * understand, so it doesn't make sense to handle the few
- * cases we do.
- */
- if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
goto change_done;
/* user-triggered change */
@@ -4171,19 +4317,13 @@ ReorderBufferCanStartStreaming(ReorderBuffer *rb)
}
/*
- * Send data of a large transaction (and its subtransactions) to the
- * output plugin, but using the stream API.
+ * This function generates or retrieves a consistent snapshot of a transaction
+ * that is currently in progress for logical replication.
*/
-static void
-ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+static Snapshot
+ReorderBufferStreamTXNSnapshot(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
Snapshot snapshot_now;
- CommandId command_id;
- Size stream_bytes;
- bool txn_is_streamed;
-
- /* We can never reach here for a subtransaction. */
- Assert(rbtxn_is_toptxn(txn));
/*
* We can't make any assumptions about base snapshot here, similar to what
@@ -4226,12 +4366,11 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (txn->base_snapshot == NULL)
{
Assert(txn->ninvalidations == 0);
- return;
+ return NULL;
}
- command_id = FirstCommandId;
snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
- txn, command_id);
+ txn, FirstCommandId);
}
else
{
@@ -4244,17 +4383,40 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* the LSN condition in the previous branch (so no need to walk
* through subxacts again). In fact, we must not do that as we may be
* using snapshot half-way through the subxact.
- */
- command_id = txn->command_id;
-
- /*
+ *
* We can't use txn->snapshot_now directly because after the last
* streaming run, we might have got some new sub-transactions. So we
* need to add them to the snapshot.
*/
snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
- txn, command_id);
+ txn, txn->command_id);
+ }
+
+ return snapshot_now;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Snapshot snapshot_now;
+ Size stream_bytes;
+ bool txn_is_streamed;
+
+ /* We can never reach here for a subtransaction. */
+ Assert(rbtxn_is_toptxn(txn));
+
+ snapshot_now = ReorderBufferStreamTXNSnapshot(rb, txn);
+
+ /* This transaction didn't make any changes to the database till now. */
+ if (snapshot_now == NULL)
+ return;
+ if (txn->snapshot_now != NULL)
+ {
/* Free the previously copied snapshot. */
Assert(txn->snapshot_now->copied);
ReorderBufferFreeSnap(rb, txn->snapshot_now);
@@ -4272,7 +4434,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* Process and send the changes to output plugin. */
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
- command_id, true);
+ snapshot_now->curcid, true);
rb->streamCount += 1;
rb->streamBytes += stream_bytes;
@@ -5461,3 +5623,158 @@ restart:
*cmax = ent->cmax;
return true;
}
+
+/*
+ * Flush mapping entries when the corresponding relations are updated in pg_class.
+ * If relid is InvalidOid then do a complete reset of the RelFileLocatorFilterCache.
+ */
+static void
+RelFileLocatorCacheInvalidateCallback(Datum arg, Oid relid)
+{
+ HASH_SEQ_STATUS status;
+ ReorderBufferRelFileLocatorEnt *entry;
+
+ if (!RelFileLocatorFilterCache)
+ return;
+
+ hash_seq_init(&status, RelFileLocatorFilterCache);
+
+ /* Slightly inefficient algorithm but not performance critical path */
+ while ((entry = (ReorderBufferRelFileLocatorEnt *) hash_seq_search(&status)) != NULL)
+ {
+ /*
+ * If relid is InvalidOid, signaling a complete reset, we must remove
+ * all entries, otherwise just remove the specific relation's entry.
+ * Always remove negative cache entries.
+ */
+ if (relid == InvalidOid || /* complete reset */
+ entry->relid == InvalidOid || /* invalid cache entry */
+ entry->relid == relid) /* individual flushed relation */
+ {
+ if (hash_search(RelFileLocatorFilterCache,
+ &entry->key,
+ HASH_REMOVE,
+ NULL) == NULL)
+ elog(ERROR, "hash table corrupted");
+ }
+ }
+}
+
+/*
+ * Context reset/delete callback for RelFileLocatorFilterCache.
+ */
+static void
+ReorderBufferMemoryResetcallback(void *arg)
+{
+ RelFileLocatorFilterCache = NULL;
+}
+
+/*
+ * Determine whether the record corresponding to the relation identified by
+ * 'rlocator' needs to be filtered and not decoded and queued in the buffer.
+ *
+ * Determining the necessity of decoding requires accessing relation
+ * information from the system catalog, which requires a historical snapshot.
+ * We cannot directly use the current snapshot of the transaction due to the
+ * presence of INTERNAL_SNAPSHOT, COMMAND_ID, or INVALIDATION records in the
+ * buffer that could modify the snapshot. A proper snapshot can only be
+ * constructed after these records are processed in ReorderBufferProcessTXN, or
+ * if we are decoding a transaction without these records. See comment on top
+ * of ReorderBufferGetRelation() to see list of relations that are not
+ * decoded by the reorderbuffer.
+ *
+ * To reduce the overhead from the system catalog access and transaction
+ * management, the results are cached in the 'RelFileLocatorFilterCache' hash
+ * table.
+ *
+ * Returns true if the relation can be filtered; otherwise, false.
+ */
+bool
+ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, RelFileLocator *rlocator,
+ bool has_tuple)
+{
+ bool found;
+ Relation relation;
+ bool using_subtxn;
+ Snapshot snapshot_now;
+ ReorderBufferTXN *txn,
+ *toptxn;
+ ReorderBufferRelFileLocatorEnt *entry;
+ ReorderBufferRelFileLocatorKey key;
+
+ Assert(RelFileLocatorFilterCache);
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+ Assert(txn);
+ toptxn = rbtxn_get_toptxn(txn);
+ rb->can_filter_change = false;
+
+ /*
+ * We cannot construct an accurate historical snapshot until all pending
+ * records in this transaction that might update the snapshot are
+ * processed.
+ */
+ if (rbtxn_has_snapshot_changes(toptxn))
+ return false;
+
+ key.reltablespace = rlocator->spcOid;
+ key.relfilenumber = rlocator->relNumber;
+ entry = hash_search(RelFileLocatorFilterCache, &key, HASH_ENTER, &found);
+
+ if (found)
+ {
+ rb->can_filter_change = entry->filterable;
+ return entry->filterable;
+ }
+
+ /* constructs a temporary historical snapshot */
+ snapshot_now = ReorderBufferStreamTXNSnapshot(rb, toptxn);
+ Assert(snapshot_now);
+
+ /* build data to be able to lookup the CommandIds of catalog tuples */
+ ReorderBufferBuildTupleCidHash(rb, toptxn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, toptxn->tuplecid_hash);
+
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("filter change by RelFileLocator");
+ else
+ StartTransactionCommand();
+
+ relation = ReorderBufferGetRelation(rb, rlocator, has_tuple);
+
+ if (RelationIsValid(relation))
+ {
+ entry->relid = RelationGetRelid(relation);
+ entry->filterable = false;
+ RelationClose(relation);
+ }
+ else
+ {
+ entry->relid = InvalidOid;
+ entry->filterable = true;
+ }
+
+ rb->can_filter_change = entry->filterable;
+
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access have
+ * persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ return entry->filterable;
+}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 517a8e3..449d8a2 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
+#define RBTXN_HAS_SNAPSHOT_CHANGES 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -215,6 +216,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_HAS_STREAMABLE_CHANGE) != 0 \
)
+/* Does this transaction make changes to the current snapshot? */
+#define rbtxn_has_snapshot_changes(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_HAS_SNAPSHOT_CHANGES) != 0 \
+)
+
/*
* Has this transaction been streamed to downstream?
*
@@ -641,6 +648,7 @@ struct ReorderBuffer
* Private memory context.
*/
MemoryContext context;
+ MemoryContextCallback relfile_callback;
/*
* Memory contexts for specific types objects
@@ -661,6 +669,12 @@ struct ReorderBuffer
/* Max-heap for sizes of all top-level and sub transactions */
pairingheap *txn_heap;
+ /* should we try to filter the change? */
+ bool can_filter_change;
+
+ /* number of changes after a failed attempt at filtering */
+ int8 unfiltered_changes_count;
+
/*
* Statistics about transactions spilled to disk.
*
@@ -761,4 +775,9 @@ extern void ReorderBufferSetRestartPoint(ReorderBuffer *rb, XLogRecPtr ptr);
extern void StartupReorderBuffer(void);
+extern bool ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, RelFileLocator *rlocator,
+ bool has_tuple);
+extern bool ReorderBufferCanFilterChanges(ReorderBuffer *rb);
+
#endif
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b6c170a..2c76247 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2434,6 +2434,8 @@ ReorderBufferIterTXNEntry
ReorderBufferIterTXNState
ReorderBufferMessageCB
ReorderBufferPrepareCB
+ReorderBufferRelFileLocatorEnt
+ReorderBufferRelFileLocatorKey
ReorderBufferRollbackPreparedCB
ReorderBufferStreamAbortCB
ReorderBufferStreamChangeCB
--
1.8.3.1
v14-0002-Introduce-a-output-plugin-callback-to-filter-cha.patchapplication/octet-stream; name=v14-0002-Introduce-a-output-plugin-callback-to-filter-cha.patchDownload
From 939a090b1a883f9aaa0672a0f3b88694af320af5 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Mon, 17 Feb 2025 05:29:06 -0500
Subject: [PATCH v14 2/2] Introduce a output plugin callback to filter changes
This new output plugin callback provides an option to logical decoding plugins to filter out
changes early. The primary purpose of the callback is to conserve memory and processing cycles
by excluding changes that are not required by output plugins.
---
doc/src/sgml/logicaldecoding.sgml | 41 ++++++++++++++-
src/backend/replication/logical/decode.c | 70 ++++++++++++++++++++++++-
src/backend/replication/logical/logical.c | 42 +++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 50 ++++++++++++++----
src/backend/replication/pgoutput/pgoutput.c | 70 +++++++++++++++++++++++++
src/include/replication/output_plugin.h | 20 +++++++
src/include/replication/reorderbuffer.h | 8 +++
src/test/subscription/t/001_rep_changes.pl | 18 ++++++-
src/test/subscription/t/013_partition.pl | 7 +++
9 files changed, 311 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 1c4ae38..a603c47 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -560,6 +560,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeCommitCB commit_cb;
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
+ LogicalDecodeFilterChangeCB filter_change_cb;
LogicalDecodeShutdownCB shutdown_cb;
LogicalDecodeFilterPrepareCB filter_prepare_cb;
LogicalDecodeBeginPrepareCB begin_prepare_cb;
@@ -582,8 +583,8 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
and <function>commit_cb</function> callbacks are required,
while <function>startup_cb</function>, <function>truncate_cb</function>,
<function>message_cb</function>, <function>filter_by_origin_cb</function>,
- and <function>shutdown_cb</function> are optional.
- If <function>truncate_cb</function> is not set but a
+ <function>shutdown_cb</function>, and <function>filter_change_cb</function>
+ are optional. If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
@@ -871,6 +872,42 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-filter-change">
+ <title>Change Filter Callback</title>
+
+ <para>
+ The optional <function>filter_change_cb</function> is called before a
+ change record is decoded to determine whether the change can be filtered
+ out.
+<programlisting>
+typedef bool (*LogicalDecodeFilterChangeCB) (struct LogicalDecodingContext *ctx,
+ Oid relid,
+ ReorderBufferChangeType change_type,
+ bool in_txn, bool *cache_valid);
+</programlisting>
+ To indicate that decoding can be skipped for the given change
+ <parameter>change_type</parameter>, return <literal>true</literal>;
+ <literal>false</literal> otherwise.
+ The <parameter>in_txn</parameter> parameter indicates whether the
+ callback is invoked within a transaction block.
+ When <parameter>in_txn</parameter> is false, and if making a decision to filter a change requires being inside a
+ transaction block, such as needing access to the catalog, set
+ <parameter>*cache_valid</parameter> to <literal>false</literal>.
+ This ensures that the callback will be reinvoked once a transaction block
+ starts. If a decision can be made immediately, set
+ <parameter>*cache_valid</parameter> to <literal>true</literal>.
+ </para>
+ <para>
+ The primary purpose of this callback function is to optimize memory usage
+ and processing efficiency by filtering out changes that are unnecessary for
+ output plugins. It enables output plugins to selectively process relevant
+ changes. Caching filtering decisions locally is recommended, as it enables
+ the callback to provide cached results without repeatedly initiating
+ transactions or querying the catalog. This approach minimizes overhead
+ and improves efficiency during the decoding process.
+ </para>
+ </sect3>
+
<sect3 id="logicaldecoding-output-plugin-message">
<title>Generic Message Callback</title>
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 978e38a..6a3388e 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -576,6 +576,17 @@ FilterByOrigin(LogicalDecodingContext *ctx, RepOriginId origin_id)
}
/*
+ * Check if filtering changes before decoding is supported and we're not suppressing filter
+ * changes currently.
+ */
+static inline bool
+FilterChangeIsEnabled(LogicalDecodingContext *ctx)
+{
+ return (ctx->callbacks.filter_change_cb != NULL &&
+ ctx->reorder->can_filter_change);
+}
+
+/*
* Handle rmgr LOGICALMSG_ID records for LogicalDecodingProcessRecord().
*/
void
@@ -915,9 +926,16 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
- if (ctx->reorder->can_filter_change &&
+ /*
+ * When filtering changes, determine if the relation associated with the change
+ * can be skipped. This could be because the relation is unlogged or because
+ * the plugin has opted to exclude this relation from decoding.
+ */
+ if (FilterChangeIsEnabled(ctx) &&
ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, &target_locator, true))
+ buf->origptr, &target_locator,
+ REORDER_BUFFER_CHANGE_INSERT,
+ true))
return;
change = ReorderBufferGetChange(ctx->reorder);
@@ -970,6 +988,18 @@ DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ /*
+ * When filtering changes, determine if the relation associated with the change
+ * can be skipped. This could be because the relation is unlogged or because
+ * the plugin has opted to exclude this relation from decoding.
+ */
+ if (FilterChangeIsEnabled(ctx) &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, &target_locator,
+ REORDER_BUFFER_CHANGE_UPDATE,
+ true))
+ return;
+
change = ReorderBufferGetChange(ctx->reorder);
change->action = REORDER_BUFFER_CHANGE_UPDATE;
change->origin_id = XLogRecGetOrigin(r);
@@ -1036,6 +1066,18 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ /*
+ * When filtering changes, determine if the relation associated with the change
+ * can be skipped. This could be because the relation is unlogged or because
+ * the plugin has opted to exclude this relation from decoding.
+ */
+ if (FilterChangeIsEnabled(ctx) &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, &target_locator,
+ REORDER_BUFFER_CHANGE_DELETE,
+ true))
+ return;
+
change = ReorderBufferGetChange(ctx->reorder);
if (xlrec->flags & XLH_DELETE_IS_SUPER)
@@ -1139,6 +1181,18 @@ DecodeMultiInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
return;
/*
+ * When filtering changes, determine if the relation associated with the change
+ * can be skipped. This could be because the relation is unlogged or because
+ * the plugin has opted to exclude this relation from decoding.
+ */
+ if (FilterChangeIsEnabled(ctx) &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, &rlocator,
+ REORDER_BUFFER_CHANGE_INSERT,
+ true))
+ return;
+
+ /*
* We know that this multi_insert isn't for a catalog, so the block should
* always have data even if a full-page write of it is taken.
*/
@@ -1229,6 +1283,18 @@ DecodeSpecConfirm(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
+ /*
+ * When filtering changes, determine if the relation associated with the change
+ * can be skipped. This could be because the relation is unlogged or because
+ * the plugin has opted to exclude this relation from decoding.
+ */
+ if (FilterChangeIsEnabled(ctx) &&
+ ReorderBufferFilterByRelFileLocator(ctx->reorder, XLogRecGetXid(r),
+ buf->origptr, &target_locator,
+ REORDER_BUFFER_CHANGE_INSERT,
+ true))
+ return;
+
change = ReorderBufferGetChange(ctx->reorder);
change->action = REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM;
change->origin_id = XLogRecGetOrigin(r);
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8ea846b..635e1de 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -74,6 +74,9 @@ static void truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+static bool filter_change_cb_wrapper(ReorderBuffer *cache, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn,
+ bool *cache_valid);
/* streaming callbacks */
static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
@@ -220,6 +223,7 @@ StartupDecodingContext(List *output_plugin_options,
/* wrap output plugin callbacks, so we can add error context information */
ctx->reorder->begin = begin_cb_wrapper;
ctx->reorder->apply_change = change_cb_wrapper;
+ ctx->reorder->filter_change = filter_change_cb_wrapper;
ctx->reorder->apply_truncate = truncate_cb_wrapper;
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
@@ -1224,6 +1228,44 @@ filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, RepOriginId origin_id)
return ret;
}
+static bool
+filter_change_cb_wrapper(ReorderBuffer *cache, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn,
+ bool *cache_valid)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+ bool ret;
+
+ Assert(!ctx->fast_forward);
+
+ /* check if the filter change callback is supported */
+ if (ctx->callbacks.filter_change_cb == NULL)
+ return false;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "filter_change";
+ state.report_location = InvalidXLogRecPtr;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = false;
+ ctx->end_xact = false;
+
+ /* do the actual work: call callback */
+ ret = ctx->callbacks.filter_change_cb(ctx, relid, change_type, in_txn, cache_valid);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
+ return ret;
+}
+
static void
message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index d2e06ca..00046a6 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -361,11 +361,12 @@ static void ReorderBufferMemoryResetcallback(void *arg);
/*
* After encountering a change that cannot be filtered out, filtering is
- * temporarily suspended. Filtering resumes after processing every 100 changes.
+ * temporarily suspended. Filtering resumes after processing CHANGES_THRESHOLD_FOR_FILTER
+ * changes.
* This strategy helps to minimize the overhead of performing a hash table
* search for each record, especially when most changes are not filterable.
*/
-#define CHANGES_THRESHOLD_FOR_FILTER 100
+#define CHANGES_THRESHOLD_FOR_FILTER 0
/*
* Allocate a new ReorderBuffer and clean out any old serialized state from
@@ -5692,9 +5693,11 @@ ReorderBufferMemoryResetcallback(void *arg)
bool
ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, RelFileLocator *rlocator,
+ ReorderBufferChangeType change_type,
bool has_tuple)
{
bool found;
+ bool cache_valid;
Relation relation;
bool using_subtxn;
Snapshot snapshot_now;
@@ -5708,7 +5711,6 @@ ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
Assert(txn);
toptxn = rbtxn_get_toptxn(txn);
- rb->can_filter_change = false;
/*
* We cannot construct an accurate historical snapshot until all pending
@@ -5725,7 +5727,25 @@ ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
if (found)
{
rb->can_filter_change = entry->filterable;
- return entry->filterable;
+
+ /*
+ * Quick return if we already know that the relation is not to be decoded.
+ * These are for special relations that are unlogged and for sequences
+ * and catalogs.
+ */
+ if (entry->filterable)
+ return true;
+
+ /* Allow the output plugin to filter relations */
+ rb->can_filter_change = rb->filter_change(rb, entry->relid, change_type,
+ false, &cache_valid);
+
+ /*
+ * If plugin had the relation ready in cache, the response is valid, else
+ * we'll need to call the plugin a second time within a transaction.
+ */
+ if (cache_valid)
+ return rb->can_filter_change;
}
/* constructs a temporary historical snapshot */
@@ -5749,18 +5769,30 @@ ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
if (RelationIsValid(relation))
{
- entry->relid = RelationGetRelid(relation);
+ if (IsToastRelation(relation))
+ {
+ Oid real_reloid = InvalidOid;
+ char *toast_name = RelationGetRelationName(relation);
+ /* pg_toast_ len is 9 */
+ char *start_ch = &toast_name[9];
+
+ real_reloid = pg_strtoint32(start_ch);
+ entry->relid = real_reloid;
+ }
+ else
+ entry->relid = RelationGetRelid(relation);
+
entry->filterable = false;
+ rb->can_filter_change = rb->filter_change(rb, entry->relid, change_type,
+ true, &cache_valid);
RelationClose(relation);
}
else
{
entry->relid = InvalidOid;
- entry->filterable = true;
+ rb->can_filter_change = entry->filterable = true;
}
- rb->can_filter_change = entry->filterable;
-
ReorderBufferFreeSnap(rb, snapshot_now);
TeardownHistoricSnapshot(false);
@@ -5776,5 +5808,5 @@ ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- return entry->filterable;
+ return rb->can_filter_change;
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 7d464f6..93eb579 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -57,6 +57,9 @@ static void pgoutput_message(LogicalDecodingContext *ctx,
Size sz, const char *message);
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static bool pgoutput_filter_change(LogicalDecodingContext *ctx, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn,
+ bool *cache_valid);
static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn);
static void pgoutput_prepare_txn(LogicalDecodingContext *ctx,
@@ -268,6 +271,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_prepared_cb = pgoutput_commit_prepared_txn;
cb->rollback_prepared_cb = pgoutput_rollback_prepared_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
+ cb->filter_change_cb = pgoutput_filter_change;
cb->shutdown_cb = pgoutput_shutdown;
/* transaction streaming */
@@ -1744,6 +1748,72 @@ pgoutput_origin_filter(LogicalDecodingContext *ctx,
}
/*
+ * Determine whether a change to the specified relation should be published.
+ *
+ * See the comments atop of LogicalDecodeFilterChangeCB for details.
+ */
+static bool
+pgoutput_filter_change(LogicalDecodingContext *ctx, Oid relid,
+ ReorderBufferChangeType change_type, bool in_txn, bool *cache_valid)
+{
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ if (in_txn)
+ {
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+ entry = get_rel_sync_entry(data, relation);
+ *cache_valid = true;
+ }
+ else
+ {
+ entry = (RelationSyncEntry *) hash_search(RelationSyncCache,
+ &relid,
+ HASH_FIND, cache_valid);
+ if (!*cache_valid)
+ return false;
+ }
+
+ /*
+ * If the pubaction is not supported by this publication then return true to say
+ * the change for this entry can be skipped.
+ */
+ switch (change_type)
+ {
+ case REORDER_BUFFER_CHANGE_INSERT:
+ if (!entry->pubactions.pubinsert)
+ {
+ elog(DEBUG1, "Filtering INSERT");
+ return true;
+ }
+ break;
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ if (!entry->pubactions.pubupdate)
+ {
+ elog(DEBUG1, "Filtering UPDATE");
+ return true;
+ }
+ break;
+ case REORDER_BUFFER_CHANGE_DELETE:
+ if (!entry->pubactions.pubdelete)
+ {
+ elog(DEBUG1, "Filtering DELETE");
+ return true;
+ }
+ break;
+ default:
+ /* allow any other changes that are not explicitly filtered */
+ return false;
+ }
+
+ return false;
+}
+
+/*
* Shutdown the output plugin.
*
* Note, we don't need to clean the data->context, data->cachectx, and
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 8d4d5b71..f82bc1d 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -97,6 +97,25 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
RepOriginId origin_id);
/*
+ * This callback is called before a change record is decoded to determine
+ * whether the change can be filtered out before entering the reorder buffer.
+ *
+ * Typically, the output plugin needs to refer to the system catalog to make
+ * this decision. To enhance efficiency, the plugin should cache the lookup
+ * result for each relation, minimizing catalog access on subsequent calls.
+ *
+ * If the callback is called with 'in_txn' set as false (the reorder
+ * buffer has not started a transaction), the output plugin should set '*cache_valid'
+ * to false to indicate that the result is not available in its internal cache.
+ * If 'in_txn' is true, the plugin can create a cache entry after querying the
+ * catalog.
+ */
+typedef bool (*LogicalDecodeFilterChangeCB) (struct LogicalDecodingContext *ctx,
+ Oid relid,
+ ReorderBufferChangeType change_type,
+ bool in_txn, bool *cache_valid);
+
+/*
* Called to shutdown an output plugin.
*/
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
@@ -222,6 +241,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeCommitCB commit_cb;
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
+ LogicalDecodeFilterChangeCB filter_change_cb;
LogicalDecodeShutdownCB shutdown_cb;
/* streaming of changes at prepare time */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 449d8a2..579a24a 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -494,6 +494,12 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* filter change callback signature */
+typedef bool (*ReorderBufferFilterChangeCB) (ReorderBuffer *rb,
+ Oid relid,
+ ReorderBufferChangeType change_type,
+ bool in_txn, bool *cache_valid);
+
/* begin prepare callback signature */
typedef void (*ReorderBufferBeginPrepareCB) (ReorderBuffer *rb,
ReorderBufferTXN *txn);
@@ -604,6 +610,7 @@ struct ReorderBuffer
*/
ReorderBufferBeginCB begin;
ReorderBufferApplyChangeCB apply_change;
+ ReorderBufferFilterChangeCB filter_change;
ReorderBufferApplyTruncateCB apply_truncate;
ReorderBufferCommitCB commit;
ReorderBufferMessageCB message;
@@ -777,6 +784,7 @@ extern void StartupReorderBuffer(void);
extern bool ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, RelFileLocator *rlocator,
+ ReorderBufferChangeType change_type,
bool has_tuple);
extern bool ReorderBufferCanFilterChanges(ReorderBuffer *rb);
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 8726fe0..bced62c 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -485,9 +485,10 @@ $node_publisher->wait_for_catchup('tap_sub');
# Check that we don't send BEGIN and COMMIT because of empty transaction
# optimization. We have to look for the DEBUG1 log messages about that, so
-# temporarily bump up the log verbosity.
+# temporarily bump up the log verbosity. Also confirm that unpublished
+# changes are filtered out after a restart.
$node_publisher->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_publisher->reload;
+$node_publisher->restart;
# Note that the current location of the log file is not grabbed immediately
# after reloading the configuration, but after sending one SQL command to
@@ -495,6 +496,8 @@ $node_publisher->reload;
$log_location = -s $node_publisher->logfile;
$node_publisher->safe_psql('postgres', "INSERT INTO tab_notrep VALUES (11)");
+$node_publisher->safe_psql('postgres', "UPDATE tab_notrep SET a = 2 WHERE a = 1");
+$node_publisher->safe_psql('postgres', "DELETE FROM tab_notrep WHERE a = 2");
$node_publisher->wait_for_catchup('tap_sub');
@@ -502,6 +505,17 @@ $logfile = slurp_file($node_publisher->logfile, $log_location);
ok($logfile =~ qr/skipped replication of an empty transaction with XID/,
'empty transaction is skipped');
+# Check that an unpublished change is filtered out.
+$logfile = slurp_file($node_publisher->logfile, $log_location);
+ok($logfile =~ qr/Filtering INSERT/,
+ 'unpublished INSERT is filtered');
+
+ok($logfile =~ qr/Filtering UPDATE/,
+ 'unpublished UPDATE is filtered');
+
+ok($logfile =~ qr/Filtering DELETE/,
+ 'unpublished DELETE is filtered');
+
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM tab_notrep");
is($result, qq(0), 'check non-replicated table is empty on subscriber');
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 14a3bea..180eaa0 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -468,6 +468,13 @@ $node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab4 (a int PRIMARY KEY)");
$node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab4_1 (a int PRIMARY KEY)");
+
+# Ensure that the subscription 'sub2' catches up with the latest changes. This
+# is necessary for the walsender to update its historic snapshot. Otherwise,
+# the walsender might retain an outdated snapshot, potentially preventing it
+# from accessing the newly created publication.
+$node_publisher->wait_for_catchup('sub2');
+
# Since we specified publish_via_partition_root in pub_all and
# pub_lower_level, all partition tables use their root tables' identity and
# schema. We set the list of publications so that the FOR ALL TABLES
--
1.8.3.1
On Fri, Feb 14, 2025 at 6:18 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Feb 12, 2025 at 10:41 AM Ajin Cherian <itsajin@gmail.com> wrote:
On Wed, Jan 29, 2025 at 9:31 AM Peter Smith <smithpb2250@gmail.com> wrote:
Hi Ajin,
Some review comments for patch v12-0001.
======
Commit message1.
Track transactions which have snapshot changes with a new flag
RBTXN_HAS_SNAPSHOT_CHANGES~
The commit message only says *what* it does, but not *why* this patch
even exists. TBH, I don't understand why this patch needs to be
separated from your patch 0002, because 0001 makes no independent use
of the flag, nor is it separately tested.Anyway, if it is going to remain separated then IMO at least the the
message should explain the intended purpose e.g. why the subsequent
patches require this flagged info and how they will use it.Fixed.
I still can't get from 0001's commit message the reason for tracking
the snapshot changes separately. Also, please find my comments for
0002's commit message.When most changes in a transaction are unfilterable, the overhead of
starting a transaction for each record is significant.Can you tell what is the exact overhead by testing it? IIRC, that was
the initial approach. It is better if you can mention in the commit
message what was overhead. It will be easier for reviewers.To reduce this overhead a hash cache of relation file locators is
created. Even then a hash search for every record before recording has
considerable overhead especially for use cases where most tables in an
instance are published.Again, can you share the link of performance data for this overhead
and if you have not published then please share it and also mention it
in commit message?
I compared the patch 1 which does not employ a hash cache and has the
overhead of starting a transaction every time the filter is checked.
I created a test setup of 10 million inserts in 3 different scenarios:
1. All inserts on unpublished tables
2. Half of the inserts on unpublished table and half on pupblished table
3. All inserts on published tables.
The percentage improvement in the new optimized patch compared to the
old patch is:
No transactions in publication: 85.39% improvement
Half transactions in publication: 72.70% improvement
All transactions in publication: 48.47% improvement
Attaching a graph to show the difference.
To further reduce this overhead a simple approach is used to suspend
filtering for a certain number of changes (100) when an unfilterable
change is encountered. In other words, continue filtering changes if
the last record was filtered out. If an unfilterable change is found,
skip filtering the next 100 changes.Can we try different thresholds for this like 10, 50, 100, 200, etc.
to decide what is a good threshold value to skip filtering changes?
Of Course this performance might vary from setup to setup but I tried
the above setup to compare the 4 different threshold levels
Conclusions:
Lower throttling values yield better performance, particularly when
transactions are included in the publication.
Increasing the throttle limit to 200 transactions causes significant
performance degradation, particularly when half or all transactions
are included.
For optimal performance, a moderate throttling value (100
transactions) may be the best balance between performance and
processing efficiency.
Attaching the graph to show the difference. I'm also attaching the
test script that I used.
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
On Thu, Feb 20, 2025 at 1:30 PM Ajin Cherian <itsajin@gmail.com> wrote:
On Fri, Feb 14, 2025 at 6:18 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Feb 12, 2025 at 10:41 AM Ajin Cherian <itsajin@gmail.com> wrote:
On Wed, Jan 29, 2025 at 9:31 AM Peter Smith <smithpb2250@gmail.com> wrote:
Hi Ajin,
Some review comments for patch v12-0001.
======
Commit message1.
Track transactions which have snapshot changes with a new flag
RBTXN_HAS_SNAPSHOT_CHANGES~
The commit message only says *what* it does, but not *why* this patch
even exists. TBH, I don't understand why this patch needs to be
separated from your patch 0002, because 0001 makes no independent use
of the flag, nor is it separately tested.Anyway, if it is going to remain separated then IMO at least the the
message should explain the intended purpose e.g. why the subsequent
patches require this flagged info and how they will use it.Fixed.
I still can't get from 0001's commit message the reason for tracking
the snapshot changes separately. Also, please find my comments for
0002's commit message.When most changes in a transaction are unfilterable, the overhead of
starting a transaction for each record is significant.Can you tell what is the exact overhead by testing it? IIRC, that was
the initial approach. It is better if you can mention in the commit
message what was overhead. It will be easier for reviewers.To reduce this overhead a hash cache of relation file locators is
created. Even then a hash search for every record before recording has
considerable overhead especially for use cases where most tables in an
instance are published.Again, can you share the link of performance data for this overhead
and if you have not published then please share it and also mention it
in commit message?I compared the patch 1 which does not employ a hash cache and has the
overhead of starting a transaction every time the filter is checked.
Just to clarify, by patch 1, I mean the first patch in this thread
(v1) posted by Lie Jie here - (1)
1 - /messages/by-id/CAGfChW5Qo2SrjJ7rU9YYtZbRaWv6v-Z8MJn=dQNx4uCSqDEOHA@mail.gmail.com
regards,
Ajin Cherian
Fujitsu Australia
Dear Ajin,
I compared the patch 1 which does not employ a hash cache and has the
overhead of starting a transaction every time the filter is checked.I created a test setup of 10 million inserts in 3 different scenarios:
1. All inserts on unpublished tables
2. Half of the inserts on unpublished table and half on pupblished table
3. All inserts on published tables.The percentage improvement in the new optimized patch compared to the
old patch is:No transactions in publication: 85.39% improvement
Half transactions in publication: 72.70% improvement
All transactions in publication: 48.47% improvementAttaching a graph to show the difference.
I could not find any comparisons with HEAD. Can you clarify the throughput/latency/memory
usage with HEAD?
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Dear Ajin,
Here are my comments. I must play with patches to understand more.
01.
```
extern bool ReorderBufferFilterByRelFileLocator(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, RelFileLocator *rlocator,
ReorderBufferChangeType change_type,
bool has_tuple);
```
Can you explain why "has_tuple is needed? All callers is set to true.
02.
```
static Relation
ReorderBufferGetRelation(ReorderBuffer *rb, RelFileLocator *rlocator,
bool has_tuple)
```
Hmm, the naming is bit confusing for me. This operation is mostly not related with
the reorder buffer. How about "GetPossibleDecodableRelation" or something?
03.
```
if (IsToastRelation(relation))
{
Oid real_reloid = InvalidOid;
char *toast_name = RelationGetRelationName(relation);
/* pg_toast_ len is 9 */
char *start_ch = &toast_name[9];
real_reloid = pg_strtoint32(start_ch);
entry->relid = real_reloid;
}
```
It is bit hacky for me. How about using sscanf like attached?
04.
IIUC, toast tables always require the filter_change() call twice, is it right?
I understood like below:
1. ReorderBufferFilterByRelFileLocator() tries to filter the change at outside the
transaction. The OID indicates the pg_toast_xxx table.
2. pgoutput_filter_change() cannot find the table from the hash. It returns false
with cache_valid=false.
3. ReorderBufferFilterByRelFileLocator() starts a transaction and get its relation.
4. The function recognizes the relation seems toast and get parent oid.
5. The function tries to filter the change in the transaction, with the parent oid.
6. pgoutput_filter_change()->get_rel_sync_entry() enters the parent relation to the
hash and return determine the filtable or not.
7. After sometime, the same table is modified. But the toast table is not stored in
the hash so that whole 1-6 steps are required.
I feel this may affect the perfomance when many toast is modified. How about skiping
the check for toasted ones? ISTM IsToastNamespace() can be used for the decision.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
On Mon, 17 Feb 2025 at 16:20, Ajin Cherian <itsajin@gmail.com> wrote:
On Mon, Feb 17, 2025 at 10:08 AM Peter Smith <smithpb2250@gmail.com> wrote:
Hi Ajin.
FYI - Patch set v13* no longer applies cleanly. Needs rebasing.
I've rebased the patch. I've also merged patch 1 into patch 2 as the
functionality of the changes in patch 1 are only n patch 2.On Mon, Feb 17, 2025 at 10:08 AM Peter Smith <smithpb2250@gmail.com> wrote:
Hi Ajin.
FYI - Patch set v13* no longer applies cleanly. Needs rebasing.
I've rebased the patch. I've also merged patch 1 into patch 2 as the
functionality of the changes in patch 1 are only in patch 2. So only 2
patches in this version.On Fri, Feb 14, 2025 at 6:18 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
Again, can you share the link of performance data for this overhead
and if you have not published then please share it and also mention it
in commit message?I will run the performance tests and post an update with the results.
Can we try different thresholds for this like 10, 50, 100, 200, etc.
to decide what is a good threshold value to skip filtering changes?Ok, will do this. For this patch, I reset the count to 0, else the
test case fails as filtering could be skipped due to throttling. I
think we need a way for the user to set this threshold via a GUC and
that can be used for testing.
Hi Ajin,
I have reviewed the v14-0001 patch, I have following comment:
1. We need to initialize the 'rb->unfiltered_changes_count = 0', initially
inside function 'ReorderBufferAllocate', otherwise it will get set to garbage
value. While debugging I found out it was setting to 128 by default, so
'++rb->unfiltered_changes_count' was setting it to '-127'. As a result it
might not enter the if condition here for some initial inserts:
+ /*
+ * If filtering was suspended and we've crossed the change threshold,
+ * attempt to filter again
+ */
+ if (!rb->can_filter_change && (++rb->unfiltered_changes_count
+ >= CHANGES_THRESHOLD_FOR_FILTER))
+ {
+ rb->can_filter_change = true;
+ rb->unfiltered_changes_count = 0;
+ }
2. After applying only 0001 patch, then I tried to insert in a published table.
I am getting following error:
postgres=# insert into s1.t1 values(15);
INSERT 0 1
postgres=# insert into s1.t1 values(15);
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back
the current transaction and exit, because another server process
exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and
repeat your command.
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
postgres=?#
Stack trace:
#0 __pthread_kill_implementation (no_tid=0, signo=6,
threadid=132913961858944) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=132913961858944) at
./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=132913961858944,
signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x000078e270c42476 in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
#4 0x000078e270c287f3 in __GI_abort () at ./stdlib/abort.c:79
#5 0x00005c9c56842f53 in ExceptionalCondition
(conditionName=0x5c9c56c10ed5 "MemoryContextIsValid(context)",
fileName=0x5c9c56c10d5f "mcxt.c",
lineNumber=1323) at assert.c:66
#6 0x00005c9c5690e99c in palloc (size=48) at mcxt.c:1323
#7 0x00005c9c55277564 in systable_beginscan
(heapRelation=0x78e2716561d8, indexId=2685, indexOK=true,
snapshot=0x0, nkeys=1, key=0x7ffd9a4446b0)
at genam.c:403
#8 0x00005c9c567cb6ca in SearchCatCacheMiss (cache=0x5c9c58af0e80,
nkeys=1, hashValue=1393055558, hashIndex=6, v1=16384, v2=0, v3=0,
v4=0)
at catcache.c:1533
#9 0x00005c9c567cb43d in SearchCatCacheInternal
(cache=0x5c9c58af0e80, nkeys=1, v1=16384, v2=0, v3=0, v4=0) at
catcache.c:1464
#10 0x00005c9c567cabd8 in SearchCatCache1 (cache=0x5c9c58af0e80,
v1=16384) at catcache.c:1332
#11 0x00005c9c5682a055 in SearchSysCache1 (cacheId=38, key1=16384) at
syscache.c:228
#12 0x00005c9c567e3b6f in get_namespace_name (nspid=16384) at lsyscache.c:3397
#13 0x00005c9c5608eb5e in logicalrep_write_namespace
(out=0x5c9c58b5d5c8, nspid=16384) at proto.c:1033
#14 0x00005c9c5608d0c2 in logicalrep_write_rel (out=0x5c9c58b5d5c8,
xid=0, rel=0x78e27167ede0, columns=0x0,
include_gencols_type=PUBLISH_GENCOLS_NONE) at proto.c:683
#15 0x000078e270fc9301 in send_relation_and_attrs
(relation=0x78e27167ede0, xid=0, ctx=0x5c9c58b4d8f0,
relentry=0x5c9c58b92278) at pgoutput.c:798
#16 0x000078e270fc8ed9 in maybe_send_schema (ctx=0x5c9c58b4d8f0,
change=0x5c9c58b8f2e0, relation=0x78e27167ede0,
relentry=0x5c9c58b92278)
at pgoutput.c:752
#17 0x000078e270fce338 in pgoutput_change (ctx=0x5c9c58b4d8f0,
txn=0x5c9c58b8b2c0, relation=0x78e27167ede0, change=0x5c9c58b8f2e0)
at pgoutput.c:1572
#18 0x00005c9c5607b265 in change_cb_wrapper (cache=0x5c9c58b4f900,
txn=0x5c9c58b8b2c0, relation=0x78e27167ede0, change=0x5c9c58b8f2e0)
at logical.c:1116
#19 0x00005c9c560a2cff in ReorderBufferApplyChange (rb=0x5c9c58b4f900,
txn=0x5c9c58b8b2c0, relation=0x78e27167ede0, change=0x5c9c58b8f2e0,
streaming=false) at reorderbuffer.c:2175
#20 0x00005c9c560a4856 in ReorderBufferProcessTXN (rb=0x5c9c58b4f900,
txn=0x5c9c58b8b2c0, commit_lsn=8380824, snapshot_now=0x5c9c58b736b8,
command_id=0, streaming=false) at reorderbuffer.c:2511
#21 0x00005c9c560a6aef in ReorderBufferReplay (txn=0x5c9c58b8b2c0,
rb=0x5c9c58b4f900, xid=752, commit_lsn=8380824, end_lsn=8380872,
commit_time=793368288662227, origin_id=0, origin_lsn=0) at
reorderbuffer.c:2973
#22 0x00005c9c560a6b71 in ReorderBufferCommit (rb=0x5c9c58b4f900,
xid=752, commit_lsn=8380824, end_lsn=8380872,
commit_time=793368288662227,
origin_id=0, origin_lsn=0) at reorderbuffer.c:2997
#23 0x00005c9c5606804a in DecodeCommit (ctx=0x5c9c58b4d8f0,
buf=0x7ffd9a445010, parsed=0x7ffd9a444e90, xid=752, two_phase=false)
at decode.c:730
#24 0x00005c9c56064d04 in xact_decode (ctx=0x5c9c58b4d8f0,
buf=0x7ffd9a445010) at decode.c:242
#25 0x00005c9c56063fe2 in LogicalDecodingProcessRecord
(ctx=0x5c9c58b4d8f0, record=0x5c9c58b4dd18) at decode.c:116
#26 0x00005c9c56135980 in XLogSendLogical () at walsender.c:3382
#27 0x00005c9c56133895 in WalSndLoop (send_data=0x5c9c56135836
<XLogSendLogical>) at walsender.c:2788
#28 0x00005c9c56130762 in StartLogicalReplication (cmd=0x5c9c58ab9a10)
at walsender.c:1496
#29 0x00005c9c56131ec3 in exec_replication_command (
cmd_string=0x5c9c58a83b90 "START_REPLICATION SLOT \"test1\"
LOGICAL 0/0 (proto_version '4', streaming 'parallel', origin 'any',
publication_names '\"pub1\"')") at walsender.c:2119
#30 0x00005c9c562abbe8 in PostgresMain (dbname=0x5c9c58abd598
"postgres", username=0x5c9c58abd580 "ubuntu") at postgres.c:4687
Thanks and Regards,
Shlok Kyal
On Thu, Feb 20, 2025 at 3:08 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
Dear Ajin,
I compared the patch 1 which does not employ a hash cache and has the
overhead of starting a transaction every time the filter is checked.I created a test setup of 10 million inserts in 3 different scenarios:
1. All inserts on unpublished tables
2. Half of the inserts on unpublished table and half on pupblished table
3. All inserts on published tables.The percentage improvement in the new optimized patch compared to the
old patch is:No transactions in publication: 85.39% improvement
Half transactions in publication: 72.70% improvement
All transactions in publication: 48.47% improvementAttaching a graph to show the difference.
I could not find any comparisons with HEAD. Can you clarify the throughput/latency/memory
usage with HEAD?
Here's the difference in latency with head. Again 10 million inserts
in 3 scenarios: All transactions on unpublished tables, half of the
transactions on unpublished tables and all transactions on published
tables
Conclusion:
The patched code with 100 transaction throttling significantly
improves performance, reducing execution time by ~69% when no
published transactions are involved, ~43% with partial published
transactions, and ~15% in all published transactions.
Attaching a graph showing the performance differences.
I will run tests comparing memory and throughput as well.
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
patched-code-vs-head.pngimage/png; name=patched-code-vs-head.pngDownload
�PNG
IHDR s � ���� sRGB ��� gAMA ���a pHYs � ��o�d ��IDATx^��|����xR��BZZ\
�����)����\.������.��p����i?�<���=�Tr��m�~�M������3��m^�aB!�B!�+���B!�B!�#�� !�B!�|��9!�B!���2'�B!��!R��B!�b>D��B!�B��H�B!�B��)sB!�B1"eN!�B!�C�� !�B!�|��9!�B!���2'�B!��!R��B!�b>D��B!�B��H��J*�j�����-EK�-����T�I<[Sz4�kz)���+y.��s8!���://o����������N<�b�x�y9^��Hg�{!^����>�=��_�]�gcV~��?�~�O�~���L���!���J��yGp�)M�#|����������zWx>��m&����Iw����� @���c����<�\�;�cN�mz��]~~�������x��G�\+ �|������_n���R[r�%m�u���[,k��}��gv��W����}E�y���\������/���^{�5����l�����}{�?��������VXX����,q������l���7��w�a�m�����j�m=#F��n���[}��}�����|��C���zk/g��9�����YII�}����]��n����{�~��������:w������.kK,��ls��1c���������������v�j+���/[��M ����?��c��]v�e���t^}�U{��g�_f�����"M���?�����_l��i6h������������O��7��E������.������Z*��;��?�`����}���6i�$+++�����k��!C�m������rx�=��SO=ew�}w�tB���;���}��[7�d�����&N�h����m��VK[���G����{� ����z�������|�>��c���+�l;������;c~��w�y�K s�-�<������1c|]��r��T>h�=�P����m���l���_�������N���;�&L�`���}�o �����2�,��� iI]B=o>��;��|���l~���Mi���6�:�?���
0���-��Y�~��'����l�
7��6��7`�<���X�������C'�h���Z�Q���OtW4
�
(�*�3�<���w_����K/��w8h�^{m[q�����F��7��?��3�����'����>�=:��~����"_�B����<G�z��|�gRQ���}���
%�?����:����>_>�w�i�Q����}g��_���������v��F�[��o���;��>�C��s��4���������[n�%��� �;����K�l�����F�.��r�����[o���Z�Lz�!�����2�`�F:(�����{��8���|G�8��|����;���lp�lC9�Mp�{��
��%�:�v����K���
�����j�F��)=t�v�m�M�6�Mr���o�<��$��{<O��[��m.��Q��������E/���������h?]tQ<x�HD�����V���o��=����=�X7n��TRY���"f�*++�V��!��������[:�}������&�gp3e��F�����7�K���������!�)��^����Cp�����O��l�_���5���L�����+���W��]w���Eg���SN��N;m�<� �O�@���,�=\�����`aN��8���8��p�H_�u�]�lD �!�����������;�2�5:t���_�A��E�"�x���l�-��3���f
�N9�f�<�H����l�����o��wZg��O>io���t�A�������m��6���M6���C��2E^Q�2� � S�#<�v�*��p/=O!<��;������X�@Zp1�r��������e.S���!��c���iH�p/�9��t?��\��$���&�uN@�a�������#���
������JI�w�J��A���L��+�#���~�
d�5�h(�������u�Y��cGo���_��29Wf��8�?��lmx?�-����d �A�?��w�3�,T�y�
�"-��QvxO���>q�Y�m�pC7��r��d*W���+w���{�2�y6��p������3���T�7\!l��uK�\�����b�UB�p����{?���O�����y�c�~�����LO�t��{�m���Jd3s���;��^���@y�`�
���+�b�f�X�%��B���J��1D�������KUl������m���~� �$Ku���[f9�����O?�g4�!`��)���p2CA���n**�����C����>�GN �N;��+c��|��q����}�������sB�OG��F��2���fS�q#;KaB��0R92��������F~�v��B�0�����7:
���{�K�x��9~1W��4�� �I{F��9�9����0
�(:�&�HsF����*iF���&�/y����q��v�i�(!���rK�t�M�z���f{I��CG�0���62r�+:F,�cV����={�2t8n��F�<�F�w�W�]e
�i;l���xS6�&?�l1SF��>(���KG�}���fn ,�7���=�G2+N���0�?��p�Q}:���tP�3=�{X��(����zH��w���qK��z(��:u�e����?�3�,Q�=yH���1���������1Y����0_��{���z/����R����$~S���k/�L7���EY�����o���`�+����ai*�C�m��`��~����������X�D���!M�t���2H������{�9S:���52�?(
�#=��]d������C�y������������4B)�!uj�1+3�Y����p�C[�������2W���Ox9H$��������=�RL:��W�
��2S��2�R��^�����Q���{�����k|F��i�X�G�bf����t���,�/��:�YExh��E�
��2G��;�������63.����]���C>�e(���d� �����:y%_��?����W�iWY�f!)c,Y�^�~!��!����Ma��!���u5ab��H�h�Y&K{�>'dr���E�?q�>�_��r���-e�$Ku ;+��3��P�� �4����,cFFi����#?�!�;3�<GZ��S��A_�OX���7�"_�Sd����a
���������7�QNp��Iw� ������b�t!4�T�A�rd�
��7'�
| F?�(��3����H��������$J~Sa���:E4
�AeNeOcOc��0~1�@C��f��i�VXa�����������N
*����s��;:l�|��WPr��r�K.���w���5
�"i���1�?�O��[�S��g�}4�� .t�D��Hc����A���(F4�aD�<�K6����Y�Ai�i�hixB�J=�5�C�Nz�o�E|��x�g�C�����F�0��O��AgI~1���(Za y��
���@�03��E��4�t�y'�A��\1sI��^��BG����o���;��t��P���:
��w��P^H:>q��3��rA~��� ���?|���L ct�(;!���#��
������i��iG�Q�C��X�v�����ne/~1�B���@~�Y���O�������t@�s�}�}�:����!M%��8SQt)��g�5�-��$rE��qtb��(k�#� �y�*R�0����^�K����O�'iO]D�i@��(R�2{��=�%o(k�#�Qp��H':�(�w���2G���L�� �a?��WXA�;H�>d��<#oP\�#Kq�m\�2�C�2��<����s�?�����5~2PHz����K���q�w���gl(�!M�g�w��i�����n��nP�C����P"�t�)t�C{����:��~��RP/i@}�;������9�/���!�-e��?e%�2?fn)������2��<�X�(a�N�e ����}���M"l�
���1�D��w�r��_����0��<�~~�/��/({�3* ��/�y��M:6�.����v�p�f�����i��NY�=����w���G����'����
���%�U������^R�(3��6�:��$� ����{�>1��2Z,���4�=�*���d�� {� w�UF��_~9�:�)�1O�J!�:\����\��r�x�5���UX)��N�F����;��~��n�������*���;��:��S���v�s���kD�;�{��R�BL����
�'\�b�a',�������\����\��a��'��a��*��/��r
��?�\�!��.��v�m���t{;�:�%���;��cG����r�v\� ���:9���>;�*O���U�)�II����)�)�yr���{�f�#��rH�5�����ulS������kD}����!�<�����������'����'o�2���������
��k�|z����[omp���:��s��!]t�wG^��w�y��+<���8����)����F��K��z���mx�`�;�uD������K=����wc8�"��Zk���f���{�9�E9s�{�����{�����Cgr��^{������SNR�m�]��/��C.H�<���s����>��3o�����(�\-���#=�"���p��k�}|� ����2C���t<��#d��t�F���y;�)�yu�e��2�\�����"Sx��<t�A
�;��m�����o�9��N�����2���S\SN�l��Q��z9w��w3u�T�/��BN��~�����\W^ye���k�?���S����O��{)��#_Q��%�C�:��#�����S����/7�q����S��o��z�2�s�w_���7��u���!\�A�\���a�Cx�S�;^{�5��u\�l8E�!,�'�G����7���m��3�{��t��L8���-yw��z9����S�S�����g��)��������������O����8�Qp��6�l���<S��;�c���rH8�b����?(<�E�L��s@:E��-�
��;Wx�����{�N����>u
z��B��)�>~�}�������/RN�lh?)��u�#���&���U�=���}]� �C���:��5b�����mLc��2Ay���"�z�GN{M[�u��;q
������g����o}8i;C�����J������
� ���?y��O�I���;�}%�N8���<v���e����;���'�<��0.�� u!�����.��kE0:��3��1���3=,9a6�Q ��Yn�h
#��F���a&7�.�#G��I��1����R�3
��W����@�m��D��$\�)gD��������G��w3�����p��_��yF�Q�L�1e���5\>=X��hV-�"L�f�?�d����"��/i�d-z������3ZM�1{FDy7#����U��Q�NF��c�<,�`t�4s
N��:p�%$!��e�K y?a�#������O[��g�'�
?��n��2ZF�]��#�����n����;�c���C�C�I� � @�)��lf
�2���,�i���t��/��3����=���+'q\���L����}d�r�����2�3;�;F�]'�A�I���'�H:��lf������Ag9��>��8R.�k�!`T�eA\��S>��kna&(����2KD��l�~f�XY@g�u(�1i��E /]Gw���O}�\�1e 3�%\@���>2B���d&�2f���@�)�r��l.�7y/������������:$�%�Sw�af�:�6 �����{X~� ���f@������?;���y���,!k,�����13G|�'m�O���<u2`���l
my������e��YUfC13���9��s��/[�]<KX#� �m�f�B� ��M���6e�����od(3��%���F��4�U�������4f�~����<����\���+���!L,�'~��q��q3����:��<�E/��O�f�C�/��LP�����?�2��@@Y6���k�cP� �*�<��p�Ig]{|���.o�2����8��E�(@�X�?t��J��(�-�D��R���a�#�v�h�����8� �� ��Wmm�oP�d4���J��_<�XvA����@���������(��"�(�����gG���n��8�����4�()�iJ����:�;+��e54�,��sM��e��������Lc�"J����'��aS�y����&\,�J_���� ��s�v@8Y�D9Bq`i#��d�S<.���?��rH�g�2�{��A>�n!��'�2���������,�y��>L���vt�P���b���N��X����J5K��~����u!�nY���m�i���1�����0�D}0+(��9���z(�ACG�0��\����� ��C~��6t��<�
~�1yKg:� �AV����+��������$=^�ucP�qe�kfiAy�,����r?�%��������s�o�(�(<�=D�BN�[������x��-{ �s�}����YvI��4�>��`m#i�=���y�]�x/rC�����������,�3a�P��e���uD
n������pn�'���������,�.�������7�5���>�6��.���eS�En�2���#�b�E:��b�b}zP��Eg���@�����+�� �iH�\Zbt5��=��c�2J��O��fGhxi��@��
��"�Uz����2�����L�}��-�
r����Tc�����t��;�O�N�)�����^f,��`��xd"��7
!
3�
�;:���`����X1��7f��*^��l�l��\��h��F|��"~\��*��M&�e�Y����
F��#�X\�=rf<���h8� �S/���H�PN9 1�,0�����,�����0.0�@q���?��A����+�g��^�>���� q�Y�@~�?�=:t�)����LP^�5&�s
��?tp�yzZ������>EF��{��|�O�7�v�3��I��Cg��S�O�M�+:��1J��JHH�N+�`��)����a�9@�Q���3hC���>d��6v�����<{i��Bh��k
��� ��DqCV(�(%$���)V0+��� a�+CsB\��'�EZ�����+e�v��s��{D��K]��p��+��I�c��X<��-�
�}vqk*�;+��G~S���_��~Lo_����9�ay K/i��4�E���* f�h<9� � ?h�Q���UKA�EC����tq�S�?�=uN��1��������t�������>���SfJ�m ~s�03�����f����
4a��\p�"�a`��C,���N�Fc�C��dS ��~�1���t/���(���o6��.���y�i�,k�7���q��W���A�\�%�����At(���t�;�,�&]��l�������L��*�_�X�Cy�u��
��p�b�b�B�(b�������G���qYH����;XvH��!,�c� ���Nd�4���A�tH:�!�P8��x�nN����-� 7�n��8��r��wR����p����������N:]����Hs��2���L180+�c���J��R�1{�3:�,�b��Y5�����2�a��:���c�)�VN2d) ���"�C}� ��(��kqY�~3�0��0C�o�"�(����+���(? /�V�4'���&�@�>4�F�#=)��d3X�b"�q�t�~���t�Y�3��<�=���8=��'.a +� ~SF�_(s
e�0�N�e�A[�����k�
��\!��}8����`�I�V��?} ������fy)�fG�x��C=M=K��d�J���4�X ��e9'����N*:�4b�������t{*j�&�a���A'�
��a����0+��k�Y�G��%�t������~3{hhx.��"�o��j:"�(�.(e,���������d-=��c��I#�,2��,�-�D�)��L��>������1���t����?!�!�q2�7���
4(`� ��-����:U�L
O&��a����v��_F2I3H�3=���44(�(�s2c��3�rN�7�����*�{��������f:F��2�N���dB�,q
"�I1�� iE����ae���A�g���P�h�Yz���1j�?���.S�){�
C��Stp)��y(���8�#,}d�����
K���0�� � W�QD�CZ�w��� f��M]��1 �'��T�jv��`G~�|#����4!���1���P��\R^��{����(�K9�� �3C�����$�=�n@������$��2�K~��s2{E�i��}%�P��{v�C��<x#^��C�D��F��,{��(7�e7�y�#��%A�df�}��:|��7��)]�kNd��'=>�42���2@�Fy�lS���B��3������w"��Wg�}�\��!�| ���Y���M��L�R����$�?�K�ez6��qH;T�i���Q�P��E�PY�H�q�3i���f�3^�1�@������e`v�0�������/����H��|b`���l�\��.�}������/���Y��X�� Nc��YfC��)����s�]�O� �P�h�)�����'(��
��������xf�@pG�I{� 2FZQw#��{�)�JS���1�S�:�gFf� ���`�pf�{T@(t���d��%uTvXfE���M:T���TL���?*).FqC�GEF#F��g��{�+>BD'��n�F��*W�K��x��Si���PS��81��AA����
�E��#����#34��'��������y%N��
�NHc��� ��N����"?P�h$ a�#���69��?����iF~O�f��/��t�x/�@q���k�m�=�l�?a��A�����g��0����;�s����3�O~� ����.�������?���;��CG��"nh i�p�b ��3y o��P��t�A��^�E�{��|'mq�~�f:����a�6~�,nP(_a�#� �^(]t��)�7+H[�H<x �;��p0�G��n:�(�����(�t>��a`&������C�J:�<�F��E}BP�P~gn�!���{�"����C�#��Q����F���� �Q��W��8�!�[�9@.�2�!yK�y �P�;�)s��<i@~�>��pR�'��A�)�c~�^�tv�{P��3d�A&��;�0:��'���#��EDxE�p!�<�����QB���t��yDg5�����;�p� uu��&!^���Q.y'iE����e8f��@����#��<���1���o�E��-�Cx��P��%�
��4D~)���td�m��A�$�(W@�a&<!����"-�=d��gO(mnI{��'���@e��
�(yE����GN�G�q��"��C�Ik����s��4�S7�F����N#��#�����A���i�=yN^�^�-�B�EH�!����y7i�� ?��4"~�(o�?��I�H�������Qr�3��� ������7����ms�Iw��;;�����%MB[���
��u�3�Isd��L> ��}�7yo�(��(�������!����<�_�P�������w�s��|s�b��
�FA�2[S���Q�P�������#L#A|�� ����-a�B2��N��-iG9�����3�-�%
���nF<�^:U��wd�g(#���`�MZ��9�D�i��/4~q�;IS�����������a���e������r�����sw�Mc�
����Gf�V��s�0����M���:S\�gV��[H/��t��.
y�X����P��-����� ���N!�=�m�M �3�gs�;���+��1����N�����Y�����4���y&,��TMeN�����w!C��5S]���s(���iN�}��Y����X<aV���PgP��c&�K��I���2%���)$y?�c�L��� Z)sB_��a_�9�`�_!�E�)c��;afS�8l;a� +~P��� ��,�g6��YP�b�2&l��R���i�����AB!X���B�_�4Yz�v�L+a����9!DTZZ!�B,x���j����Rh6S��Y��*w!�b��6>\b�A�\+�o�� ��_�/!D2p( '
!�����!e���\�D�a�z6�� !���i���!�"{�� ��<7:B4/(s|�J��
a�Y�����L����My&n��7��MG��R��K.�����j�t�a�%����B$�Q>$,�<��� 7���^}�U�������Q��8N2��|��������m|W�o��'�����r�-���|��� |x��_��=����g}��w�6�|s�����!�X�"�,�B4)s��Y)s|�3|pT�4�|$�s����hI(�tF�8� eN������������#���/�����������3J��W_m�{�W����(y|[��C���Z��<�">�~�QG���.k7�t�p��?�i�?��W Q����J����N��g��'�|b'�|�W���9!�G�\+�1e���.�Z1���|h����;Hk*X���c��V�f(td����
�.J��o���������]s�5�������l����������@-��>���`�_����]z��~�c��6��/���[n9����������oXl��l����'b�s�=v���7���r����!e��B�gR�%c���\�N�a�KUD�C���������Qk�oA�`��������2��������ng�}����O�:o��w�����
���o��N:�$k����GX���=���&M�d[m��=���~��w�}�g�P�P��=�\�������>�VZi�&��(��";t ����\4�bs��Rre�i�t@9��b~#S� �,�`�.�J?���p���/��+����?����!+=��_��B�RK>o����"�"{���YCe�\W����_�cn������{�q���B$2�%�b~'�?=z����t��cf
p���zw����{�%��n���/��s'N�^�z�%����q��J��6�"{
�tDf��x�������VYe_��&���r�������q����fw
�e�]��'4���(_8.�%��F����<��_���G��l�&�����j�%����'u���{��_������1!��2:�������������{v���������asJx�����4�����h} �,�m��]d#���?���+��_��m��u�'Pr� ��9u�%��������B�3���m��a��O/��[u�U����������~���P8�����7����X��"��F5���L�c!D����V
��i������G��h5��?���+���`��9���|rd�w���2d�m���7�
�E]d/�������FO�w����{6g�����m���~�6
��<��;4l��C��I^�c��?9,������Q����}Z��]x&��2��ry��P������'��t�\4v/�>������i�}S@���n���`���m�C=��v(q���J<1���2@�r���_���K%Q���=E�x��6q�5��m�$`I&m&��cf��
a�>B4)s��}�����k>e��sg��1Q�����o�@�2�1�{������s��g�����(y����r�)��j��{���]v�e~#xX���]w����s�G4�0r���t(z�
����~������q�E���t�e?���`m���~�d�LH�������G>�2���������?���!�A�'D�H��<;
�hqq2W���e,�c#v���KF!=|����]:t�8f�w������M6�d��"���k�����f����>��]w�u^y;��S�rP�=�y�r�r�-~_���h����
B�/0k}�UW�A���?&���#��t���=K��;����`
���h8
�g�� �\9��3��Yg��q�g ��q����//����,�B$�-��)sb���u�,Ya����0��6ac�� �D���Fo>>���Y������8,w��?.t:�)��(o_~�����O��#���{������1�c��=���\���Az8���a�y������q��k����}��7���zYd�3{n�yg���b������b���o���+�
��9��#d��=��C����M!�G����
:]j���r�
��c ��g�y����x����9�Q�� ��U��H#X��{8��ip����F� �\�!���c�|����n���w����'b~����2��Y2d����/��,ke�k�e��&�/3u��a�7�00�\�d9b�3�>*���>���z3�@N6�_�N��_B�������.lw =�MF��h��i�k�s5+89���1�bY#'\�p� ^y�Byb��t.����>���9��f�mN�3�X����>"�~8:�tJ��23�"�RKf�����Y���`X�� _�,r�&3��� ���(�� 2���\ ��1p�r�r(q�.+���=���~�����N����v�m���{��m�
6�3��>��w�
\$D���^|����l��[�5���>M�MF��R����|X����������zc����,��r������g������3g�g�������W^���dF��c������r�=����������;f�P���[:k���_�����[��X������k���w`�t
"Qe��=E��S�XF��D:(r�:�4��A>� ������U�CRpD:��>:�R�x��~ �������s�'