[BUG] wrong refresh when ALTER SUBSCRIPTION ADD/DROP PUBLICATION
Hi hackers,
When testing some other logical replication related patches, I found two
unexpected behaviours about ALTER SUBSCRIPTION ADD/DROP PUBLICATION.
(1)
when I execute the following sqls[1], the data of tables that registered to
'pub' wasn't copied to the subscriber side which means tablesync worker didn't
start.
-----sub originally had 2 pub nodes(pub,pub2)
ALTER SUBSCRIPTION sub drop PUBLICATION pub;
ALTER SUBSCRIPTION sub add PUBLICATION pub;
-----
(2)
And when I execute the following sqls, the data of table registered to 'pub2'
are synced again.
-----sub originally had 2 pub nodes(pub,pub2)
ALTER SUBSCRIPTION sub drop PUBLICATION pub;
ALTER SUBSCRIPTION sub REFRESH PUBLICATION;
-----
After looking into this problem, I think the reason is the
[alter sub add/drop publication] misused the function
AlterSubscription_refresh().
For DROP cases:
Currently, in function AlterSubscription_refresh(), it will first fetch the
target tables from the target publication, and also fetch the tables in
subscriber side from pg_subscription_rel. Then it will check each table from
local pg_subscription_rel, if the table does not exists in the tables fetched
from the target publication then drop it.
The logic above only works for SET PUBLICATION. However, When DROP PUBLICATION,
the tables fetched from target publication is actually the tables that need to
be dropped. If reuse the above logic, it will drop the wrong table which result
in unexpected behavioud in (1) and (2).(ADD PUBLICATION have similar problem).
So, I think we'd better fix this problem. I tried add some additional check in
AlterSubscription_refresh() which can avoid the problem like the attached
patch. Not sure do we need to further refactor.
Best regards,
houzj
Attachments:
0001-fix-ALTER-SUB-ADD-DROP-PUBLICATION.patchapplication/octet-stream; name=0001-fix-ALTER-SUB-ADD-DROP-PUBLICATION.patchDownload
From 411cf2a1af7960b7604b4120e5f5061734f3c796 Mon Sep 17 00:00:00 2001
From: Houzhijie <Houzhijie@foxmail.com>
Date: Mon, 2 Aug 2021 20:09:29 +0800
Subject: [PATCH] fix ALTER SUB ADD DROP PUBLICATION
---
src/backend/commands/subscriptioncmds.c | 76 +++++++++++++++++--------
1 file changed, 51 insertions(+), 25 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 22ae982328..9addfdd869 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -625,7 +625,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
}
static void
-AlterSubscription_refresh(Subscription *sub, bool copy_data)
+AlterSubscription_refresh(Subscription *sub, bool copy_data,
+ AlterSubscriptionType type)
{
char *err;
List *pubrel_names;
@@ -636,6 +637,7 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
int off;
int remove_rel_len;
Relation rel = NULL;
+ int ntables_to_drop = 0;
typedef struct SubRemoveRels
{
Oid relid;
@@ -644,6 +646,11 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
SubRemoveRels *sub_remove_rels;
WalReceiverConn *wrconn;
+ Assert(type == ALTER_SUBSCRIPTION_SET_PUBLICATION ||
+ type == ALTER_SUBSCRIPTION_ADD_PUBLICATION ||
+ type == ALTER_SUBSCRIPTION_DROP_PUBLICATION ||
+ type == ALTER_SUBSCRIPTION_REFRESH);
+
/* Load the library providing us libpq calls. */
load_file("libpqwalreceiver", false);
@@ -675,8 +682,10 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
subrel_local_oids[off++] = relstate->relid;
}
- qsort(subrel_local_oids, list_length(subrel_states),
- sizeof(Oid), oid_cmp);
+
+ if (type != ALTER_SUBSCRIPTION_DROP_PUBLICATION)
+ qsort(subrel_local_oids, list_length(subrel_states),
+ sizeof(Oid), oid_cmp);
/*
* Rels that we want to remove from subscription and drop any slots
@@ -700,22 +709,25 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
Oid relid;
relid = RangeVarGetRelid(rv, AccessShareLock, false);
-
- /* Check for supported relkind. */
- CheckSubscriptionRelkind(get_rel_relkind(relid),
- rv->schemaname, rv->relname);
-
pubrel_local_oids[off++] = relid;
- if (!bsearch(&relid, subrel_local_oids,
- list_length(subrel_states), sizeof(Oid), oid_cmp))
+ if (type != ALTER_SUBSCRIPTION_DROP_PUBLICATION)
{
- AddSubscriptionRelState(sub->oid, relid,
- copy_data ? SUBREL_STATE_INIT : SUBREL_STATE_READY,
- InvalidXLogRecPtr);
- ereport(DEBUG1,
- (errmsg_internal("table \"%s.%s\" added to subscription \"%s\"",
- rv->schemaname, rv->relname, sub->name)));
+ /* Check for supported relkind. */
+ CheckSubscriptionRelkind(get_rel_relkind(relid),
+ rv->schemaname, rv->relname);
+
+ if (!bsearch(&relid, subrel_local_oids,
+ list_length(subrel_states), sizeof(Oid), oid_cmp))
+ {
+ AddSubscriptionRelState(sub->oid, relid,
+ copy_data ? SUBREL_STATE_INIT :
+ SUBREL_STATE_READY,
+ InvalidXLogRecPtr);
+ ereport(DEBUG1,
+ (errmsg_internal("table \"%s.%s\" added to subscription \"%s\"",
+ rv->schemaname, rv->relname, sub->name)));
+ }
}
}
@@ -723,16 +735,30 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
* Next remove state for tables we should not care about anymore using
* the data we collected above
*/
- qsort(pubrel_local_oids, list_length(pubrel_names),
- sizeof(Oid), oid_cmp);
+ if (type != ALTER_SUBSCRIPTION_ADD_PUBLICATION)
+ {
+ qsort(pubrel_local_oids, list_length(pubrel_names),
+ sizeof(Oid), oid_cmp);
+ ntables_to_drop = list_length(subrel_states);
+ }
remove_rel_len = 0;
- for (off = 0; off < list_length(subrel_states); off++)
+ for (off = 0; off < ntables_to_drop; off++)
{
Oid relid = subrel_local_oids[off];
-
- if (!bsearch(&relid, pubrel_local_oids,
- list_length(pubrel_names), sizeof(Oid), oid_cmp))
+ bool drop_table = false;
+
+ if (type == ALTER_SUBSCRIPTION_SET_PUBLICATION ||
+ type == ALTER_SUBSCRIPTION_REFRESH)
+ drop_table = !bsearch(&relid, pubrel_local_oids,
+ list_length(pubrel_names),
+ sizeof(Oid), oid_cmp);
+ else if (type == ALTER_SUBSCRIPTION_DROP_PUBLICATION)
+ drop_table = bsearch(&relid, pubrel_local_oids,
+ list_length(pubrel_names),
+ sizeof(Oid), oid_cmp);
+
+ if (drop_table)
{
char state;
XLogRecPtr statelsn;
@@ -1019,7 +1045,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
/* Make sure refresh sees the new list of publications. */
sub->publications = stmt->publication;
- AlterSubscription_refresh(sub, opts.copy_data);
+ AlterSubscription_refresh(sub, opts.copy_data, stmt->kind);
}
break;
@@ -1070,7 +1096,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
/* Only refresh the added/dropped list of publications. */
sub->publications = stmt->publication;
- AlterSubscription_refresh(sub, opts.copy_data);
+ AlterSubscription_refresh(sub, opts.copy_data, stmt->kind);
}
break;
@@ -1112,7 +1138,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION ... REFRESH");
- AlterSubscription_refresh(sub, opts.copy_data);
+ AlterSubscription_refresh(sub, opts.copy_data, stmt->kind);
break;
}
--
2.28.0.windows.1
Hi,
On Mon, Aug 2, 2021 at 10:52 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
Hi hackers,
When testing some other logical replication related patches, I found two
unexpected behaviours about ALTER SUBSCRIPTION ADD/DROP PUBLICATION.(1)
when I execute the following sqls[1], the data of tables that registered to
'pub' wasn't copied to the subscriber side which means tablesync worker didn't
start.-----sub originally had 2 pub nodes(pub,pub2)
ALTER SUBSCRIPTION sub drop PUBLICATION pub;
ALTER SUBSCRIPTION sub add PUBLICATION pub;
-----(2)
And when I execute the following sqls, the data of table registered to 'pub2'
are synced again.-----sub originally had 2 pub nodes(pub,pub2)
ALTER SUBSCRIPTION sub drop PUBLICATION pub;
ALTER SUBSCRIPTION sub REFRESH PUBLICATION;
-----
I could reproduce this issue by the above steps.
After looking into this problem, I think the reason is the
[alter sub add/drop publication] misused the function
AlterSubscription_refresh().For DROP cases:
Currently, in function AlterSubscription_refresh(), it will first fetch the
target tables from the target publication, and also fetch the tables in
subscriber side from pg_subscription_rel. Then it will check each table from
local pg_subscription_rel, if the table does not exists in the tables fetched
from the target publication then drop it.The logic above only works for SET PUBLICATION. However, When DROP PUBLICATION,
the tables fetched from target publication is actually the tables that need to
be dropped. If reuse the above logic, it will drop the wrong table which result
in unexpected behavioud in (1) and (2).(ADD PUBLICATION have similar problem).
Yes. For example, suppose that the publisher has two publications pub1
and pub2 for table1 and table2, respecitively. And we create a
subscription subscribing to those two publications.
postgres(1:74454)=# \dRs sub
List of subscriptions
Name | Owner | Enabled | Publication
------+----------+---------+-------------
sub | masahiko | t | {pub1,pub2}
(1 row)
postgres(1:74454)=# select srsubid, srrelid::regclass::text,
srsubstate, srsublsn from pg_subscription_rel;
srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
16394 | table1 | r | 0/170F388
16394 | table2 | r | 0/170F388
(2 rows)
After dropping pub1 from the subscription, 'table2' associated with
'pub2' is removed:
postgres(1:74454)=# alter subscription sub drop publication pub1;
ALTER SUBSCRIPTION
postgres(1:74454)=# select srsubid, srrelid::regclass::text,
srsubstate, srsublsn from pg_subscription_rel;
srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
16394 | table1 | r | 0/170F388
(1 row)
So, I think we'd better fix this problem. I tried add some additional check in
AlterSubscription_refresh() which can avoid the problem like the attached
patch. Not sure do we need to further refactor.
I've not looked at the patch deeply yet but I think that the following
one line change seems to fix the cases you reported, although I migit
be missing something:
diff --git a/src/backend/commands/subscriptioncmds.c
b/src/backend/commands/subscriptioncmds.c
index 22ae982328..c74d6d51d6 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1068,7 +1068,7 @@ AlterSubscription(ParseState *pstate,
AlterSubscriptionStmt *stmt,
PreventInTransactionBlock(isTopLevel, "ALTER
SUBSCRIPTION with refresh");
/* Only refresh the added/dropped list of publications. */
- sub->publications = stmt->publication;
+ sub->publications = publist;
AlterSubscription_refresh(sub, opts.copy_data);
}
Also, I think we need to add some regression tests for this.
Currently, subscription.sql has tests for ADD/DROP PUBLICATION but
they don't check pg_subscription_rel.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
On Wednesday, August 4, 2021 1:47 PM Masahiko Sawada <sawada.mshk@gmail.com>
On Mon, Aug 2, 2021 at 10:52 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:Hi hackers,
When testing some other logical replication related patches, I found
two unexpected behaviours about ALTER SUBSCRIPTION ADD/DROPPUBLICATION.
(1)
when I execute the following sqls[1], the data of tables that
registered to 'pub' wasn't copied to the subscriber side which means
tablesync worker didn't start.-----sub originally had 2 pub nodes(pub,pub2) ALTER SUBSCRIPTION sub
drop PUBLICATION pub; ALTER SUBSCRIPTION sub add PUBLICATION pub;
-----(2)
And when I execute the following sqls, the data of table registered to 'pub2'
are synced again.-----sub originally had 2 pub nodes(pub,pub2) ALTER SUBSCRIPTION sub
drop PUBLICATION pub; ALTER SUBSCRIPTION sub REFRESH PUBLICATION;
-----I could reproduce this issue by the above steps.
Thanks for looking into this problem.
I've not looked at the patch deeply yet but I think that the following one line
change seems to fix the cases you reported, although I migit be missing
something:diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 22ae982328..c74d6d51d6 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -1068,7 +1068,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");/* Only refresh the added/dropped list of publications. */ - sub->publications = stmt->publication; + sub->publications = publist;AlterSubscription_refresh(sub, opts.copy_data);
}
I considered the same fix before, but with the above fix, it seems refresh both
existsing publications and new publication, which most of people didn't like in
previous discussion[1]/messages/by-id/MEYP282MB166921C8622675A5C0701C31B6BB0@MEYP282MB1669.AUSP282.PROD.OUTLOOK.COM. From the doc[2]https://www.postgresql.org/docs/14/sql-altersubscription.html By default, this command will also act like REFRESH PUBLICATION, except that in case of ADD or DROP, only the added or dropped publications are refreshed., IIRC, Currently, the ADD/DROP
PUBLICATION is supposed to only refresh the new publication.
[1]: /messages/by-id/MEYP282MB166921C8622675A5C0701C31B6BB0@MEYP282MB1669.AUSP282.PROD.OUTLOOK.COM
[2]: https://www.postgresql.org/docs/14/sql-altersubscription.html By default, this command will also act like REFRESH PUBLICATION, except that in case of ADD or DROP, only the added or dropped publications are refreshed.
By default, this command will also act like REFRESH PUBLICATION, except that in
case of ADD or DROP, only the added or dropped publications are refreshed.
Also, I think we need to add some regression tests for this.
Currently, subscription.sql has tests for ADD/DROP PUBLICATION but they don't
check pg_subscription_rel.
Currently, the subscription.sql doesn't have actual tables to
be subscribed which means the pg_subscription_rel is empty. I am not sure where
to place the testcases, temporarily placed in 001_rep_changes.pl.
Best regards,
houzj
Attachments:
v2-0001-fix-ALTER-SUB-ADD-DROP-PUBLICATION.patchapplication/octet-stream; name=v2-0001-fix-ALTER-SUB-ADD-DROP-PUBLICATION.patchDownload
From 78b8e3e959d83e898ad5befaf55fe46c6fdd6f20 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <HouZhijie@foxmail.com>
Date: Wed, 4 Aug 2021 15:33:15 +0800
Subject: [PATCH] fix-ALTER-SUB-ADD-DROP-PUBLICATION
---
src/backend/commands/subscriptioncmds.c | 76 +++++++++++++++-------
src/test/subscription/t/001_rep_changes.pl | 21 +++++-
2 files changed, 71 insertions(+), 26 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5157f44058..68a128ced0 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -606,7 +606,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
}
static void
-AlterSubscription_refresh(Subscription *sub, bool copy_data)
+AlterSubscription_refresh(Subscription *sub, bool copy_data,
+ AlterSubscriptionType type)
{
char *err;
List *pubrel_names;
@@ -617,6 +618,7 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
int off;
int remove_rel_len;
Relation rel = NULL;
+ int ntables_to_drop = 0;
typedef struct SubRemoveRels
{
Oid relid;
@@ -625,6 +627,11 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
SubRemoveRels *sub_remove_rels;
WalReceiverConn *wrconn;
+ Assert(type == ALTER_SUBSCRIPTION_SET_PUBLICATION ||
+ type == ALTER_SUBSCRIPTION_ADD_PUBLICATION ||
+ type == ALTER_SUBSCRIPTION_DROP_PUBLICATION ||
+ type == ALTER_SUBSCRIPTION_REFRESH);
+
/* Load the library providing us libpq calls. */
load_file("libpqwalreceiver", false);
@@ -656,8 +663,10 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
subrel_local_oids[off++] = relstate->relid;
}
- qsort(subrel_local_oids, list_length(subrel_states),
- sizeof(Oid), oid_cmp);
+
+ if (type != ALTER_SUBSCRIPTION_DROP_PUBLICATION)
+ qsort(subrel_local_oids, list_length(subrel_states),
+ sizeof(Oid), oid_cmp);
/*
* Rels that we want to remove from subscription and drop any slots
@@ -681,22 +690,25 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
Oid relid;
relid = RangeVarGetRelid(rv, AccessShareLock, false);
-
- /* Check for supported relkind. */
- CheckSubscriptionRelkind(get_rel_relkind(relid),
- rv->schemaname, rv->relname);
-
pubrel_local_oids[off++] = relid;
- if (!bsearch(&relid, subrel_local_oids,
- list_length(subrel_states), sizeof(Oid), oid_cmp))
+ if (type != ALTER_SUBSCRIPTION_DROP_PUBLICATION)
{
- AddSubscriptionRelState(sub->oid, relid,
- copy_data ? SUBREL_STATE_INIT : SUBREL_STATE_READY,
- InvalidXLogRecPtr);
- ereport(DEBUG1,
- (errmsg_internal("table \"%s.%s\" added to subscription \"%s\"",
- rv->schemaname, rv->relname, sub->name)));
+ /* Check for supported relkind. */
+ CheckSubscriptionRelkind(get_rel_relkind(relid),
+ rv->schemaname, rv->relname);
+
+ if (!bsearch(&relid, subrel_local_oids,
+ list_length(subrel_states), sizeof(Oid), oid_cmp))
+ {
+ AddSubscriptionRelState(sub->oid, relid,
+ copy_data ? SUBREL_STATE_INIT :
+ SUBREL_STATE_READY,
+ InvalidXLogRecPtr);
+ ereport(DEBUG1,
+ (errmsg_internal("table \"%s.%s\" added to subscription \"%s\"",
+ rv->schemaname, rv->relname, sub->name)));
+ }
}
}
@@ -704,16 +716,30 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
* Next remove state for tables we should not care about anymore using
* the data we collected above
*/
- qsort(pubrel_local_oids, list_length(pubrel_names),
- sizeof(Oid), oid_cmp);
+ if (type != ALTER_SUBSCRIPTION_ADD_PUBLICATION)
+ {
+ qsort(pubrel_local_oids, list_length(pubrel_names),
+ sizeof(Oid), oid_cmp);
+ ntables_to_drop = list_length(subrel_states);
+ }
remove_rel_len = 0;
- for (off = 0; off < list_length(subrel_states); off++)
+ for (off = 0; off < ntables_to_drop; off++)
{
Oid relid = subrel_local_oids[off];
-
- if (!bsearch(&relid, pubrel_local_oids,
- list_length(pubrel_names), sizeof(Oid), oid_cmp))
+ bool drop_table = false;
+
+ if (type == ALTER_SUBSCRIPTION_SET_PUBLICATION ||
+ type == ALTER_SUBSCRIPTION_REFRESH)
+ drop_table = !bsearch(&relid, pubrel_local_oids,
+ list_length(pubrel_names),
+ sizeof(Oid), oid_cmp);
+ else if (type == ALTER_SUBSCRIPTION_DROP_PUBLICATION)
+ drop_table = bsearch(&relid, pubrel_local_oids,
+ list_length(pubrel_names),
+ sizeof(Oid), oid_cmp);
+
+ if (drop_table)
{
char state;
XLogRecPtr statelsn;
@@ -994,7 +1020,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
/* Make sure refresh sees the new list of publications. */
sub->publications = stmt->publication;
- AlterSubscription_refresh(sub, opts.copy_data);
+ AlterSubscription_refresh(sub, opts.copy_data, stmt->kind);
}
break;
@@ -1045,7 +1071,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
/* Only refresh the added/dropped list of publications. */
sub->publications = stmt->publication;
- AlterSubscription_refresh(sub, opts.copy_data);
+ AlterSubscription_refresh(sub, opts.copy_data, stmt->kind);
}
break;
@@ -1087,7 +1113,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION ... REFRESH");
- AlterSubscription_refresh(sub, opts.copy_data);
+ AlterSubscription_refresh(sub, opts.copy_data, stmt->kind);
break;
}
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 0c84d87873..f04df8b8d9 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -6,7 +6,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 32;
+use Test::More tests => 34;
# Initialize publisher node
my $node_publisher = PostgresNode->new('publisher');
@@ -262,6 +262,25 @@ $result =
$node_subscriber->safe_psql('postgres', "SELECT count(*) FROM temp1");
is($result, qq(1), 'check rows on subscriber with multiple publications');
+# Test changing the list of subscribed publications
+# Removes publications from the list of publications
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub_temp1 DROP PUBLICATION tap_pub_temp1");
+
+# pg_subscription_rel should only have one row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel t1, pg_subscription t2 WHERE t1.srsubid = t2.oid AND t2.subname = 'tap_sub_temp1'");
+is($result, qq(1),
+ 'check one relation was removed from subscribed list');
+
+# Add additional publications
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub_temp1 ADD PUBLICATION tap_pub_temp1");
+
+# pg_subscription_rel should only have two rows
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel t1, pg_subscription t2 WHERE t1.srsubid = t2.oid AND t2.subname = 'tap_sub_temp1'");
+is($result, qq(2),
+ 'check one more relation was subscribed');
+
# Drop subscription as we don't need it anymore
$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub_temp1");
--
2.27.0
On Wed, Aug 4, 2021 at 5:06 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
On Wednesday, August 4, 2021 1:47 PM Masahiko Sawada <sawada.mshk@gmail.com>
On Mon, Aug 2, 2021 at 10:52 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:Hi hackers,
When testing some other logical replication related patches, I found
two unexpected behaviours about ALTER SUBSCRIPTION ADD/DROPPUBLICATION.
(1)
when I execute the following sqls[1], the data of tables that
registered to 'pub' wasn't copied to the subscriber side which means
tablesync worker didn't start.-----sub originally had 2 pub nodes(pub,pub2) ALTER SUBSCRIPTION sub
drop PUBLICATION pub; ALTER SUBSCRIPTION sub add PUBLICATION pub;
-----(2)
And when I execute the following sqls, the data of table registered to 'pub2'
are synced again.-----sub originally had 2 pub nodes(pub,pub2) ALTER SUBSCRIPTION sub
drop PUBLICATION pub; ALTER SUBSCRIPTION sub REFRESH PUBLICATION;
-----I could reproduce this issue by the above steps.
Thanks for looking into this problem.
I've not looked at the patch deeply yet but I think that the following one line
change seems to fix the cases you reported, although I migit be missing
something:diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 22ae982328..c74d6d51d6 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -1068,7 +1068,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");/* Only refresh the added/dropped list of publications. */ - sub->publications = stmt->publication; + sub->publications = publist;AlterSubscription_refresh(sub, opts.copy_data);
}I considered the same fix before, but with the above fix, it seems refresh both
existsing publications and new publication, which most of people didn't like in
previous discussion[1]. From the doc[2], IIRC, Currently, the ADD/DROP
PUBLICATION is supposed to only refresh the new publication.
What do you mean by refreshing both existing and new publications? I
dropped and created one publication out of three publications that the
subscription is subscribing to but the corresponding entries in
pg_subscription_rel for tables associated with the other two
publications were not changed and the table sync workers also were not
started for them.
Also, I think we need to add some regression tests for this.
Currently, subscription.sql has tests for ADD/DROP PUBLICATION but they don't
check pg_subscription_rel.Currently, the subscription.sql doesn't have actual tables to
be subscribed which means the pg_subscription_rel is empty. I am not sure where
to place the testcases, temporarily placed in 001_rep_changes.pl.
Right. Adding tests to 001_rep_changes.pl seems good to me.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
On Wednesday, August 4, 2021 7:00 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote
On Wed, Aug 4, 2021 at 5:06 PM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
On Wednesday, August 4, 2021 1:47 PM Masahiko Sawada <sawada.mshk@gmail.com>
I've not looked at the patch deeply yet but I think that the
following one line change seems to fix the cases you reported,
although I migit be missing
something:diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 22ae982328..c74d6d51d6 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -1068,7 +1068,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");/* Only refresh the added/dropped list of publications.
*/
- sub->publications = stmt->publication; + sub->publications = publist;AlterSubscription_refresh(sub, opts.copy_data);
}I considered the same fix before, but with the above fix, it seems
refresh both existsing publications and new publication, which most of
people didn't like in previous discussion[1]. From the doc[2], IIRC,
Currently, the ADD/DROP PUBLICATION is supposed to only refresh the newpublication.
What do you mean by refreshing both existing and new publications? I dropped
and created one publication out of three publications that the subscription is
subscribing to but the corresponding entries in pg_subscription_rel for tables
associated with the other two publications were not changed and the table sync
workers also were not started for them.
+ sub->publications = publist;
- sub->publications = stmt->publication;
With the above fix, When ADD/DROP PUBLICATION, I meant the table that doesn't
belong to the dropped or added publication could be updated in
pg_subscription_rel.
I can see the effect in the following example:
1)-publisher-
CREATE TABLE test,test2,test3
CREATE PUBLICATION pub FOR TABLE test;
CREATE PUBLICATION pub2 FOR TABLE test2;
2)-subscriber-
CREATE TABLE test,test2,test3
CREATE SUBSCRIPTION sub CONNECTION 'dbname=postgres port=10000' PUBLICATION pub,pub2;
select * from pg_subscription_rel;
-[ RECORD 1 ]---------
srsubid | 16393
srrelid | 16387
srsubstate | r
srsublsn | 0/150A2D8
-[ RECORD 2 ]---------
srsubid | 16393
srrelid | 16384
srsubstate | r
srsublsn | 0/150A310
3)-publisher-
alter publication pub add table test3;
4)-subscriber-
ALTER SUBSCRIPTION sub drop PUBLICATION pub2;
select * from pg_subscription_rel;
-[ RECORD 1 ]---------
srsubid | 16393
srrelid | 16384
srsubstate | r
srsublsn | 0/150A310
-[ RECORD 2 ]---------
srsubid | 16393
srrelid | 16390
srsubstate | r
srsublsn |
I can see the [RECORD 2] which is the new registered table in 'pub2' (see step 3)
has been added to the pg_subscription_rel. Based pervious discussion and
document, that table seems shouldn't be refreshed when drop another
publication.
Best regards,
houzj
On Wed, Aug 4, 2021 at 9:19 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
On Wednesday, August 4, 2021 7:00 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote
On Wed, Aug 4, 2021 at 5:06 PM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
On Wednesday, August 4, 2021 1:47 PM Masahiko Sawada <sawada.mshk@gmail.com>
I've not looked at the patch deeply yet but I think that the
following one line change seems to fix the cases you reported,
although I migit be missing
something:diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 22ae982328..c74d6d51d6 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -1068,7 +1068,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");/* Only refresh the added/dropped list of publications.
*/
- sub->publications = stmt->publication; + sub->publications = publist;AlterSubscription_refresh(sub, opts.copy_data);
}I considered the same fix before, but with the above fix, it seems
refresh both existsing publications and new publication, which most of
people didn't like in previous discussion[1]. From the doc[2], IIRC,
Currently, the ADD/DROP PUBLICATION is supposed to only refresh the newpublication.
What do you mean by refreshing both existing and new publications? I dropped
and created one publication out of three publications that the subscription is
subscribing to but the corresponding entries in pg_subscription_rel for tables
associated with the other two publications were not changed and the table sync
workers also were not started for them.+ sub->publications = publist; - sub->publications = stmt->publication; With the above fix, When ADD/DROP PUBLICATION, I meant the table that doesn't belong to the dropped or added publication could be updated in pg_subscription_rel.I can see the effect in the following example:
1)-publisher-
CREATE TABLE test,test2,test3
CREATE PUBLICATION pub FOR TABLE test;
CREATE PUBLICATION pub2 FOR TABLE test2;
2)-subscriber-
CREATE TABLE test,test2,test3
CREATE SUBSCRIPTION sub CONNECTION 'dbname=postgres port=10000' PUBLICATION pub,pub2;
select * from pg_subscription_rel;
-[ RECORD 1 ]---------
srsubid | 16393
srrelid | 16387
srsubstate | r
srsublsn | 0/150A2D8
-[ RECORD 2 ]---------
srsubid | 16393
srrelid | 16384
srsubstate | r
srsublsn | 0/150A3103)-publisher-
alter publication pub add table test3;
4)-subscriber-
ALTER SUBSCRIPTION sub drop PUBLICATION pub2;
select * from pg_subscription_rel;
-[ RECORD 1 ]---------
srsubid | 16393
srrelid | 16384
srsubstate | r
srsublsn | 0/150A310
-[ RECORD 2 ]---------
srsubid | 16393
srrelid | 16390
srsubstate | r
srsublsn |I can see the [RECORD 2] which is the new registered table in 'pub2' (see step 3)
has been added to the pg_subscription_rel. Based pervious discussion and
document, that table seems shouldn't be refreshed when drop another
publication.
Thanks for the explanation! I understood the problem.
I've reviewed v2 patch. Here are some comments:
+ if (type == ALTER_SUBSCRIPTION_SET_PUBLICATION ||
+ type == ALTER_SUBSCRIPTION_REFRESH)
+ drop_table = !bsearch(&relid, pubrel_local_oids,
+ list_length(pubrel_names),
+ sizeof(Oid), oid_cmp);
+ else if (type == ALTER_SUBSCRIPTION_DROP_PUBLICATION)
+ drop_table = bsearch(&relid, pubrel_local_oids,
+ list_length(pubrel_names),
+ sizeof(Oid), oid_cmp);
IIUC, in DROP PUBLICATION cases, we don't need to refer to the tables
in the publication on the publisher that we're dropping in order to
check whether to remove the relation. If we drop a table from the
publication on the publisher before dropping the publication from the
subscription on the subscriber, the pg_subscription_rel entry for the
table remains. For example, please consider the following case:
On publisher and subscriber:
create table t1 (a int);
create table t2 (a int);
create table t3 (a int);
On publisher:
create publication pub12 for table t1, t2;
create publication pub3 for table t3;
On subscriber:
create subscription sub connection 'dbname=postgres' publication pub12, pub3;
On publisher:
alter publication pub12 drop table t2;
On subscriber:
alter subscription sub drop publication pub12;
select srsubid, srrelid::regclass::text, srsubstate, srsublsn from
pg_subscription_rel;
srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
16393 | t3 | r | 0/1707CE0
16393 | t2 | r | 0/1707CE0
With v2 patch, the pg_subscription_rel has the entry for 't2' but it
should not exist.
But that doesn't mean that drop_table should always be true in DROP
PULIBCATION cases. Since the tables associated with two publications
can overlap, we cannot remove the relation from pg_subscription_rel if
it is also associated with the remaining publication. For example:
On publisher and subscriber:
create table t1 (a int);
create table t2 (a int);
create table t3 (a int);
On publisher:
create publication pub12 for table t1, t2;
create publication pub13 for table t1, t3;
On subscriber:
create subscription sub connection 'dbname=postgres' publication pub12, pub13;
On subscriber:
alter subscription sub drop publication pub12;
select srsubid, srrelid::regclass::text, srsubstate, srsublsn from
pg_subscription_rel;
srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
16393 | t3 | r | 0/17080B0
(1 row)
With v2 patch, the pg_subscription_rel has only one entry for t3 but
it should have also the one for t1 since it's still subscribing to
pub13.
To summary, I think that what we want to do in DROP SUBSCRIPTION cases
is to drop relations from pg_subscription_rel that are no longer
included in the set of after-calculated publications. In the latter
example, when ALTER SUBSCRIPTION ... DROP PUBLICATION pub12, we should
compare the current set of relations associated with {pub12, pub13} to
the set of relcations associated with pub13, not pub12. Then we can
find out t2 is no longer associated with the after-calculated
publication (i.g., pub13) so remove it.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
On Thu, Aug 5, 2021 at 2:08 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Wed, Aug 4, 2021 at 9:19 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:On Wednesday, August 4, 2021 7:00 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote
On Wed, Aug 4, 2021 at 5:06 PM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
On Wednesday, August 4, 2021 1:47 PM Masahiko Sawada <sawada.mshk@gmail.com>
I've not looked at the patch deeply yet but I think that the
following one line change seems to fix the cases you reported,
although I migit be missing
something:diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 22ae982328..c74d6d51d6 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -1068,7 +1068,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");/* Only refresh the added/dropped list of publications.
*/
- sub->publications = stmt->publication; + sub->publications = publist;AlterSubscription_refresh(sub, opts.copy_data);
}I considered the same fix before, but with the above fix, it seems
refresh both existsing publications and new publication, which most of
people didn't like in previous discussion[1]. From the doc[2], IIRC,
Currently, the ADD/DROP PUBLICATION is supposed to only refresh the newpublication.
What do you mean by refreshing both existing and new publications? I dropped
and created one publication out of three publications that the subscription is
subscribing to but the corresponding entries in pg_subscription_rel for tables
associated with the other two publications were not changed and the table sync
workers also were not started for them.+ sub->publications = publist; - sub->publications = stmt->publication; With the above fix, When ADD/DROP PUBLICATION, I meant the table that doesn't belong to the dropped or added publication could be updated in pg_subscription_rel.I can see the effect in the following example:
1)-publisher-
CREATE TABLE test,test2,test3
CREATE PUBLICATION pub FOR TABLE test;
CREATE PUBLICATION pub2 FOR TABLE test2;
2)-subscriber-
CREATE TABLE test,test2,test3
CREATE SUBSCRIPTION sub CONNECTION 'dbname=postgres port=10000' PUBLICATION pub,pub2;
select * from pg_subscription_rel;
-[ RECORD 1 ]---------
srsubid | 16393
srrelid | 16387
srsubstate | r
srsublsn | 0/150A2D8
-[ RECORD 2 ]---------
srsubid | 16393
srrelid | 16384
srsubstate | r
srsublsn | 0/150A3103)-publisher-
alter publication pub add table test3;
4)-subscriber-
ALTER SUBSCRIPTION sub drop PUBLICATION pub2;
select * from pg_subscription_rel;
-[ RECORD 1 ]---------
srsubid | 16393
srrelid | 16384
srsubstate | r
srsublsn | 0/150A310
-[ RECORD 2 ]---------
srsubid | 16393
srrelid | 16390
srsubstate | r
srsublsn |I can see the [RECORD 2] which is the new registered table in 'pub2' (see step 3)
has been added to the pg_subscription_rel. Based pervious discussion and
document, that table seems shouldn't be refreshed when drop another
publication.Thanks for the explanation! I understood the problem.
I've reviewed v2 patch. Here are some comments:
Regarding regression tests, since ADD/DROP PUBLICATION logic could be
complicated it seems to me that we can add tests to a new file, say
alter_sub_pub.pl, that includes some scenarios as I mentioned in the
previous message.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
On Thursday, August 5, 2021 1:09 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote
I've reviewed v2 patch. Here are some comments:
+ if (type == ALTER_SUBSCRIPTION_SET_PUBLICATION || + type == ALTER_SUBSCRIPTION_REFRESH) + drop_table = !bsearch(&relid, pubrel_local_oids, + list_length(pubrel_names), + sizeof(Oid), oid_cmp); + else if (type == ALTER_SUBSCRIPTION_DROP_PUBLICATION) + drop_table = bsearch(&relid, pubrel_local_oids, + list_length(pubrel_names), + sizeof(Oid), oid_cmp);IIUC, in DROP PUBLICATION cases, we don't need to refer to the tables in the
publication on the publisher that we're dropping in order to check whether to
remove the relation. If we drop a table from the publication on the publisher
before dropping the publication from the subscription on the subscriber, the
pg_subscription_rel entry for the table remains. For example, please consider the
following case:On publisher and subscriber:
create table t1 (a int);
create table t2 (a int);
create table t3 (a int);On publisher:
create publication pub12 for table t1, t2; create publication pub3 for table t3;On subscriber:
create subscription sub connection 'dbname=postgres' publication pub12, pub3;On publisher:
alter publication pub12 drop table t2;On subscriber:
alter subscription sub drop publication pub12; select srsubid,
srrelid::regclass::text, srsubstate, srsublsn from pg_subscription_rel;srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
16393 | t3 | r | 0/1707CE0
16393 | t2 | r | 0/1707CE0With v2 patch, the pg_subscription_rel has the entry for 't2' but it should not
exist.But that doesn't mean that drop_table should always be true in DROP
PULIBCATION cases. Since the tables associated with two publications can
overlap, we cannot remove the relation from pg_subscription_rel if it is also
associated with the remaining publication. For example:On publisher and subscriber:
create table t1 (a int);
create table t2 (a int);
create table t3 (a int);On publisher:
create publication pub12 for table t1, t2; create publication pub13 for table t1, t3;On subscriber:
create subscription sub connection 'dbname=postgres' publication pub12,
pub13;On subscriber:
alter subscription sub drop publication pub12; select srsubid,
srrelid::regclass::text, srsubstate, srsublsn from pg_subscription_rel; srsubid |
srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
16393 | t3 | r | 0/17080B0
(1 row)With v2 patch, the pg_subscription_rel has only one entry for t3 but it should
have also the one for t1 since it's still subscribing to pub13.To summary, I think that what we want to do in DROP SUBSCRIPTION cases is to
drop relations from pg_subscription_rel that are no longer included in the set of
after-calculated publications. In the latter example, when ALTER
SUBSCRIPTION ... DROP PUBLICATION pub12, we should compare the current
set of relations associated with {pub12, pub13} to the set of relcations associated
with pub13, not pub12. Then we can find out t2 is no longer associated with the
after-calculated publication (i.g., pub13) so remove it.
Thanks for the review. You are right, I missed this point.
Attach new version patch which fix these problems(Also add a new pl-test).
Best regards,
Houzj
Attachments:
v3-0001-fix-ALTER-SUB-ADD-DROP-PUBLICATION.patchapplication/octet-stream; name=v3-0001-fix-ALTER-SUB-ADD-DROP-PUBLICATION.patchDownload
From ac986d987b1eae6dae320deb76686cac2d904bf2 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <HouZhijie@foxmail.com>
Date: Thu, 5 Aug 2021 18:29:26 +0800
Subject: [PATCH] 0001-fix-ALTER-SUB-ADD-DROP-PUBLICATION
Currently, in function AlterSubscription_refresh(), it will compare
the tables from the target publication with the tables in local
pg_subscription_rel. And then add the tables to pg_subscription_rel
which not exists yet, and delele the tables from pg_subscription_rel
which doesn't exists in target publication.
But When ALTER SUB ADD/DROP PUBLICATION, the code only pass added or dropped
publications as the target publication to AlterSubscription_refresh() which
could result in wrong record in pg_subscription_rel and unexpected table sync.
Fix the use of AlterSubscription_refresh() by passing correct publications
to it and distinguish the refresh logic of ADD and DROP.
---
src/backend/commands/subscriptioncmds.c | 71 +++++++----
src/test/subscription/t/024_alter_sub_pub.pl | 124 +++++++++++++++++++
2 files changed, 171 insertions(+), 24 deletions(-)
create mode 100644 src/test/subscription/t/024_alter_sub_pub.pl
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5157f44058..6b14ef1fbc 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -606,7 +606,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
}
static void
-AlterSubscription_refresh(Subscription *sub, bool copy_data)
+AlterSubscription_refresh(Subscription *sub, bool copy_data,
+ AlterSubscriptionType type)
{
char *err;
List *pubrel_names;
@@ -617,6 +618,7 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
int off;
int remove_rel_len;
Relation rel = NULL;
+ int ntables_to_drop = 0;
typedef struct SubRemoveRels
{
Oid relid;
@@ -625,6 +627,11 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
SubRemoveRels *sub_remove_rels;
WalReceiverConn *wrconn;
+ Assert(type == ALTER_SUBSCRIPTION_SET_PUBLICATION ||
+ type == ALTER_SUBSCRIPTION_ADD_PUBLICATION ||
+ type == ALTER_SUBSCRIPTION_DROP_PUBLICATION ||
+ type == ALTER_SUBSCRIPTION_REFRESH);
+
/* Load the library providing us libpq calls. */
load_file("libpqwalreceiver", false);
@@ -656,8 +663,14 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
subrel_local_oids[off++] = relstate->relid;
}
- qsort(subrel_local_oids, list_length(subrel_states),
- sizeof(Oid), oid_cmp);
+
+ /*
+ * Don't need to lookup oid in subrel_local_oids if we are DROPing
+ * PUBLICATION, so skip the sort.
+ */
+ if (type != ALTER_SUBSCRIPTION_DROP_PUBLICATION)
+ qsort(subrel_local_oids, list_length(subrel_states),
+ sizeof(Oid), oid_cmp);
/*
* Rels that we want to remove from subscription and drop any slots
@@ -681,34 +694,41 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data)
Oid relid;
relid = RangeVarGetRelid(rv, AccessShareLock, false);
-
- /* Check for supported relkind. */
- CheckSubscriptionRelkind(get_rel_relkind(relid),
- rv->schemaname, rv->relname);
-
pubrel_local_oids[off++] = relid;
- if (!bsearch(&relid, subrel_local_oids,
- list_length(subrel_states), sizeof(Oid), oid_cmp))
+ if (type != ALTER_SUBSCRIPTION_DROP_PUBLICATION)
{
- AddSubscriptionRelState(sub->oid, relid,
- copy_data ? SUBREL_STATE_INIT : SUBREL_STATE_READY,
- InvalidXLogRecPtr);
- ereport(DEBUG1,
- (errmsg_internal("table \"%s.%s\" added to subscription \"%s\"",
- rv->schemaname, rv->relname, sub->name)));
+ /* Check for supported relkind. */
+ CheckSubscriptionRelkind(get_rel_relkind(relid),
+ rv->schemaname, rv->relname);
+
+ if (!bsearch(&relid, subrel_local_oids,
+ list_length(subrel_states), sizeof(Oid), oid_cmp))
+ {
+ AddSubscriptionRelState(sub->oid, relid,
+ copy_data ? SUBREL_STATE_INIT :
+ SUBREL_STATE_READY,
+ InvalidXLogRecPtr);
+ ereport(DEBUG1,
+ (errmsg_internal("table \"%s.%s\" added to subscription \"%s\"",
+ rv->schemaname, rv->relname, sub->name)));
+ }
}
}
/*
* Next remove state for tables we should not care about anymore using
- * the data we collected above
+ * the data we collected above, if we are not ADDing PUBLICATION.
*/
- qsort(pubrel_local_oids, list_length(pubrel_names),
- sizeof(Oid), oid_cmp);
+ if (type != ALTER_SUBSCRIPTION_ADD_PUBLICATION)
+ {
+ qsort(pubrel_local_oids, list_length(pubrel_names),
+ sizeof(Oid), oid_cmp);
+ ntables_to_drop = list_length(subrel_states);
+ }
remove_rel_len = 0;
- for (off = 0; off < list_length(subrel_states); off++)
+ for (off = 0; off < ntables_to_drop; off++)
{
Oid relid = subrel_local_oids[off];
@@ -994,7 +1014,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
/* Make sure refresh sees the new list of publications. */
sub->publications = stmt->publication;
- AlterSubscription_refresh(sub, opts.copy_data);
+ AlterSubscription_refresh(sub, opts.copy_data, stmt->kind);
}
break;
@@ -1043,9 +1063,12 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");
/* Only refresh the added/dropped list of publications. */
- sub->publications = stmt->publication;
+ if (stmt->kind == ALTER_SUBSCRIPTION_ADD_PUBLICATION)
+ sub->publications = stmt->publication;
+ else
+ sub->publications = publist;
- AlterSubscription_refresh(sub, opts.copy_data);
+ AlterSubscription_refresh(sub, opts.copy_data, stmt->kind);
}
break;
@@ -1087,7 +1110,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION ... REFRESH");
- AlterSubscription_refresh(sub, opts.copy_data);
+ AlterSubscription_refresh(sub, opts.copy_data, stmt->kind);
break;
}
diff --git a/src/test/subscription/t/024_alter_sub_pub.pl b/src/test/subscription/t/024_alter_sub_pub.pl
new file mode 100644
index 0000000000..974182aebe
--- /dev/null
+++ b/src/test/subscription/t/024_alter_sub_pub.pl
@@ -0,0 +1,124 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This test checks that constraints work on subscriber
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 8;
+
+# Initialize publisher node
+my $node_publisher = PostgresNode->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgresNode->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create tables on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE temp1 (a int)");
+$node_publisher->safe_psql('postgres', "CREATE TABLE temp2 (a int)");
+$node_publisher->safe_psql('postgres', "CREATE TABLE temp3 (a int)");
+
+# Create tables on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE temp1 (a int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE temp2 (a int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE temp3 (a int)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_temp1 FOR TABLE temp1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_temp2 FOR TABLE temp2");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_temp3 FOR TABLE temp3, temp2");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub_temp1 CONNECTION '$publisher_connstr' PUBLICATION tap_pub_temp1, tap_pub_temp2, tap_pub_temp3"
+);
+
+# check initial pg_subscription_rel
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel");
+is($result, qq(3),
+ 'three relation was subscribed');
+
+# Test changing the list of subscribed publications
+# Removes table from publication
+$node_publisher->safe_psql('postgres', "ALTER PUBLICATION tap_pub_temp1 DROP TABLE temp1");
+
+# Remove publications from the list of publications
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub_temp1 DROP PUBLICATION tap_pub_temp1");
+
+# temp1 should have been deleted from pg_subscription_rel
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel");
+
+is($result, qq(2),
+ 'check one relation was removed from subscribed list');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel where srrelid::regclass::text = 'temp1'");
+is($result, qq(0),
+ 'check relation temp1 was removed from subscribed list');
+
+
+# Removes publications from the list of publications
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub_temp1 DROP PUBLICATION tap_pub_temp3");
+
+# one row should have been deleted from pg_subscription_rel
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel");
+is($result, qq(1),
+ 'check one relation was removed from subscribed list');
+
+# temp3 should have been deleted from pg_subscription_rel
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel where srrelid::regclass::text = 'temp3'");
+is($result, qq(0),
+ 'check relation temp1 was removed from subscribed list');
+
+# temp2 should still exists
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel where srrelid::regclass::text = 'temp2'");
+is($result, qq(1),
+ 'check relation temp1 was removed from subscribed list');
+
+# Readd table to publication
+$node_publisher->safe_psql('postgres', "ALTER PUBLICATION tap_pub_temp1 ADD TABLE temp1");
+
+# Add additional publications
+$node_subscriber->safe_psql('postgres', "ALTER SUBSCRIPTION tap_sub_temp1 ADD PUBLICATION tap_pub_temp1");
+
+# pg_subscription_rel should have two rows(temp1, temp2)
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel");
+is($result, qq(2),
+ 'check one more relation was subscribed');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel where srrelid::regclass::text = 'temp1'");
+is($result, qq(1),
+ 'check relation temp1 was added to subscribed list');
+
+# Drop subscription as we don't need it anymore
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub_temp1");
+
+# Drop publications as we don't need them anymore
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_temp1");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_temp2");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_temp3");
+
+# Clean up the tables on both publisher and subscriber as we don't need them
+$node_publisher->safe_psql('postgres', "DROP TABLE temp1");
+$node_publisher->safe_psql('postgres', "DROP TABLE temp2");
+$node_publisher->safe_psql('postgres', "DROP TABLE temp3");
+$node_subscriber->safe_psql('postgres', "DROP TABLE temp1");
+$node_subscriber->safe_psql('postgres', "DROP TABLE temp2");
+$node_subscriber->safe_psql('postgres', "DROP TABLE temp3");
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
--
2.27.0
On Thu, Aug 5, 2021 at 11:40 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
On Thursday, August 5, 2021 1:09 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote
I've reviewed v2 patch. Here are some comments:
+ if (type == ALTER_SUBSCRIPTION_SET_PUBLICATION || + type == ALTER_SUBSCRIPTION_REFRESH) + drop_table = !bsearch(&relid, pubrel_local_oids, + list_length(pubrel_names), + sizeof(Oid), oid_cmp); + else if (type == ALTER_SUBSCRIPTION_DROP_PUBLICATION) + drop_table = bsearch(&relid, pubrel_local_oids, + list_length(pubrel_names), + sizeof(Oid), oid_cmp);IIUC, in DROP PUBLICATION cases, we don't need to refer to the tables in the
publication on the publisher that we're dropping in order to check whether to
remove the relation. If we drop a table from the publication on the publisher
before dropping the publication from the subscription on the subscriber, the
pg_subscription_rel entry for the table remains. For example, please consider the
following case:On publisher and subscriber:
create table t1 (a int);
create table t2 (a int);
create table t3 (a int);On publisher:
create publication pub12 for table t1, t2; create publication pub3 for table t3;On subscriber:
create subscription sub connection 'dbname=postgres' publication pub12, pub3;On publisher:
alter publication pub12 drop table t2;On subscriber:
alter subscription sub drop publication pub12; select srsubid,
srrelid::regclass::text, srsubstate, srsublsn from pg_subscription_rel;srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
16393 | t3 | r | 0/1707CE0
16393 | t2 | r | 0/1707CE0With v2 patch, the pg_subscription_rel has the entry for 't2' but it should not
exist.But that doesn't mean that drop_table should always be true in DROP
PULIBCATION cases. Since the tables associated with two publications can
overlap, we cannot remove the relation from pg_subscription_rel if it is also
associated with the remaining publication. For example:On publisher and subscriber:
create table t1 (a int);
create table t2 (a int);
create table t3 (a int);On publisher:
create publication pub12 for table t1, t2; create publication pub13 for table t1, t3;On subscriber:
create subscription sub connection 'dbname=postgres' publication pub12,
pub13;On subscriber:
alter subscription sub drop publication pub12; select srsubid,
srrelid::regclass::text, srsubstate, srsublsn from pg_subscription_rel; srsubid |
srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
16393 | t3 | r | 0/17080B0
(1 row)With v2 patch, the pg_subscription_rel has only one entry for t3 but it should
have also the one for t1 since it's still subscribing to pub13.To summary, I think that what we want to do in DROP SUBSCRIPTION cases is to
drop relations from pg_subscription_rel that are no longer included in the set of
after-calculated publications. In the latter example, when ALTER
SUBSCRIPTION ... DROP PUBLICATION pub12, we should compare the current
set of relations associated with {pub12, pub13} to the set of relcations associated
with pub13, not pub12. Then we can find out t2 is no longer associated with the
after-calculated publication (i.g., pub13) so remove it.Thanks for the review. You are right, I missed this point.
Attach new version patch which fix these problems(Also add a new pl-test).
Thank you for updating the patch!
Here are some comments:
I think that it still could happen that DROP PULIBCATION misses
dropping relations. Please consider the following case:
On publisher and subscriber:
create table t1 (a int);
create table t2 (a int);
create table t3 (a int);
On publisher:
create publication pub12 for table t1, t2;
create publication pub3 for table t3;
On subscriber:
create subscription sub connection 'dbname=postgres' publication pub12, pub3;
On publisher:
alter publication pub3 add table t1;
On subscriber:
alter subscription sub drop publication pub12;
select srsubid, srrelid::regclass::text, srsubstate, srsublsn from
pg_subscription_rel;
srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
16393 | t3 | r | 0/1707CE0
16393 | t1 | r | 0/1707D18
With v3 patch, pg_subscription_rel has entries for t3 and t1. But if
ALTER SUBSCRIPTION ... DROP PUBLICATION pub12 refreshes only relations
associated with pub12, t1 should be removed and be added when we
refresh pub3.
---
-AlterSubscription_refresh(Subscription *sub, bool copy_data)
+AlterSubscription_refresh(Subscription *sub, bool copy_data,
+ AlterSubscriptionType type)
I think it's better to add comments to this function. The meaning of
*sub varies depending on 'type'. For example, only in ADD PUBLICATION
cases, 'sub' has only newly-added publications to this function. In
other cases, 'sub' has a new set of publications.
---
+ if (type != ALTER_SUBSCRIPTION_DROP_PUBLICATION)
{
- AddSubscriptionRelState(sub->oid, relid,
- copy_data ? SUBREL_STATE_INIT
: SUBREL_STATE_READY,
- InvalidXLogRecPtr);
- ereport(DEBUG1,
- (errmsg_internal("table \"%s.%s\" added to
subscription \"%s\"",
- rv->schemaname, rv->relname,
sub->name)));
+ /* Check for supported relkind. */
+ CheckSubscriptionRelkind(get_rel_relkind(relid),
+ rv->schemaname, rv->relname);
I think CheckSubscriptionRelkind() is necessary even in
non-DROP-PUBLICATION cases.
---
+ if (type != ALTER_SUBSCRIPTION_ADD_PUBLICATION)
+ {
+ qsort(pubrel_local_oids, list_length(pubrel_names),
+ sizeof(Oid), oid_cmp);
+ ntables_to_drop = list_length(subrel_states);
+ }
I think we can break here in ADD PUBLICATION cases, which seems more
readable to me. That way, we don't need ntables_to_drop. Also, we
palloc the memory for sub_remove_rels even in ADD PUBLICATION cases
but I think we don't need to do that.
---
+# Create tables on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE temp1 (a int)");
+$node_publisher->safe_psql('postgres', "CREATE TABLE temp2 (a int)");
+$node_publisher->safe_psql('postgres', "CREATE TABLE temp3 (a int)");
+
+# Create tables on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE temp1 (a int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE temp2 (a int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE temp3 (a int)");
Can we execute multiple statements in one safe_psql?
---
+# check initial pg_subscription_rel
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel");
+is($result, qq(3),
+ 'three relation was subscribed');
s/three relation/three relations/
---
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel");
+
+is($result, qq(2),
+ 'check one relation was removed from subscribed list');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*) FROM pg_subscription_rel where
srrelid::regclass::text = 'temp1'");
+is($result, qq(0),
+ 'check relation temp1 was removed from subscribed list');
At some places, the test checks the number of total tuples and then
does an existence check. I think we can check the contents of
pg_subscription_rel instead. For example, in the above checks, we can
check if pg_subscription_rel has entries for temp2 and temp3.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
On Fri, Aug 6, 2021 at 5:09 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Thu, Aug 5, 2021 at 11:40 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:To summary, I think that what we want to do in DROP SUBSCRIPTION cases is to
drop relations from pg_subscription_rel that are no longer included in the set of
after-calculated publications. In the latter example, when ALTER
SUBSCRIPTION ... DROP PUBLICATION pub12, we should compare the current
set of relations associated with {pub12, pub13} to the set of relcations associated
with pub13, not pub12. Then we can find out t2 is no longer associated with the
after-calculated publication (i.g., pub13) so remove it.Thanks for the review. You are right, I missed this point.
Attach new version patch which fix these problems(Also add a new pl-test).Thank you for updating the patch!
Here are some comments:
I think that it still could happen that DROP PULIBCATION misses
dropping relations. Please consider the following case:On publisher and subscriber:
create table t1 (a int);
create table t2 (a int);
create table t3 (a int);On publisher:
create publication pub12 for table t1, t2;
create publication pub3 for table t3;On subscriber:
create subscription sub connection 'dbname=postgres' publication pub12, pub3;On publisher:
alter publication pub3 add table t1;On subscriber:
alter subscription sub drop publication pub12;
select srsubid, srrelid::regclass::text, srsubstate, srsublsn from
pg_subscription_rel;
srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
16393 | t3 | r | 0/1707CE0
16393 | t1 | r | 0/1707D18With v3 patch, pg_subscription_rel has entries for t3 and t1. But if
ALTER SUBSCRIPTION ... DROP PUBLICATION pub12 refreshes only relations
associated with pub12, t1 should be removed and be added when we
refresh pub3.
But, isn't this happening because of your suggestion to compare the
current set of relations with relations from publications that doesn't
need to be removed? Do we have better ideas to fix or shall we just
say that we will refresh based on whatever current set of relations
are present in publication to be dropped?
One more thing, I think it would be better to write a separate refresh
function for add/drop rather than adding checks in the current refresh
functionality. We can extract common functionality code which can be
called from the different refresh functions.
--
With Regards,
Amit Kapila.
On Fri, Aug 6, 2021 at 10:09 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 6, 2021 at 5:09 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
But, isn't this happening because of your suggestion to compare the
current set of relations with relations from publications that doesn't
need to be removed? Do we have better ideas to fix or shall we just
say that we will refresh based on whatever current set of relations
are present in publication to be dropped?
The other options could be that (a) for drop publication, we refresh
all the publications, (b) for both add/drop publication, we refresh
all the publications.
Any better ideas?
As this is introduced in PG-14 via the below commit, I am adding
authors and committer to see if they have any better ideas.
commit 82ed7748b710e3ddce3f7ebc74af80fe4869492f
Author: Peter Eisentraut <peter@eisentraut.org>
Date: Tue Apr 6 10:44:26 2021 +0200
ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION
Author: Japin Li <japinli@hotmail.com>
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
--
With Regards,
Amit Kapila.
On Fri, Aug 6, 2021 at 1:39 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 6, 2021 at 5:09 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Thu, Aug 5, 2021 at 11:40 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:To summary, I think that what we want to do in DROP SUBSCRIPTION cases is to
drop relations from pg_subscription_rel that are no longer included in the set of
after-calculated publications. In the latter example, when ALTER
SUBSCRIPTION ... DROP PUBLICATION pub12, we should compare the current
set of relations associated with {pub12, pub13} to the set of relcations associated
with pub13, not pub12. Then we can find out t2 is no longer associated with the
after-calculated publication (i.g., pub13) so remove it.Thanks for the review. You are right, I missed this point.
Attach new version patch which fix these problems(Also add a new pl-test).Thank you for updating the patch!
Here are some comments:
I think that it still could happen that DROP PULIBCATION misses
dropping relations. Please consider the following case:On publisher and subscriber:
create table t1 (a int);
create table t2 (a int);
create table t3 (a int);On publisher:
create publication pub12 for table t1, t2;
create publication pub3 for table t3;On subscriber:
create subscription sub connection 'dbname=postgres' publication pub12, pub3;On publisher:
alter publication pub3 add table t1;On subscriber:
alter subscription sub drop publication pub12;
select srsubid, srrelid::regclass::text, srsubstate, srsublsn from
pg_subscription_rel;
srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
16393 | t3 | r | 0/1707CE0
16393 | t1 | r | 0/1707D18With v3 patch, pg_subscription_rel has entries for t3 and t1. But if
ALTER SUBSCRIPTION ... DROP PUBLICATION pub12 refreshes only relations
associated with pub12, t1 should be removed and be added when we
refresh pub3.But, isn't this happening because of your suggestion to compare the
current set of relations with relations from publications that doesn't
need to be removed?
Hmm yes, it cannot cover all cases. I had somehow misunderstood that
the subscriber knows which relations are associated with which
publications. Given that the subscriber doesn’t know which relations
are associated with which publications (and the set of subscribed
relations on the subscriber becomes out of date once the publication
is updated), I think it's impossible to achieve that DROP PUBLICATION
drops relations that are associated with only the publication being
dropped.
Do we have better ideas to fix or shall we just
say that we will refresh based on whatever current set of relations
are present in publication to be dropped?
I don't have better ideas. It seems to me that it's prudent that we
accept the approach in v3 patch and refresh all publications in DROP
PUBLICATION cases.
One more thing, I think it would be better to write a separate refresh
function for add/drop rather than adding checks in the current refresh
functionality. We can extract common functionality code which can be
called from the different refresh functions.
+1
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
Hi,
Sorry for the late reply. Having read this thread, the problem is caused by
misunderstanding the AlterSubscription_refresh(). My apologies.
On Fri, 06 Aug 2021 at 14:12, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Fri, Aug 6, 2021 at 1:39 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 6, 2021 at 5:09 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Thu, Aug 5, 2021 at 11:40 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:To summary, I think that what we want to do in DROP SUBSCRIPTION cases is to
drop relations from pg_subscription_rel that are no longer included in the set of
after-calculated publications. In the latter example, when ALTER
SUBSCRIPTION ... DROP PUBLICATION pub12, we should compare the current
set of relations associated with {pub12, pub13} to the set of relcations associated
with pub13, not pub12. Then we can find out t2 is no longer associated with the
after-calculated publication (i.g., pub13) so remove it.Thanks for the review. You are right, I missed this point.
Attach new version patch which fix these problems(Also add a new pl-test).Thank you for updating the patch!
Here are some comments:
I think that it still could happen that DROP PULIBCATION misses
dropping relations. Please consider the following case:On publisher and subscriber:
create table t1 (a int);
create table t2 (a int);
create table t3 (a int);On publisher:
create publication pub12 for table t1, t2;
create publication pub3 for table t3;On subscriber:
create subscription sub connection 'dbname=postgres' publication pub12, pub3;On publisher:
alter publication pub3 add table t1;On subscriber:
alter subscription sub drop publication pub12;
select srsubid, srrelid::regclass::text, srsubstate, srsublsn from
pg_subscription_rel;
srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
16393 | t3 | r | 0/1707CE0
16393 | t1 | r | 0/1707D18With v3 patch, pg_subscription_rel has entries for t3 and t1. But if
ALTER SUBSCRIPTION ... DROP PUBLICATION pub12 refreshes only relations
associated with pub12, t1 should be removed and be added when we
refresh pub3.But, isn't this happening because of your suggestion to compare the
current set of relations with relations from publications that doesn't
need to be removed?Hmm yes, it cannot cover all cases. I had somehow misunderstood that
the subscriber knows which relations are associated with which
publications. Given that the subscriber doesn’t know which relations
are associated with which publications (and the set of subscribed
relations on the subscriber becomes out of date once the publication
is updated), I think it's impossible to achieve that DROP PUBLICATION
drops relations that are associated with only the publication being
dropped.Do we have better ideas to fix or shall we just
say that we will refresh based on whatever current set of relations
are present in publication to be dropped?I don't have better ideas. It seems to me that it's prudent that we
accept the approach in v3 patch and refresh all publications in DROP
PUBLICATION cases.
Maybe refreshing all publications in PUBLICATION cases is simple way to
fix the problem.
One more thing, I think it would be better to write a separate refresh
function for add/drop rather than adding checks in the current refresh
functionality. We can extract common functionality code which can be
called from the different refresh functions.+1
Agreed.
Regards,
--
Regrads,
Japin Li.
ChengDu WenWu Information Technology Co.,Ltd.
On Fri, Aug 6, 2021 at 2:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 6, 2021 at 10:09 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 6, 2021 at 5:09 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
But, isn't this happening because of your suggestion to compare the
current set of relations with relations from publications that doesn't
need to be removed? Do we have better ideas to fix or shall we just
say that we will refresh based on whatever current set of relations
are present in publication to be dropped?The other options could be that (a) for drop publication, we refresh
all the publications, (b) for both add/drop publication, we refresh
all the publications.Any better ideas?
As this is introduced in PG-14 via the below commit, I am adding
authors and committer to see if they have any better ideas.commit 82ed7748b710e3ddce3f7ebc74af80fe4869492f
Author: Peter Eisentraut <peter@eisentraut.org>
Date: Tue Apr 6 10:44:26 2021 +0200ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION
Author: Japin Li <japinli@hotmail.com>
Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
I've added this to Open Items.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
On Fri, Aug 6, 2021 at 9:57 PM Japin Li <japinli@hotmail.com> wrote:
Hmm yes, it cannot cover all cases. I had somehow misunderstood that
the subscriber knows which relations are associated with which
publications. Given that the subscriber doesn’t know which relations
are associated with which publications (and the set of subscribed
relations on the subscriber becomes out of date once the publication
is updated), I think it's impossible to achieve that DROP PUBLICATION
drops relations that are associated with only the publication being
dropped.Do we have better ideas to fix or shall we just
say that we will refresh based on whatever current set of relations
are present in publication to be dropped?I don't have better ideas. It seems to me that it's prudent that we
accept the approach in v3 patch and refresh all publications in DROP
PUBLICATION cases.Maybe refreshing all publications in PUBLICATION cases is simple way to
fix the problem.
Do you mean to say that do it for both Add and Drop or just for Drop?
Actually, doing it both will make the behavior consistent but doing it
just for Drop might be preferable by some users. I think it is better
to be consistent here but I am fine if you and others feel it is
better to refresh all publications only in Drop case.
--
With Regards,
Amit Kapila.
On Aug 7, 2021, at 1:35 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 6, 2021 at 9:57 PM Japin Li <japinli@hotmail.com> wrote:
Hmm yes, it cannot cover all cases. I had somehow misunderstood that
the subscriber knows which relations are associated with which
publications. Given that the subscriber doesn’t know which relations
are associated with which publications (and the set of subscribed
relations on the subscriber becomes out of date once the publication
is updated), I think it's impossible to achieve that DROP PUBLICATION
drops relations that are associated with only the publication being
dropped.Do we have better ideas to fix or shall we just
say that we will refresh based on whatever current set of relations
are present in publication to be dropped?I don't have better ideas. It seems to me that it's prudent that we
accept the approach in v3 patch and refresh all publications in DROP
PUBLICATION cases.Maybe refreshing all publications in PUBLICATION cases is simple way to
fix the problem.Do you mean to say that do it for both Add and Drop or just for Drop?
Actually, doing it both will make the behavior consistent but doing it
just for Drop might be preferable by some users. I think it is better
to be consistent here but I am fine if you and others feel it is
better to refresh all publications only in Drop case.
I prefer to refresh all PUBLICATIONS for both ADD and DROP operations, I think
this is more consistent and makes the code simple.
--
Best regards
Japin Li
On Sat, Aug 7, 2021 1:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 6, 2021 at 9:57 PM Japin Li <japinli@hotmail.com> wrote:
Hmm yes, it cannot cover all cases. I had somehow misunderstood that
the subscriber knows which relations are associated with which
publications. Given that the subscriber doesn’t know which relations
are associated with which publications (and the set of subscribed
relations on the subscriber becomes out of date once the publication
is updated), I think it's impossible to achieve that DROP
PUBLICATION drops relations that are associated with only the
publication being dropped.Do we have better ideas to fix or shall we just say that we will
refresh based on whatever current set of relations are present in
publication to be dropped?I don't have better ideas. It seems to me that it's prudent that we
accept the approach in v3 patch and refresh all publications in DROP
PUBLICATION cases.Maybe refreshing all publications in PUBLICATION cases is simple way
to fix the problem.Do you mean to say that do it for both Add and Drop or just for Drop?
Actually, doing it both will make the behavior consistent but doing it just for
Drop might be preferable by some users. I think it is better to be consistent here
but I am fine if you and others feel it is better to refresh all publications only in
Drop case.
Personally, I also think it will be better to make the behavior consistent.
Attach the new version patch make both ADD and DROP behave the same as SET PUBLICATION
which refresh all the publications.
Best regards,
houzj
Attachments:
v4-0001-fix-ALTER-SUBSCRIPTION-ADD-DROP-PUBLICATION.patchapplication/octet-stream; name=v4-0001-fix-ALTER-SUBSCRIPTION-ADD-DROP-PUBLICATION.patchDownload
From 6fd398473ee27dd72f6b17d766a7bc12ff86b24f Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@fujitsu.com>
Date: Sat, 7 Aug 2021 19:38:49 +0800
Subject: [PATCH] fix ALTER SUBSCRIPTION ADD DROP PUBLICATION
Currently, in function AlterSubscription_refresh(), it will compare
the tables from the target publication with the tables in local
pg_subscription_rel. And then add the tables to pg_subscription_rel
which not exists yet, and delele the tables from pg_subscription_rel
which doesn't exists in target publication.
But When ALTER SUB ADD/DROP PUBLICATION, the code only pass added or dropped
publications as the target publication to AlterSubscription_refresh() which
could result in wrong record in pg_subscription_rel and unexpected table sync.
Given that the subscriber doesn't know which relations are associated with
which publications(and the set of subscribed relations on the subscriber
becomes out of date once the publication is updated), It's impossible to
achieve that DROP PUBLICATION drops relations that are associated with only
the publication being dropped.
So, change the behaviour of ADD/DROP PUBLICATION to refresh all publications
like SET PUBLICATION.
---
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
src/backend/commands/subscriptioncmds.c | 8 ++-
src/test/regress/expected/subscription.out | 3 --
src/test/regress/sql/subscription.sql | 3 --
src/test/subscription/t/001_rep_changes.pl | 57 +++++++++++++++++++++-
5 files changed, 60 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index a6f994450d..76eb011c74 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -111,9 +111,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
publications, and <literal>DROP</literal> removes the publications from
the list of publications. See <xref linkend="sql-createsubscription"/>
for more information. By default, this command will also act like
- <literal>REFRESH PUBLICATION</literal>, except that in case of
- <literal>ADD</literal> or <literal>DROP</literal>, only the added or
- dropped publications are refreshed.
+ <literal>REFRESH PUBLICATION</literal>.
</para>
<para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5157f44058..8aaa735926 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1006,9 +1006,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
List *publist;
bool isadd = stmt->kind == ALTER_SUBSCRIPTION_ADD_PUBLICATION;
- supported_opts = SUBOPT_REFRESH;
- if (isadd)
- supported_opts |= SUBOPT_COPY_DATA;
+ supported_opts = SUBOPT_REFRESH | SUBOPT_COPY_DATA;
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1042,8 +1040,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");
- /* Only refresh the added/dropped list of publications. */
- sub->publications = stmt->publication;
+ /* Refresh the new list of publications. */
+ sub->publications = publist;
AlterSubscription_refresh(sub, opts.copy_data);
}
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 77b4437b69..15a1ac6398 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -230,9 +230,6 @@ ERROR: cannot drop all the publications from a subscription
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
ERROR: publication "testpub3" is not in subscription "regress_testsub"
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-ERROR: unrecognized subscription parameter: "copy_data"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index d42104c191..7faa935a2a 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -171,9 +171,6 @@ ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub, testpub1, testpub2
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 0c84d87873..62e5b169f6 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -6,7 +6,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 32;
+use Test::More tests => 34;
# Initialize publisher node
my $node_publisher = PostgresNode->new('publisher');
@@ -178,6 +178,61 @@ is( $node_subscriber->safe_psql(
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_full SELECT generate_series(1,10)");
+# Test behaviour of ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION
+# The list of publication should be refreshed
+
+# Setup logical replication
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_ins_for_readd (a int)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_ins_for_readd SELECT generate_series(1,10)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_for_readd FOR TABLE tab_ins_for_readd");
+
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_ins_for_readd (a int)");
+
+# Add tap_pub_for_readd to publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub ADD PUBLICATION tap_pub_for_readd");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_ins_for_readd is copied to subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_ins_for_readd");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Drop tap_pub_for_readd from publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DROP PUBLICATION tap_pub_for_readd");
+
+# Readd tap_pub_for_readd to publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub ADD PUBLICATION tap_pub_for_readd");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_ins_for_readd was copied to subscriber again
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_ins_for_readd");
+is($result, qq(20|1|10), 'check initial data is copied to subscriber');
+
+# Clean unused tables and publication
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DROP PUBLICATION tap_pub_for_readd");
+$node_subscriber->safe_psql('postgres', "DROP TABLE tab_ins_for_readd");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_for_readd");
+$node_publisher->safe_psql('postgres', "DROP TABLE tab_ins_for_readd");
+
# Test behaviour of ALTER PUBLICATION ... DROP TABLE
#
# When a publisher drops a table from publication, it should also stop sending
--
2.18.4
On Sat, Aug 7, 2021 at 6:53 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
On Sat, Aug 7, 2021 1:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 6, 2021 at 9:57 PM Japin Li <japinli@hotmail.com> wrote:
Do you mean to say that do it for both Add and Drop or just for Drop?
Actually, doing it both will make the behavior consistent but doing it just for
Drop might be preferable by some users. I think it is better to be consistent here
but I am fine if you and others feel it is better to refresh all publications only in
Drop case.Personally, I also think it will be better to make the behavior consistent.
Attach the new version patch make both ADD and DROP behave the same as SET PUBLICATION
which refresh all the publications.
There is a bug reported on pgsql-bugs [1]/messages/by-id/17132-6a702189086c6243@postgresql.org which seems to be happening
due to the same reason. Can you please test that the other bug is also
fixed with your patch?
[1]: /messages/by-id/17132-6a702189086c6243@postgresql.org
--
With Regards,
Amit Kapila.
On Monday, August 9, 2021 11:10 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Sat, Aug 7, 2021 at 6:53 PM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com>
wrote:On Sat, Aug 7, 2021 1:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 6, 2021 at 9:57 PM Japin Li <japinli@hotmail.com> wrote:
Do you mean to say that do it for both Add and Drop or just for Drop?
Actually, doing it both will make the behavior consistent but doing
it just for Drop might be preferable by some users. I think it is
better to be consistent here but I am fine if you and others feel it
is better to refresh all publications only in Drop case.Personally, I also think it will be better to make the behavior consistent.
Attach the new version patch make both ADD and DROP behave the same as
SET PUBLICATION which refresh all the publications.There is a bug reported on pgsql-bugs [1] which seems to be happening due to
the same reason. Can you please test that the other bug is also fixed with your
patch?
Thanks for the reminder, I have checked and tested the reported bug.
The reported bug is that when drop and then re-add a publication on subscriber side,
the table in the publication wasn't synced. And after applying the patch here, the table
will be synced as expected if re-added(behaves the same as SET PUBLICATION).
Best regards,
Hou zj
On Sat, Aug 7, 2021 at 2:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 6, 2021 at 9:57 PM Japin Li <japinli@hotmail.com> wrote:
Hmm yes, it cannot cover all cases. I had somehow misunderstood that
the subscriber knows which relations are associated with which
publications. Given that the subscriber doesn’t know which relations
are associated with which publications (and the set of subscribed
relations on the subscriber becomes out of date once the publication
is updated), I think it's impossible to achieve that DROP PUBLICATION
drops relations that are associated with only the publication being
dropped.Do we have better ideas to fix or shall we just
say that we will refresh based on whatever current set of relations
are present in publication to be dropped?I don't have better ideas. It seems to me that it's prudent that we
accept the approach in v3 patch and refresh all publications in DROP
PUBLICATION cases.Maybe refreshing all publications in PUBLICATION cases is simple way to
fix the problem.Actually, doing it both will make the behavior consistent but doing it
just for Drop might be preferable by some users.
I agree that ADD/DROP PUBLICATION is convenient to just add/drop
publications instead of providing the complete publication list. But
I'm concerned that during developing this feature most people didn't
like the behavior of refreshing new and existing publications. Also,
there was pointing out that there will be an overhead of a SQL with
more publications when AlterSubscription_refresh() is called with all
the existing publications. If we feel this behavior is unnatural, the
users will feel the same. I personally think there is a benefit only
in terms of syntax. And we can work on the part of refreshing only
publications being added/dropped on v15 development. But it might be
better to consider also if there will be use cases for users to
add/remove publications in the subscription even with such downsides.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
On Tue, Aug 10, 2021 at 8:05 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Sat, Aug 7, 2021 at 2:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 6, 2021 at 9:57 PM Japin Li <japinli@hotmail.com> wrote:
Hmm yes, it cannot cover all cases. I had somehow misunderstood that
the subscriber knows which relations are associated with which
publications. Given that the subscriber doesn’t know which relations
are associated with which publications (and the set of subscribed
relations on the subscriber becomes out of date once the publication
is updated), I think it's impossible to achieve that DROP PUBLICATION
drops relations that are associated with only the publication being
dropped.Do we have better ideas to fix or shall we just
say that we will refresh based on whatever current set of relations
are present in publication to be dropped?I don't have better ideas. It seems to me that it's prudent that we
accept the approach in v3 patch and refresh all publications in DROP
PUBLICATION cases.Maybe refreshing all publications in PUBLICATION cases is simple way to
fix the problem.Actually, doing it both will make the behavior consistent but doing it
just for Drop might be preferable by some users.I agree that ADD/DROP PUBLICATION is convenient to just add/drop
publications instead of providing the complete publication list. But
I'm concerned that during developing this feature most people didn't
like the behavior of refreshing new and existing publications. Also,
there was pointing out that there will be an overhead of a SQL with
more publications when AlterSubscription_refresh() is called with all
the existing publications.
True, but I think at that time we didn't know this design problem with
'drop publication'.
If we feel this behavior is unnatural, the
users will feel the same. I personally think there is a benefit only
in terms of syntax.
Right, and I think in this particular case, the new syntax is much
easier to use for users.
And we can work on the part of refreshing only
publications being added/dropped on v15 development.
Yeah, unless we change design drastically we might not be able to do a
refresh for dropped publications, for add it is possible. It seems
most of the people responded on this thread that we can be consistent
in terms of refreshing for add/drop at this stage but we can still go
with another option where we can refresh only newly added publications
for add but for drop refresh all publications.
Peter E., do you have any opinion on this matter?
--
With Regards,
Amit Kapila.
On 10.08.21 05:22, Amit Kapila wrote:
Yeah, unless we change design drastically we might not be able to do a
refresh for dropped publications, for add it is possible. It seems
most of the people responded on this thread that we can be consistent
in terms of refreshing for add/drop at this stage but we can still go
with another option where we can refresh only newly added publications
for add but for drop refresh all publications.Peter E., do you have any opinion on this matter?
Refresh everything for both ADD and DROP makes sense to me.
On Sat, Aug 7, 2021 at 6:53 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
Personally, I also think it will be better to make the behavior consistent.
Attach the new version patch make both ADD and DROP behave the same as SET PUBLICATION
which refresh all the publications.
I think we can have tests in the separate test file (alter_sub_pub.pl)
like you earlier had in one of the versions. Use some meaningful names
for tables instead of temp1, temp2 as you had in the previous version.
Otherwise, the code changes look good to me.
--
With Regards,
Amit Kapila.
On Wed, Aug 18, 2021 at 7:54 PM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:
On 10.08.21 05:22, Amit Kapila wrote:
Yeah, unless we change design drastically we might not be able to do a
refresh for dropped publications, for add it is possible. It seems
most of the people responded on this thread that we can be consistent
in terms of refreshing for add/drop at this stage but we can still go
with another option where we can refresh only newly added publications
for add but for drop refresh all publications.Peter E., do you have any opinion on this matter?
Refresh everything for both ADD and DROP makes sense to me.
Thanks for your suggestion.
--
With Regards,
Amit Kapila.
On Mon, Aug 23, 2021 at 1:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Sat, Aug 7, 2021 at 6:53 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:Personally, I also think it will be better to make the behavior consistent.
Attach the new version patch make both ADD and DROP behave the same as SET PUBLICATION
which refresh all the publications.I think we can have tests in the separate test file (alter_sub_pub.pl)
like you earlier had in one of the versions. Use some meaningful names
for tables instead of temp1, temp2 as you had in the previous version.
Otherwise, the code changes look good to me.
- supported_opts = SUBOPT_REFRESH;
- if (isadd)
- supported_opts |= SUBOPT_COPY_DATA;
+ supported_opts = SUBOPT_REFRESH | SUBOPT_COPY_DATA;
I think that the currently the doc says copy_data option can be
specified except in DROP PUBLICATION case, which needs to be fixed
corresponding the above change.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
On Mon, Aug 23, 2021 12:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Sat, Aug 7, 2021 at 6:53 PM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
Personally, I also think it will be better to make the behavior consistent.
Attach the new version patch make both ADD and DROP behave the same as
SET PUBLICATION which refresh all the publications.I think we can have tests in the separate test file (alter_sub_pub.pl) like you
earlier had in one of the versions. Use some meaningful names for tables
instead of temp1, temp2 as you had in the previous version.
Otherwise, the code changes look good to me.
Thanks for the comment.
Attach new version patch which did the following changes.
* move the tests to a separate test file (024_alter_sub_pub.pl)
* adjust the document of copy_data option
* add tab-complete for copy_data option when ALTER DROP publication.
Best regards,
Hou zj
Attachments:
v5-0001-Fix-Alter-Subscription-Add-Drop-Publication-refresh-.patchapplication/octet-stream; name=v5-0001-Fix-Alter-Subscription-Add-Drop-Publication-refresh-.patchDownload
From 98d500330c19b6b3f6a0aa1487ed88aa9288749f Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@cn.fujitsu.com>
Date: Mon, 23 Aug 2021 17:03:03 +0800
Subject: [PATCH] Fix Alter Subscription Add/Drop Publication refresh behavior.
The current refresh behavior tries to just refresh added/dropped
publications but that leads to removing wrong tables from subscription. We
can't refresh just the dropped publication because it is quite possible
that some of the tables are removed from publication by that time and now
those will remain as part of the subscription. Also, there is a chance
that the tables that were part of the publication being dropped are also
part of another publication, so we can't remove those.
So, we decided that by default, add/drop commands will also act like
REFRESH PUBLICATION which means they will refresh all the publications. We
can keep the old behavior for "add publication" but it is better to be
consistent with "drop publication".
Author: Hou Zhijie
Reviewed-by: Masahiko Sawada, Amit Kapila
Backpatch-through: 14, where it was introduced
Discussion: https://postgr.es/m/OS0PR01MB5716935D4C2CC85A6143073F94EF9@OS0PR01MB5716.jpnprd01.prod.outlook.com
---
doc/src/sgml/ref/alter_subscription.sgml | 7 +-
src/backend/commands/subscriptioncmds.c | 8 +--
src/bin/psql/tab-complete.c | 8 +--
src/test/regress/expected/subscription.out | 3 -
src/test/regress/sql/subscription.sql | 3 -
src/test/subscription/t/024_alter_sub_pub.pl | 99 ++++++++++++++++++++++++++++
6 files changed, 106 insertions(+), 22 deletions(-)
create mode 100644 src/test/subscription/t/024_alter_sub_pub.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index a6f9944..835be0d 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -111,9 +111,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
publications, and <literal>DROP</literal> removes the publications from
the list of publications. See <xref linkend="sql-createsubscription"/>
for more information. By default, this command will also act like
- <literal>REFRESH PUBLICATION</literal>, except that in case of
- <literal>ADD</literal> or <literal>DROP</literal>, only the added or
- dropped publications are refreshed.
+ <literal>REFRESH PUBLICATION</literal>.
</para>
<para>
@@ -134,8 +132,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</variablelist>
Additionally, refresh options as described
- under <literal>REFRESH PUBLICATION</literal> may be specified,
- except in the case of <literal>DROP PUBLICATION</literal>.
+ under <literal>REFRESH PUBLICATION</literal> may be specified.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5157f44..8aaa735 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1006,9 +1006,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
List *publist;
bool isadd = stmt->kind == ALTER_SUBSCRIPTION_ADD_PUBLICATION;
- supported_opts = SUBOPT_REFRESH;
- if (isadd)
- supported_opts |= SUBOPT_COPY_DATA;
+ supported_opts = SUBOPT_REFRESH | SUBOPT_COPY_DATA;
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1042,8 +1040,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");
- /* Only refresh the added/dropped list of publications. */
- sub->publications = stmt->publication;
+ /* Refresh the new list of publications. */
+ sub->publications = publist;
AlterSubscription_refresh(sub, opts.copy_data);
}
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 0750f70..b48d193 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1675,14 +1675,10 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny))
COMPLETE_WITH("WITH (");
- /* ALTER SUBSCRIPTION <name> ADD|SET PUBLICATION <name> WITH ( */
+ /* ALTER SUBSCRIPTION <name> ADD|DROP|SET PUBLICATION <name> WITH ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
- TailMatches("ADD|SET", "PUBLICATION", MatchAny, "WITH", "("))
+ TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny, "WITH", "("))
COMPLETE_WITH("copy_data", "refresh");
- /* ALTER SUBSCRIPTION <name> DROP PUBLICATION <name> WITH ( */
- else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
- TailMatches("DROP", "PUBLICATION", MatchAny, "WITH", "("))
- COMPLETE_WITH("refresh");
/* ALTER SCHEMA <name> */
else if (Matches("ALTER", "SCHEMA", MatchAny))
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 77b4437..15a1ac6 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -230,9 +230,6 @@ ERROR: cannot drop all the publications from a subscription
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
ERROR: publication "testpub3" is not in subscription "regress_testsub"
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-ERROR: unrecognized subscription parameter: "copy_data"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index d42104c..7faa935 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -171,9 +171,6 @@ ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub, testpub1, testpub2
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
diff --git a/src/test/subscription/t/024_alter_sub_pub.pl b/src/test/subscription/t/024_alter_sub_pub.pl
new file mode 100644
index 0000000..3550941
--- /dev/null
+++ b/src/test/subscription/t/024_alter_sub_pub.pl
@@ -0,0 +1,99 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This test checks behaviour of ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+# Initialize publisher node
+my $node_publisher = PostgresNode->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgresNode->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create tables on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_readd_refresh (a int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_readd_refresh SELECT generate_series(1,10)");
+
+# Create tables on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_readd_refresh (a int)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_readd FOR TABLE tab_readd_refresh");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub_readd, tap_pub"
+);
+
+# Wait for initial table sync to finish
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_readd_refresh is copied to subscriber
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_readd_refresh");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Create a new table on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_drop_refresh (a int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_drop_refresh SELECT generate_series(1,10)");
+
+# Create a new table on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_drop_refresh (a int)");
+
+# Add the table to publication
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION tap_pub ADD TABLE tab_drop_refresh");
+
+# Drop tap_pub_readd from publication list
+# the publication list should be refreshed
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DROP PUBLICATION tap_pub_readd");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_drop_refresh was copied to subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_drop_refresh");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Re-add tap_pub_readd to publication list
+# the publication list should be refreshed again
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub ADD PUBLICATION tap_pub_readd");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_readd_refresh was copied to subscriber again
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_readd_refresh");
+is($result, qq(20|1|10), 'check initial data is copied to subscriber');
+
+# shutdown
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
--
2.7.2.windows.1
On Mon, Aug 23, 2021 1:18 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Mon, Aug 23, 2021 at 1:59 PM Amit Kapila <amit.kapila16@gmail.com>
wrote:On Sat, Aug 7, 2021 at 6:53 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:Personally, I also think it will be better to make the behavior consistent.
Attach the new version patch make both ADD and DROP behave the same
as SET PUBLICATION which refresh all the publications.I think we can have tests in the separate test file (alter_sub_pub.pl)
like you earlier had in one of the versions. Use some meaningful names
for tables instead of temp1, temp2 as you had in the previous version.
Otherwise, the code changes look good to me.- supported_opts = SUBOPT_REFRESH; - if (isadd) - supported_opts |= SUBOPT_COPY_DATA; + supported_opts = SUBOPT_REFRESH | SUBOPT_COPY_DATA;I think that the currently the doc says copy_data option can be specified except
in DROP PUBLICATION case, which needs to be fixed corresponding the above
change.
Thanks for the comment.
Fixed in the new version patch.
Best regards,
Hou zj
On Mon, Aug 23, 2021 at 2:45 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
On Mon, Aug 23, 2021 12:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Sat, Aug 7, 2021 at 6:53 PM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
Personally, I also think it will be better to make the behavior consistent.
Attach the new version patch make both ADD and DROP behave the same as
SET PUBLICATION which refresh all the publications.I think we can have tests in the separate test file (alter_sub_pub.pl) like you
earlier had in one of the versions. Use some meaningful names for tables
instead of temp1, temp2 as you had in the previous version.
Otherwise, the code changes look good to me.Thanks for the comment.
Attach new version patch which did the following changes.* move the tests to a separate test file (024_alter_sub_pub.pl)
* adjust the document of copy_data option
* add tab-complete for copy_data option when ALTER DROP publication.
Thanks, the patch looks good to me. I have made some cosmetic changes
in the attached version. It makes the test cases easier to understand.
I am planning to push this tomorrow unless there are more comments or
suggestions.
--
With Regards,
Amit Kapila.
Attachments:
v6-0001-Fix-Alter-Subscription-Add-Drop-Publication-refre.patchapplication/octet-stream; name=v6-0001-Fix-Alter-Subscription-Add-Drop-Publication-refre.patchDownload
From bd16fa3b8c7e30db04d30f89244b3a0f3dcea061 Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Mon, 23 Aug 2021 17:05:07 +0530
Subject: [PATCH v6] Fix Alter Subscription Add/Drop Publication refresh
behavior.
The current refresh behavior tries to just refresh added/dropped
publications but that leads to removing wrong tables from subscription. We
can't refresh just the dropped publication because it is quite possible
that some of the tables are removed from publication by that time and now
those will remain as part of the subscription. Also, there is a chance
that the tables that were part of the publication being dropped are also
part of another publication, so we can't remove those.
So, we decided that by default, add/drop commands will also act like
REFRESH PUBLICATION which means they will refresh all the publications. We
can keep the old behavior for "add publication" but it is better to be
consistent with "drop publication".
Author: Hou Zhijie
Reviewed-by: Masahiko Sawada, Amit Kapila
Backpatch-through: 14, where it was introduced
Discussion: https://postgr.es/m/OS0PR01MB5716935D4C2CC85A6143073F94EF9@OS0PR01MB5716.jpnprd01.prod.outlook.com
---
doc/src/sgml/ref/alter_subscription.sgml | 7 +-
src/backend/commands/subscriptioncmds.c | 9 +-
src/bin/psql/tab-complete.c | 8 +-
src/test/regress/expected/subscription.out | 3 -
src/test/regress/sql/subscription.sql | 3 -
src/test/subscription/t/024_add_drop_pub.pl | 98 +++++++++++++++++++++
6 files changed, 105 insertions(+), 23 deletions(-)
create mode 100644 src/test/subscription/t/024_add_drop_pub.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index a6f994450d..835be0d2a4 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -111,9 +111,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
publications, and <literal>DROP</literal> removes the publications from
the list of publications. See <xref linkend="sql-createsubscription"/>
for more information. By default, this command will also act like
- <literal>REFRESH PUBLICATION</literal>, except that in case of
- <literal>ADD</literal> or <literal>DROP</literal>, only the added or
- dropped publications are refreshed.
+ <literal>REFRESH PUBLICATION</literal>.
</para>
<para>
@@ -134,8 +132,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</variablelist>
Additionally, refresh options as described
- under <literal>REFRESH PUBLICATION</literal> may be specified,
- except in the case of <literal>DROP PUBLICATION</literal>.
+ under <literal>REFRESH PUBLICATION</literal> may be specified.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5157f44058..c47ba26369 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1006,10 +1006,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
List *publist;
bool isadd = stmt->kind == ALTER_SUBSCRIPTION_ADD_PUBLICATION;
- supported_opts = SUBOPT_REFRESH;
- if (isadd)
- supported_opts |= SUBOPT_COPY_DATA;
-
+ supported_opts = SUBOPT_REFRESH | SUBOPT_COPY_DATA;
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1042,8 +1039,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");
- /* Only refresh the added/dropped list of publications. */
- sub->publications = stmt->publication;
+ /* Refresh the new list of publications. */
+ sub->publications = publist;
AlterSubscription_refresh(sub, opts.copy_data);
}
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 0750f70273..b48d193595 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1675,14 +1675,10 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny))
COMPLETE_WITH("WITH (");
- /* ALTER SUBSCRIPTION <name> ADD|SET PUBLICATION <name> WITH ( */
+ /* ALTER SUBSCRIPTION <name> ADD|DROP|SET PUBLICATION <name> WITH ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
- TailMatches("ADD|SET", "PUBLICATION", MatchAny, "WITH", "("))
+ TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny, "WITH", "("))
COMPLETE_WITH("copy_data", "refresh");
- /* ALTER SUBSCRIPTION <name> DROP PUBLICATION <name> WITH ( */
- else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
- TailMatches("DROP", "PUBLICATION", MatchAny, "WITH", "("))
- COMPLETE_WITH("refresh");
/* ALTER SCHEMA <name> */
else if (Matches("ALTER", "SCHEMA", MatchAny))
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 77b4437b69..15a1ac6398 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -230,9 +230,6 @@ ERROR: cannot drop all the publications from a subscription
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
ERROR: publication "testpub3" is not in subscription "regress_testsub"
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-ERROR: unrecognized subscription parameter: "copy_data"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index d42104c191..7faa935a2a 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -171,9 +171,6 @@ ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub, testpub1, testpub2
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
diff --git a/src/test/subscription/t/024_add_drop_pub.pl b/src/test/subscription/t/024_add_drop_pub.pl
new file mode 100644
index 0000000000..24493a9c4e
--- /dev/null
+++ b/src/test/subscription/t/024_add_drop_pub.pl
@@ -0,0 +1,98 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This test checks behaviour of ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+# Initialize publisher node
+my $node_publisher = PostgresNode->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgresNode->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_1 (a int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_1 SELECT generate_series(1,10)");
+
+# Create table on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_1 (a int)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_1 FOR TABLE tab_1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_2");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub_1, tap_pub_2"
+);
+
+# Wait for initial table sync to finish
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_1 is copied to subscriber
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_1");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Create a new table on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_2 (a int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_2 SELECT generate_series(1,10)");
+
+# Create a new table on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_2 (a int)");
+
+# Add the table to publication
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION tap_pub_2 ADD TABLE tab_2");
+
+# Dropping tap_pub_1 will refresh the entire publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DROP PUBLICATION tap_pub_1");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_drop_refresh was copied to subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_2");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Re-adding tap_pub_1 will refresh the entire publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub ADD PUBLICATION tap_pub_1");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_1 was copied to subscriber again
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_1");
+is($result, qq(20|1|10), 'check initial data is copied to subscriber');
+
+# shutdown
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
--
2.28.0.windows.1
On Mon, Aug 23, 2021 8:01 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Aug 23, 2021 at 2:45 PM houzj.fnst@fujitsu.com<houzj.fnst@fujitsu.com> wrote:
On Mon, Aug 23, 2021 12:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Sat, Aug 7, 2021 at 6:53 PM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
Personally, I also think it will be better to make the behavior consistent.
Attach the new version patch make both ADD and DROP behave the
same as SET PUBLICATION which refresh all the publications.I think we can have tests in the separate test file
(alter_sub_pub.pl) like you earlier had in one of the versions. Use
some meaningful names for tables instead of temp1, temp2 as you had in theprevious version.
Otherwise, the code changes look good to me.
Thanks for the comment.
Attach new version patch which did the following changes.* move the tests to a separate test file (024_alter_sub_pub.pl)
* adjust the document of copy_data option
* add tab-complete for copy_data option when ALTER DROP publication.Thanks, the patch looks good to me. I have made some cosmetic changes in the
attached version. It makes the test cases easier to understand.
I am planning to push this tomorrow unless there are more comments or
suggestions.
Thanks ! The patch looks good to me.
I noticed that the patch cannot be applied to PG14-stable,
so I attach a separate patch for back-patch(I didn’t change the patch for HEAD branch).
Best regards,
Hou zj
Attachments:
v7-0001-PG14-Fix-Alter-Subscription-Add-Drop-Publication-refre_patchapplication/octet-stream; name=v7-0001-PG14-Fix-Alter-Subscription-Add-Drop-Publication-refre_patchDownload
From 478c73a0943cfd1b2bf7683384dd067a920f3fc7 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@cn.fujitsu.com>
Date: Mon, 23 Aug 2021 21:49:30 +0800
Subject: [PATCH] Fix Alter Subscription Add/Drop Publication refresh behavior.
The current refresh behavior tries to just refresh added/dropped
publications but that leads to removing wrong tables from subscription. We
can't refresh just the dropped publication because it is quite possible
that some of the tables are removed from publication by that time and now
those will remain as part of the subscription. Also, there is a chance
that the tables that were part of the publication being dropped are also
part of another publication, so we can't remove those.
So, we decided that by default, add/drop commands will also act like
REFRESH PUBLICATION which means they will refresh all the publications. We
can keep the old behavior for "add publication" but it is better to be
consistent with "drop publication".
Author: Hou Zhijie
Reviewed-by: Masahiko Sawada, Amit Kapila
Backpatch-through: 14, where it was introduced
Discussion: https://postgr.es/m/OS0PR01MB5716935D4C2CC85A6143073F94EF9@OS0PR01MB5716.jpnprd01.prod.outlook.com
---
doc/src/sgml/ref/alter_subscription.sgml | 7 +-
src/backend/commands/subscriptioncmds.c | 5 +-
src/bin/psql/tab-complete.c | 8 +--
src/test/regress/expected/subscription.out | 3 -
src/test/regress/sql/subscription.sql | 3 -
src/test/subscription/t/021_alter_sub_pub.pl | 98 ++++++++++++++++++++++++++++
6 files changed, 104 insertions(+), 20 deletions(-)
create mode 100644 src/test/subscription/t/021_alter_sub_pub.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index b3d1731..acccde9 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -106,9 +106,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
publications, and <literal>DROP</literal> removes the publications from
the list of publications. See <xref linkend="sql-createsubscription"/>
for more information. By default, this command will also act like
- <literal>REFRESH PUBLICATION</literal>, except that in case of
- <literal>ADD</literal> or <literal>DROP</literal>, only the added or
- dropped publications are refreshed.
+ <literal>REFRESH PUBLICATION</literal>.
</para>
<para>
@@ -129,8 +127,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</variablelist>
Additionally, refresh options as described
- under <literal>REFRESH PUBLICATION</literal> may be specified,
- except in the case of <literal>DROP PUBLICATION</literal>.
+ under <literal>REFRESH PUBLICATION</literal> may be specified.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e096d5f..0b3a17e 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -960,8 +960,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
NULL, NULL, /* no "enabled" */
NULL, /* no "create_slot" */
NULL, NULL, /* no "slot_name" */
- isadd ? ©_data : NULL, /* for drop, no
- * "copy_data" */
+ ©_data,
NULL, /* no "synchronous_commit" */
&refresh,
NULL, NULL, /* no "binary" */
@@ -987,7 +986,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");
/* Only refresh the added/dropped list of publications. */
- sub->publications = stmt->publication;
+ sub->publications = publist;
AlterSubscription_refresh(sub, copy_data);
}
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 0ebd5aa..38af568 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1675,14 +1675,10 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny))
COMPLETE_WITH("WITH (");
- /* ALTER SUBSCRIPTION <name> ADD|SET PUBLICATION <name> WITH ( */
+ /* ALTER SUBSCRIPTION <name> ADD|DROP|SET PUBLICATION <name> WITH ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
- TailMatches("ADD|SET", "PUBLICATION", MatchAny, "WITH", "("))
+ TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny, "WITH", "("))
COMPLETE_WITH("copy_data", "refresh");
- /* ALTER SUBSCRIPTION <name> DROP PUBLICATION <name> WITH ( */
- else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
- TailMatches("DROP", "PUBLICATION", MatchAny, "WITH", "("))
- COMPLETE_WITH("refresh");
/* ALTER SCHEMA <name> */
else if (Matches("ALTER", "SCHEMA", MatchAny))
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 4ac7259..590d7f1 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -230,9 +230,6 @@ ERROR: cannot drop all the publications from a subscription
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
ERROR: publication "testpub3" is not in subscription "regress_testsub"
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-ERROR: unrecognized subscription parameter: "copy_data"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index c9d1bd8..855a341 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -171,9 +171,6 @@ ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub, testpub1, testpub2
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
diff --git a/src/test/subscription/t/021_alter_sub_pub.pl b/src/test/subscription/t/021_alter_sub_pub.pl
new file mode 100644
index 0000000..104eddb
--- /dev/null
+++ b/src/test/subscription/t/021_alter_sub_pub.pl
@@ -0,0 +1,98 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This test checks behaviour of ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+# Initialize publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_1 (a int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_1 SELECT generate_series(1,10)");
+
+# Create table on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_1 (a int)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_1 FOR TABLE tab_1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_2");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub_1, tap_pub_2"
+);
+
+# Wait for initial table sync to finish
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_1 is copied to subscriber
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_1");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Create a new table on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_2 (a int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_2 SELECT generate_series(1,10)");
+
+# Create a new table on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_2 (a int)");
+
+# Add the table to publication
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION tap_pub_2 ADD TABLE tab_2");
+
+# Dropping tap_pub_1 will refresh the entire publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DROP PUBLICATION tap_pub_1");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_drop_refresh was copied to subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_2");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Re-adding tap_pub_1 will refresh the entire publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub ADD PUBLICATION tap_pub_1");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_1 was copied to subscriber again
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_1");
+is($result, qq(20|1|10), 'check initial data is copied to subscriber');
+
+# shutdown
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
--
2.7.2.windows.1
v7-0001-PG15-Fix-Alter-Subscription-Add-Drop-Publication-refre.patchapplication/octet-stream; name=v7-0001-PG15-Fix-Alter-Subscription-Add-Drop-Publication-refre.patchDownload
From bd16fa3b8c7e30db04d30f89244b3a0f3dcea061 Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Mon, 23 Aug 2021 17:05:07 +0530
Subject: [PATCH v7] Fix Alter Subscription Add/Drop Publication refresh
behavior.
The current refresh behavior tries to just refresh added/dropped
publications but that leads to removing wrong tables from subscription. We
can't refresh just the dropped publication because it is quite possible
that some of the tables are removed from publication by that time and now
those will remain as part of the subscription. Also, there is a chance
that the tables that were part of the publication being dropped are also
part of another publication, so we can't remove those.
So, we decided that by default, add/drop commands will also act like
REFRESH PUBLICATION which means they will refresh all the publications. We
can keep the old behavior for "add publication" but it is better to be
consistent with "drop publication".
Author: Hou Zhijie
Reviewed-by: Masahiko Sawada, Amit Kapila
Backpatch-through: 14, where it was introduced
Discussion: https://postgr.es/m/OS0PR01MB5716935D4C2CC85A6143073F94EF9@OS0PR01MB5716.jpnprd01.prod.outlook.com
---
doc/src/sgml/ref/alter_subscription.sgml | 7 +-
src/backend/commands/subscriptioncmds.c | 9 +-
src/bin/psql/tab-complete.c | 8 +-
src/test/regress/expected/subscription.out | 3 -
src/test/regress/sql/subscription.sql | 3 -
src/test/subscription/t/024_add_drop_pub.pl | 98 +++++++++++++++++++++
6 files changed, 105 insertions(+), 23 deletions(-)
create mode 100644 src/test/subscription/t/024_add_drop_pub.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index a6f994450d..835be0d2a4 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -111,9 +111,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
publications, and <literal>DROP</literal> removes the publications from
the list of publications. See <xref linkend="sql-createsubscription"/>
for more information. By default, this command will also act like
- <literal>REFRESH PUBLICATION</literal>, except that in case of
- <literal>ADD</literal> or <literal>DROP</literal>, only the added or
- dropped publications are refreshed.
+ <literal>REFRESH PUBLICATION</literal>.
</para>
<para>
@@ -134,8 +132,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</variablelist>
Additionally, refresh options as described
- under <literal>REFRESH PUBLICATION</literal> may be specified,
- except in the case of <literal>DROP PUBLICATION</literal>.
+ under <literal>REFRESH PUBLICATION</literal> may be specified.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5157f44058..c47ba26369 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1006,10 +1006,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
List *publist;
bool isadd = stmt->kind == ALTER_SUBSCRIPTION_ADD_PUBLICATION;
- supported_opts = SUBOPT_REFRESH;
- if (isadd)
- supported_opts |= SUBOPT_COPY_DATA;
-
+ supported_opts = SUBOPT_REFRESH | SUBOPT_COPY_DATA;
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1042,8 +1039,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");
- /* Only refresh the added/dropped list of publications. */
- sub->publications = stmt->publication;
+ /* Refresh the new list of publications. */
+ sub->publications = publist;
AlterSubscription_refresh(sub, opts.copy_data);
}
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 0750f70273..b48d193595 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1675,14 +1675,10 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny))
COMPLETE_WITH("WITH (");
- /* ALTER SUBSCRIPTION <name> ADD|SET PUBLICATION <name> WITH ( */
+ /* ALTER SUBSCRIPTION <name> ADD|DROP|SET PUBLICATION <name> WITH ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
- TailMatches("ADD|SET", "PUBLICATION", MatchAny, "WITH", "("))
+ TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny, "WITH", "("))
COMPLETE_WITH("copy_data", "refresh");
- /* ALTER SUBSCRIPTION <name> DROP PUBLICATION <name> WITH ( */
- else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
- TailMatches("DROP", "PUBLICATION", MatchAny, "WITH", "("))
- COMPLETE_WITH("refresh");
/* ALTER SCHEMA <name> */
else if (Matches("ALTER", "SCHEMA", MatchAny))
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 77b4437b69..15a1ac6398 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -230,9 +230,6 @@ ERROR: cannot drop all the publications from a subscription
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
ERROR: publication "testpub3" is not in subscription "regress_testsub"
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-ERROR: unrecognized subscription parameter: "copy_data"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index d42104c191..7faa935a2a 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -171,9 +171,6 @@ ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub, testpub1, testpub2
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
diff --git a/src/test/subscription/t/024_add_drop_pub.pl b/src/test/subscription/t/024_add_drop_pub.pl
new file mode 100644
index 0000000000..24493a9c4e
--- /dev/null
+++ b/src/test/subscription/t/024_add_drop_pub.pl
@@ -0,0 +1,98 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This test checks behaviour of ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+# Initialize publisher node
+my $node_publisher = PostgresNode->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgresNode->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_1 (a int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_1 SELECT generate_series(1,10)");
+
+# Create table on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_1 (a int)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_1 FOR TABLE tab_1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_2");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub_1, tap_pub_2"
+);
+
+# Wait for initial table sync to finish
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_1 is copied to subscriber
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_1");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Create a new table on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_2 (a int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_2 SELECT generate_series(1,10)");
+
+# Create a new table on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_2 (a int)");
+
+# Add the table to publication
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION tap_pub_2 ADD TABLE tab_2");
+
+# Dropping tap_pub_1 will refresh the entire publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DROP PUBLICATION tap_pub_1");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_drop_refresh was copied to subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_2");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Re-adding tap_pub_1 will refresh the entire publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub ADD PUBLICATION tap_pub_1");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_1 was copied to subscriber again
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_1");
+is($result, qq(20|1|10), 'check initial data is copied to subscriber');
+
+# shutdown
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
--
2.28.0.windows.1
On Mon, Aug 23, 2021 at 11:05 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
On Mon, Aug 23, 2021 8:01 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Aug 23, 2021 at 2:45 PM houzj.fnst@fujitsu.com<houzj.fnst@fujitsu.com> wrote:
On Mon, Aug 23, 2021 12:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Sat, Aug 7, 2021 at 6:53 PM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
Personally, I also think it will be better to make the behavior consistent.
Attach the new version patch make both ADD and DROP behave the
same as SET PUBLICATION which refresh all the publications.I think we can have tests in the separate test file
(alter_sub_pub.pl) like you earlier had in one of the versions. Use
some meaningful names for tables instead of temp1, temp2 as you had in theprevious version.
Otherwise, the code changes look good to me.
Thanks for the comment.
Attach new version patch which did the following changes.* move the tests to a separate test file (024_alter_sub_pub.pl)
* adjust the document of copy_data option
* add tab-complete for copy_data option when ALTER DROP publication.Thanks, the patch looks good to me. I have made some cosmetic changes in the
attached version. It makes the test cases easier to understand.
I am planning to push this tomorrow unless there are more comments or
suggestions.Thanks ! The patch looks good to me.
I noticed that the patch cannot be applied to PG14-stable,
so I attach a separate patch for back-patch(I didn’t change the patch for HEAD branch).
Thanks. The patch for HEAD looks good to me. Regarding the patch for
PG14, I think we need to update the comment as well:
@@ -987,7 +986,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt,
bool isTopLevel)
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");
/* Only refresh the added/dropped list of publications. */
- sub->publications = stmt->publication;
+ sub->publications = publist;
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
-----Original Message-----
From: Masahiko Sawada <sawada.mshk@gmail.com>
Sent: Tuesday, August 24, 2021 8:52 AMThanks. The patch for HEAD looks good to me. Regarding the patch for PG14, I
think we need to update the comment as well:@@ -987,7 +986,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool
isTopLevel)
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with
refresh");/* Only refresh the added/dropped list of publications. */ - sub->publications = stmt->publication; + sub->publications = publist;
Thanks for the comment, attach new version patch which fixed it.
Best regards,
Hou zj
Attachments:
v8-0001-PG14-Fix-Alter-Subscription-Add-Drop-Publication-refre_patchapplication/octet-stream; name=v8-0001-PG14-Fix-Alter-Subscription-Add-Drop-Publication-refre_patchDownload
From 607e99ad8a2312ec0e7bac67a64473b661923538 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@cn.fujitsu.com>
Date: Tue, 24 Aug 2021 08:55:58 +0800
Subject: [PATCH] Fix Alter Subscription Add/Drop Publication refresh behavior.
The current refresh behavior tries to just refresh added/dropped
publications but that leads to removing wrong tables from subscription. We
can't refresh just the dropped publication because it is quite possible
that some of the tables are removed from publication by that time and now
those will remain as part of the subscription. Also, there is a chance
that the tables that were part of the publication being dropped are also
part of another publication, so we can't remove those.
So, we decided that by default, add/drop commands will also act like
REFRESH PUBLICATION which means they will refresh all the publications. We
can keep the old behavior for "add publication" but it is better to be
consistent with "drop publication".
Author: Hou Zhijie
Reviewed-by: Masahiko Sawada, Amit Kapila
Backpatch-through: 14, where it was introduced
Discussion: https://postgr.es/m/OS0PR01MB5716935D4C2CC85A6143073F94EF9@OS0PR01MB5716.jpnprd01.prod.outlook.com
---
doc/src/sgml/ref/alter_subscription.sgml | 7 +-
src/backend/commands/subscriptioncmds.c | 7 +-
src/bin/psql/tab-complete.c | 8 +--
src/test/regress/expected/subscription.out | 3 -
src/test/regress/sql/subscription.sql | 3 -
src/test/subscription/t/021_alter_sub_pub.pl | 98 ++++++++++++++++++++++++++++
6 files changed, 105 insertions(+), 21 deletions(-)
create mode 100644 src/test/subscription/t/021_alter_sub_pub.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index b3d1731..acccde9 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -106,9 +106,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
publications, and <literal>DROP</literal> removes the publications from
the list of publications. See <xref linkend="sql-createsubscription"/>
for more information. By default, this command will also act like
- <literal>REFRESH PUBLICATION</literal>, except that in case of
- <literal>ADD</literal> or <literal>DROP</literal>, only the added or
- dropped publications are refreshed.
+ <literal>REFRESH PUBLICATION</literal>.
</para>
<para>
@@ -129,8 +127,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</variablelist>
Additionally, refresh options as described
- under <literal>REFRESH PUBLICATION</literal> may be specified,
- except in the case of <literal>DROP PUBLICATION</literal>.
+ under <literal>REFRESH PUBLICATION</literal> may be specified.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e096d5f..1719f04 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -960,8 +960,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
NULL, NULL, /* no "enabled" */
NULL, /* no "create_slot" */
NULL, NULL, /* no "slot_name" */
- isadd ? ©_data : NULL, /* for drop, no
- * "copy_data" */
+ ©_data,
NULL, /* no "synchronous_commit" */
&refresh,
NULL, NULL, /* no "binary" */
@@ -986,8 +985,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool isTopLevel)
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");
- /* Only refresh the added/dropped list of publications. */
- sub->publications = stmt->publication;
+ /* Refresh the new list of publications. */
+ sub->publications = publist;
AlterSubscription_refresh(sub, copy_data);
}
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 0ebd5aa..38af568 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1675,14 +1675,10 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny))
COMPLETE_WITH("WITH (");
- /* ALTER SUBSCRIPTION <name> ADD|SET PUBLICATION <name> WITH ( */
+ /* ALTER SUBSCRIPTION <name> ADD|DROP|SET PUBLICATION <name> WITH ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
- TailMatches("ADD|SET", "PUBLICATION", MatchAny, "WITH", "("))
+ TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny, "WITH", "("))
COMPLETE_WITH("copy_data", "refresh");
- /* ALTER SUBSCRIPTION <name> DROP PUBLICATION <name> WITH ( */
- else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
- TailMatches("DROP", "PUBLICATION", MatchAny, "WITH", "("))
- COMPLETE_WITH("refresh");
/* ALTER SCHEMA <name> */
else if (Matches("ALTER", "SCHEMA", MatchAny))
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 4ac7259..590d7f1 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -230,9 +230,6 @@ ERROR: cannot drop all the publications from a subscription
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
ERROR: publication "testpub3" is not in subscription "regress_testsub"
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-ERROR: unrecognized subscription parameter: "copy_data"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index c9d1bd8..855a341 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -171,9 +171,6 @@ ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub, testpub1, testpub2
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
diff --git a/src/test/subscription/t/021_alter_sub_pub.pl b/src/test/subscription/t/021_alter_sub_pub.pl
new file mode 100644
index 0000000..104eddb
--- /dev/null
+++ b/src/test/subscription/t/021_alter_sub_pub.pl
@@ -0,0 +1,98 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This test checks behaviour of ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+# Initialize publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_1 (a int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_1 SELECT generate_series(1,10)");
+
+# Create table on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_1 (a int)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_1 FOR TABLE tab_1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_2");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub_1, tap_pub_2"
+);
+
+# Wait for initial table sync to finish
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_1 is copied to subscriber
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_1");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Create a new table on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_2 (a int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_2 SELECT generate_series(1,10)");
+
+# Create a new table on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_2 (a int)");
+
+# Add the table to publication
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION tap_pub_2 ADD TABLE tab_2");
+
+# Dropping tap_pub_1 will refresh the entire publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DROP PUBLICATION tap_pub_1");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_drop_refresh was copied to subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_2");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Re-adding tap_pub_1 will refresh the entire publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub ADD PUBLICATION tap_pub_1");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_1 was copied to subscriber again
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_1");
+is($result, qq(20|1|10), 'check initial data is copied to subscriber');
+
+# shutdown
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
--
2.7.2.windows.1
v8-0001-PG15-Fix-Alter-Subscription-Add-Drop-Publication-refre.patchapplication/octet-stream; name=v8-0001-PG15-Fix-Alter-Subscription-Add-Drop-Publication-refre.patchDownload
From bd16fa3b8c7e30db04d30f89244b3a0f3dcea061 Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Mon, 23 Aug 2021 17:05:07 +0530
Subject: [PATCH v7] Fix Alter Subscription Add/Drop Publication refresh
behavior.
The current refresh behavior tries to just refresh added/dropped
publications but that leads to removing wrong tables from subscription. We
can't refresh just the dropped publication because it is quite possible
that some of the tables are removed from publication by that time and now
those will remain as part of the subscription. Also, there is a chance
that the tables that were part of the publication being dropped are also
part of another publication, so we can't remove those.
So, we decided that by default, add/drop commands will also act like
REFRESH PUBLICATION which means they will refresh all the publications. We
can keep the old behavior for "add publication" but it is better to be
consistent with "drop publication".
Author: Hou Zhijie
Reviewed-by: Masahiko Sawada, Amit Kapila
Backpatch-through: 14, where it was introduced
Discussion: https://postgr.es/m/OS0PR01MB5716935D4C2CC85A6143073F94EF9@OS0PR01MB5716.jpnprd01.prod.outlook.com
---
doc/src/sgml/ref/alter_subscription.sgml | 7 +-
src/backend/commands/subscriptioncmds.c | 9 +-
src/bin/psql/tab-complete.c | 8 +-
src/test/regress/expected/subscription.out | 3 -
src/test/regress/sql/subscription.sql | 3 -
src/test/subscription/t/024_add_drop_pub.pl | 98 +++++++++++++++++++++
6 files changed, 105 insertions(+), 23 deletions(-)
create mode 100644 src/test/subscription/t/024_add_drop_pub.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index a6f994450d..835be0d2a4 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -111,9 +111,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
publications, and <literal>DROP</literal> removes the publications from
the list of publications. See <xref linkend="sql-createsubscription"/>
for more information. By default, this command will also act like
- <literal>REFRESH PUBLICATION</literal>, except that in case of
- <literal>ADD</literal> or <literal>DROP</literal>, only the added or
- dropped publications are refreshed.
+ <literal>REFRESH PUBLICATION</literal>.
</para>
<para>
@@ -134,8 +132,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</variablelist>
Additionally, refresh options as described
- under <literal>REFRESH PUBLICATION</literal> may be specified,
- except in the case of <literal>DROP PUBLICATION</literal>.
+ under <literal>REFRESH PUBLICATION</literal> may be specified.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5157f44058..c47ba26369 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -1006,10 +1006,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
List *publist;
bool isadd = stmt->kind == ALTER_SUBSCRIPTION_ADD_PUBLICATION;
- supported_opts = SUBOPT_REFRESH;
- if (isadd)
- supported_opts |= SUBOPT_COPY_DATA;
-
+ supported_opts = SUBOPT_REFRESH | SUBOPT_COPY_DATA;
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1042,8 +1039,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with refresh");
- /* Only refresh the added/dropped list of publications. */
- sub->publications = stmt->publication;
+ /* Refresh the new list of publications. */
+ sub->publications = publist;
AlterSubscription_refresh(sub, opts.copy_data);
}
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 0750f70273..b48d193595 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1675,14 +1675,10 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny))
COMPLETE_WITH("WITH (");
- /* ALTER SUBSCRIPTION <name> ADD|SET PUBLICATION <name> WITH ( */
+ /* ALTER SUBSCRIPTION <name> ADD|DROP|SET PUBLICATION <name> WITH ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
- TailMatches("ADD|SET", "PUBLICATION", MatchAny, "WITH", "("))
+ TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny, "WITH", "("))
COMPLETE_WITH("copy_data", "refresh");
- /* ALTER SUBSCRIPTION <name> DROP PUBLICATION <name> WITH ( */
- else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
- TailMatches("DROP", "PUBLICATION", MatchAny, "WITH", "("))
- COMPLETE_WITH("refresh");
/* ALTER SCHEMA <name> */
else if (Matches("ALTER", "SCHEMA", MatchAny))
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 77b4437b69..15a1ac6398 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -230,9 +230,6 @@ ERROR: cannot drop all the publications from a subscription
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
ERROR: publication "testpub3" is not in subscription "regress_testsub"
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-ERROR: unrecognized subscription parameter: "copy_data"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index d42104c191..7faa935a2a 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -171,9 +171,6 @@ ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub, testpub1, testpub2
-- fail - publication does not exist in subscription
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub3 WITH (refresh = false);
--- fail - do not support copy_data option
-ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1 WITH (refresh = false, copy_data = true);
-
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
diff --git a/src/test/subscription/t/024_add_drop_pub.pl b/src/test/subscription/t/024_add_drop_pub.pl
new file mode 100644
index 0000000000..24493a9c4e
--- /dev/null
+++ b/src/test/subscription/t/024_add_drop_pub.pl
@@ -0,0 +1,98 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This test checks behaviour of ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+# Initialize publisher node
+my $node_publisher = PostgresNode->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgresNode->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_1 (a int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_1 SELECT generate_series(1,10)");
+
+# Create table on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_1 (a int)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_1 FOR TABLE tab_1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_2");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub_1, tap_pub_2"
+);
+
+# Wait for initial table sync to finish
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_1 is copied to subscriber
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_1");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Create a new table on publisher
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab_2 (a int)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_2 SELECT generate_series(1,10)");
+
+# Create a new table on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_2 (a int)");
+
+# Add the table to publication
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION tap_pub_2 ADD TABLE tab_2");
+
+# Dropping tap_pub_1 will refresh the entire publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub DROP PUBLICATION tap_pub_1");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_drop_refresh was copied to subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_2");
+is($result, qq(10|1|10), 'check initial data is copied to subscriber');
+
+# Re-adding tap_pub_1 will refresh the entire publication list
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub ADD PUBLICATION tap_pub_1");
+
+# Wait for initial table sync to finish
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->wait_for_catchup('tap_sub');
+
+# Check the initial data of tab_1 was copied to subscriber again
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT count(*), min(a), max(a) FROM tab_1");
+is($result, qq(20|1|10), 'check initial data is copied to subscriber');
+
+# shutdown
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
--
2.28.0.windows.1
On Tue, Aug 24, 2021 at 10:01 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
-----Original Message-----
From: Masahiko Sawada <sawada.mshk@gmail.com>
Sent: Tuesday, August 24, 2021 8:52 AMThanks. The patch for HEAD looks good to me. Regarding the patch for PG14, I
think we need to update the comment as well:@@ -987,7 +986,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool
isTopLevel)
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with
refresh");/* Only refresh the added/dropped list of publications. */ - sub->publications = stmt->publication; + sub->publications = publist;Thanks for the comment, attach new version patch which fixed it.
Thank you for updating the patch! Looks good to me.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
On Tue, Aug 24, 2021 at 7:13 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Tue, Aug 24, 2021 at 10:01 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:-----Original Message-----
From: Masahiko Sawada <sawada.mshk@gmail.com>
Sent: Tuesday, August 24, 2021 8:52 AMThanks. The patch for HEAD looks good to me. Regarding the patch for PG14, I
think we need to update the comment as well:@@ -987,7 +986,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool
isTopLevel)
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with
refresh");/* Only refresh the added/dropped list of publications. */ - sub->publications = stmt->publication; + sub->publications = publist;Thanks for the comment, attach new version patch which fixed it.
Thank you for updating the patch! Looks good to me.
Pushed!
--
With Regards,
Amit Kapila.
On Tue, Aug 24, 2021 at 5:38 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Tue, Aug 24, 2021 at 7:13 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Tue, Aug 24, 2021 at 10:01 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:-----Original Message-----
From: Masahiko Sawada <sawada.mshk@gmail.com>
Sent: Tuesday, August 24, 2021 8:52 AMThanks. The patch for HEAD looks good to me. Regarding the patch for PG14, I
think we need to update the comment as well:@@ -987,7 +986,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt, bool
isTopLevel)
PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION with
refresh");/* Only refresh the added/dropped list of publications. */ - sub->publications = stmt->publication; + sub->publications = publist;Thanks for the comment, attach new version patch which fixed it.
Thank you for updating the patch! Looks good to me.
Pushed!
Thanks! I've marked this open item as resolved.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/