[BUG] Unexpected action when publishing partition tables
Hi
I met a problem when using logical replication. Maybe it's a bug in logical replication.
When publishing a partition table without replica identity, update
or delete operation can be successful in some cases.
For example:
create table tbl1 (a int) partition by range ( a );
create table tbl1_part1 partition of tbl1 for values from (1) to (101);
create table tbl1_part2 partition of tbl1 for values from (101) to (200);
insert into tbl1 select generate_series(1, 10);
delete from tbl1 where a=1;
create publication pub for table tbl1;
delete from tbl1 where a=2;
The last DELETE statement can be executed successfully, but it should report
error message about missing a replica identity.
I found this problem on HEAD and I could reproduce this problem at PG13 and
PG14. (Logical replication of partition table was introduced in PG13.)
Regards
Tang
On Mon, Sep 6, 2021 at 1:49 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
From Mon, Sep 6, 2021 3:59 PM tanghy.fnst@fujitsu.com <tanghy.fnst@fujitsu.com> wrote:
I met a problem when using logical replication. Maybe it's a bug in logical
replication.
When publishing a partition table without replica identity, update
or delete operation can be successful in some cases.For example:
create table tbl1 (a int) partition by range ( a );
create table tbl1_part1 partition of tbl1 for values from (1) to (101);
create table tbl1_part2 partition of tbl1 for values from (101) to (200);
insert into tbl1 select generate_series(1, 10);
delete from tbl1 where a=1;
create publication pub for table tbl1;
delete from tbl1 where a=2;The last DELETE statement can be executed successfully, but it should report
error message about missing a replica identity.I found this problem on HEAD and I could reproduce this problem at PG13 and
PG14. (Logical replication of partition table was introduced in PG13.)
Adding Amit L and Peter E who were involved in this work (commit:
17b9e7f9) to see if they have opinions on this matter.
I can reproduce this bug.
I think the reason is it didn't invalidate all the leaf partitions' relcache
when add a partitioned table to the publication, so the publication info was
not rebuilt.The following code only invalidate the target table:
---
PublicationAddTables
publication_add_relation
/* Invalidate relcache so that publication info is rebuilt. */
CacheInvalidateRelcache(targetrel);
---In addition, this problem can happen in both ADD TABLE, DROP
TABLE, and SET TABLE cases, so we need to invalidate the leaf partitions'
recache in all these cases.
Few comments:
=============
{
@@ -664,7 +673,13 @@ PublicationDropTables(Oid pubid, List *rels, bool
missing_ok)
ObjectAddressSet(obj, PublicationRelRelationId, prid);
performDeletion(&obj, DROP_CASCADE, 0);
+
+ relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_LEAF,
+ relid);
}
+
+ /* Invalidate relcache so that publication info is rebuilt. */
+ InvalidatePublicationRels(relids);
}
We already register the invalidation for the main table in
RemovePublicationRelById which is called via performDeletion. I think
it is better to perform invalidation for partitions at that place.
Similarly is there a reason for not doing invalidations of partitions
in publication_add_relation()?
--
With Regards,
Amit Kapila.
From Tues, Sep 7, 2021 12:02 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Sep 6, 2021 at 1:49 PM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
I can reproduce this bug.
I think the reason is it didn't invalidate all the leaf partitions'
relcache when add a partitioned table to the publication, so the
publication info was not rebuilt.The following code only invalidate the target table:
---
PublicationAddTables
publication_add_relation
/* Invalidate relcache so that publication info is rebuilt. */
CacheInvalidateRelcache(targetrel);
---In addition, this problem can happen in both ADD TABLE, DROP TABLE,
and SET TABLE cases, so we need to invalidate the leaf partitions'
recache in all these cases.Few comments:
=============
{
@@ -664,7 +673,13 @@ PublicationDropTables(Oid pubid, List *rels, bool
missing_ok)ObjectAddressSet(obj, PublicationRelRelationId, prid); performDeletion(&obj, DROP_CASCADE, 0); + + relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_LEAF, + relid); } + + /* Invalidate relcache so that publication info is rebuilt. */ + InvalidatePublicationRels(relids); }We already register the invalidation for the main table in
RemovePublicationRelById which is called via performDeletion. I think it is
better to perform invalidation for partitions at that place.
Similarly is there a reason for not doing invalidations of partitions in
publication_add_relation()?
Thanks for the comment. I originally intended to reduce the number of invalid
message when add/drop serval tables while each table has lots of partitions which
could exceed the MAX_RELCACHE_INVAL_MSGS. But that seems a rare case, so ,
I changed the code as suggested.
Attach new version patches which addressed the comment.
Best regards,
Hou zj
Attachments:
v2-0001-Made-the-existing-relation-cache-invalidation-an.patchapplication/octet-stream; name=v2-0001-Made-the-existing-relation-cache-invalidation-an.patchDownload
From da591c9c967c693fd507d52a79357ca17aa0034a Mon Sep 17 00:00:00 2001
From: "Vigneshwaran C" <vignesh21@gmail.com>
Date: Mon, 6 Sep 2021 13:37:11 +0800
Subject: [PATCH 1/2] refactor relation cache invalidation code
Made the existing relation cache invalidation code into a function. Also
made getting the relations based on the publication partition option for a
specified relation into a function. This will be used in the later
"FOR ALL TABLES IN SCHEMA" implementation patch.
---
src/backend/catalog/pg_publication.c | 66 ++++++++++++++++++++--------------
src/backend/commands/publicationcmds.c | 42 +++++++++++-----------
src/include/commands/publicationcmds.h | 5 +++
3 files changed, 66 insertions(+), 47 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 2a2fe03..a587e09 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -136,6 +136,42 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(result);
}
+/*
+ * Gets the relations based on the publication partition option for a specified
+ * relation.
+ */
+static List *
+GetPubPartitionOptionRelations(List *result, PublicationPartOpt pub_partopt,
+ Oid relid)
+{
+ if (get_rel_relkind(relid) == RELKIND_PARTITIONED_TABLE &&
+ pub_partopt != PUBLICATION_PART_ROOT)
+ {
+ List *all_parts = find_all_inheritors(relid, NoLock,
+ NULL);
+
+ if (pub_partopt == PUBLICATION_PART_ALL)
+ result = list_concat(result, all_parts);
+ else if (pub_partopt == PUBLICATION_PART_LEAF)
+ {
+ ListCell *lc;
+
+ foreach(lc, all_parts)
+ {
+ Oid partOid = lfirst_oid(lc);
+
+ if (get_rel_relkind(partOid) != RELKIND_PARTITIONED_TABLE)
+ result = lappend_oid(result, partOid);
+ }
+ }
+ else
+ Assert(false);
+ }
+ else
+ result = lappend_oid(result, relid);
+
+ return result;
+}
/*
* Insert new publication / relation mapping.
@@ -241,7 +277,7 @@ GetRelationPublications(Oid relid)
/*
* Gets list of relation oids for a publication.
*
- * This should only be used for normal publications, the FOR ALL TABLES
+ * This should only be used FOR TABLE publications, the FOR ALL TABLES
* should use GetAllTablesPublicationRelations().
*/
List *
@@ -270,32 +306,8 @@ GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt)
Form_pg_publication_rel pubrel;
pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
-
- if (get_rel_relkind(pubrel->prrelid) == RELKIND_PARTITIONED_TABLE &&
- pub_partopt != PUBLICATION_PART_ROOT)
- {
- List *all_parts = find_all_inheritors(pubrel->prrelid, NoLock,
- NULL);
-
- if (pub_partopt == PUBLICATION_PART_ALL)
- result = list_concat(result, all_parts);
- else if (pub_partopt == PUBLICATION_PART_LEAF)
- {
- ListCell *lc;
-
- foreach(lc, all_parts)
- {
- Oid partOid = lfirst_oid(lc);
-
- if (get_rel_relkind(partOid) != RELKIND_PARTITIONED_TABLE)
- result = lappend_oid(result, partOid);
- }
- }
- else
- Assert(false);
- }
- else
- result = lappend_oid(result, pubrel->prrelid);
+ result = GetPubPartitionOptionRelations(result, pub_partopt,
+ pubrel->prrelid);
}
systable_endscan(scan);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 8487eeb..e1d17f6 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -45,9 +45,6 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
-/* Same as MAXNUMMESSAGES in sinvaladt.c */
-#define MAX_RELCACHE_INVAL_MSGS 4096
-
static List *OpenTableList(List *tables);
static void CloseTableList(List *rels);
static void PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
@@ -324,23 +321,7 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
List *relids = GetPublicationRelations(pubform->oid,
PUBLICATION_PART_ALL);
- /*
- * We don't want to send too many individual messages, at some point
- * it's cheaper to just reset whole relcache.
- */
- if (list_length(relids) < MAX_RELCACHE_INVAL_MSGS)
- {
- ListCell *lc;
-
- foreach(lc, relids)
- {
- Oid relid = lfirst_oid(lc);
-
- CacheInvalidateRelcacheByRelid(relid);
- }
- }
- else
- CacheInvalidateRelcacheAll();
+ InvalidatePublicationRels(relids);
}
ObjectAddressSet(obj, PublicationRelationId, pubform->oid);
@@ -351,6 +332,27 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
}
/*
+ * Invalidate the relations.
+ */
+void
+InvalidatePublicationRels(List *relids)
+{
+ /*
+ * We don't want to send too many individual messages, at some point it's
+ * cheaper to just reset whole relcache.
+ */
+ if (list_length(relids) < MAX_RELCACHE_INVAL_MSGS)
+ {
+ ListCell *lc;
+
+ foreach(lc, relids)
+ CacheInvalidateRelcacheByRelid(lfirst_oid(lc));
+ }
+ else
+ CacheInvalidateRelcacheAll();
+}
+
+/*
* Add or remove table to/from publication.
*/
static void
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index a3fa2ac..fa47d6d 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -17,6 +17,10 @@
#include "catalog/objectaddress.h"
#include "parser/parse_node.h"
+#include "utils/inval.h"
+
+/* Same as MAXNUMMESSAGES in sinvaladt.c */
+#define MAX_RELCACHE_INVAL_MSGS 4096
extern ObjectAddress CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt);
extern void AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt);
@@ -24,5 +28,6 @@ extern void RemovePublicationRelById(Oid proid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
+extern void InvalidatePublicationRels(List *relids);
#endif /* PUBLICATIONCMDS_H */
--
2.7.2.windows.1
v2-0002-fix-publication-invalidation.patchapplication/octet-stream; name=v2-0002-fix-publication-invalidation.patchDownload
From 5f1263808925c092b76db7c453d6671ec7ac03b7 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@cn.fujitsu.com>
Date: Tue, 7 Sep 2021 13:51:21 +0800
Subject: [PATCH] fix publication invalidation
Invalidate all partitions contained in the respective partition
trees when add or drop a partitioned table from publication.
---
src/backend/catalog/pg_publication.c | 17 ++++++++++++++---
src/backend/commands/publicationcmds.c | 14 ++++++++++++--
src/include/catalog/pg_publication.h | 3 +++
src/test/regress/expected/publication.out | 11 ++++++++++-
src/test/regress/sql/publication.sql | 9 ++++++++-
5 files changed, 47 insertions(+), 7 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index fa47f70..c6ff2d4 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -31,6 +31,7 @@
#include "catalog/pg_publication.h"
#include "catalog/pg_publication_rel.h"
#include "catalog/pg_type.h"
+#include "commands/publicationcmds.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "utils/array.h"
@@ -140,7 +141,7 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
* Gets the relations based on the publication partition option for a specified
* relation.
*/
-static List *
+List *
GetPubPartitionOptionRelations(List *result, PublicationPartOpt pub_partopt,
Oid relid)
{
@@ -189,6 +190,7 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
referenced;
+ List *relids = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -244,8 +246,17 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
/* Close the table. */
table_close(rel, RowExclusiveLock);
- /* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel->relation);
+ /*
+ * Invalidate relcache so that publication info is rebuilt.
+ *
+ * For partitioned table contained in the publication, we must
+ * invalidate all partitions contained in the respective partition
+ * trees, not just the one explicitly mentioned in the publication.
+ */
+ relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_ALL,
+ relid);
+
+ InvalidatePublicationRels(relids);
return myself;
}
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 9ce412a..8689907 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -485,6 +485,7 @@ RemovePublicationRelById(Oid proid)
Relation rel;
HeapTuple tup;
Form_pg_publication_rel pubrel;
+ List *relids = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -496,8 +497,17 @@ RemovePublicationRelById(Oid proid)
pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
- /* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcacheByRelid(pubrel->prrelid);
+ /*
+ * Invalidate relcache so that publication info is rebuilt.
+ *
+ * For partitioned table contained in the publication, we must
+ * invalidate all partitions contained in the respective partition
+ * trees, not just the one explicitly mentioned in the publication.
+ */
+ relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_ALL,
+ pubrel->prrelid);
+
+ InvalidatePublicationRels(relids);
CatalogTupleDelete(rel, &tup->t_self);
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 561266a..82f2536 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -111,6 +111,9 @@ typedef enum PublicationPartOpt
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern List *GetPubPartitionOptionRelations(List *result,
+ PublicationPartOpt pub_partopt,
+ Oid relid);
extern bool is_publishable_relation(Relation rel);
extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 4a5ef0b..6523909 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -126,10 +126,12 @@ CREATE PUBLICATION testpub_forparted;
CREATE PUBLICATION testpub_forparted1;
RESET client_min_messages;
CREATE TABLE testpub_parted1 (LIKE testpub_parted);
+CREATE TABLE testpub_parted2 (LIKE testpub_parted);
ALTER PUBLICATION testpub_forparted1 SET (publish='insert');
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted2 FOR VALUES IN (2);
-- works despite missing REPLICA IDENTITY, because updates are not replicated
UPDATE testpub_parted1 SET a = 1;
-ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
-- only parent is listed as being in publication, not the partition
ALTER PUBLICATION testpub_forparted ADD TABLE testpub_parted;
\dRp+ testpub_forparted
@@ -156,6 +158,13 @@ ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
Tables:
"public.testpub_parted"
+-- still fail, because parent's publication replicates updates
+UPDATE testpub_parted2 SET a = 2;
+ERROR: cannot update table "testpub_parted2" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub_forparted DROP TABLE testpub_parted;
+-- works again, because update is no longer replicated
+UPDATE testpub_parted2 SET a = 2;
DROP TABLE testpub_parted1;
DROP PUBLICATION testpub_forparted, testpub_forparted1;
-- fail - view
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index d844075..20cb1c4 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -76,10 +76,12 @@ CREATE PUBLICATION testpub_forparted;
CREATE PUBLICATION testpub_forparted1;
RESET client_min_messages;
CREATE TABLE testpub_parted1 (LIKE testpub_parted);
+CREATE TABLE testpub_parted2 (LIKE testpub_parted);
ALTER PUBLICATION testpub_forparted1 SET (publish='insert');
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted2 FOR VALUES IN (2);
-- works despite missing REPLICA IDENTITY, because updates are not replicated
UPDATE testpub_parted1 SET a = 1;
-ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
-- only parent is listed as being in publication, not the partition
ALTER PUBLICATION testpub_forparted ADD TABLE testpub_parted;
\dRp+ testpub_forparted
@@ -90,6 +92,11 @@ ALTER TABLE testpub_parted DETACH PARTITION testpub_parted1;
UPDATE testpub_parted1 SET a = 1;
ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
\dRp+ testpub_forparted
+-- still fail, because parent's publication replicates updates
+UPDATE testpub_parted2 SET a = 2;
+ALTER PUBLICATION testpub_forparted DROP TABLE testpub_parted;
+-- works again, because update is no longer replicated
+UPDATE testpub_parted2 SET a = 2;
DROP TABLE testpub_parted1;
DROP PUBLICATION testpub_forparted, testpub_forparted1;
--
2.7.2.windows.1
---
PublicationAddTables
publication_add_relation
/* Invalidate relcache so that publication info is rebuilt. */
CacheInvalidateRelcache(targetrel);
---In addition, this problem can happen in both ADD TABLE, DROP TABLE,
and SET TABLE cases, so we need to invalidate the leaf partitions'
recache in all these cases.Few comments:
=============
{
@@ -664,7 +673,13 @@ PublicationDropTables(Oid pubid, List *rels, bool
missing_ok)ObjectAddressSet(obj, PublicationRelRelationId, prid); performDeletion(&obj, DROP_CASCADE, 0); + + relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_LEAF, + relid); } + + /* Invalidate relcache so that publication info is rebuilt. */ + InvalidatePublicationRels(relids); }We already register the invalidation for the main table in
RemovePublicationRelById which is called via performDeletion. I think it is
better to perform invalidation for partitions at that place.
Similarly is there a reason for not doing invalidations of partitions in
publication_add_relation()?Thanks for the comment. I originally intended to reduce the number of invalid
message when add/drop serval tables while each table has lots of partitions which
could exceed the MAX_RELCACHE_INVAL_MSGS. But that seems a rare case, so ,
I changed the code as suggested.Attach new version patches which addressed the comment.
Thanks for your patch. I confirmed that the problem I reported was fixed.
Besides, Your v2 patch also fixed an existing a problem about "DROP PUBLICATION" on HEAD.
(Publication was dropped but it still reported errors about replica identity when trying to
update or delete a partition table.)
Regards
Tang
On Tue, Sep 7, 2021 at 11:38 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
From Tues, Sep 7, 2021 12:02 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Sep 6, 2021 at 1:49 PM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
I can reproduce this bug.
I think the reason is it didn't invalidate all the leaf partitions'
relcache when add a partitioned table to the publication, so the
publication info was not rebuilt.The following code only invalidate the target table:
---
PublicationAddTables
publication_add_relation
/* Invalidate relcache so that publication info is rebuilt. */
CacheInvalidateRelcache(targetrel);
---In addition, this problem can happen in both ADD TABLE, DROP TABLE,
and SET TABLE cases, so we need to invalidate the leaf partitions'
recache in all these cases.Few comments:
=============
{
@@ -664,7 +673,13 @@ PublicationDropTables(Oid pubid, List *rels, bool
missing_ok)ObjectAddressSet(obj, PublicationRelRelationId, prid); performDeletion(&obj, DROP_CASCADE, 0); + + relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_LEAF, + relid); } + + /* Invalidate relcache so that publication info is rebuilt. */ + InvalidatePublicationRels(relids); }We already register the invalidation for the main table in
RemovePublicationRelById which is called via performDeletion. I think it is
better to perform invalidation for partitions at that place.
Similarly is there a reason for not doing invalidations of partitions in
publication_add_relation()?Thanks for the comment. I originally intended to reduce the number of invalid
message when add/drop serval tables while each table has lots of partitions which
could exceed the MAX_RELCACHE_INVAL_MSGS. But that seems a rare case, so ,
I changed the code as suggested.Attach new version patches which addressed the comment.
Thanks for fixing this issue. The bug gets fixed by the patch, I did
not find any issues in my testing.
I just had one minor comment:
We could clean the table at the end by calling DROP TABLE testpub_parted2:
+-- still fail, because parent's publication replicates updates
+UPDATE testpub_parted2 SET a = 2;
+ERROR: cannot update table "testpub_parted2" because it does not
have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub_forparted DROP TABLE testpub_parted;
+-- works again, because update is no longer replicated
+UPDATE testpub_parted2 SET a = 2;
Regards,
Vignesh
On Tuesday, September 14, 2021 10:41 PM vignesh C <vignesh21@gmail.com> wrote:
On Tue, Sep 7, 2021 at 11:38 AM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
Attach new version patches which addressed the comment.
Thanks for fixing this issue. The bug gets fixed by the patch, I did not find any
issues in my testing.
I just had one minor comment:We could clean the table at the end by calling DROP TABLE testpub_parted2: +-- still fail, because parent's publication replicates updates UPDATE +testpub_parted2 SET a = 2; +ERROR: cannot update table "testpub_parted2" because it does not have a replica identity and publishes updates +HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE. +ALTER PUBLICATION testpub_forparted DROP TABLE testpub_parted; +-- works again, because update is no longer replicated UPDATE +testpub_parted2 SET a = 2;
Thanks for the comment.
Attach new version patches which clean the table at the end.
Best regards,
Hou zj
Attachments:
v3-0001-Made-the-existing-relation-cache-invalidation-an.patchapplication/octet-stream; name=v3-0001-Made-the-existing-relation-cache-invalidation-an.patchDownload
From da591c9c967c693fd507d52a79357ca17aa0034a Mon Sep 17 00:00:00 2001
From: "Vigneshwaran C" <vignesh21@gmail.com>
Date: Mon, 6 Sep 2021 13:37:11 +0800
Subject: [PATCH 1/2] refactor relation cache invalidation code
Made the existing relation cache invalidation code into a function. Also
made getting the relations based on the publication partition option for a
specified relation into a function. This will be used in the later
"FOR ALL TABLES IN SCHEMA" implementation patch.
---
src/backend/catalog/pg_publication.c | 66 ++++++++++++++++++++--------------
src/backend/commands/publicationcmds.c | 42 +++++++++++-----------
src/include/commands/publicationcmds.h | 5 +++
3 files changed, 66 insertions(+), 47 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 2a2fe03..a587e09 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -136,6 +136,42 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(result);
}
+/*
+ * Gets the relations based on the publication partition option for a specified
+ * relation.
+ */
+static List *
+GetPubPartitionOptionRelations(List *result, PublicationPartOpt pub_partopt,
+ Oid relid)
+{
+ if (get_rel_relkind(relid) == RELKIND_PARTITIONED_TABLE &&
+ pub_partopt != PUBLICATION_PART_ROOT)
+ {
+ List *all_parts = find_all_inheritors(relid, NoLock,
+ NULL);
+
+ if (pub_partopt == PUBLICATION_PART_ALL)
+ result = list_concat(result, all_parts);
+ else if (pub_partopt == PUBLICATION_PART_LEAF)
+ {
+ ListCell *lc;
+
+ foreach(lc, all_parts)
+ {
+ Oid partOid = lfirst_oid(lc);
+
+ if (get_rel_relkind(partOid) != RELKIND_PARTITIONED_TABLE)
+ result = lappend_oid(result, partOid);
+ }
+ }
+ else
+ Assert(false);
+ }
+ else
+ result = lappend_oid(result, relid);
+
+ return result;
+}
/*
* Insert new publication / relation mapping.
@@ -241,7 +277,7 @@ GetRelationPublications(Oid relid)
/*
* Gets list of relation oids for a publication.
*
- * This should only be used for normal publications, the FOR ALL TABLES
+ * This should only be used FOR TABLE publications, the FOR ALL TABLES
* should use GetAllTablesPublicationRelations().
*/
List *
@@ -270,32 +306,8 @@ GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt)
Form_pg_publication_rel pubrel;
pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
-
- if (get_rel_relkind(pubrel->prrelid) == RELKIND_PARTITIONED_TABLE &&
- pub_partopt != PUBLICATION_PART_ROOT)
- {
- List *all_parts = find_all_inheritors(pubrel->prrelid, NoLock,
- NULL);
-
- if (pub_partopt == PUBLICATION_PART_ALL)
- result = list_concat(result, all_parts);
- else if (pub_partopt == PUBLICATION_PART_LEAF)
- {
- ListCell *lc;
-
- foreach(lc, all_parts)
- {
- Oid partOid = lfirst_oid(lc);
-
- if (get_rel_relkind(partOid) != RELKIND_PARTITIONED_TABLE)
- result = lappend_oid(result, partOid);
- }
- }
- else
- Assert(false);
- }
- else
- result = lappend_oid(result, pubrel->prrelid);
+ result = GetPubPartitionOptionRelations(result, pub_partopt,
+ pubrel->prrelid);
}
systable_endscan(scan);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 8487eeb..e1d17f6 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -45,9 +45,6 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
-/* Same as MAXNUMMESSAGES in sinvaladt.c */
-#define MAX_RELCACHE_INVAL_MSGS 4096
-
static List *OpenTableList(List *tables);
static void CloseTableList(List *rels);
static void PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
@@ -324,23 +321,7 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
List *relids = GetPublicationRelations(pubform->oid,
PUBLICATION_PART_ALL);
- /*
- * We don't want to send too many individual messages, at some point
- * it's cheaper to just reset whole relcache.
- */
- if (list_length(relids) < MAX_RELCACHE_INVAL_MSGS)
- {
- ListCell *lc;
-
- foreach(lc, relids)
- {
- Oid relid = lfirst_oid(lc);
-
- CacheInvalidateRelcacheByRelid(relid);
- }
- }
- else
- CacheInvalidateRelcacheAll();
+ InvalidatePublicationRels(relids);
}
ObjectAddressSet(obj, PublicationRelationId, pubform->oid);
@@ -351,6 +332,27 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
}
/*
+ * Invalidate the relations.
+ */
+void
+InvalidatePublicationRels(List *relids)
+{
+ /*
+ * We don't want to send too many individual messages, at some point it's
+ * cheaper to just reset whole relcache.
+ */
+ if (list_length(relids) < MAX_RELCACHE_INVAL_MSGS)
+ {
+ ListCell *lc;
+
+ foreach(lc, relids)
+ CacheInvalidateRelcacheByRelid(lfirst_oid(lc));
+ }
+ else
+ CacheInvalidateRelcacheAll();
+}
+
+/*
* Add or remove table to/from publication.
*/
static void
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index a3fa2ac..fa47d6d 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -17,6 +17,10 @@
#include "catalog/objectaddress.h"
#include "parser/parse_node.h"
+#include "utils/inval.h"
+
+/* Same as MAXNUMMESSAGES in sinvaladt.c */
+#define MAX_RELCACHE_INVAL_MSGS 4096
extern ObjectAddress CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt);
extern void AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt);
@@ -24,5 +28,6 @@ extern void RemovePublicationRelById(Oid proid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
+extern void InvalidatePublicationRels(List *relids);
#endif /* PUBLICATIONCMDS_H */
--
2.7.2.windows.1
v3-0002-Invalidate-all-partitions.patchapplication/octet-stream; name=v3-0002-Invalidate-all-partitions.patchDownload
From bddf695e17becbc39bde4593f63a90c64dd97ae3 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@fujitsu.com>
Date: Thu, 16 Sep 2021 09:27:29 +0800
Subject: [PATCH] Invalidate all partitions for partitioned table in publication.
Updates/Deletes on a partition were allowed even without replica identity
after the parent table was added to a publiaction. This would later lead to
an error on subscribers.
The reason was that we were not invalidating the partition's relcache and the
publication information for partitions was not getting rebuilt. Similarly, we
were not invalidating the partitions' relcache after dropping a partitioned
table from publication which will prohibit Updates/Deletes on its partition
without replica identity even without any publication.
---
src/backend/catalog/pg_publication.c | 17 ++++++++++++++---
src/backend/commands/publicationcmds.c | 14 ++++++++++++--
src/include/catalog/pg_publication.h | 3 +++
src/test/regress/expected/publication.out | 13 +++++++++++--
src/test/regress/sql/publication.sql | 11 +++++++++--
5 files changed, 49 insertions(+), 9 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index fa47f7052e..c6ff2d4acf 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -31,6 +31,7 @@
#include "catalog/pg_publication.h"
#include "catalog/pg_publication_rel.h"
#include "catalog/pg_type.h"
+#include "commands/publicationcmds.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "utils/array.h"
@@ -140,7 +141,7 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
* Gets the relations based on the publication partition option for a specified
* relation.
*/
-static List *
+List *
GetPubPartitionOptionRelations(List *result, PublicationPartOpt pub_partopt,
Oid relid)
{
@@ -189,6 +190,7 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
referenced;
+ List *relids = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -244,8 +246,17 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
/* Close the table. */
table_close(rel, RowExclusiveLock);
- /* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel->relation);
+ /*
+ * Invalidate relcache so that publication info is rebuilt.
+ *
+ * For partitioned table contained in the publication, we must
+ * invalidate all partitions contained in the respective partition
+ * trees, not just the one explicitly mentioned in the publication.
+ */
+ relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_ALL,
+ relid);
+
+ InvalidatePublicationRels(relids);
return myself;
}
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 945df49078..89972fa3cc 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -490,6 +490,7 @@ RemovePublicationRelById(Oid proid)
Relation rel;
HeapTuple tup;
Form_pg_publication_rel pubrel;
+ List *relids = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -501,8 +502,17 @@ RemovePublicationRelById(Oid proid)
pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
- /* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcacheByRelid(pubrel->prrelid);
+ /*
+ * Invalidate relcache so that publication info is rebuilt.
+ *
+ * For partitioned table contained in the publication, we must
+ * invalidate all partitions contained in the respective partition
+ * trees, not just the one explicitly mentioned in the publication.
+ */
+ relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_ALL,
+ pubrel->prrelid);
+
+ InvalidatePublicationRels(relids);
CatalogTupleDelete(rel, &tup->t_self);
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 561266aa3e..82f2536c65 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -111,6 +111,9 @@ typedef enum PublicationPartOpt
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern List *GetPubPartitionOptionRelations(List *result,
+ PublicationPartOpt pub_partopt,
+ Oid relid);
extern bool is_publishable_relation(Relation rel);
extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index cad1b374be..82bce9be09 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -126,10 +126,12 @@ CREATE PUBLICATION testpub_forparted;
CREATE PUBLICATION testpub_forparted1;
RESET client_min_messages;
CREATE TABLE testpub_parted1 (LIKE testpub_parted);
+CREATE TABLE testpub_parted2 (LIKE testpub_parted);
ALTER PUBLICATION testpub_forparted1 SET (publish='insert');
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted2 FOR VALUES IN (2);
-- works despite missing REPLICA IDENTITY, because updates are not replicated
UPDATE testpub_parted1 SET a = 1;
-ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
-- only parent is listed as being in publication, not the partition
ALTER PUBLICATION testpub_forparted ADD TABLE testpub_parted;
\dRp+ testpub_forparted
@@ -156,7 +158,14 @@ ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
Tables:
"public.testpub_parted"
-DROP TABLE testpub_parted1;
+-- still fail, because parent's publication replicates updates
+UPDATE testpub_parted2 SET a = 2;
+ERROR: cannot update table "testpub_parted2" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub_forparted DROP TABLE testpub_parted;
+-- works again, because update is no longer replicated
+UPDATE testpub_parted2 SET a = 2;
+DROP TABLE testpub_parted1, testpub_parted2;
DROP PUBLICATION testpub_forparted, testpub_forparted1;
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 04b34ee299..e5745d575b 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -76,10 +76,12 @@ CREATE PUBLICATION testpub_forparted;
CREATE PUBLICATION testpub_forparted1;
RESET client_min_messages;
CREATE TABLE testpub_parted1 (LIKE testpub_parted);
+CREATE TABLE testpub_parted2 (LIKE testpub_parted);
ALTER PUBLICATION testpub_forparted1 SET (publish='insert');
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted2 FOR VALUES IN (2);
-- works despite missing REPLICA IDENTITY, because updates are not replicated
UPDATE testpub_parted1 SET a = 1;
-ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
-- only parent is listed as being in publication, not the partition
ALTER PUBLICATION testpub_forparted ADD TABLE testpub_parted;
\dRp+ testpub_forparted
@@ -90,7 +92,12 @@ ALTER TABLE testpub_parted DETACH PARTITION testpub_parted1;
UPDATE testpub_parted1 SET a = 1;
ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
\dRp+ testpub_forparted
-DROP TABLE testpub_parted1;
+-- still fail, because parent's publication replicates updates
+UPDATE testpub_parted2 SET a = 2;
+ALTER PUBLICATION testpub_forparted DROP TABLE testpub_parted;
+-- works again, because update is no longer replicated
+UPDATE testpub_parted2 SET a = 2;
+DROP TABLE testpub_parted1, testpub_parted2;
DROP PUBLICATION testpub_forparted, testpub_forparted1;
-- Test cache invalidation FOR ALL TABLES publication
--
2.18.4
On Thu, Sep 16, 2021 at 7:15 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
On Tuesday, September 14, 2021 10:41 PM vignesh C <vignesh21@gmail.com> wrote:
On Tue, Sep 7, 2021 at 11:38 AM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
Thanks for the comment.
Attach new version patches which clean the table at the end.
+ * For partitioned table contained in the publication, we must
+ * invalidate all partitions contained in the respective partition
+ * trees, not just the one explicitly mentioned in the publication.
Can we slightly change the above comment as: "For the partitioned
tables, we must invalidate all partitions contained in the respective
partition hierarchies, not just the one explicitly mentioned in the
publication. This is required because we implicitly publish the child
tables when the parent table is published."
Apart from this, the patch looks good to me.
I think we need to back-patch this till v13. What do you think? If
yes, then can you please prepare and test the patches for
back-branches? Does anyone else have opinions on back-patching this?
I think this is not a show-stopper bug, so even if we decide to
back-patch, I will do it next week after 14 RC1.
--
With Regards,
Amit Kapila.
On Thursday, September 16, 2021 6:05 PM Amit Kapila <amit.kapila16@gmail.com>
On Tuesday, September 14, 2021 10:41 PM vignesh C <vignesh21@gmail.com> wrote:
On Tue, Sep 7, 2021 at 11:38 AM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
Thanks for the comment.
Attach new version patches which clean the table at the end.+ * For partitioned table contained in the publication, we must + * invalidate all partitions contained in the respective partition + * trees, not just the one explicitly mentioned in the publication.Can we slightly change the above comment as: "For the partitioned tables, we
must invalidate all partitions contained in the respective partition hierarchies,
not just the one explicitly mentioned in the publication. This is required
because we implicitly publish the child tables when the parent table is
published."Apart from this, the patch looks good to me.
I think we need to back-patch this till v13. What do you think?
I agreed.
Attach patches for back-branch, each has passed regression tests and pgindent.
Best regards,
Hou zj
Attachments:
HEAD-Invalidate-all-partitions-for-partitioned-table-in-p.patchapplication/octet-stream; name=HEAD-Invalidate-all-partitions-for-partitioned-table-in-p.patchDownload
From 70ae849dab3b7a36d5607832d3abb4a5d1b4b1c7 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@cn.fujitsu.com>
Date: Fri, 17 Sep 2021 13:35:21 +0800
Subject: [PATCH] Invalidate all partitions for partitioned table in
publication
Updates/Deletes on a partition were allowed even without replica identity
after the parent table was added to a publiaction. This would later lead to
an error on subscribers.The reason was that we were not invalidating the
partition's relcache and the publication information for partitions was not
getting rebuilt. Similarly, we were not invalidating the partitions' relcache
after dropping a partitioned table from publication which will prohibit
Updates/Deletes on its partition without replica identity even without any
publication.
In addition to invalidating all partitions, made the existing relation cache
invalidation code into a function. Also made getting the relations based on the
publication partition option for a specified relation into a function.
---
src/backend/catalog/pg_publication.c | 82 ++++++++++++++++++++-----------
src/backend/commands/publicationcmds.c | 57 ++++++++++++---------
src/include/catalog/pg_publication.h | 3 ++
src/include/commands/publicationcmds.h | 5 ++
src/test/regress/expected/publication.out | 13 ++++-
src/test/regress/sql/publication.sql | 11 ++++-
6 files changed, 116 insertions(+), 55 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index d6fddd6..9cd0c82 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -31,6 +31,7 @@
#include "catalog/pg_publication.h"
#include "catalog/pg_publication_rel.h"
#include "catalog/pg_type.h"
+#include "commands/publicationcmds.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "utils/array.h"
@@ -136,6 +137,42 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(result);
}
+/*
+ * Gets the relations based on the publication partition option for a specified
+ * relation.
+ */
+List *
+GetPubPartitionOptionRelations(List *result, PublicationPartOpt pub_partopt,
+ Oid relid)
+{
+ if (get_rel_relkind(relid) == RELKIND_PARTITIONED_TABLE &&
+ pub_partopt != PUBLICATION_PART_ROOT)
+ {
+ List *all_parts = find_all_inheritors(relid, NoLock,
+ NULL);
+
+ if (pub_partopt == PUBLICATION_PART_ALL)
+ result = list_concat(result, all_parts);
+ else if (pub_partopt == PUBLICATION_PART_LEAF)
+ {
+ ListCell *lc;
+
+ foreach(lc, all_parts)
+ {
+ Oid partOid = lfirst_oid(lc);
+
+ if (get_rel_relkind(partOid) != RELKIND_PARTITIONED_TABLE)
+ result = lappend_oid(result, partOid);
+ }
+ }
+ else
+ Assert(false);
+ }
+ else
+ result = lappend_oid(result, relid);
+
+ return result;
+}
/*
* Insert new publication / relation mapping.
@@ -153,6 +190,7 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
referenced;
+ List *relids = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -208,8 +246,18 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
/* Close the table. */
table_close(rel, RowExclusiveLock);
- /* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel->relation);
+ /*
+ * Invalidate relcache so that publication info is rebuilt.
+ *
+ * For the partitioned tables, we must invalidate all partitions contained
+ * in the respective partition hierarchies, not just the one explicitly
+ * mentioned in the publication. This is required because we implicitly
+ * publish the child tables when the parent table is published.
+ */
+ relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_ALL,
+ relid);
+
+ InvalidatePublicationRels(relids);
return myself;
}
@@ -241,7 +289,7 @@ GetRelationPublications(Oid relid)
/*
* Gets list of relation oids for a publication.
*
- * This should only be used for normal publications, the FOR ALL TABLES
+ * This should only be used FOR TABLE publications, the FOR ALL TABLES
* should use GetAllTablesPublicationRelations().
*/
List *
@@ -270,32 +318,8 @@ GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt)
Form_pg_publication_rel pubrel;
pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
-
- if (get_rel_relkind(pubrel->prrelid) == RELKIND_PARTITIONED_TABLE &&
- pub_partopt != PUBLICATION_PART_ROOT)
- {
- List *all_parts = find_all_inheritors(pubrel->prrelid, NoLock,
- NULL);
-
- if (pub_partopt == PUBLICATION_PART_ALL)
- result = list_concat(result, all_parts);
- else if (pub_partopt == PUBLICATION_PART_LEAF)
- {
- ListCell *lc;
-
- foreach(lc, all_parts)
- {
- Oid partOid = lfirst_oid(lc);
-
- if (get_rel_relkind(partOid) != RELKIND_PARTITIONED_TABLE)
- result = lappend_oid(result, partOid);
- }
- }
- else
- Assert(false);
- }
- else
- result = lappend_oid(result, pubrel->prrelid);
+ result = GetPubPartitionOptionRelations(result, pub_partopt,
+ pubrel->prrelid);
}
systable_endscan(scan);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 30929da..9c7f916 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -45,9 +45,6 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
-/* Same as MAXNUMMESSAGES in sinvaladt.c */
-#define MAX_RELCACHE_INVAL_MSGS 4096
-
static List *OpenTableList(List *tables);
static void CloseTableList(List *rels);
static void PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
@@ -329,23 +326,7 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
List *relids = GetPublicationRelations(pubform->oid,
PUBLICATION_PART_ALL);
- /*
- * We don't want to send too many individual messages, at some point
- * it's cheaper to just reset whole relcache.
- */
- if (list_length(relids) < MAX_RELCACHE_INVAL_MSGS)
- {
- ListCell *lc;
-
- foreach(lc, relids)
- {
- Oid relid = lfirst_oid(lc);
-
- CacheInvalidateRelcacheByRelid(relid);
- }
- }
- else
- CacheInvalidateRelcacheAll();
+ InvalidatePublicationRels(relids);
}
ObjectAddressSet(obj, PublicationRelationId, pubform->oid);
@@ -356,6 +337,27 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
}
/*
+ * Invalidate the relations.
+ */
+void
+InvalidatePublicationRels(List *relids)
+{
+ /*
+ * We don't want to send too many individual messages, at some point it's
+ * cheaper to just reset whole relcache.
+ */
+ if (list_length(relids) < MAX_RELCACHE_INVAL_MSGS)
+ {
+ ListCell *lc;
+
+ foreach(lc, relids)
+ CacheInvalidateRelcacheByRelid(lfirst_oid(lc));
+ }
+ else
+ CacheInvalidateRelcacheAll();
+}
+
+/*
* Add or remove table to/from publication.
*/
static void
@@ -488,6 +490,7 @@ RemovePublicationRelById(Oid proid)
Relation rel;
HeapTuple tup;
Form_pg_publication_rel pubrel;
+ List *relids = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -499,8 +502,18 @@ RemovePublicationRelById(Oid proid)
pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
- /* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcacheByRelid(pubrel->prrelid);
+ /*
+ * Invalidate relcache so that publication info is rebuilt.
+ *
+ * For the partitioned tables, we must invalidate all partitions contained
+ * in the respective partition hierarchies, not just the one explicitly
+ * mentioned in the publication. This is required because we implicitly
+ * publish the child tables when the parent table is published.
+ */
+ relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_ALL,
+ pubrel->prrelid);
+
+ InvalidatePublicationRels(relids);
CatalogTupleDelete(rel, &tup->t_self);
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 561266a..82f2536 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -111,6 +111,9 @@ typedef enum PublicationPartOpt
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern List *GetPubPartitionOptionRelations(List *result,
+ PublicationPartOpt pub_partopt,
+ Oid relid);
extern bool is_publishable_relation(Relation rel);
extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index c98d519..77a299b 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -17,6 +17,10 @@
#include "catalog/objectaddress.h"
#include "parser/parse_node.h"
+#include "utils/inval.h"
+
+/* Same as MAXNUMMESSAGES in sinvaladt.c */
+#define MAX_RELCACHE_INVAL_MSGS 4096
extern ObjectAddress CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt);
extern void AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt);
@@ -25,5 +29,6 @@ extern void RemovePublicationRelById(Oid proid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
+extern void InvalidatePublicationRels(List *relids);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index cad1b37..82bce9b 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -126,10 +126,12 @@ CREATE PUBLICATION testpub_forparted;
CREATE PUBLICATION testpub_forparted1;
RESET client_min_messages;
CREATE TABLE testpub_parted1 (LIKE testpub_parted);
+CREATE TABLE testpub_parted2 (LIKE testpub_parted);
ALTER PUBLICATION testpub_forparted1 SET (publish='insert');
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted2 FOR VALUES IN (2);
-- works despite missing REPLICA IDENTITY, because updates are not replicated
UPDATE testpub_parted1 SET a = 1;
-ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
-- only parent is listed as being in publication, not the partition
ALTER PUBLICATION testpub_forparted ADD TABLE testpub_parted;
\dRp+ testpub_forparted
@@ -156,7 +158,14 @@ ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
Tables:
"public.testpub_parted"
-DROP TABLE testpub_parted1;
+-- still fail, because parent's publication replicates updates
+UPDATE testpub_parted2 SET a = 2;
+ERROR: cannot update table "testpub_parted2" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub_forparted DROP TABLE testpub_parted;
+-- works again, because update is no longer replicated
+UPDATE testpub_parted2 SET a = 2;
+DROP TABLE testpub_parted1, testpub_parted2;
DROP PUBLICATION testpub_forparted, testpub_forparted1;
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 04b34ee..e5745d5 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -76,10 +76,12 @@ CREATE PUBLICATION testpub_forparted;
CREATE PUBLICATION testpub_forparted1;
RESET client_min_messages;
CREATE TABLE testpub_parted1 (LIKE testpub_parted);
+CREATE TABLE testpub_parted2 (LIKE testpub_parted);
ALTER PUBLICATION testpub_forparted1 SET (publish='insert');
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted2 FOR VALUES IN (2);
-- works despite missing REPLICA IDENTITY, because updates are not replicated
UPDATE testpub_parted1 SET a = 1;
-ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
-- only parent is listed as being in publication, not the partition
ALTER PUBLICATION testpub_forparted ADD TABLE testpub_parted;
\dRp+ testpub_forparted
@@ -90,7 +92,12 @@ ALTER TABLE testpub_parted DETACH PARTITION testpub_parted1;
UPDATE testpub_parted1 SET a = 1;
ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
\dRp+ testpub_forparted
-DROP TABLE testpub_parted1;
+-- still fail, because parent's publication replicates updates
+UPDATE testpub_parted2 SET a = 2;
+ALTER PUBLICATION testpub_forparted DROP TABLE testpub_parted;
+-- works again, because update is no longer replicated
+UPDATE testpub_parted2 SET a = 2;
+DROP TABLE testpub_parted1, testpub_parted2;
DROP PUBLICATION testpub_forparted, testpub_forparted1;
-- Test cache invalidation FOR ALL TABLES publication
--
2.7.2.windows.1
PG14-Invalidate-all-partitions-for-partitioned-table-in-p.patchapplication/octet-stream; name=PG14-Invalidate-all-partitions-for-partitioned-table-in-p.patchDownload
From 1bab59788056c21060e6a1de41d1f42831544294 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@cn.fujitsu.com>
Date: Fri, 17 Sep 2021 11:43:16 +0800
Subject: [PATCH] Invalidate all partitions for partitioned table in
publication.
Updates/Deletes on a partition were allowed even without replica identity
after the parent table was added to a publiaction. This would later lead to
an error on subscribers.The reason was that we were not invalidating the
partition's relcache and the publication information for partitions was not
getting rebuilt. Similarly, we were not invalidating the partitions' relcache
after dropping a partitioned table from publication which will prohibit
Updates/Deletes on its partition without replica identity even without any
publication.
In addition to invalidating all partitions, made the existing relation cache
invalidation code into a function. Also made getting the relations based on the
publication partition option for a specified relation into a function.
---
src/backend/catalog/pg_publication.c | 82 ++++++++++++++++++++-----------
src/backend/commands/publicationcmds.c | 57 ++++++++++++---------
src/include/catalog/pg_publication.h | 3 ++
src/include/commands/publicationcmds.h | 5 ++
src/test/regress/expected/publication.out | 13 ++++-
src/test/regress/sql/publication.sql | 11 ++++-
6 files changed, 116 insertions(+), 55 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 61bed1d..5d7e1b1 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -31,6 +31,7 @@
#include "catalog/pg_publication.h"
#include "catalog/pg_publication_rel.h"
#include "catalog/pg_type.h"
+#include "commands/publicationcmds.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "utils/array.h"
@@ -136,6 +137,42 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(result);
}
+/*
+ * Gets the relations based on the publication partition option for a specified
+ * relation.
+ */
+List *
+GetPubPartitionOptionRelations(List *result, PublicationPartOpt pub_partopt,
+ Oid relid)
+{
+ if (get_rel_relkind(relid) == RELKIND_PARTITIONED_TABLE &&
+ pub_partopt != PUBLICATION_PART_ROOT)
+ {
+ List *all_parts = find_all_inheritors(relid, NoLock,
+ NULL);
+
+ if (pub_partopt == PUBLICATION_PART_ALL)
+ result = list_concat(result, all_parts);
+ else if (pub_partopt == PUBLICATION_PART_LEAF)
+ {
+ ListCell *lc;
+
+ foreach(lc, all_parts)
+ {
+ Oid partOid = lfirst_oid(lc);
+
+ if (get_rel_relkind(partOid) != RELKIND_PARTITIONED_TABLE)
+ result = lappend_oid(result, partOid);
+ }
+ }
+ else
+ Assert(false);
+ }
+ else
+ result = lappend_oid(result, relid);
+
+ return result;
+}
/*
* Insert new publication / relation mapping.
@@ -153,6 +190,7 @@ publication_add_relation(Oid pubid, Relation targetrel,
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
referenced;
+ List *relids = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -208,8 +246,18 @@ publication_add_relation(Oid pubid, Relation targetrel,
/* Close the table. */
table_close(rel, RowExclusiveLock);
- /* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel);
+ /*
+ * Invalidate relcache so that publication info is rebuilt.
+ *
+ * For the partitioned tables, we must invalidate all partitions contained
+ * in the respective partition hierarchies, not just the one explicitly
+ * mentioned in the publication. This is required because we implicitly
+ * publish the child tables when the parent table is published.
+ */
+ relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_ALL,
+ relid);
+
+ InvalidatePublicationRels(relids);
return myself;
}
@@ -241,7 +289,7 @@ GetRelationPublications(Oid relid)
/*
* Gets list of relation oids for a publication.
*
- * This should only be used for normal publications, the FOR ALL TABLES
+ * This should only be used FOR TABLE publications, the FOR ALL TABLES
* should use GetAllTablesPublicationRelations().
*/
List *
@@ -270,32 +318,8 @@ GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt)
Form_pg_publication_rel pubrel;
pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
-
- if (get_rel_relkind(pubrel->prrelid) == RELKIND_PARTITIONED_TABLE &&
- pub_partopt != PUBLICATION_PART_ROOT)
- {
- List *all_parts = find_all_inheritors(pubrel->prrelid, NoLock,
- NULL);
-
- if (pub_partopt == PUBLICATION_PART_ALL)
- result = list_concat(result, all_parts);
- else if (pub_partopt == PUBLICATION_PART_LEAF)
- {
- ListCell *lc;
-
- foreach(lc, all_parts)
- {
- Oid partOid = lfirst_oid(lc);
-
- if (get_rel_relkind(partOid) != RELKIND_PARTITIONED_TABLE)
- result = lappend_oid(result, partOid);
- }
- }
- else
- Assert(false);
- }
- else
- result = lappend_oid(result, pubrel->prrelid);
+ result = GetPubPartitionOptionRelations(result, pub_partopt,
+ pubrel->prrelid);
}
systable_endscan(scan);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 8797101..7ee8825 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -45,9 +45,6 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
-/* Same as MAXNUMMESSAGES in sinvaladt.c */
-#define MAX_RELCACHE_INVAL_MSGS 4096
-
static List *OpenTableList(List *tables);
static void CloseTableList(List *rels);
static void PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
@@ -330,23 +327,7 @@ AlterPublicationOptions(AlterPublicationStmt *stmt, Relation rel,
List *relids = GetPublicationRelations(pubform->oid,
PUBLICATION_PART_ALL);
- /*
- * We don't want to send too many individual messages, at some point
- * it's cheaper to just reset whole relcache.
- */
- if (list_length(relids) < MAX_RELCACHE_INVAL_MSGS)
- {
- ListCell *lc;
-
- foreach(lc, relids)
- {
- Oid relid = lfirst_oid(lc);
-
- CacheInvalidateRelcacheByRelid(relid);
- }
- }
- else
- CacheInvalidateRelcacheAll();
+ InvalidatePublicationRels(relids);
}
ObjectAddressSet(obj, PublicationRelationId, pubform->oid);
@@ -357,6 +338,27 @@ AlterPublicationOptions(AlterPublicationStmt *stmt, Relation rel,
}
/*
+ * Invalidate the relations.
+ */
+void
+InvalidatePublicationRels(List *relids)
+{
+ /*
+ * We don't want to send too many individual messages, at some point it's
+ * cheaper to just reset whole relcache.
+ */
+ if (list_length(relids) < MAX_RELCACHE_INVAL_MSGS)
+ {
+ ListCell *lc;
+
+ foreach(lc, relids)
+ CacheInvalidateRelcacheByRelid(lfirst_oid(lc));
+ }
+ else
+ CacheInvalidateRelcacheAll();
+}
+
+/*
* Add or remove table to/from publication.
*/
static void
@@ -512,6 +514,7 @@ RemovePublicationRelById(Oid proid)
Relation rel;
HeapTuple tup;
Form_pg_publication_rel pubrel;
+ List *relids = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -523,8 +526,18 @@ RemovePublicationRelById(Oid proid)
pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
- /* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcacheByRelid(pubrel->prrelid);
+ /*
+ * Invalidate relcache so that publication info is rebuilt.
+ *
+ * For the partitioned tables, we must invalidate all partitions contained
+ * in the respective partition hierarchies, not just the one explicitly
+ * mentioned in the publication. This is required because we implicitly
+ * publish the child tables when the parent table is published.
+ */
+ relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_ALL,
+ pubrel->prrelid);
+
+ InvalidatePublicationRels(relids);
CatalogTupleDelete(rel, &tup->t_self);
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 5955ba0..c876085 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -107,6 +107,9 @@ extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern bool is_publishable_relation(Relation rel);
extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
bool if_not_exists);
+extern List *GetPubPartitionOptionRelations(List *result,
+ PublicationPartOpt pub_partopt,
+ Oid relid);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index e713df0..5573bc5 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -17,6 +17,10 @@
#include "catalog/objectaddress.h"
#include "nodes/parsenodes.h"
+#include "utils/inval.h"
+
+/* Same as MAXNUMMESSAGES in sinvaladt.c */
+#define MAX_RELCACHE_INVAL_MSGS 4096
extern ObjectAddress CreatePublication(CreatePublicationStmt *stmt);
extern void AlterPublication(AlterPublicationStmt *stmt);
@@ -25,5 +29,6 @@ extern void RemovePublicationRelById(Oid proid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
+extern void InvalidatePublicationRels(List *relids);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index c299702..f278593 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -124,10 +124,12 @@ CREATE PUBLICATION testpub_forparted;
CREATE PUBLICATION testpub_forparted1;
RESET client_min_messages;
CREATE TABLE testpub_parted1 (LIKE testpub_parted);
+CREATE TABLE testpub_parted2 (LIKE testpub_parted);
ALTER PUBLICATION testpub_forparted1 SET (publish='insert');
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted2 FOR VALUES IN (2);
-- works despite missing REPLICA IDENTITY, because updates are not replicated
UPDATE testpub_parted1 SET a = 1;
-ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
-- only parent is listed as being in publication, not the partition
ALTER PUBLICATION testpub_forparted ADD TABLE testpub_parted;
\dRp+ testpub_forparted
@@ -154,7 +156,14 @@ ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
Tables:
"public.testpub_parted"
-DROP TABLE testpub_parted1;
+-- still fail, because parent's publication replicates updates
+UPDATE testpub_parted2 SET a = 2;
+ERROR: cannot update table "testpub_parted2" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub_forparted DROP TABLE testpub_parted;
+-- works again, because update is no longer replicated
+UPDATE testpub_parted2 SET a = 2;
+DROP TABLE testpub_parted1, testpub_parted2;
DROP PUBLICATION testpub_forparted, testpub_forparted1;
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 04b34ee..e5745d5 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -76,10 +76,12 @@ CREATE PUBLICATION testpub_forparted;
CREATE PUBLICATION testpub_forparted1;
RESET client_min_messages;
CREATE TABLE testpub_parted1 (LIKE testpub_parted);
+CREATE TABLE testpub_parted2 (LIKE testpub_parted);
ALTER PUBLICATION testpub_forparted1 SET (publish='insert');
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted2 FOR VALUES IN (2);
-- works despite missing REPLICA IDENTITY, because updates are not replicated
UPDATE testpub_parted1 SET a = 1;
-ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
-- only parent is listed as being in publication, not the partition
ALTER PUBLICATION testpub_forparted ADD TABLE testpub_parted;
\dRp+ testpub_forparted
@@ -90,7 +92,12 @@ ALTER TABLE testpub_parted DETACH PARTITION testpub_parted1;
UPDATE testpub_parted1 SET a = 1;
ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
\dRp+ testpub_forparted
-DROP TABLE testpub_parted1;
+-- still fail, because parent's publication replicates updates
+UPDATE testpub_parted2 SET a = 2;
+ALTER PUBLICATION testpub_forparted DROP TABLE testpub_parted;
+-- works again, because update is no longer replicated
+UPDATE testpub_parted2 SET a = 2;
+DROP TABLE testpub_parted1, testpub_parted2;
DROP PUBLICATION testpub_forparted, testpub_forparted1;
-- Test cache invalidation FOR ALL TABLES publication
--
2.7.2.windows.1
PG13-Invalidate-all-partitions-for-partitioned-table-in-p.patchapplication/octet-stream; name=PG13-Invalidate-all-partitions-for-partitioned-table-in-p.patchDownload
From 1bab59788056c21060e6a1de41d1f42831544294 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@cn.fujitsu.com>
Date: Fri, 17 Sep 2021 11:43:16 +0800
Subject: [PATCH] Invalidate all partitions for partitioned table in
publication.
Updates/Deletes on a partition were allowed even without replica identity
after the parent table was added to a publiaction. This would later lead to
an error on subscribers.The reason was that we were not invalidating the
partition's relcache and the publication information for partitions was not
getting rebuilt. Similarly, we were not invalidating the partitions' relcache
after dropping a partitioned table from publication which will prohibit
Updates/Deletes on its partition without replica identity even without any
publication.
In addition to invalidating all partitions, made the existing relation cache
invalidation code into a function. Also made getting the relations based on the
publication partition option for a specified relation into a function.
---
src/backend/catalog/pg_publication.c | 82 ++++++++++++++++++++-----------
src/backend/commands/publicationcmds.c | 57 ++++++++++++---------
src/include/catalog/pg_publication.h | 3 ++
src/include/commands/publicationcmds.h | 5 ++
src/test/regress/expected/publication.out | 13 ++++-
src/test/regress/sql/publication.sql | 11 ++++-
6 files changed, 116 insertions(+), 55 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 61bed1d..5d7e1b1 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -31,6 +31,7 @@
#include "catalog/pg_publication.h"
#include "catalog/pg_publication_rel.h"
#include "catalog/pg_type.h"
+#include "commands/publicationcmds.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "utils/array.h"
@@ -136,6 +137,42 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(result);
}
+/*
+ * Gets the relations based on the publication partition option for a specified
+ * relation.
+ */
+List *
+GetPubPartitionOptionRelations(List *result, PublicationPartOpt pub_partopt,
+ Oid relid)
+{
+ if (get_rel_relkind(relid) == RELKIND_PARTITIONED_TABLE &&
+ pub_partopt != PUBLICATION_PART_ROOT)
+ {
+ List *all_parts = find_all_inheritors(relid, NoLock,
+ NULL);
+
+ if (pub_partopt == PUBLICATION_PART_ALL)
+ result = list_concat(result, all_parts);
+ else if (pub_partopt == PUBLICATION_PART_LEAF)
+ {
+ ListCell *lc;
+
+ foreach(lc, all_parts)
+ {
+ Oid partOid = lfirst_oid(lc);
+
+ if (get_rel_relkind(partOid) != RELKIND_PARTITIONED_TABLE)
+ result = lappend_oid(result, partOid);
+ }
+ }
+ else
+ Assert(false);
+ }
+ else
+ result = lappend_oid(result, relid);
+
+ return result;
+}
/*
* Insert new publication / relation mapping.
@@ -153,6 +190,7 @@ publication_add_relation(Oid pubid, Relation targetrel,
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
referenced;
+ List *relids = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -208,8 +246,18 @@ publication_add_relation(Oid pubid, Relation targetrel,
/* Close the table. */
table_close(rel, RowExclusiveLock);
- /* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel);
+ /*
+ * Invalidate relcache so that publication info is rebuilt.
+ *
+ * For the partitioned tables, we must invalidate all partitions contained
+ * in the respective partition hierarchies, not just the one explicitly
+ * mentioned in the publication. This is required because we implicitly
+ * publish the child tables when the parent table is published.
+ */
+ relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_ALL,
+ relid);
+
+ InvalidatePublicationRels(relids);
return myself;
}
@@ -241,7 +289,7 @@ GetRelationPublications(Oid relid)
/*
* Gets list of relation oids for a publication.
*
- * This should only be used for normal publications, the FOR ALL TABLES
+ * This should only be used FOR TABLE publications, the FOR ALL TABLES
* should use GetAllTablesPublicationRelations().
*/
List *
@@ -270,32 +318,8 @@ GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt)
Form_pg_publication_rel pubrel;
pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
-
- if (get_rel_relkind(pubrel->prrelid) == RELKIND_PARTITIONED_TABLE &&
- pub_partopt != PUBLICATION_PART_ROOT)
- {
- List *all_parts = find_all_inheritors(pubrel->prrelid, NoLock,
- NULL);
-
- if (pub_partopt == PUBLICATION_PART_ALL)
- result = list_concat(result, all_parts);
- else if (pub_partopt == PUBLICATION_PART_LEAF)
- {
- ListCell *lc;
-
- foreach(lc, all_parts)
- {
- Oid partOid = lfirst_oid(lc);
-
- if (get_rel_relkind(partOid) != RELKIND_PARTITIONED_TABLE)
- result = lappend_oid(result, partOid);
- }
- }
- else
- Assert(false);
- }
- else
- result = lappend_oid(result, pubrel->prrelid);
+ result = GetPubPartitionOptionRelations(result, pub_partopt,
+ pubrel->prrelid);
}
systable_endscan(scan);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 8797101..7ee8825 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -45,9 +45,6 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
-/* Same as MAXNUMMESSAGES in sinvaladt.c */
-#define MAX_RELCACHE_INVAL_MSGS 4096
-
static List *OpenTableList(List *tables);
static void CloseTableList(List *rels);
static void PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
@@ -330,23 +327,7 @@ AlterPublicationOptions(AlterPublicationStmt *stmt, Relation rel,
List *relids = GetPublicationRelations(pubform->oid,
PUBLICATION_PART_ALL);
- /*
- * We don't want to send too many individual messages, at some point
- * it's cheaper to just reset whole relcache.
- */
- if (list_length(relids) < MAX_RELCACHE_INVAL_MSGS)
- {
- ListCell *lc;
-
- foreach(lc, relids)
- {
- Oid relid = lfirst_oid(lc);
-
- CacheInvalidateRelcacheByRelid(relid);
- }
- }
- else
- CacheInvalidateRelcacheAll();
+ InvalidatePublicationRels(relids);
}
ObjectAddressSet(obj, PublicationRelationId, pubform->oid);
@@ -357,6 +338,27 @@ AlterPublicationOptions(AlterPublicationStmt *stmt, Relation rel,
}
/*
+ * Invalidate the relations.
+ */
+void
+InvalidatePublicationRels(List *relids)
+{
+ /*
+ * We don't want to send too many individual messages, at some point it's
+ * cheaper to just reset whole relcache.
+ */
+ if (list_length(relids) < MAX_RELCACHE_INVAL_MSGS)
+ {
+ ListCell *lc;
+
+ foreach(lc, relids)
+ CacheInvalidateRelcacheByRelid(lfirst_oid(lc));
+ }
+ else
+ CacheInvalidateRelcacheAll();
+}
+
+/*
* Add or remove table to/from publication.
*/
static void
@@ -512,6 +514,7 @@ RemovePublicationRelById(Oid proid)
Relation rel;
HeapTuple tup;
Form_pg_publication_rel pubrel;
+ List *relids = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -523,8 +526,18 @@ RemovePublicationRelById(Oid proid)
pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
- /* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcacheByRelid(pubrel->prrelid);
+ /*
+ * Invalidate relcache so that publication info is rebuilt.
+ *
+ * For the partitioned tables, we must invalidate all partitions contained
+ * in the respective partition hierarchies, not just the one explicitly
+ * mentioned in the publication. This is required because we implicitly
+ * publish the child tables when the parent table is published.
+ */
+ relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_ALL,
+ pubrel->prrelid);
+
+ InvalidatePublicationRels(relids);
CatalogTupleDelete(rel, &tup->t_self);
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 5955ba0..c876085 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -107,6 +107,9 @@ extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern bool is_publishable_relation(Relation rel);
extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
bool if_not_exists);
+extern List *GetPubPartitionOptionRelations(List *result,
+ PublicationPartOpt pub_partopt,
+ Oid relid);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index e713df0..5573bc5 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -17,6 +17,10 @@
#include "catalog/objectaddress.h"
#include "nodes/parsenodes.h"
+#include "utils/inval.h"
+
+/* Same as MAXNUMMESSAGES in sinvaladt.c */
+#define MAX_RELCACHE_INVAL_MSGS 4096
extern ObjectAddress CreatePublication(CreatePublicationStmt *stmt);
extern void AlterPublication(AlterPublicationStmt *stmt);
@@ -25,5 +29,6 @@ extern void RemovePublicationRelById(Oid proid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
+extern void InvalidatePublicationRels(List *relids);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index c299702..f278593 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -124,10 +124,12 @@ CREATE PUBLICATION testpub_forparted;
CREATE PUBLICATION testpub_forparted1;
RESET client_min_messages;
CREATE TABLE testpub_parted1 (LIKE testpub_parted);
+CREATE TABLE testpub_parted2 (LIKE testpub_parted);
ALTER PUBLICATION testpub_forparted1 SET (publish='insert');
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted2 FOR VALUES IN (2);
-- works despite missing REPLICA IDENTITY, because updates are not replicated
UPDATE testpub_parted1 SET a = 1;
-ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
-- only parent is listed as being in publication, not the partition
ALTER PUBLICATION testpub_forparted ADD TABLE testpub_parted;
\dRp+ testpub_forparted
@@ -154,7 +156,14 @@ ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
Tables:
"public.testpub_parted"
-DROP TABLE testpub_parted1;
+-- still fail, because parent's publication replicates updates
+UPDATE testpub_parted2 SET a = 2;
+ERROR: cannot update table "testpub_parted2" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub_forparted DROP TABLE testpub_parted;
+-- works again, because update is no longer replicated
+UPDATE testpub_parted2 SET a = 2;
+DROP TABLE testpub_parted1, testpub_parted2;
DROP PUBLICATION testpub_forparted, testpub_forparted1;
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 04b34ee..e5745d5 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -76,10 +76,12 @@ CREATE PUBLICATION testpub_forparted;
CREATE PUBLICATION testpub_forparted1;
RESET client_min_messages;
CREATE TABLE testpub_parted1 (LIKE testpub_parted);
+CREATE TABLE testpub_parted2 (LIKE testpub_parted);
ALTER PUBLICATION testpub_forparted1 SET (publish='insert');
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
+ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted2 FOR VALUES IN (2);
-- works despite missing REPLICA IDENTITY, because updates are not replicated
UPDATE testpub_parted1 SET a = 1;
-ALTER TABLE testpub_parted ATTACH PARTITION testpub_parted1 FOR VALUES IN (1);
-- only parent is listed as being in publication, not the partition
ALTER PUBLICATION testpub_forparted ADD TABLE testpub_parted;
\dRp+ testpub_forparted
@@ -90,7 +92,12 @@ ALTER TABLE testpub_parted DETACH PARTITION testpub_parted1;
UPDATE testpub_parted1 SET a = 1;
ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
\dRp+ testpub_forparted
-DROP TABLE testpub_parted1;
+-- still fail, because parent's publication replicates updates
+UPDATE testpub_parted2 SET a = 2;
+ALTER PUBLICATION testpub_forparted DROP TABLE testpub_parted;
+-- works again, because update is no longer replicated
+UPDATE testpub_parted2 SET a = 2;
+DROP TABLE testpub_parted1, testpub_parted2;
DROP PUBLICATION testpub_forparted, testpub_forparted1;
-- Test cache invalidation FOR ALL TABLES publication
--
2.7.2.windows.1
On Fri, Sep 17, 2021 at 11:36 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
On Thursday, September 16, 2021 6:05 PM Amit Kapila <amit.kapila16@gmail.com>
On Tuesday, September 14, 2021 10:41 PM vignesh C <vignesh21@gmail.com> wrote:
On Tue, Sep 7, 2021 at 11:38 AM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
Thanks for the comment.
Attach new version patches which clean the table at the end.+ * For partitioned table contained in the publication, we must + * invalidate all partitions contained in the respective partition + * trees, not just the one explicitly mentioned in the publication.Can we slightly change the above comment as: "For the partitioned tables, we
must invalidate all partitions contained in the respective partition hierarchies,
not just the one explicitly mentioned in the publication. This is required
because we implicitly publish the child tables when the parent table is
published."Apart from this, the patch looks good to me.
I think we need to back-patch this till v13. What do you think?
I agreed.
Attach patches for back-branch, each has passed regression tests and pgindent.
Thanks, your patches look good to me. I'll push them sometime next
week after Tuesday unless there are any comments.
--
With Regards,
Amit Kapila.
On Fri, Sep 17, 2021 at 4:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
Thanks, your patches look good to me. I'll push them sometime next
week after Tuesday unless there are any comments.
Pushed.
--
With Regards,
Amit Kapila.
On Fri, Sep 17, 2021 at 7:38 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Sep 17, 2021 at 11:36 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:On Thursday, September 16, 2021 6:05 PM Amit Kapila <amit.kapila16@gmail.com>
On Tuesday, September 14, 2021 10:41 PM vignesh C <vignesh21@gmail.com> wrote:
On Tue, Sep 7, 2021 at 11:38 AM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
Thanks for the comment.
Attach new version patches which clean the table at the end.+ * For partitioned table contained in the publication, we must + * invalidate all partitions contained in the respective partition + * trees, not just the one explicitly mentioned in the publication.Can we slightly change the above comment as: "For the partitioned tables, we
must invalidate all partitions contained in the respective partition hierarchies,
not just the one explicitly mentioned in the publication. This is required
because we implicitly publish the child tables when the parent table is
published."Apart from this, the patch looks good to me.
I think we need to back-patch this till v13. What do you think?
I agreed.
Attach patches for back-branch, each has passed regression tests and pgindent.
Thanks, your patches look good to me. I'll push them sometime next
week after Tuesday unless there are any comments.
Thanks Amit, Tang, and Hou for this.
Sorry that I didn't comment on this earlier, but I think either
GetPubPartitionOptionRelations() or InvalidatePublicationRels()
introduced in the commit 4548c76738b should lock the partitions, just
like to the parent partitioned table would be, before invalidating
them. There may be some hazards to invalidating a relation without
locking it.
For example, maybe add a 'lockmode' parameter to
GetPubPartitionOptionRelations() which it passes down to
find_all_inheritors() instead of NoLock as now. And make all sites
except GetPublicationRelations() pass ShareUpdateExclusiveLock for it.
Maybe like the attached.
--
Amit Langote
EDB: http://www.enterprisedb.com
Attachments:
lock-publication-partitions-before-inval.patchapplication/octet-stream; name=lock-publication-partitions-before-inval.patchDownload
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 9cd0c82f93..5c3767da81 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -140,15 +140,19 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
/*
* Gets the relations based on the publication partition option for a specified
* relation.
+ *
+ * This also locks the partitions, presumably with the same lockmode as one
+ * with which the original relation (given by 'relid') would have already been
+ * locked by the caller.
*/
List *
GetPubPartitionOptionRelations(List *result, PublicationPartOpt pub_partopt,
- Oid relid)
+ Oid relid, int lockmode)
{
if (get_rel_relkind(relid) == RELKIND_PARTITIONED_TABLE &&
pub_partopt != PUBLICATION_PART_ROOT)
{
- List *all_parts = find_all_inheritors(relid, NoLock,
+ List *all_parts = find_all_inheritors(relid, lockmode,
NULL);
if (pub_partopt == PUBLICATION_PART_ALL)
@@ -255,7 +259,7 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
* publish the child tables when the parent table is published.
*/
relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_ALL,
- relid);
+ relid, ShareUpdateExclusiveLock);
InvalidatePublicationRels(relids);
@@ -318,8 +322,14 @@ GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt)
Form_pg_publication_rel pubrel;
pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
+
+ /*
+ * No need to lock the partitions (if any) either because it is
+ * unnecessary or the caller does its own locking.
+ */
result = GetPubPartitionOptionRelations(result, pub_partopt,
- pubrel->prrelid);
+ pubrel->prrelid,
+ NoLock);
}
systable_endscan(scan);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 9c7f91611d..52f1ab2cb0 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -511,7 +511,8 @@ RemovePublicationRelById(Oid proid)
* publish the child tables when the parent table is published.
*/
relids = GetPubPartitionOptionRelations(relids, PUBLICATION_PART_ALL,
- pubrel->prrelid);
+ pubrel->prrelid,
+ ShareUpdateExclusiveLock);
InvalidatePublicationRels(relids);
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 82f2536c65..fa61df5695 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -113,7 +113,7 @@ extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern List *GetPubPartitionOptionRelations(List *result,
PublicationPartOpt pub_partopt,
- Oid relid);
+ Oid relid, int lockmode);
extern bool is_publishable_relation(Relation rel);
extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
On Thu, Oct 7, 2021 at 12:39 PM Amit Langote <amitlangote09@gmail.com> wrote:
On Fri, Sep 17, 2021 at 7:38 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Sep 17, 2021 at 11:36 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:On Thursday, September 16, 2021 6:05 PM Amit Kapila <amit.kapila16@gmail.com>
On Tuesday, September 14, 2021 10:41 PM vignesh C <vignesh21@gmail.com> wrote:
On Tue, Sep 7, 2021 at 11:38 AM houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com> wrote:
Thanks for the comment.
Attach new version patches which clean the table at the end.+ * For partitioned table contained in the publication, we must + * invalidate all partitions contained in the respective partition + * trees, not just the one explicitly mentioned in the publication.Can we slightly change the above comment as: "For the partitioned tables, we
must invalidate all partitions contained in the respective partition hierarchies,
not just the one explicitly mentioned in the publication. This is required
because we implicitly publish the child tables when the parent table is
published."Apart from this, the patch looks good to me.
I think we need to back-patch this till v13. What do you think?
I agreed.
Attach patches for back-branch, each has passed regression tests and pgindent.
Thanks, your patches look good to me. I'll push them sometime next
week after Tuesday unless there are any comments.Thanks Amit, Tang, and Hou for this.
Sorry that I didn't comment on this earlier, but I think either
GetPubPartitionOptionRelations() or InvalidatePublicationRels()
introduced in the commit 4548c76738b should lock the partitions, just
like to the parent partitioned table would be, before invalidating
them. There may be some hazards to invalidating a relation without
locking it.
I see your point but then on the same lines didn't the existing code
"for all tables" case (where we call CacheInvalidateRelcacheAll()
without locking all relations) have a similar problem. Also, in your
patch, you are assuming that the callers of GetPublicationRelations()
will lock all the relations but what when it gets called from
AlterPublicationOptions()?
--
With Regards,
Amit Kapila.
Hi Amit,
On Fri, Oct 8, 2021 at 12:47 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Thu, Oct 7, 2021 at 12:39 PM Amit Langote <amitlangote09@gmail.com> wrote:
Sorry that I didn't comment on this earlier, but I think either
GetPubPartitionOptionRelations() or InvalidatePublicationRels()
introduced in the commit 4548c76738b should lock the partitions, just
like to the parent partitioned table would be, before invalidating
them. There may be some hazards to invalidating a relation without
locking it.I see your point but then on the same lines didn't the existing code
"for all tables" case (where we call CacheInvalidateRelcacheAll()
without locking all relations) have a similar problem.
There might be. I checked to see how other callers/modules use
CacheInvalidateRelcacheAll(), though it seems that only the functions
in publicationcmds.c use it or really was invented in 665d1fad99e for
use by publication commands.
Maybe I need to look harder than I've done for any examples of hazard.
Also, in your
patch, you are assuming that the callers of GetPublicationRelations()
will lock all the relations but what when it gets called from
AlterPublicationOptions()?
Ah, my bad. I hadn't noticed that one for some reason.
Now that you mention it, I do find it somewhat concerning (on the
similar grounds as what prompted my previous email) that
AlterPublicationOptions() does away with any locking on the affected
relations.
Anyway, I'll think a bit more about the possible hazards of not doing
the locking and will reply again if there's indeed a problem(s) that
needs to be fixed.
--
Amit Langote
EDB: http://www.enterprisedb.com
On Wed, Oct 13, 2021 at 9:11 AM Amit Langote <amitlangote09@gmail.com> wrote:
Hi Amit,
On Fri, Oct 8, 2021 at 12:47 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Thu, Oct 7, 2021 at 12:39 PM Amit Langote <amitlangote09@gmail.com> wrote:
Sorry that I didn't comment on this earlier, but I think either
GetPubPartitionOptionRelations() or InvalidatePublicationRels()
introduced in the commit 4548c76738b should lock the partitions, just
like to the parent partitioned table would be, before invalidating
them. There may be some hazards to invalidating a relation without
locking it.I see your point but then on the same lines didn't the existing code
"for all tables" case (where we call CacheInvalidateRelcacheAll()
without locking all relations) have a similar problem.There might be. I checked to see how other callers/modules use
CacheInvalidateRelcacheAll(), though it seems that only the functions
in publicationcmds.c use it or really was invented in 665d1fad99e for
use by publication commands.Maybe I need to look harder than I've done for any examples of hazard.
Also, in your
patch, you are assuming that the callers of GetPublicationRelations()
will lock all the relations but what when it gets called from
AlterPublicationOptions()?Ah, my bad. I hadn't noticed that one for some reason.
Now that you mention it, I do find it somewhat concerning (on the
similar grounds as what prompted my previous email) that
AlterPublicationOptions() does away with any locking on the affected
relations.Anyway, I'll think a bit more about the possible hazards of not doing
the locking and will reply again if there's indeed a problem(s) that
needs to be fixed.
I think you can try to reproduce the problem via the debugger. You can
stop before calling GetPubPartitionOptionRelations in
publication_add_relation() in session-1 and then from another session
(say session-2) try to delete one of the partition table (without
replica identity). Then stop in session-2 somewhere after acquiring
lock to the corresponding partition relation. Now, continue in
session-1 and invalidate the rels and let it complete the command. I
think session-2 will complete the update without processing the
invalidations.
If the above is true, then, this breaks the following behavior
specified in the documentation: "The tables added to a publication
that publishes UPDATE and/or DELETE operations must have REPLICA
IDENTITY defined. Otherwise, those operations will be disallowed on
those tables.". Also, I think such updates won't be replicated on
subscribers as there is no replica identity or primary key column.
--
With Regards,
Amit Kapila.
On Mon, Oct 18, 2021 at 12:24 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Oct 13, 2021 at 9:11 AM Amit Langote <amitlangote09@gmail.com> wrote:
Anyway, I'll think a bit more about the possible hazards of not doing
the locking and will reply again if there's indeed a problem(s) that
needs to be fixed.I think you can try to reproduce the problem via the debugger. You can
stop before calling GetPubPartitionOptionRelations in
publication_add_relation() in session-1 and then from another session
(say session-2) try to delete one of the partition table (without
replica identity). Then stop in session-2 somewhere after acquiring
lock to the corresponding partition relation. Now, continue in
session-1 and invalidate the rels and let it complete the command. I
think session-2 will complete the update without processing the
invalidations.
In the last sentence, it should be delete rather than update.
--
With Regards,
Amit Kapila.